'Ship it first, fix it later' — we've heard this line many times. There's nothing wrong with it; if you wait until every detail is settled, the project may not ship for half a year. What's dangerous is the other extreme: going live means switching 100% of traffic at once, with no way back when something breaks.
So every time we hear this line, we take it with one addition: 'OK — then let's settle the gray-release plan and rollback first.'
Four Steps, Not Two
Many people think a gray release simply means 'give 10% of users the new version first.' Who those 10% are, what you watch, and how long you watch matter more than the number 10 itself.
Step one: internal accounts for 2 days. Only a handful of employee IDs — ours and the client's project team — can reach the new version. We use it to run the paths automated tests can't cover: abandoning a form halfway, submitting twice, opening a new page from an old bookmark.
Step two: roll out to 10% by the last digit of employee IDs. Pick one digit, say 3. This group stays fixed, so nobody bounces between the new and old versions from one day to the next. We watch error logs and support tickets, not 'feelings.'
Step three: roll out to 3 stores. This step is for projects with offline scenarios. The same feature can run smoothly at headquarters and fail on the shared computer at a store, with a different scanner, on a different network. Of the 3 stores, we deliberately picked the one with the worst network.
Leave one observation day between steps. Never roll out on consecutive days, and never on weekends — nobody is watching.
Step four: full rollout.
How Traffic Splitting Works
It's not complicated. Nginx splits traffic by a flag in the cookie: requests with gray=on go to the new-version upstream, the rest go to the old one. The flag is issued by the login API based on employee ID, so no frontend routing changes. Two versions of the code run on two machines with one shared database, which means schema changes must stay backward compatible: adding fields is fine; dropping fields or changing their meaning is not. We've written this rule into our launch checklist.
Rollback Is One Line of Config
After full rollout, we keep the old version running for at least 7 days. If something really breaks, we set the Nginx weight back to 0 and we're done — no repackaging, no redeploy, no waking someone up at midnight to patch code. We've used it once for real, on a style issue that only appeared in IE. From discovery to rollback: 4 minutes.
The Plain Truth
'Ship first, fix later' only holds up if the 'fix' is cheap enough. How cheap? Cheap enough that the on-duty person can hit rollback without asking permission. If you can't get there, don't rush to full rollout.
