Our gray release script was written separately from the Jenkins pipeline we set up in 2020. It ramps traffic by the last digit of employee IDs and had exactly one action — advance: 10%, 30%, 50%, 100%, always forward. It ran two years without incident, so no one ever thought about going backward.
August 21
Half an hour after ramping to 50%, support forwarded two screenshots: buttons squashed together on older Android phones. The cause was a flex pattern in the new version that older WebView versions didn't support — our test devices were all too new to catch it.
Simple fix: roll back. Except the script had no rollback.
The Manual 12 Minutes
The person on duty dug through docs to find the Nginx config location, edited the weights, reloaded, then re-checked the logs to confirm traffic was back on the old version. Twelve minutes in all — not long, but during those 12 minutes nobody could be sure everything had fully reverted, because the script normally refreshes the gray release flags in Redis as a side effect, and manually editing weights doesn't. Sure enough, a few scattered users were still on the new version, and we had to clear the flags separately.
The problem wasn't speed — it was that rolling back and advancing weren't the same operation, and things fell through the gap between them.
The Line We Added
We added a rollback command that also takes a ratio: drop straight to 0, or back to 10% and keep a small cohort under observation. The key is that it runs the same code path as advance: adjust Nginx weights, refresh the Redis gray release flags, write a release log line, and post a message to the group chat. All four, every time.
While we were at it, we made "current gray release ratio" a queryable command instead of something you guess from config files. Before, your best option was finding whoever last posted a screenshot in the group.
Two More Guards Added Alongside
- After advancing to 100%, keep the old version running for 7 days. This used to live in documentation only; now the script enforces it.
- Run the 5 curl health checks before every ramp step; if any fail, stay at the current ratio instead of pushing on. Previously they only ran at the 100% step.
Afterward
We rewrote the flex pattern and re-ran the ramp — 10%, 30%, 50% — this time with an older Android device added to the test fleet.
The script now has both advance and rollback, side by side on the first line of the README. When someone asked why we'd write code for an action we'd "probably never use," I showed them the chat log from those 12 minutes on August 21.
