Deploys used to run on a paper note taped to the right edge of the ops monitor. Five steps: pull the code, edit the config, back up, reload, click through 5 pages. Handwritten, amended three times in two years — the last revision squeezed "don't forget to clear the cache" into the corner.
In April, we moved it into Jenkins.
What the Pipeline Looks Like
One job, 7 stages: pull the specified branch, install dependencies, run tests, package, upload to the target machine, back up the current version, switch the symlink and reload. Artifact filenames carry the Git short hash — something like app-20200409-a3f7c2.tar.gz — stored in a releases directory, with current as a symlink pointing at the active one.
Rolling back means picking the previous artifact and re-pointing the symlink — no rebuild, no fresh pull. It's a different thing from the Nginx-weight gray release we set up before; dumber, but good enough.
The First Run Went Sideways
The backup step was a shell script, and we'd forgotten set -e. The backup actually failed, but the script kept going and switched the new version in. Fortunately it was staging, so production was untouched — but watching the log stay green all the way down was unnerving.
After fixing it we added two things: set -euo pipefail at the top of every script, and an immediate check after backup that the artifact size is non-zero — otherwise the whole pipeline halts.
Config Never Enters the Repo
For a while people kept asking where the database password lived. It never went into Git; it sits in a file on the target machine readable only by the deploy account, and Jenkins holds just an SSH private key. Config changes go through a separate approval, never mixed in with releases.
Those 5 Pages
The 5 pages we used to click by hand became 5 curl checks: homepage returns 200, the login endpoint returns the expected fields, static assets are fetchable, no API returns a 500, and the health check endpoint responds. Any failure stops the pipeline before the symlink switch.
We don't test business correctness, only "is it alive." The business part still gets clicked by humans — we didn't dare cut that.
The Paper Note
The paper note wasn't thrown away — it's under the keyboard now. Once, when the Jenkins service died on its own, someone followed it step by step. All five steps were still there, and the handwriting was still legible.
