Safe deploys on one small server: staging, backups and automatic rollback

This site runs on one server that it shares with a dozen other projects. There is no cluster, no load balancer and no second machine to fail over to. A bad release here means the site is down until someone notices.

So the release process has one job: a broken change must not reach visitors, and if it does, the server must put the old version back without me.

This is how it works, and what happened the first time the rollback fired for real.

The path from push to production

A push to the main branch starts two jobs. The second only runs if the first passes.

Validate runs in CI, away from the server:

1. Type-check, lint, build and unit tests.
2. Integration tests against a real PostgreSQL database.
3. A release rehearsal. CI builds two Docker images, one from the previous commit and one from the new commit, then walks through a release between them: migrate the database, check that uploaded media survives a container being recreated, take a backup, restore it, and roll the image back.

The rehearsal is the part I would keep if I could keep only one. Unit tests tell you the code works. The rehearsal tells you the upgrade works, which is a different question.

Deploy then connects to the server and asks it to release one exact commit.

The server only accepts one sentence

The CI job holds an SSH key, and a leaked CI key is a common way to lose a server. This key cannot open a shell. It is tied to a forced command, a small script that reads the request and accepts exactly one shape:

read -r action commit_sha branch_name extra <<<"${SSH_ORIGINAL_COMMAND:-}"

if [[ "$action" != "deploy" || -n "${extra:-}" \
   || ! "$commit_sha" =~ ^[0-9a-f]{40}$ \
   || ! "$branch_name" =~ ^(main|master)$ ]]; then
  echo "Only an exact portfolio deployment command is permitted." >&2
  exit 64
fi

It then checks that the commit is the current tip of that branch, takes a lock so two releases can never overlap, and runs the release script from that same commit. Anyone holding the key can do one thing: ask the server to deploy what is already on the main branch.

The release itself

The release script runs on the server, in this order:

1. Refuse to start if the checkout has local changes, or if the Compose file or the nginx configuration does not validate.
2. Back up the database, and tag the running image as the rollback image.
3. Build the candidate image.
4. Back up again, this time the database and uploaded media together, so the two always match.
5. Migrate. Migrations are additive only. The old code must keep working against the new schema, or an image rollback would not be a real rollback.
6. Stage the candidate. Start the new image on a spare local port, with the production database and media, and request the health check, the homepage, the project list, a case study and the sitemap.
7. Cut over. Point the production tag at the candidate and recreate the containers. Wait for Docker to report them healthy.
8. Check again, first on the internal port, then through nginx on the public address.

Every step after the backup runs under one error trap:

rollback_on_error() {
  if [[ "$nginx_changed" == "1" && "$deployment_complete" != "1" ]]; then
    sudo cp "$nginx_backup" "$NGINX_SITE_CONFIG"
    sudo nginx -t && sudo systemctl reload nginx
  fi
  if [[ "$cutover_started" == "1" && "$deployment_complete" != "1" ]]; then
    docker image tag "$old_image_id" "$latest_image"
    "${compose[@]}" up -d --no-build --force-recreate app worker
  fi
}
trap rollback_on_error ERR

If anything fails before the cutover, production was never touched. If anything fails after it, the previous image and the previous nginx configuration go back.

The day the rollback fired

The first real rollback came when I moved the site to a new domain. That release changed the nginx configuration so that a new hostname served the site and the old one redirected to it.

Validation passed. The candidate passed. The cutover passed. Then the very last check, a request to the new public address, failed with a TLS error: the certificate did not match the hostname.

The script did what it was built to do. It restored the old nginx file, put the old image back, and the site kept serving on the old domain. The whole event was a red mark in CI and nothing else.

The cause was a race in my own script:

sudo systemctl reload nginx
curl --fail --silent --show-error "${PUBLIC_ORIGIN}/api/healthz" >/dev/null

systemctl reload returns as soon as nginx has been told to reload. The new worker processes take a moment to start accepting connections. My request arrived in that moment, reached a worker still running the old configuration, and was answered with the certificate of a different site on the same server.

Every earlier release had the same race. It never mattered, because the hostnames had never changed, so the old workers gave the right answer too.

The fix was one line. Wait for the address to answer instead of asking once:

sudo systemctl reload nginx
wait_for_url "${PUBLIC_ORIGIN}/api/healthz" 30

The second release went through.

What the rollback did not cover

The rollback restored the image and the nginx file. It did not restore one thing I had changed by hand before the release: an environment variable holding the site's public address.

For the few minutes between the two releases, the old code was running with the new address. Visitors saw nothing wrong, but sign-in links for the admin area pointed at a domain that was not serving yet.

Image rollback covers what is inside the image. Configuration that lives beside it needs its own plan: either version it with the release, or change it in a way that both the old and the new code can run with.

What this setup does not do

• It is not zero downtime. Recreating the container leaves a gap of roughly fifteen seconds while the new one starts. For a portfolio that is fine. For a checkout page it would not be.
• It does not survive losing the server. The backups are kept for fourteen days, on the same machine. A second copy somewhere else is the obvious next step.
• It only deploys the newest commit. Push three commits a few minutes apart and the first two runs stop at the deploy step, because they are no longer the tip of the branch. That is by design, and it still surprised me the first time.

What I would tell someone building the same thing

• Rehearse the upgrade, not just the code. Build the previous version and the new one, and release from one to the other in CI.
• Give the deploy key one sentence it is allowed to say.
• Stage the candidate against production data before it takes traffic.
• Make the script undo its own work, and test that path on purpose. Mine was first tested by accident.
• When a check follows a reload or a restart, wait for the service. Do not ask once.