Deploys that refuse no connections

A single binary makes deploying simple: copy the new file and restart. But the restart itself has a gap. The old process stops listening, the new one isn’t listening yet, and any visitor who arrives in between gets “connection refused”. It only lasts a moment, and it happens on every deploy.

This post covers how a Renox app closes that gap with tools Linux already has, and the other details that make a restart safe: shutting down gracefully, migrations, and backups.

What rnx make:deploy writes

rnx build          # dist/<app>: views, CSS, JS, translations and migrations inside
rnx make:deploy    # deploy/: a systemd service and socket, Litestream, a README; plus a Dockerfile

The service unit runs migrations before each start, and gives running jobs time to finish when it stops:

[Service]
User=shop
WorkingDirectory=/opt/shop
EnvironmentFile=/opt/shop/.env
ExecStartPre=/opt/shop/shop migrate
ExecStart=/opt/shop/shop
Restart=on-failure
# Renox finishes running jobs on SIGTERM (up to 30 s).
KillSignal=SIGTERM
TimeoutStopSec=40

systemd holds the port

The trick is to stop the app from owning its listening socket. With socket activation, systemd opens the port and hands it to the app when the app starts:

[Socket]
ListenStream=127.0.0.1:3000
# Connections that may wait while the app restarts.
Backlog=4096

The socket stays open across restarts. A connection that arrives while the app is restarting waits in the kernel’s backlog until the new process takes over the socket, instead of being refused. The app sees LISTEN_FDS in its environment and uses that socket instead of binding APP_HOST:APP_PORT itself. No code in the app has to change for this.

To switch an existing server over, do it once:

sudo cp deploy/shop.socket /etc/systemd/system/ && sudo systemctl daemon-reload
sudo systemctl stop shop                 # it holds the port; the socket takes it over
sudo systemctl enable --now shop.socket
sudo systemctl start shop

After that, every deploy is a copy and a restart:

scp dist/shop server:/opt/shop/shop.new
ssh server 'mv /opt/shop/shop.new /opt/shop/shop && sudo systemctl restart shop'

The mv replaces the file in one step, so the service never starts a half-copied binary.

Renox’s documentation measured the difference on a laptop, with its hello example and requests sent back to back through three restarts. Without the socket, 280 of 1,288 requests were refused. With it, all 1,350 were answered.

Stopping gracefully

On SIGTERM, a Renox app stops taking requests and waits up to 30 seconds for running jobs to finish. That’s why the unit gives it TimeoutStopSec=40. A job that is still running when the process is killed anyway (out of memory, SIGKILL) isn’t lost. It is retried after 15 minutes (longer for jobs with a longer timeout), and once it has used up its attempts it goes to failed_jobs, where queue:retry can run it again.

Migrations both versions accept

For a short moment during a restart, the new schema exists while the old code is still finishing its requests. With several servers, that moment lasts longer. So a migration has to work with both versions of the code:

• Add columns as nullable or with a default, and backfill them in a job or a command.
• Rename or drop a column over two deploys: first stop reading it, then drop it.
• On PostgreSQL, add indexes with CREATE INDEX CONCURRENTLY. A file with a -- renox:no-transaction line runs outside a transaction, which that statement needs.

Each migration otherwise runs in a transaction, so a migration that fails leaves nothing behind. If several servers run migrate at the same time, they take turns, and the later ones find nothing left to do.

Backups while it runs

The database is storage/app.db, a file. Copying it with cp while the app writes to it can produce a broken copy. There are two safe ways:

sqlite3 storage/app.db ".backup backup.db"   # a consistent one-off copy, while the app runs

For continuous backups, the generated litestream.yml streams every change to S3-compatible storage (AWS S3, Cloudflare R2, MinIO):

dbs:
  - path: /opt/shop/storage/app.db
    replicas:
      - type: s3
        bucket: CHANGE-ME
        path: shop

Litestream copies the write-ahead log as it grows, so a restore loses seconds of writes rather than everything since last night’s copy.

When even a pause is too much

Socket activation turns refused connections into a short wait. If the app starts slowly, or must keep a warm cache, run two copies on two ports behind Caddy and restart them one at a time. The generated deploy/README.md has the Caddyfile (lb_policy first with health_uri /health). Both copies can share one SQLite file on the same machine. Set CACHE_STORE=database so they share rate limits and locks, and SESSION_DRIVER=database if sessions are too large for a cookie.

For most apps one copy is enough: one binary, one socket, one database file, and one command to deploy.