← projectsplatform lab
individual infrastructure lab / sep 2026

Running a service beyond the first deployment. A small API and database, with automated releases, monitoring and recovery exercises on a single Linux host.

linux / docker / github actions / postgresql / prometheus / grafana

the project

I used a Surface running Ubuntu to explore the operational side of software: how a change reaches a running service, how to tell whether it works, and what happens when the machine restarts.

The workload is deliberately small—a Flask API backed by PostgreSQL. Docker Compose runs the services, GitHub Actions publishes releases, and Prometheus and Grafana provide monitoring. The application stays on the local network, with S3 used for off-host backups.

I developed the lab with substantial AI assistance and carried out the host investigations and validation. Three failures shaped the final setup.

a commit arrived before its image

The host could detect a new commit while CI was still building its container image. Treating that commit as ready to deploy meant requesting a release that did not yet exist.

The deployment check now waits for the image tagged with the exact commit, keeping the current application running until it is available. After replacing the API container, it waits for the application to confirm it can reach PostgreSQL.

A deployment exercise confirmed that sequence: retain the old release, wait for publication, then tolerate a brief startup failure before readiness succeeds. There is still a short interruption while the single API container is replaced.

a healthy reading could be stale

The readiness metric looked useful, but it only updated when someone requested the health endpoint. Monitoring could keep collecting an old success without testing the database connection again.

I changed monitoring to actively request the database-aware endpoint through a Blackbox probe. Each check now exercises the dependency it reports on. It probes the API directly; access through the Caddy proxy is checked separately.

restarting containers was not enough

After a reboot, Caddy tried to bind its LAN address before Wi-Fi had acquired it. The service needed startup ordering at the host level, beyond container restart settings.

I added a systemd unit to reconcile the Compose stack after Docker and network-online. Repeating the reboot check confirmed that the address binding and HTTP access recovered without the original error.

recovery and next steps

I also restored a SQL backup into an isolated database and verified the expected table while the application remained ready. That established a working database restore, with the procedure documented alongside the deployment and operating instructions.

The lab remains a single-host system. The next recovery exercise is broader: restore the scheduled backup format from S3 onto a fresh host, then test rollback from an unhealthy release. Those checks would extend the evidence beyond the successful deployment, reboot and SQL restore already recorded.