Hetzner Deployment — Runbook & Observability Gaps

Status: Runbook for issue #168 Phase H. Scope: doc-only. Code/CI follow-ups (granular health endpoints, workflow rollback input, Railway-workflow deletion) are tracked as explicit gaps in §6.

This document is the operational pair to infra/hetzner/README.md. The infra README is for standing up the environment; this doc is for running, deploying, rolling back, debugging, and recovering it.

Companion docs: ENV_VARS.md, NAMING.md, SECRETS.md.

---

1. Architecture (TL;DR)

Hetzner Cloud (nbg1)
├── API server (CPX32, 4 vCPU / 8 GB, 10.0.1.10)
│   ├── Coolify orchestration (:8000)
│   ├── Traefik (SSL termination, :80/:443)
│   ├── Platform API (:3000, network_mode: host)
│   └── Dashboard (:5000)
└── Database server (CPX21, 10.0.1.20)
    ├── PostgreSQL 17 (private network only)
    └── Daily backups → /opt/backups + optional S3

Full diagram and provisioning details: infra/hetzner/README.md.

Front door: GitHub Actions workflow → Coolify webhook → container restart. Migrations run via SSH because the database isn’t reachable from GitHub Actions runners (private network).

1.1 SSH access

Every manual recipe below (rollback §3, restore §5, incident response §7) starts with an SSH session. The deploy key lives at ~/.ssh/platform-hetzner (public-key comment platform-hetzner-deploy) and authorises root on both servers.

# API server (Coolify, Traefik, app containers) — the public IP of platform-production-api.
ssh -i ~/.ssh/platform-hetzner root@$API_SERVER_IP

# DB server (Postgres, backups) — no public IP; reach it from the API server
# (jump host) or over the 10.0.1.20 private network.
ssh -i ~/.ssh/platform-hetzner -J root@$API_SERVER_IP root@10.0.1.20

Tips:

• Add -o IdentitiesOnly=yes so SSH offers only this key — avoids Too many authentication failures when your agent holds several keys (this is what makes a non-interactive connection work reliably).
• For scripts add -o BatchMode=yes -o ConnectTimeout=10; run a one-off remote command by appending it: ssh -i ~/.ssh/platform-hetzner root@$API_SERVER_IP 'docker ps'.
• ~/.ssh/config shortcut so you can just ssh platform-api:

  Host platform-api
    HostName <API_SERVER_IP>
    User root
    IdentityFile ~/.ssh/platform-hetzner
    IdentitiesOnly yes

Still root-only — narrowing to a scoped deploy user is tracked in §6.6. The Hetzner Console (VNC) is the fallback if SSH itself is down (§7.8).

---

2. Deploy

2.1 The happy path

GitHub Actions ──webhook──► Coolify ──builds image──► starts API container
                                                              │
                                          docker-entrypoint.sh ┘
                                          └─ drizzle-kit migrate ─► then bun server

Trigger from GitHub Actions → “Deploy to Hetzner” → Run workflow. Inputs:

• environment: production | staging (default production)
• skip_deploy: bool (default false) — no-op, reserved

Workflow file: .github/workflows/deploy-hetzner.yml. Required secrets/vars enumerated in ENV_VARS.md §9.

Migrations run at container startup, not in CI. The release image bakes apps/api/lib/drizzle/migrations + a standalone drizzle.migrate.config.ts, and apps/api/docker-entrypoint.sh runs drizzle-kit migrate before starting the server. This is deliberate: the GitHub Actions runner cannot reach the private database, but the container can. A failed migration aborts boot (set -e), so the deploy health check (§2.2) catches it and Coolify keeps the previous container.

Two gotchas baked into the image, do not “simplify” them away:

• drizzle-kit v1 lives in apps/api/node_modules; the root node_modules ships an incompatible v0 (0.31.x) that cannot read the v1 migration format. The Dockerfile copies the workspace-local modules for this reason.
• the migrate config reads DATABASE_URL straight from process.env (no full env-validation chain) so the image needs no app source to migrate.

2.2 Built-in gates

The workflow performs:

1. Pre-flight check — fails fast if required secrets/vars are missing.
2. Deploy — POST to Coolify webhook with Bearer token; Coolify builds + starts the container, which migrates itself at boot.
3. Wait 60s for Coolify build + container restart.
4. Health check API — 30 attempts × 10s against https://$API_DOMAIN/api/health (post-Phase C this returns {status, commit, llmProvider, enrichEnabled, exportEnabled, db}). Because a failed migration aborts boot, a green health check also means migrations applied.
5. Health check Dashboard — non-fatal; warns if https://$DASHBOARD_DOMAIN doesn’t return 200.
6. Summary — markdown report posted to the Actions run.

2.3 What’s NOT in the workflow today

Gap	Workaround	Tracked in
Automated rollback to previous image on failed health check	Manual SSH (§3)	§6
Pre-deploy schema diff preview	Run drizzle-kit generate locally first	—
Slack / Discord notification on success/failure	Read the email Actions sends	—
Staging environment distinct from production	environment: staging input exists but maps to same infra	§6

2.4 Build memory & the swapfile (OOM safeguard)

Coolify builds both services (api + dashboard) with docker compose build on the API server, while the previous containers keep serving. On the 8 GB CPX32 a from-scratch build of both bundles can exceed RAM and the kernel OOM-kills bun build mid-flight — the deploy then fails at Dockerfile:197 RUN bun run build even though the build is clean in isolation. (First seen when the org-features PR grew both bundles; image builds fine locally on amd64 + arm64.)

A 4 GB swapfile is provisioned on the API server as the safeguard:

ssh -i ~/.ssh/platform-hetzner root@$API_SERVER_IP
fallocate -l 4G /swapfile && chmod 600 /swapfile && mkswap /swapfile && swapon /swapfile
echo '/swapfile none swap sw 0 0' >> /etc/fstab   # persist across reboots
free -h                                            # expect: Swap 4.0Gi

Re-trigger a deploy after the host change via the Coolify API (no new commit needed): curl -H "Authorization: Bearer $COOLIFY_API_TOKEN" "$COOLIFY_BASE_URL/api/v1/deploy?uuid=$APP_UUID". A layer-cached rebuild peaks ~3 GB; a cold rebuild is what risks the OOM, so the swap is the durable fix until builds move off the host (a registry-pull deploy would eliminate host build load entirely).

---

3. Rollback (currently manual)

Coolify keeps the previous image as :previous after each deploy. To roll back:

# 1. SSH to the API server.
ssh root@$API_SERVER_IP

# 2. List recent images for the API service.
docker images | grep platform-api

# 3. Re-tag the previous image as current and restart.
docker tag platform-api:previous platform-api:current
cd /data/coolify/applications/<app-id>   # ask Coolify UI for the path
docker compose up -d --no-deps api

# 4. If you also need to revert migrations: not supported in-place.
#    Restore the database from the most recent backup (§5.2).

# 5. Verify.
curl -s "https://$API_DOMAIN/api/health" | jq .

Operator note: the :previous tag is only as good as the last successful deploy. If two bad deploys land in a row, both Coolify and the rollback recipe will point at broken images. In that case, find a known-good tag in docker images --format '{{.Tag}} {{.CreatedAt}}' and use that.

Follow-up (§6.1): turn this into a rollback_to_image input on deploy-hetzner.yml so it runs from the Actions UI without SSH.

---

4. Healthchecks

4.1 What exists today (post-Phase C / #173)

GET /api/health
→ {
    "status": "ok" | "degraded",
    "llmProvider": "cline" | "anthropic" | "openrouter" | "none",
    "enrichEnabled": bool,
    "exportEnabled": bool,
    "db": "up" | "down"
  }

Pings the Postgres pool with SELECT 1; status degrades to "degraded" when db is down. Used by the workflow’s deploy-time health check (§2.2 step 5).

4.2 What still needs to land (§6.2 follow-up)

The original Phase H plan called for two granular endpoints to let monitoring distinguish “API alive but db down” from “API alive but pg-boss broken.” Sketch:

GET /api/health/db
→ { "ok": bool, "latencyMs": number }    # pings pool; fails fast on connect

GET /api/health/jobs
→ { "ok": bool, "boss": "up"|"down" }    # pings pg-boss schema; checks worker

Sketch of caller use:

• External uptime monitor (UptimeRobot / Better Stack) polls all three at 30s intervals.
• Coolify healthcheck (if introduced) uses /api/health/db since that’s the only one that matters for “should I restart the container.”

---

5. Backups & restore

5.1 Current state (from infra/hetzner/cloud-init/postgres-init.yaml)

• Daily cron at 03:00 UTC on the database server runs /opt/backup/backup.sh.
• Backups land in /opt/backups/platform_YYYYMMDD_HHMMSS.sql.gz.
• Retention: ${backup_retention_days} (default 7) — older files are pruned.
• Off-site (optional): if s3_backup_enabled = true in terraform.tfvars, the backup is aws s3 cp’d to the configured bucket after local dump.
• Log: appended to /var/log/backup.log.

5.2 Restore recipe

# 1. SSH to db server (note: db server, not API server).
ssh root@$(terraform output -raw database_server_ip)

# 2. List available backups.
ls -lh /opt/backups/

# 3. Restore (this is destructive — confirms via existing script).
/opt/backup/restore.sh /opt/backups/platform_20240115_030000.sql.gz

# 4. Restart API to clear connection pool to old data.
ssh root@$API_SERVER_IP "docker restart \$(docker ps -q -f name=api)"

# 5. Verify with /api/health (db should be 'up') and a known query.
curl -s "https://$API_DOMAIN/api/health" | jq .

5.3 Known gaps

Gap	Impact	Mitigation
No tested RTO	Restore time is unknown; could be 1 min or 30 min depending on backup size	Schedule a quarterly restore drill into staging
pg-boss schema is in the same DB and backed up with it ✅	(Not a gap — confirmed via cloud-init)	—
Coolify config is NOT included in /opt/backup/backup.sh	If Coolify host is lost, app config rebuild is manual	Export Coolify config to git periodically
No off-site by default — s3_backup_enabled = false in terraform.tfvars.example	Hetzner disk failure = total backup loss	Enable S3 backups before going live with real users
Backup script is not version-controlled beyond the cloud-init template	Drift between what’s running and what cloud-init says	Periodically cat /opt/backup/backup.sh and diff against template

---

6. Concrete follow-up actions (the “next PRs”)

These were deliberately scoped out of this PR (per “doc only”). Listed here so the next engineer has the punch list.

6.1 Add rollback input to deploy workflow

Add to deploy-hetzner.yml’s workflow_dispatch.inputs:

rollback_to_image:
  description: "Image tag to roll back to (e.g. 'previous' or 'sha-abc123'). Skips deploy step."
  required: false
  type: string

When set, skip the migrate + deploy jobs and just SSH + retag + restart + healthcheck. ~30 lines of workflow YAML.

6.2 Add /api/health/db and /api/health/jobs

Two small Hono routes. /api/health/db already mostly exists inside /api/health (the db field); split it into its own endpoint with timing. /api/health/jobs needs a pg-boss reachability check — easiest is to SELECT count(*) FROM pgboss.job with a short timeout.

6.3 Adminer hardening — IP allowlist

Currently Adminer is on the public internet behind Traefik with no IP restriction (per infra/hetzner/README.md:165–196). Add a Traefik IP allowlist middleware to docker-compose.coolify.yml:

labels:
  - "traefik.http.middlewares.adminer-ipallowlist.ipallowlist.sourcerange=OFFICE_IP/32,HOME_IP/32"
  - "traefik.http.routers.adminer.middlewares=adminer-ipallowlist"

Operator must provide the IP list. If no static IP exists, switch to basic-auth middleware instead.

6.4 Deprecate Railway workflow (decision pending)

.github/workflows/deploy-railway.yml exists alongside deploy-hetzner.yml. The Railway-only secrets (RAILWAY_TOKEN) still occupy a slot in repo settings. Action: confirm Hetzner is the only production target, then:

1. Delete .github/workflows/deploy-railway.yml.
2. Remove RAILWAY_TOKEN from GitHub Actions secrets.
3. Remove references from infra/ or README.md if any.

Blocker for this: operator confirmation that no Railway environment is still receiving traffic.

6.5 Observability stack

Currently we have no central log aggregation, no metrics dashboard, no alerting beyond Coolify’s container view. The plan recommends one of:

• Self-hosted on the API server: Vector → Loki → Grafana, with Prometheus for metrics. Adds ~1GB RAM overhead; runs in the same docker-compose. Free.
• Hosted: Better Stack (logs + uptime) or Axiom (logs + traces). $20–50/mo for small workloads. Zero server-side work.

Recommendation: start with Better Stack ($24/mo for 5GB logs + 50 monitors) because:

• Free monitor count gets us multi-region uptime checks against /api/health/* immediately.
• Log aggregation comes via a single Vector container forwarding everything.
• Cancel-anytime if we outgrow it.

Decision deferred to a follow-up issue — recommend filing once §6.2 lands.

6.6 Narrow CI SSH key scope

ENV_VARS.md §9 flags HETZNER_SSH_PRIVATE_KEY as root SSH — highest blast radius in the CI surface. Mitigation:

1. Create a deploy user on each Hetzner server with sudo scoped to docker commands only.
2. Generate a fresh keypair for that user.
3. Rotate the GitHub Actions secret.
4. Revoke the root key.

This is a one-evening task; flag with the infra team.

---

7. Incident response cheat-sheet

When something breaks in production, walk this list in order:

1. Check /api/health — if it returns at all, the container is up. The fields tell you which subsystem is down.
2. If db: "down" — SSH to db server, check systemctl status postgresql. Most common cause: disk full (Coolify writes a lot of logs). Check df -h.
3. If enrichEnabled: false — LLM provider key missing or rotated. Cross-reference ENV_VARS.md §7.
4. If exportEnabled: false — Notion token missing or revoked. Same.
5. If /api/health doesn’t respond at all — Coolify lost the container. SSH to API server, docker ps -a | grep api. If exited, docker logs <id> --tail=200.
6. If recent deploy is the cause — roll back per §3.
7. If db is corrupt — restore from latest backup per §5.2. RTO unknown (see §5.3); expect ~10 min minimum.
8. If SSH itself is down — Hetzner Console gives you VNC. Login is the deploy user (or root if §6.6 isn’t done yet).

Escalation:

• Hetzner support: https://console.hetzner.cloud (24/7 for paid accounts).
• Coolify support: GitHub issues only — not real-time. Treat as community.

---

8. Open questions

• Is deploy-railway.yml still load-bearing? (§6.4)
• Are there Coolify-only env values not in docker-compose.coolify.yml? (Cross-ref SECRETS.md §6.)
• Move from Coolify + docker-compose to k3s? If yes, this runbook gets a major rewrite. If no, invest in §6.5 observability.
• How big should the disk be on the API server before we need to worry? Currently CPX31 ships with ~80GB SSD; Coolify + images + logs might fill that faster than expected. Set up a disk-usage alert via §6.5.

---

9. References

• Provisioning: infra/hetzner/README.md
• Deploy workflow: .github/workflows/deploy-hetzner.yml
• Backup script source: infra/hetzner/cloud-init/postgres-init.yaml
• Env inventory: docs/devops/ENV_VARS.md
• Secrets store & rotation runbook: docs/devops/SECRETS.md
• Original audit + follow-up issue: #168