Skip to content
viite.aiViite Platform

Secrets Management — Decision & Runbook

Status: Decision recorded for issue #168 Phase G. Implementation deferred. Companion docs: ENV_VARS.md (inventory), NAMING.md (naming convention).

This document captures which secrets exist, where they live today, where they should live going forward, and how to rotate them. The implementation that backs this decision (sops + age tooling, CI integration, dev UX) is a separate work item — not in this PR.

1. Decision: sops + age

Section titled “1. Decision: sops + age”

Adopt sops with age keys for Terraform state, Hetzner provisioning vars, and any secrets that should be version-controlled. Keep GitHub Secrets for CI credentials and Coolify env UI for live application secrets — sops doesn’t replace those, it complements them.

Why sops + age

Section titled “Why sops + age”
Concernsops + age
Version control of encrypted stateYes — files are git-committable; diffs visible in PRs
Key rotationRe-encrypt with new recipient; old key revoked separately
Multi-recipient (team)Native — list age public keys in .sops.yaml
CI integrationSingle secret (SOPS_AGE_KEY) injected via GitHub Actions; sops decrypts in-place
Offline-friendlyNo SaaS dependency; works on a plane
Learning curveMinimal — sops -e file.yaml, sops -d file.yaml

Alternatives considered

Section titled “Alternatives considered”
OptionWhy not
1Password CLI (op run)Excellent dev UX, but Hetzner box can’t reach 1Password’s API without storing a service account token locally — circular. Better as a complement for dev-machine UX than a backend.
Doppler / Infisical / Hosted SaaSAdds vendor lock-in and SaaS cost for a small team. Audit logs are nice but we don’t need them yet.
Status quo (Coolify UI + .tfvars in gitignore)Works but: (a) tfvars are plaintext on someone’s laptop, (b) no audit trail, (c) onboarding requires DM’ing secrets, (d) loss of laptop = re-rotation of every key.
HashiCorp VaultOperational overhead too high for current team size. Reconsider if we hit ~15 engineers or a compliance audit.

Dev UX (separate from the backend)

Section titled “Dev UX (separate from the backend)”

For local dev, continue with plaintext .env files in .gitignore. Onboarding flow:

  1. New engineer installs sops + age, generates their own age key.
  2. Existing maintainer adds their public age key to .sops.yaml.
  3. Re-encrypts current encrypted files for the new recipient.
  4. Engineer decrypts infra/hetzner/secrets.yaml → fills their apps/api/.env manually for first-run.

Future improvement: wrap the decrypt step in bun scripts/dev-secrets.ts so the engineer just runs bun run secrets:pull.

2. Current state inventory (2026-05-22)

Section titled “2. Current state inventory (2026-05-22)”

2.1 Where secrets live today

Section titled “2.1 Where secrets live today”
LocationContentsPlaintext?Encrypted?
apps/api/.env (local dev)DATABASE_URL, AUTH_SECRET, LLM provider keys, Notion token, Gmail OAuth, Resend✅❌
infra/hetzner/terraform.tfvars (local)Hetzner API token, postgres password, better_auth_secret, anthropic key, S3 keys✅❌
GitHub Actions secretsHETZNER_SSH_PRIVATE_KEY, COOLIFY_WEBHOOK_URL, COOLIFY_API_TOKEN, RAILWAY_TOKEN, NPM_TOKEN, GITLEAKS_LICENSEn/a (managed)✅ (GitHub-encrypted at rest)
Coolify env UIProduction runtime env (DATABASE_URL, AUTH_SECRET, NOTION_INTEGRATION_TOKEN, etc.)n/a (Coolify-managed)✅ (encrypted in Coolify’s DB)

2.2 Risks today

Section titled “2.2 Risks today”
  • Laptop loss = full rotation needed for everything in apps/api/.env and infra/hetzner/terraform.tfvars.
  • Onboarding requires DM’ing 10+ values; mistakes are silent (a typo in AUTH_SECRET only surfaces when sessions break days later).
  • No audit trail for who pulled which secret when.
  • CI surface is broad: HETZNER_SSH_PRIVATE_KEY is root SSH (Phase H deliverable: narrow to deploy user).

3. Scope of the new system

Section titled “3. Scope of the new system”

3.1 In scope for sops + age

Section titled “3.1 In scope for sops + age”
  • infra/hetzner/terraform.tfvars → committed as infra/hetzner/terraform.tfvars.sops.yaml
  • Any future bootstrap scripts that need secrets at provision time
  • Shared dev-onboarding .env template that contains placeholder + real OAuth client IDs (the non-secret-but-painful-to-rebuild bits)

3.2 Out of scope (intentionally)

Section titled “3.2 Out of scope (intentionally)”
  • GitHub Actions secrets — already encrypted, already audited, already team-shared. Adding sops here is duplication.
  • Coolify production env — Coolify owns this; sops would compete with it. The runbook is “operator edits in Coolify UI”; never commit production env values.
  • Local apps/api/.env — keep gitignored plaintext. The onboarding flow decrypts the shared bootstrap doc and engineer fills their own file. (Reason: every engineer’s dev .env has slightly different values — Gmail OAuth pointing at their own account, etc. A shared encrypted file would imply we all share the same dev secrets.)

4. Rotation runbook

Section titled “4. Rotation runbook”

4.1 Outstanding rotations (post-#171 + #169 leaks)

Section titled “4.1 Outstanding rotations (post-#171 + #169 leaks)”

Two recent PRs closed credential leaks; the credentials they exposed must still be rotated as an operator action:

Exposed via PR #171’s fix (/api/config endpoint was returning these to authenticated clients):

  • NOTION_INTEGRATION_TOKEN
  • GMAIL_REFRESH_TOKEN
  • TWITTER_BEARER_TOKEN

Exposed via PR #169’s fix (boot stdout was logging the entire parsed env):

  • BETTER_AUTH_SECRET / AUTH_SECRET
  • CLINE_API_KEY
  • ANTHROPIC_API_KEY
  • OPEN_ROUTER_API_KEY
  • GMAIL_CLIENT_SECRET
  • RESEND_API_KEY

Rotation steps for each:

  1. Generate a new value at the provider (Notion → new integration, Gmail → revoke + recreate OAuth refresh token, etc.).
  2. Update Coolify env UI on staging → smoke test → production.
  3. Update local apps/api/.env for any engineer who has one.
  4. Confirm production logs show no errors post-rotation; hit /api/health (post-#173 expanded version) to verify subsystems still up.
  5. For AUTH_SECRET: rotating invalidates all existing sessions — users get logged out and need to log in again. Schedule outside business hours.

Action owner: infra team. Not blocked by this PR — track in a separate operations issue. Cross-reference: [memory: devops_rationalization_2026-05-21.md].

4.2 General rotation procedure (for any future leak)

Section titled “4.2 General rotation procedure (for any future leak)”
  1. Generate new secret at the provider.
  2. Update Coolify (staging first, validate, then production).
  3. Update infra/hetzner/terraform.tfvars (or the sops-encrypted equivalent once Phase G implementation lands).
  4. Update GitHub Secrets if the value is also referenced in CI.
  5. Revoke the old secret at the provider — don’t leave it active “for safety.”
  6. Smoke test: hit /api/health and one happy-path endpoint per subsystem touched.
  7. Document the rotation in docs/devops/ROTATIONS.md (new file when first rotation happens — date, key, reason).

5. Rollout sketch (when implementing)

Section titled “5. Rollout sketch (when implementing)”

Not in scope for this PR. Sketched here so the next person picking this up doesn’t reinvent the plan.

G.1 — Bootstrap sops + age (1–2 days)

Section titled “G.1 — Bootstrap sops + age (1–2 days)”
  • Install sops + age on each maintainer’s laptop.
  • Each maintainer generates an age keypair; commits their public key to a new .sops.yaml recipient list at repo root.
  • Add bun run secrets:edit script that wraps sops infra/hetzner/secrets.yaml.

G.2 — Migrate terraform.tfvars (1 day)

Section titled “G.2 — Migrate terraform.tfvars (1 day)”
  • Move sensitive values from terraform.tfvars to infra/hetzner/secrets.sops.yaml (encrypted).
  • Update infra/hetzner/main.tf to read from the decrypted file via a sops_file data source (terraform-sops provider).
  • Add infra/hetzner/terraform.tfvars to .gitignore (currently gitignored; confirm).
  • Document the workflow in infra/hetzner/README.md.

G.3 — CI integration (0.5 days)

Section titled “G.3 — CI integration (0.5 days)”
  • Add SOPS_AGE_KEY to GitHub Actions secrets.
  • Any workflow that needs decrypted values (none today — production deploy is Coolify-driven) installs sops + decrypts in-step.
  • Verify CI still passes; document the new step.

G.4 — Onboarding doc (0.5 days)

Section titled “G.4 — Onboarding doc (0.5 days)”
  • Add docs/devops/ONBOARDING.md with the new-engineer flow.
  • Update root README.md to point at it for first-day setup.

G.5 — Optional: 1Password CLI wrapper (1 day, follow-up)

Section titled “G.5 — Optional: 1Password CLI wrapper (1 day, follow-up)”
  • Wrap the dev decrypt step in a bun run secrets:pull that pulls from 1Password Personal if available, falls back to sops otherwise. Keeps the muscle memory familiar for engineers who already use 1Password.

6. Open questions

Section titled “6. Open questions”
  • Coolify-only secrets (set in UI, not in docker-compose.coolify.yml): someone with Coolify access should export and reconcile against ENV_VARS.md before Phase G implementation starts. If there are values living only in Coolify, sops can’t help with them — they’re already “encrypted at rest” by Coolify.
  • Keep .tfvars.example as the documented shape, with secrets.sops.yaml carrying values only? Or merge both into one encrypted source-of-truth? Recommend keeping .example for documentation and shape (no values).
  • Backups: are encrypted secrets included in the postgres backup story? They shouldn’t be (they’re not in postgres), but worth confirming as part of Phase H.

7. References

Section titled “7. References”