Secrets Management — Decision & Runbook

Status: Decision recorded for issue #168 Phase G. Implementation deferred. Companion docs: ENV_VARS.md (inventory), NAMING.md (naming convention).

This document captures which secrets exist, where they live today, where they should live going forward, and how to rotate them. The implementation that backs this decision (sops + age tooling, CI integration, dev UX) is a separate work item — not in this PR.

---

1. Decision: sops + age

Adopt sops with age keys for Terraform state, Hetzner provisioning vars, and any secrets that should be version-controlled. Keep GitHub Secrets for CI credentials and Coolify env UI for live application secrets — sops doesn’t replace those, it complements them.

Why sops + age

Concern	sops + age
Version control of encrypted state	Yes — files are git-committable; diffs visible in PRs
Key rotation	Re-encrypt with new recipient; old key revoked separately
Multi-recipient (team)	Native — list age public keys in .sops.yaml
CI integration	Single secret (SOPS_AGE_KEY) injected via GitHub Actions; sops decrypts in-place
Offline-friendly	No SaaS dependency; works on a plane
Learning curve	Minimal — sops -e file.yaml, sops -d file.yaml

Alternatives considered

Option	Why not
1Password CLI (op run)	Excellent dev UX, but Hetzner box can’t reach 1Password’s API without storing a service account token locally — circular. Better as a complement for dev-machine UX than a backend.
Doppler / Infisical / Hosted SaaS	Adds vendor lock-in and SaaS cost for a small team. Audit logs are nice but we don’t need them yet.
Status quo (Coolify UI + .tfvars in gitignore)	Works but: (a) tfvars are plaintext on someone’s laptop, (b) no audit trail, (c) onboarding requires DM’ing secrets, (d) loss of laptop = re-rotation of every key.
HashiCorp Vault	Operational overhead too high for current team size. Reconsider if we hit ~15 engineers or a compliance audit.

Dev UX (separate from the backend)

For local dev, continue with plaintext .env files in .gitignore. Onboarding flow:

1. New engineer installs sops + age, generates their own age key.
2. Existing maintainer adds their public age key to .sops.yaml.
3. Re-encrypts current encrypted files for the new recipient.
4. Engineer decrypts infra/hetzner/secrets.yaml → fills their apps/api/.env manually for first-run.

Future improvement: wrap the decrypt step in bun scripts/dev-secrets.ts so the engineer just runs bun run secrets:pull.

---

2. Current state inventory (2026-05-22)

2.1 Where secrets live today

Location	Contents	Plaintext?	Encrypted?
apps/api/.env (local dev)	DATABASE_URL, AUTH_SECRET, LLM provider keys, Notion token, Gmail OAuth, Resend	✅	❌
infra/hetzner/terraform.tfvars (local)	Hetzner API token, postgres password, better_auth_secret, anthropic key, S3 keys	✅	❌
GitHub Actions secrets	HETZNER_SSH_PRIVATE_KEY, COOLIFY_WEBHOOK_URL, COOLIFY_API_TOKEN, GITLEAKS_LICENSE (no npm secret: publishing uses Trusted Publishing, NPM_PUBLIC_TOKEN deleted 2026-09-14 — §4.2)	n/a (managed)	✅ (GitHub-encrypted at rest)
Coolify env UI	Production runtime env (DATABASE_URL, AUTH_SECRET, NOTION_INTEGRATION_TOKEN, etc.)	n/a (Coolify-managed)	✅ (encrypted in Coolify’s DB)

2.2 Risks today

• Laptop loss = full rotation needed for everything in apps/api/.env and infra/hetzner/terraform.tfvars.
• Onboarding requires DM’ing 10+ values; mistakes are silent (a typo in AUTH_SECRET only surfaces when sessions break days later).
• No audit trail for who pulled which secret when.
• CI surface is broad: HETZNER_SSH_PRIVATE_KEY is root SSH (Phase H deliverable: narrow to deploy user).

---

3. Scope of the new system

3.1 In scope for sops + age

• infra/hetzner/terraform.tfvars → committed as infra/hetzner/terraform.tfvars.sops.yaml
• Any future bootstrap scripts that need secrets at provision time
• Shared dev-onboarding .env template that contains placeholder + real OAuth client IDs (the non-secret-but-painful-to-rebuild bits)

3.2 Out of scope (intentionally)

• GitHub Actions secrets — already encrypted, already audited, already team-shared. Adding sops here is duplication.
• Coolify production env — Coolify owns this; sops would compete with it. The runbook is “operator edits in Coolify UI”; never commit production env values.
• Local apps/api/.env — keep gitignored plaintext. The onboarding flow decrypts the shared bootstrap doc and engineer fills their own file. (Reason: every engineer’s dev .env has slightly different values — Gmail OAuth pointing at their own account, etc. A shared encrypted file would imply we all share the same dev secrets.)

---

4. Rotation runbook

4.1 Outstanding rotations (post-#171 + #169 leaks)

Two recent PRs closed credential leaks; the credentials they exposed must still be rotated as an operator action:

Exposed via PR #171’s fix (/api/config endpoint was returning these to authenticated clients):

• NOTION_INTEGRATION_TOKEN
• GMAIL_REFRESH_TOKEN
• TWITTER_BEARER_TOKEN

Exposed via PR #169’s fix (boot stdout was logging the entire parsed env):

• BETTER_AUTH_SECRET / AUTH_SECRET
• CLINE_API_KEY
• ANTHROPIC_API_KEY
• GMAIL_CLIENT_SECRET
• RESEND_API_KEY

Rotation steps for each:

1. Generate a new value at the provider (Notion → new integration, Gmail → revoke + recreate OAuth refresh token, etc.).
2. Update Coolify env UI on staging → smoke test → production.
3. Update local apps/api/.env for any engineer who has one.
4. Confirm production logs show no errors post-rotation; hit /api/health (post-#173 expanded version) to verify subsystems still up.
5. For AUTH_SECRET: rotating invalidates all existing sessions — users get logged out and need to log in again. Schedule outside business hours.

Action owner: infra team. Not blocked by this PR — track in a separate operations issue. Cross-reference: [memory: devops_rationalization_2026-05-21.md].

4.2 npm publishing: Trusted Publishing, no token

There is no npm publish secret to rotate. publish-packages.yml publishes the four public artifacts, @viite-ai/platform-utils, @viite-ai/platform-domain, @viite-ai/platform-sdk and the unscoped viite-cli, with npm Trusted Publishing. The job has permissions: id-token: write. Each npm publish trades the job’s GitHub OIDC token for a short-lived credential that npm scopes to that one package. Nothing is stored in GitHub, Coolify, apps/api/.env or on a laptop.

Why it replaced a token: NPM_PUBLIC_TOKEN was a granular access token with “bypass 2FA” and direct publish access. npm deprecated that combination on 2026-09-14. From January 2027 such a token can only stage a version, which a human then approves with 2FA. viite-cli pins the three @viite-ai/* packages at the exact version, so four separately approved staged versions per release would reopen the partial-publish failure below.

What each package needs on npmjs.com (package → Settings → Trusted Publisher → GitHub Actions). All fields are case-sensitive:

Field	Value
Organization or user	abeauvois
Repository	platform
Workflow filename	publish-packages.yml — the workflow that runs npm publish. Not release.yml: it dispatches this workflow through the API, which starts a separate run.
Environment	empty
Allowed actions	allow npm publish (direct), not stage-only

What the workflow relies on, and why:

• npm publish, not bun publish. bun does not implement npm’s OIDC exchange (oven-sh/bun#22423).
• npm is pinned (NPM_VERSION, ≥ 11.5.1). Older npm has no trusted publishing, and the runner’s Node 24 can bundle one.
• No provenance. npm adds provenance automatically only for a public repository. abeauvois/platform is private, and npm refuses --provenance there even for public packages, so the flag is not passed.
• No token may be configured in the job. A failed OIDC exchange does not fail npm publish. npm logs it at verbose and falls back to any configured token. The “Refuse a configured npm token” step makes a green run mean OIDC worked.
• Without a files field, npm pack packs by .gitignore, which excludes dist/. bun did not. platform-domain’s files field exists for that reason. scripts/publish-packages.sh refuses to publish a tarball that is missing its own main / types / bin / exports targets.

If a publish fails with ENEEDAUTH (or E404 on the PUT), npm could not match the run to that package’s trusted publisher. npm publish says only “need auth”; the script also prints npm’s oidc log lines. To ask npm directly, without building or publishing anything, run the diagnose-only mode. It prints each package’s exchange result, HTTP 201 … accepted or HTTP 404 … package not found, and the OIDC claims npm matched:

gh workflow run publish-packages.yml --ref main -f diagnose_oidc=true

package not found means npm has no trusted publisher for that package matching those claims. On 2026-09-14 that was the state of @viite-ai/platform-utils while the other three were set up, and the v2.0.1 publish stopped on it with nothing published. Read or set a package’s publisher from a terminal. Both commands need a browser approval, and the unscoped package is viite-cli, with no @:

npx -y npm@11.19.1 trust list @viite-ai/platform-utils
npx -y npm@11.19.1 trust github @viite-ai/platform-utils --file publish-packages.yml --repo abeauvois/platform --allow-publish

Renaming publish-packages.yml breaks all four packages at once.

Verify a change here with a real release (gh workflow run release.yml), not a bare gh workflow run publish-packages.yml. The bare dispatch skips every version that is already published, so it never reaches npm publish. scripts/publish-packages.sh batches four packages, and a partial publish leaves the CLI uninstallable, because viite-cli depends on the exact versions of the three published before it. The script stops at the first package it cannot confirm.

NPM_PUBLIC_TOKEN is retired. v2.0.1 (2026-09-14) was the first release published through OIDC, with no token in the job, and the repo secret was deleted afterwards. There is no token-based rollback any more: if trusted publishing breaks, fix the publisher (above), don’t re-add a token. A bypass-2FA granular token loses direct publish in January 2027 anyway. If the viite-ai account still lists the old token under Access Tokens, revoke it.

Optional hardening: per package, Settings → Publishing access → “Require two-factor authentication and disallow tokens”. The viite-ai account has 2FA disabled, so this also blocks a manual publish from a laptop until 2FA is on.

Retired: NPM_TOKEN and GitHub Packages. A second token used to sit here, authenticating npm.pkg.github.com for a private @abeauvois/* scope. The only package still published there was @abeauvois/platform-ui, which nothing installed from a registry — apps/dashboard takes it as workspace:*. The registry, the token, the repo .npmrc and the packages are all gone; platform-ui is now marked "private": true. If you find a document telling you to rotate NPM_TOKEN, it is out of date.

4.3 General rotation procedure (for any future leak)

1. Generate new secret at the provider.
2. Update Coolify (staging first, validate, then production).
3. Update infra/hetzner/terraform.tfvars (or the sops-encrypted equivalent once Phase G implementation lands).
4. Update GitHub Secrets if the value is also referenced in CI.
5. Revoke the old secret at the provider — don’t leave it active “for safety.”
6. Smoke test: hit /api/health and one happy-path endpoint per subsystem touched.
7. Document the rotation in docs/devops/ROTATIONS.md (new file when first rotation happens — date, key, reason).

---

5. Rollout sketch (when implementing)

Not in scope for this PR. Sketched here so the next person picking this up doesn’t reinvent the plan.

G.1 — Bootstrap sops + age (1–2 days)

• Install sops + age on each maintainer’s laptop.
• Each maintainer generates an age keypair; commits their public key to a new .sops.yaml recipient list at repo root.
• Add bun run secrets:edit script that wraps sops infra/hetzner/secrets.yaml.

G.2 — Migrate terraform.tfvars (1 day)

• Move sensitive values from terraform.tfvars to infra/hetzner/secrets.sops.yaml (encrypted).
• Update infra/hetzner/main.tf to read from the decrypted file via a sops_file data source (terraform-sops provider).
• Add infra/hetzner/terraform.tfvars to .gitignore (currently gitignored; confirm).
• Document the workflow in infra/hetzner/README.md.

G.3 — CI integration (0.5 days)

• Add SOPS_AGE_KEY to GitHub Actions secrets.
• Any workflow that needs decrypted values (none today — production deploy is Coolify-driven) installs sops + decrypts in-step.
• Verify CI still passes; document the new step.

G.4 — Onboarding doc (0.5 days)

• Add docs/devops/ONBOARDING.md with the new-engineer flow.
• Update root README.md to point at it for first-day setup.

G.5 — Optional: 1Password CLI wrapper (1 day, follow-up)

• Wrap the dev decrypt step in a bun run secrets:pull that pulls from 1Password Personal if available, falls back to sops otherwise. Keeps the muscle memory familiar for engineers who already use 1Password.

---

6. Open questions

• Coolify-only secrets (set in UI, not in docker-compose.coolify.yml): someone with Coolify access should export and reconcile against ENV_VARS.md before Phase G implementation starts. If there are values living only in Coolify, sops can’t help with them — they’re already “encrypted at rest” by Coolify.
• Keep .tfvars.example as the documented shape, with secrets.sops.yaml carrying values only? Or merge both into one encrypted source-of-truth? Recommend keeping .example for documentation and shape (no values).
• Backups: are encrypted secrets included in the postgres backup story? They shouldn’t be (they’re not in postgres), but worth confirming as part of Phase H.

---

7. References

• sops: https://github.com/getsops/sops
• age: https://github.com/FiloSottile/age
• terraform-sops provider: https://registry.terraform.io/providers/carlpett/sops/latest/docs
• 1Password CLI: https://developer.1password.com/docs/cli
• Original audit comment with the secrets-store decision matrix: #168 comment