Unified Job Architecture

Post-refactor reference for the Job subsystem. Migration complete through Phase H — the system executes end-to-end.

Why

Three overlapping concepts (Task, BackgroundTask, Workflow) were collapsed into a single Job DDD aggregate. Progress tracking, step orchestration, and queue integration now live in one place with a clean hexagonal split.

Layers

packages/platform-domain/src/jobs/    ← pure domain (no infra)
├── entities/
│   ├── Job.ts              aggregate root
│   ├── JobStep.ts          data-shaped step + JobStepResult
│   ├── JobPreset.ts        single source of truth for preset → steps mapping
│   ├── JobTemplate.ts      reusable spec (userId + preset/steps + inputs)
│   └── JobSchedule.ts      cron + template; constructor validates name + cron
└── ports/
    ├── IJobRepository.ts            persistence port (incl. findInFlightBySchedule)
    ├── IJobRunner.ts                dispatch port (incl. schedule / unschedule)
    ├── IJobScheduleRepository.ts    schedule persistence port
    └── IWorkflowStep.ts             runtime step contract (BaseWorkflowStep)

apps/api/jobs/                        ← application + infra
├── usecases/
│   ├── commands/   Create / Cancel / UpdateJobStatus / AddStepResult
│   │               + CreateJobSchedule / DeleteJobSchedule / RunScheduleNow
│   └── queries/    GetJob / ListJobs / GetJobSchedule / ListJobSchedules
├── routes/         createJobsApp({ jobRepository, jobRunner, jobScheduleRepository }) — Hono factory
├── adapters/       DrizzleJobRepository, DrizzleJobScheduleRepository,
│                   PgBossStarter, PgBossJobRunner,
│                   InMemoryJobRunner, InMemoryJobScheduleRepository
└── workers/
    ├── jobWorkflow.worker.ts   pg-boss worker callback (discriminates scheduled vs immediate fires)
    ├── processJob.ts           shared step loop (used by worker AND InMemoryJobRunner)
    ├── stepRegistry.ts         name → IWorkflowStep lookup
    └── steps/
        ├── runRead.ts          shared engine for the atomic source-read steps
        ├── ReadGmailStep.ts     read_gmail (read_classifieds / read_drive alongside)
        └── ReadUrlStep.ts       read_url — fetches the pages linked in upstream items

Aggregate: Job<T>

Immutable transitions return a new Job instance. Public surface:

Member	Description
id, type, steps, userId, createdAt	identity + structure
status, progress, currentStep, itemProgress, result, updatedAt	live state
metadata: JobMetadata	value object with .data / .get / .set
start(), complete(result), fail(error)	state transitions
updateProgress(...), updateStepProgress(...), addMetadata(...)	progress updates
isPending() / isRunning() / isCompleted() / isFailed()	predicates
hasErrors(), getErrorCount()	error inspection

Errors live on job.result?.errors, not on the job itself. Use serializeJob() in routes/job.routes.ts as the canonical wire shape.

Use cases

One file per action:

• Commands: CreateJobUsecase (optional IJobRunner), CancelJobUsecase, UpdateJobStatusUsecase, AddStepResultUsecase
• Queries: GetJobUsecase, ListJobsUsecase

Each takes an IJobRepository (and the create command optionally an IJobRunner) in its constructor. Routes instantiate per-request — DI container is intentionally out of scope at current scale.

Ports & adapters

Port	Adapter	Status
IJobRepository	DrizzleJobRepository	live
IJobRunner<T>	PgBossJobRunner (default), InMemoryJobRunner (fallback / tests)	live
IJobScheduleRepository	DrizzleJobScheduleRepository, InMemoryJobScheduleRepository (tests / fallback)	live
IWorkflowStep<T> / BaseWorkflowStep<T>	concrete steps under apps/api/jobs/workers/steps/ (e.g. ReadGmailStep), discovered via StepRegistry	live

apps/api/index.ts selects PgBossJobRunner when startJobQueue(...) succeeds; otherwise falls back to InMemoryJobRunner (dev mode without postgres, or tests). Both runners share the same processJob() step loop.

Presets

JobPreset (in domain) holds the canonical preset → JobStep[] mapping plus a per-preset input schema (InputFieldSchema: type / required / label / optional default, where a default may itself be a ${env:KEY} placeholder). CreateJobUsecase resolves steps from it server-side, so presets and the runtime can’t drift.

Preset metadata is exposed for the dashboard’s create-schedule form via GET /api/jobs/presets (SDK: JobClient.listPresets()), returning { name, label, status: 'ready' | 'blocked', steps, inputs } per preset. status: 'blocked' flags a preset whose steps include an unregistered executor.

Registered presets: veilleToBoard, socialVeille, calendarToTasks, captureComments, gmailTriage, getSuppliersInvoices, agentResearch, implementTask, plus the cross-tenant operations presets (generateInvoices, trialLifecycleEmails) — full list + step gates in packages/platform-domain/src/jobs/entities/JobPreset.ts. There is no publish preset: socialPublish/blogPublish/emailPublish were each a lone publish step (and the first two were the same one), so publishing is triggered by the row’s lifecycle status and run either by the one-off job the dashboard’s “Publish now” enqueues or by a custom flow carrying the publish step. The blog preset went the same way as veille — the composer builds that chain directly. Presets carrying ${input:...} placeholders (gmailTriage, agentResearch, implementTask, …) collect those values from the schedule form — and they stay editable per-schedule afterwards; inputless presets resolve only ${env:...}. There’s no hardcoded preset for a Gmail → Notion dispatch anymore — the dashboard’s “Build custom flow” composer (scheduled-jobs.new.tsx) lets a user assemble that chain (or any other) directly, since the API already accepts an arbitrary template.steps array.

Why a preset shows “(coming soon)” in the dashboard

A preset is returned with status: 'blocked' — rendered as a disabled ”… (coming soon)” option in /scheduled-jobs/new — when any step in its chain has no registered executor (stepRegistry.has(name), checked in GET /api/jobs/presets). In practice almost every step now registers unconditionally at boot (see the gate table below), so status is essentially always 'ready'; the meaningful per-user signal GET /api/jobs/presets returns is the channels array — which of the chain’s required channel schemes (gmail, llm, notion, …) this caller has actually connected. A preset whose steps are all registered but whose required channel isn’t connected still shows as ready in the picker; it fails clearly at run time (or, for the dashboard’s own guidance, the channels[].connected flag) rather than being hidden.

Step → registration gate:

Step	Registered when
read_gmail, read_drive	always — atomic per-source read steps sharing the runRead engine; need a connected gmail:// channel at run time
read_notion	always — reads every row of the owner’s connected notion:// database (url/tags/description already structured, so no enrich needed downstream); resolves its own channel rather than the shared gmail:// credential runRead binds
read_url	always — second-hop fetch: scrapes the URLs found in upstream items into page-content refs
analyze, enrich	always — resolve their IContentEnricher per run from the owner’s connected llm:// channel; fail clearly at run time with none connected
dispatch	always — dashboard/sales/link are internal schemes (always deliverable, no channel needed); the post/notion/email targets are all registered too, but writing to one needs a connected connector of that scheme at run time
tag	always — needs a connected gmail:// channel at run time
calendar_to_tasks	always — needs a connected gmail:// channel (Calendar scope) at run time
triage	always — needs a connected LLM channel at run time (flagged Gmail thread → kanban Task, or → dispatch rows with emit: 'rows')
agent, implement_task	always — need a connected LLM channel at run time; implement_task additionally needs AGENT_WORKSPACE_ROOT for its read-mostly code tools
publish	always — routed per row, not per lane: a blog row is a pure state flip (no OAuth gate), a social row needs an OAuth-connected platform (X_CLIENT_ID + a connected account) or it bounces on its own
update_link	always (DB-backed)
watch	always (DB-backed observations); the optional LLM-confirm wires only when a connected LLM channel is present
collect_invoices	always (DB-backed); the optional LLM tiebreak wires only when a connected LLM channel is present — without one it runs gate-only
deliver_webhook	always (DB-backed, self-sourcing)

Preset → step chain (from packages/platform-domain/src/jobs/entities/JobPreset.ts; all ready given the gate table above — the channel each needs is what actually varies per user):

Preset (dashboard label)	Steps	Needs at run time
veilleToBoard (Veille to board)	read_gmail → read_url → enrich → dispatch[dashboard] → tag	a connected gmail:// channel and an LLM channel; enriched links → kanban todo cards
socialVeille (Gmail → X draft)	read_gmail → read_url → enrich → dispatch[social] → tag	a connected gmail:// channel and an LLM channel; composes drafts, no posting
calendarToTasks (Calendar to tasks)	calendar_to_tasks	a connected gmail:// channel with Calendar scope; one kanban Task per upcoming event, idempotent on event id
gmailTriage (Gmail triage)	read_gmail → triage → tag	a connected gmail:// channel and an LLM channel; a routing skill may add a route:<x> tag (also serves the “admin process”)
getSuppliersInvoices (Get suppliers invoices)	read_gmail → collect_invoices → tag	a connected gmail:// channel; an LLM channel is OPTIONAL — deterministic rules decide first and the model is consulted only for mail they can’t call. Files entity: 'link' cards tagged from:invoices, which /api/accounting lists. Nothing is archived: a card carries the supplier’s link plus a Gmail permalink, never a copy
agentResearch (Agent research)	agent	an LLM channel on an anthropic/openrouter provider (the tool-loop is unavailable on a Cline-only provider)
implementTask (Implement task)	implement_task	an LLM channel and AGENT_WORKSPACE_ROOT; proposes a dry-run diff on tagged in-progress Tasks, never applies it
captureComments (Capture comments)	capture_comments	at least one connected LinkedIn / Facebook Page account — capture_comments is the one preset step that is NOT registered unconditionally (no connectable comment reader ⇒ dropped), so this preset can genuinely show blocked
generateInvoices (Generate invoices)	generate_invoices	STRIPE_SECRET_KEY + a connected Stripe account with chargesEnabled per selling org. Cross-tenant: run behind exactly ONE system-wide schedule
trialLifecycleEmails (Trial reminder emails)	trial_lifecycle_emails	RESEND_API_KEY + SMTP_FROM. Cross-tenant, same shape as generateInvoices; idempotent via a lifecycle_emails ledger row per (org, kind)

There is no hardcoded Gmail → Notion (or any other single-shot ingestion → dispatch) preset — the dashboard’s “Build custom flow” composer assembles an arbitrary template.steps chain, including a read_notion → dispatch(link://…) flow with no enrich step needed. Quotes are no longer a separate pipeline: they’re sales Deals living directly on the Task kanban (destination.scheme = 'sales'), so the old quote_requests / quotesToBoard presets and their read_quote_requests / mark_quote_requests_reviewed steps are gone.

Domain boundary

Enforced two ways:

1. packages/platform-domain/package.json — only zod as runtime dep. No drizzle-orm, pg-boss, hono, or Node built-ins.
2. packages/platform-domain/eslint.config.js — no-restricted-imports blocks drizzle-orm, pg-boss, hono, fs/http/https/path, node:*, and any apps/* path.

Run bun run lint inside packages/platform-domain/ to check.

Step authoring

To add a new step (e.g. analyze, enrich, dispatch):

1. Implement BaseWorkflowStep<T> under apps/api/jobs/workers/steps/:
   • name must match the string used in JobPreset (e.g. 'analyze')
   • doExecute(context) reads per-job config from context.metadata.stepConfig and userId from context.metadata.userId (both injected by processJob before each step runs)
   • Return a StepResult<T> with the updated context and a continue flag — continue: false stops the workflow mid-job without failing it
2. Register the step in apps/api/index.ts against the shared StepRegistry. Steps not in the registry cause the worker to fail the job with "No executor registered for step 'X'" — this is intentional fail-fast behavior.
3. Per-step results are persisted via IJobRepository.addStepResult automatically — no extra work needed.

Registered steps today (in apps/api/index.ts):

• read_gmail / read_classifieds / read_drive — always registered. Atomic per-source read steps (one source each) sharing the runRead helper, which resolves the owner’s channel + per-run reader factory (Gmail/Drive), records a deferred watermark advance (see “Watermarks advance only on job success” below — it no longer advances inline), maps BaseContent → ReadItemRef, and merges counts into context.metadata.read. They replace the old sourceFrom-dispatching read. Message selection is fully manual: the Gmail presets no longer expose structured gmailLabel/fromEmail inputs — a single required extraQuery input carries raw Gmail search syntax (label:Triage, from:someone@x.com, to:me, has:attachment, …) and is the whole inclusion filter. It never forces from:me, so a label-watch also catches received mail (a message you tag but didn’t send); add from: yourself to restrict to one sender. The -label:<processedLabel> exclusion is appended automatically and is what keeps re-runs idempotent (and what makes runRead drop the watermark’s after: bound — see below).

• Watermarks advance only on job success. A read step stashes its intended advance on context.metadata.pendingWatermarks; processJob commits them (via an injected commitWatermarks) only when the whole job completes — a mid-chain failure (e.g. dispatch halting on “no LLM configured”) leaves the cursor where it was, so the message is re-read on the next run. Idempotency across those retries is the source’s own guard (Gmail’s -label:<processedLabel> exclusion), not the watermark. This prevents the class of bug where a message is read once, a later step fails, and the advanced cursor then hides it from every future run.

• read_url — always registered. Second-hop fetch: in fromItems mode it extracts the http(s) URLs from each upstream item’s content (or uses an explicit urls list), scrapes each via IWebScraper, and appends page-content ReadItemRefs (fetchedPage: true, carrying parentSourceMessageId/parentUrl so provenance survives the 1→N fan-out). Never replaces upstream items, so tag idempotency (which reads metadata.read) is untouched.

• read_notion — always registered (apps/api/notion/jobs/ReadNotionStep.ts). Reads every row of the owner’s connected notion:// database (integration token + database id, the same channel the notion dispatch target writes through) — no watermark/incremental filtering, since a curated database is small and the downstream link dispatch target is idempotent per-URL. Shapes rows directly into both context.items (standard ReadItemRefs) and context.metadata.enrich.rows (DispatchRow[]), so a read_notion → dispatch(link://...) flow needs no enrich step in between — the source already has structured url/tags/description. Column names are configurable via stepConfig.columnMap (defaults: Name / URL / Tags / Description).

• analyze — always registered (apps/api/jobs/workers/steps/AnalyzeStep.ts). Cheap per-item keep/drop pre-filter (no scraping) that runs before enrich pays the fan-out cost. Steerable: config.skill takes an enricher-kind skill (ref by id or slug, optional @<version> pin) whose body REPLACES the free-text instruction and is declared authoritative to the classifier — a selection rule (“only receipts”) otherwise loses to the generic relevance rubric, the failure tasks/lessons.md records for triage. Resolution is shared with triage (apps/api/skill/resolveSkillSteering.ts): a missing/disabled/mis-kinded/version-drifted skill never runs, it falls back to the free text and records the reason in context.metadata.analyze.skill. The unsteered prompt is byte-identical to what it was before the step became skillable.

• enrich — registered when an LLM channel is available. Pure synthesis now: for each content ref (the read_url page refs when present, else the raw refs), IContentEnricher.analyzeContents its rawContent into a {title,url,summary,tags} row, skipping empty analyses. No scraping or URL discovery — that moved to read_url. Emits rows into context.metadata.enrich.rows; maxRowsPerJob default 50.

• dispatch — the step formerly named export (and its port formerly IExportTarget) in earlier revisions of this doc; both were renamed in code and this doc now matches. Registered when any target is available (in practice always — see the gate table above). Reads context.metadata.enrich.rows, routes to a Map<string, IDispatchTarget> keyed by stepConfig.destination’s URI scheme (canonical schemes: DESTINATION_SCHEMES in packages/platform-domain/src/connector/Destination.ts). buildJobs assembles the map from all available targets: notion → NotionDatabaseWriter, email → EmailDigestExportTarget, dashboard → BoardTaskTarget (kanban Tasks, sourceType: 'dispatch'), link → LinkDispatchTarget (channel='link' Tasks, sourceType: 'link' — the rows backing the dashboard’s /links inbox), plus one shared DraftDispatchTarget registered under every key returned by DestinationHandlerRegistry.drafters() (the post and email kinds), routing config.destination to that handler’s draft axis. dashboard, sales, and link are internal schemes: no connected channel is required (only dashboard and link have a target — sales is internal with no target, a Task just “lives” there), and the dashboard’s “Build custom flow” composer offers dashboard/link directly in the Destination picker’s “Board & Links” group (previously only reachable via the raw-JSON step-config fallback). DispatchStep threads context.metadata.userId/orgId into every target’s config, plus a derived pipelineName (the job’s scheduleName slug, or its preset) that BoardTaskTarget/LinkDispatchTarget stamp as a from:<pipelineName> provenance tag (e.g. from:notion-to-board) via the shared pipelineTag() helper — falls back to the generic from:pipeline for an unnamed one-off run.

• publish — the unified self-sourcing publish worker (apps/api/tasks/jobs/PublishStep.ts; reached by a custom flow carrying the step, or by the one-off job the dashboard’s “Publish now”/“Send now” enqueues — no preset points here). Ignores enrich.rows and selects its own due rows (trigger status + scheduledAt <= now) from the shared ITaskLifecycleRepository — the same DrizzleTaskRepository instance the draft target writes through, since drafts and posts are both tasks rows. Dispatches by DestinationKind to an IDestinationHandler’s publish axis (PostDestinationHandler / EmailDestinationHandler, resolved via DestinationHandlerRegistry.publishers()). Per row, in order: claim (repository.claim — an atomic triggerStatus → claimStatus conditional UPDATE; winning it is what makes publishing single-shot, and a lost claim is skipped rather than double-published), publish, then commit + afterCommit (e.g. blog→social cross-link), run outside the failure path so a cross-link failure can’t undo an already-committed publish. A per-row failure routes that row to needs_review (a human must re-approve — no silent retry loop) and the pass continues.

  The social/blog gate is per-row, not per-lane. Social and blog are one post handler now, so there is deliberately no “drop the social publisher when nothing is connectable” step — doing that would silently disable blog publishing too. The handler instead refuses an individual row whose platform has no publisher, with a warning-level PublishError that bounces just that row.

• triage — always registered (apps/api/triage/jobs/TriageStep.ts). Asks an ITriageDecider (LLM, resolved from the owner’s dashboard:// channel) whether each read item needs a follow-up, and turns actionable ones into a kanban Task keyed manual:<userId>:<gmail message id>. config.emit chooses what happens to a verdict: task (the default, and what every preset uses) creates that card here; rows instead appends a DispatchRow to metadata.enrich.rows so a downstream dispatch routes it — to Links, Notion, a draft, or back to the board — which is what makes read_gmail → triage → dispatch → tag a legal chain. The row carries the same title/tags/priority/dueDate/source and the same idempotency key, so a schedule can switch modes without duplicating anything, and BoardTaskTarget prefers those row-level values over its own defaults. The mode is explicit rather than inferred because DispatchStep treats an empty row batch as a success: a card-creating triage sitting in front of a dispatch would report a green run nightly while exporting nothing — validateJobChain rejects that pairing (triage_emits_tasks) at build time. A decider failure is not a verdict. The deciders fail soft (triageFailSoft → a “Review: ” placeholder) and mark that decision degraded: true; the step then (a) writes the placeholder under a distinct …:failsoft key so the canonical key stays free — a placeholder holding the canonical key would read as “already triaged” forever, which is how a Cline credit exhaustion once permanently poisoned 29 threads — (b) adds the message to context.metadata.tagExclusions so tag leaves it unlabelled and the next read retries it, and (c) after DEGRADED_HALT_THRESHOLD (5) consecutive failures aborts the pass entirely (continue: false), because a run where every decision fails is an outage, not N follow-ups. When a later pass does judge the thread, its untouched placeholder (still todo, still tagged triage:degraded) is deleted. The decider is given the subject from ReadItemRef.metadata.subject — without it every placeholder is titled the identical, useless “Review: flagged email thread”.

• tagExclusions — the “don’t mark this one processed” channel (apps/api/jobs/workers/steps/tagExclusions.ts). tag labels every message the read step pulled and the next read excludes that label, which makes any unprocessed message permanently invisible. A middle step that could not process an item adds its provider message id to context.metadata.tagExclusions; TagStep filters those out before labelling. Step-agnostic by design (ids only, no reason) — tag must not learn about triage.

Placeholder-resolved step config: CreateJobUsecase walks each step’s config and replaces whole-string placeholders in three passes — ${input:fieldName} first (user-supplied schedule/job inputs, with preset defaults filled by applyInputDefaults), then ${env:KEY} (env-derived values; unknown env keys are left intact — fail-loud at the worker), then ${setting:KEY} (the creating user’s settings, resolved per-job by a settingsResolver keyed on userId; an unset setting resolves to '' so a downstream consumer applies its own fallback). This lets preset definitions in pure domain reference input-, env-, and per-user-setting-derived values without coupling the domain layer to those stores. Mixed-string templating ("${env:X}/foo") is intentionally unsupported — step config is data, not templates.

No built-in preset uses ${setting:...} today (the one consumer, the quote_requests preset, was removed when quotes became sales Deals on the Task kanban), but it’s still live in CreateJobUsecase for any custom email:// dispatch flow: point to at ${setting:notificationEmail} (the user’s app:notifications.defaultEmail setting) and EmailDigestExportTarget falls back to NOTIFICATION_EMAIL when that resolves empty.

Scheduling

Recurring fires of any preset (or explicit step list) on a cron, shipped in tasks/scheduled-jobs.md Phases 1–4. The dashboard management UI shipped separately (tasks/scheduled-jobs-ui.md) — see Dashboard surface below.

Shape

A JobSchedule is a long-lived row that spawns Jobs; each cron tick materializes a fresh Job aggregate that flows through the same processJob() loop. There is no “scheduled job” type — every fire produces a normal Job that shows up in GET /api/jobs alongside one-shot submissions. Schedules and Jobs are linked by Job.metadata.scheduleName.

Schedule names are namespaced server-side as {userId}.{slug}. Users supply only the slug; the namespace prevents cross-tenant collisions in pg-boss’s flat schedule table.

HTTP surface (under /api/jobs/schedules/)

Method	Path	Notes
POST	/	Create + register cron. Validator enforces slug + cron + template (preset XOR steps)
GET	/	List my schedules
GET	/:name	Read one
DELETE	/:name	Unschedule + delete row
POST	/:name/run	Fire-once now, skips the overlap guard (manual trigger is allowed to overlap)

Preset metadata for the create form is served by a sibling endpoint, GET /api/jobs/presets (not under /schedules). Jobs materialized by a schedule are filterable via GET /api/jobs?scheduleName={userId}.{slug}.

Worker dispatch

jobWorkflow.worker.ts discriminates on payload shape:

• { jobId, userId } → existing immediate-fire path via processJob.
• { __scheduledFire, scheduleName, template } → handleScheduledFire:
  1. Query IJobRepository.findInFlightBySchedule(scheduleName, userId).
  2. If found, log + skip (overlap policy — see below).
  3. Otherwise materialize via templateToJobCreate + CreateJobUsecase (with null runner, so we don’t re-enqueue onto pg-boss and loop), best-effort touch(lastFiredAt) on the schedule, then processJob inline.

Overlap policy — skip-if-previous-still-running

Locked decision (tasks/scheduled-jobs.md §0 D4). A scheduled fire that arrives while a prior fire is still pending or running is logged and skipped, not queued. Rationale: not every preset has per-step idempotency (gmailTriage happens to via Gmail-label dedup, but analyze and future dispatch targets don’t necessarily), so the failure mode “schedule slips” is preferable to “two writers race the same Notion dispatch.”

Hot lookup is backed by a partial index — jobs(user_id, (metadata->>'scheduleName')) WHERE status IN ('pending','running') — added in the create_job_schedules migration.

Don’t add bypass paths to the worker without explicit reason. The route layer’s POST /:name/run is the only intentional bypass (manual trigger).

pg-boss schedule integration

PgBossJobRunner.schedule(name, cron, template) calls boss.schedule(JOB_WORKFLOW_QUEUE, cron, payload, { key: name }). The key is what makes multiple schedules coexist on a single queue in pg-boss v12 — without it the queue would only hold one schedule total. unschedule(name) calls boss.unschedule(queue, name) and swallows not-found (idempotent like cancel).

Usecase invariants

Both write-side schedule usecases enforce an ordering that prevents orphan state:

• Create — row first, then runner. If the runner fails, the row is rolled back. Never leave a job_schedules entry the cron-side doesn’t know about.
• Delete — runner first, then row. If the runner fails, the row stays so an operator can retry the DELETE. Never orphan a cron registration without a row to find it again.

Inferred Job type

templateToJobCreate (alongside CreateJobUsecase) resolves template.preset via JobPreset.stepsFor(...) and derives JobTypes from the step count — single-step vs workflow-of-steps. Every cron fire re-runs preset resolution + placeholder substitution, so changing an env var or a preset definition propagates automatically without re-saving schedules.

Retiring a preset

Preset resolution distinguishes absent from unknown. No preset name (a custom-steps template) falls back to DEFAULT_STEPS; a name that isn’t in the registry throws UnknownJobPresetError from both JobPreset.stepsFor and JobPreset.inputsFor. It used to fall back in both cases, which meant deleting a preset silently converted every schedule pointing at it into a green-but-useless read + analyze run.

So deleting a preset from JobPreset.ts is safe by construction — the stored schedules that referenced it become loudly broken rather than quietly wrong:

• POST /api/jobs/schedules refuses to create one (400), so the state can only arise from a retirement.
• reconcileJobSchedules (boot) does not register its cron and reports it in failures; when the retired preset was a SYSTEM one, systemSchedulesHealthy on /api/health goes false.
• A scheduled fire logs [jobs] schedule '<name>' cannot fire: … at error level and skips. Deliberately not rethrown — a permanently-broken template would otherwise be retried by pg-boss on every tick. Other schedules are unaffected.
• GET /api/jobs/schedules/:name still returns 200, with a templateError string the dashboard’s schedule detail page renders as a destructive banner (and which disables “Run now”). POST /api/jobs/schedules/:name/run 400s with the same message.

The fix for an operator is always the same: delete the schedule and recreate it against a registered preset.

Dashboard surface

The scheduling management UI lives in apps/dashboard (tasks/scheduled-jobs-ui.md):

• /scheduled-jobs — lists schedules (cron rendered human-readably via cronstrue, last-fired, manual run now).
• /scheduled-jobs/new — create form; preset picker fed by GET /api/jobs/presets, with input fields rendered from each preset’s inputs schema (status: 'blocked' presets shown as unavailable).
• /scheduled-jobs/$name — detail; schedule config plus the Jobs this schedule has materialized (via the ?scheduleName= filter), and delete.

SDK (JobClient): createSchedule, listSchedules, getSchedule, deleteSchedule, runScheduleNow, listPresets, list({ scheduleName }). Pause/edit of an existing schedule is not yet implemented (edit = delete + recreate).

Monitoring

GET /api/jobs/queue-stats — admin-only endpoint (gated by the ADMIN_EMAILS allowlist; 403 for everyone else) that returns live queue depth plus a 24-hour rollup per queue.

Response shape: { success: true, data: QueueStats[] } where each QueueStats has queue, pending, active, completed24h, failed24h, failureRate24h, avgDurationMs24h. SDK: JobClient.getQueueStats().

Port IJobQueueStats (packages/platform-domain/src/jobs/ports/IJobQueueStats.ts) keeps the domain infra-free. Adapters: PgBossQueueStats (reads pgboss.job + pgboss.archive read-only via Drizzle raw SQL; silently returns [] if the schema is not yet migrated) and InMemoryQueueStats (always [], used when pg-boss is unavailable). The dashboard home renders these stats as a “Queue health” card; a 403 response hides the card without breaking the page.

No ADMIN_EMAILS (and no NOTIFICATION_EMAIL fallback) set → gate denies everyone (403) → card hidden. The pgboss schema is runtime-managed by pg-boss — never add a drizzle migration for it; this adapter only reads it.

End-to-end UAT

Two scripts target the runtime path:

apps/api/scripts/uat-jobs.ts posts a single-step read job (read_gmail / read_classifieds / read_drive by --source) and polls until terminal. Use after touching the read/runner/registry path.

# Requires API on :3000 and a renewable session (~/.platform-cli/cookies.txt).
# Auto-renews via apps/api/scripts/renew-session.ts on 401.
bun run api:uat                                  # gmail, last 30 days
bun run api:uat --source gmail --days 7
bun run api:uat --api-url http://localhost:3000 --timeout 60
bun run api:uat --no-poll                        # submit and exit

Exit codes: 0 on completed, 1 on failed/timeout, 2 on env errors (API unreachable, renew failure, bad args).

Open work

Genuinely open product gaps — not structural debt:

• analyze needs a connected LLM channel. analyze is always registered but resolves its IContentEnricher per run from the owner’s llm:// channel; any job whose steps include it (today only the DEFAULT_STEPS fallback used when NO preset name is given) fails at that step with no channel connected.
• Schedule pause/edit not implemented. The dashboard scheduling UI (list/create/detail, tasks/scheduled-jobs-ui.md) shipped, but editing an existing schedule means delete + recreate; there’s no pause toggle yet.
• No timezone-aware scheduling. pg-boss schedules fire in UTC. Add a timezone field to JobSchedule only when a user asks.
• Multi-instance scheduler scaling. pg-boss dispatches scheduled ticks from the singleton API instance that holds the boss connection — fine today. If we scale the dispatcher horizontally, the overlap-guard race window widens; revisit with an advisory lock keyed on scheduleName.
• No domain event dispatch. JobEventBus was removed in Phase G (it had zero subscribers). When jobs need to publish lifecycle events (e.g. for the activity log), add an IEventDispatcher port + adapter rather than a private pub/sub class.
• Credit/gamification not gated. Running a Gmail → enrich → Notion job today (whether from veilleToBoard-style presets or the custom-flow composer) consumes LLM + Notion API quota without any credit-deduction check. When credit pricing is agreed, wire creditService.canExecuteTrade() into CreateJobUsecase.