Automation architecture

Two engines run work, and they are split on the one axis that matters: where the work executes.

• The workflow engine (packages/platform-domain/src/workflows/, apps/api/workflow/) — an n8n-style graph of nodes on an infinite canvas (/workflows), with one universal item shape on every edge, named outputs, triggers drawn as nodes, and a per-node run view. Everything the platform executes itself runs here: crons, event triggers, the Run button, and the one-off internal chains.
• The runs control plane (apps/api/runs/, apps/cli/commands/agent/) — work executed on the USER’s machine. The platform runs nothing: it queues a runs row, viite agent watch claims it under a lease, and streams a transcript back. Its design record is tasks/spike-coding-cards.md.

There is no longer a third engine. The Job engine — Job<T>, processJob, two runners, the job-workflow queue, /api/jobs/* and both tables — is deleted. Its two capabilities already existed as nodes, it ran the same step instances through the same AI gateway, and it survived only because three dashboard call sites still POSTed to it. Those three now enqueue ad-hoc workflow runs; see “One-off runs” in Part 1. The decision record is tasks/spike-reduce-engines.md.

When touching a node, this doc is the reference — do not re-derive it from code.

Part 1 — The workflow engine

Why a second engine, not an extension

The old engine’s two hardest problems were artefacts of its own shape system. Steps communicated through a shared mutable WorkflowContext — items plus a metadata bag of side channels (enrich.rows, read.sourceMessageIdsByAdapter, tagExclusions) that no edge could draw and no validator could derive rules from. That worked for a straight line and cannot work for a graph: two branches sharing one mutable context clobber each other, and a side channel is invisible on a canvas. The rewrite deletes the shape system rather than generalizing it.

One item on every edge

interface WorkflowItem {
    /** Identity in the source system — Gmail message id, Notion page id. */
    externalId?: string;
    /** Where it came from, for provenance and tag-style write-backs. */
    source?: { scheme: string; host?: string };
    /** The record. What conditions read, what expressions address. */
    json: Record<string, unknown>;
}

ReadItemRef and DispatchRow collapse into conventional keys inside json (title, url, summary, tags, rawContent, …). Any node can connect to any node; validity becomes “does this node find the fields it needs?”, answered at run time with the missing field named — the n8n trade, taken deliberately, and the reason the run view has to be good. Source readers flatten their metadata alongside the envelope fields (a condition reads {{ $json.subject }}, not a nested bag), writing url/contentType last so a source can never shadow them.

Graph, nodes, named outputs

interface WorkflowNode { id; type; name; params; position; runOnFailure? }
interface WorkflowEdge { from: { node, output }; to: { node, input } }

A node implements INode.execute(items, params, context) → NodeResult, whose outputs is keyed by output name — { main: [...] } for most, { true: [...], false: [...] } for the If node, { actionable, dismissed } for triage. Named outputs are what made If/Else expressible at all, and they replace every side channel: what a step could only count, a node routes. enrich emits main/declined, analyze main/dropped, triage actionable/dismissed, collect_invoices invoices/rejected, best_of_tech main/picks. An item a node could NOT judge (a thrown call, a degraded decider) leaves on neither output — it reaches no downstream tag, stays unlabelled at the source, and is re-read next run. That is the old tagExclusions retry contract, structural instead of a set of ids in a bag.

A NodeResult also carries halt (success: true = clean early stop of this branch; false = fail the run), usage (LLM spend, reported not appended — the old slice arithmetic charged one fork for the other’s tokens), watermarks (cursor advances, committed only if the whole run succeeds), incomplete (a cap bit, so a partial run cannot pass for a complete one) and signalSamples (per-item condition verdicts for the run view).

Node descriptors (NodeDescriptor) are the design-time half: category (trigger/ingest/enrich/decide/act), mode (deterministic/ai/hybrid — drives the credits gate and the badge), outputs, inputs (an InputFieldSchema map the inspector renders), emits (declared json keys, the field picker’s pre-first-run half), skillable.

The executor — runWorkflow (apps/api/workflow/engine/)

Kahn-ordered walk with three load-bearing semantics:

• A node with no live input is SKIPPED, not run empty. Run it empty and a dispatch on a condition’s untaken branch logs “no rows”, returns success, and every branching run reports work it never did — the silent-green-run failure the old shape system existed to prevent. skipped is a first-class outcome (never rendered red), with a recorded skipReason (no_input / upstream_skipped / upstream_failed / branch_halted / cancelled). Consequence worth teaching users: an edge means “and use what came out”, not merely “and then” — a self-sourcing node chained after an empty read skips; to say “do both”, fan out from one parent.
• runOnFailure closers run whenever the pass did not reach them. A node flagged runOnFailure owns a subject’s terminal write (update_link, update_contact). It executes with an empty batch and context.runError set in TWO cases: after a failure (the run stays failed), and at the end of a completed run that skipped it for no_input / upstream_skipped — the case a green run used to strand: read_url RAN and fetched nothing, enrich skipped, the closer skipped behind it, and the link sat on enriching with a 142 ms completed run behind it. The runError then carries the nearest empty-handed ancestor’s own sentence (“Fetch the page: Fetched 0 pages (1 failed: https://…)”), so the card names the cause. A closer behind a deliberately halted branch stays skipped — the halt already gave its verdict. Its skipped row is replaced by the ran row, so the run view shows one row that says it happened.
• Deferred effects. Usage is charged per node as it completes; watermarks accumulate and commit only on a fully successful run, so a mid-graph failure leaves the source cursor untouched and the batch is retried. Idempotency across those retries is the import ledger (below), backed by the source’s own guard where it has one (Gmail’s -label:<processed> exclusion).

The NodeRegistry maps type → executor, fail-fast like the old StepRegistry: an unregistered type stops the run rather than skipping. Registration is environment gating (buildNodeRegistry in apps/api/workflow/buildWorkflows.ts): a node whose integration is not configured is never registered, never offered by GET /api/workflows/nodes, and the canvas cannot draw it. The work nodes share the SAME dependencies the old steps use — one ingestion API, one watermark store, one connector registry, one decider factory, one task repository, one dispatch target map, one digest reader set — handed over by buildWork, so the two engines cannot disagree about credentials, cursors, or rows.

Triggers — the graph is the binding

Three kinds, one dumb TriggerNode (trigger.manual / trigger.schedule / trigger.event): a trigger just forwards its seed items; the difference is entirely in how each FIRES. resolveWorkflowTriggers(graph) reads the binding off the graph — no subscription table to drift from the canvas, which is the failure job_schedules.templateError had to exist for. A cron lives on the node it starts.

Event triggers ride the in-process event bus: WorkflowTriggerDispatcher guards with isDomainEventType (the bus also carries per-request activity events), looks up listeners via a GIN-indexed jsonb containment query that is only ever a pre-filter (survivors are re-checked with the same resolveWorkflowTriggers the canvas uses), and hands each run to a durable enqueuer — the bus is not durable across replicas. The event vocabulary (task.created, task.status_changed, link.captured, …) and each event’s declared payload fields are exposed at GET /api/workflows/event-types, feeding the condition builder’s dropdown.

Saving a broken graph is legal — work in progress must be storable. Only activation validates: resolveWorkflowGraph (cycles, dangling endpoints, duplicate ids/names) must pass and at least one trigger must exist. Create is never active, whatever the body asks — drawing a workflow is not consent to run it.

Conditions and expressions

interface Condition { left: string; operator: Operator; right?: unknown }
interface ConditionGroup { combinator: 'and' | 'or'; conditions: Condition[] }

Thirteen operators (is, contains, gt, matches, …), case/accent-folded string matching lifted from invoiceGate, numeric coercion for form-borne strings, and — the part users trust — a signals audit trail recording which predicate fired per item, persisted per run and rendered by the run view as “why did this item go down the false branch”.

{{ $json.field }} addresses the current item; {{ $node['Name'].json.field }} a named upstream node’s output. Resolved per item at run time — deliberately a different syntax from the ${input:}/${env:}/${setting:}/${skill:} family, which resolves once at job creation in the old engine. Same-looking, different time, different scope.

The node vocabulary (24 types)

Node	Category · mode	Outputs	Notes / gating
trigger.manual / .schedule / .event	trigger · det	main	always registered; .schedule takes cron, .event takes eventType
if	decide · det	true / false	one condition group per node; the only node holding conditions
read_gmail / read_drive	ingest · det	main	share performRead; watermark scoped to the workflow id; items carry externalId = provider message id
read_url	ingest · det	main	second-hop page fetch; canonical-URL dedupe, boilerplate/thin-page drops, all counted
read_notion	ingest · det	main	one shape now — the step used to write items AND rows for the same records
agent	enrich · ai	main	tool-loop (scrape + create_task); needs channels
enrich	enrich · ai	main / declined	rows as items; a FAILURE leaves on neither output (retry by construction)
analyze	enrich · ai	main / dropped	partitions instead of dropping — a dropped item can finally be tagged, ending the re-read-forever bug
triage	decide · ai	actionable / dismissed	cards created in-node (idempotent manual:<userId>:<messageId>), every terminal verdict written to the import ledger; verdict rides the items’ json (taskId, triageTitle, …); fail-soft placeholder + 5-consecutive-failure halt shared with the old step
collect_invoices	decide · hybrid	invoices / rejected	free gate first, LLM only for the ambiguous band; gate-only without channels
best_of_tech	ingest · hybrid	main / picks	files the digest card; picks is dispatch-shaped; the brief param opts into page-grounded briefs (cost, never shape); picks: "showhn:3, github:2" sets a per-board quota (each board its own shortlist, a board the model shorts filled on score and reported unjudged, ≤ 10 total, output in quota order); picks never repeat a story: before shortlisting, every scored entry whose storyKey(url) matches the sourceUrl of one of the owner’s 500 most recently touched post rows (any status, archived included) is dropped, and picks within one run are distinct by storyKey across boards; main reports alreadyPosted
dispatch	act · det	main	routes by destination URI scheme to the shared target map; passes input through so tag can follow; autoPublish (post destinations only) resolves publishTimes in tz to upcoming slots (upcomingPublishSlots: 12am = midnight, a passed time = tomorrow) and hands them over as publishSlots — the drafter arms each NEW post reviewed + dueDate once the destination’s full rules pass; a skipped source or a rule-breaking post takes no slot, posts past the slots stay drafts
tag	act · det	main	labels the messages that reached it (ids off the edge, not a bag) — wiring both verdict outputs of a filter into one tag is the old flag-once behaviour, drawn
update_link	act · det	main	writes the first input row back onto the link card; flag runOnFailure so an upstream abort — or a completed run that never reached it — marks it failed with the real reason
find_linkedin_profile	ingest · det	main / declined	the contact acquisition run’s first hop: a web search restricted to linkedin.com/in (keyless DuckDuckGo behind the IWebSearch port), the top profile read best-effort through the scraper; declined = not found or search unavailable, still carrying the card so the next hop can work from its notes
enrich_contact	enrich · ai	main / declined	one JSON answer per person — name parts, company, role, location, profile (only a candidate URL), a summary and the category tags to file them under; a thrown call leaves on neither, all-thrown halts
update_contact	act · det	main	writes findings onto the contact card filling empty fields only (typed beats found), merges tags, sets enriched/failed; flag runOnFailure like update_link (runs on failure AND on a completed run that skipped it)
read_tasks	ingest · det	main	the AI-slop cleanup run’s first hop: searches the board, or — when an upstream task.* event named cards by id — hydrates exactly those instead; drops cards already carrying skipTag (writing:deslopped) before they reach the model
remove_ai_slop	enrich · ai	main / clean / declined	runs the deterministic detectSlop scan first — zero warnings leaves on clean with no model call and no channel required; a scored field is rewritten and judged by judgeRewrite (facts survived, length sane, warnings did not rise); a judged-bad answer leaves on declined with the reason, a thrown call leaves on neither (retried next run)
update_task	act · det	main	writes the rewritten field back onto the card and tags it (writing:deslopped / writing:deslop-declined) so the same card is judged once; the body cap is enforced with the same constraintsFor(socialPlatformForHost(...)) rule a hand edit is held to
log_activity	act · det	main	appends the first input’s text (or the field named by bodyKey) to a CRM card’s timeline (task_activities) as an agent-authored entry of kind (default system); the domain validator decides — an empty human-kind body or an over-long one halts the node rather than landing a broken line; flag runOnFailure so a halted upstream still leaves a line naming the reason

watch and the classifieds source are deleted, not ported — the generic If node over a time-series read replaces the bespoke comparison (apps/api/observation/ stays; social_audience writes through it).

The import ledger — what stops a message being imported twice

workflow_imports (IWorkflowImportLedger), keyed (org_id, workflow_id, source, external_id) with the uniqueness enforced by the index rather than a read-then-write check, so two concurrent ticks cannot both pass.

It exists because the two older guards each have a blind spot. A source-side exclusion (-label:<processed>) lives in a field the user owns: editing it discards every “already handled” mark at once, and since a label-driven Gmail read deliberately drops the watermark’s after: bound (runRead’s labelIsGuard), the next run re-reads a full unbounded page — capped at 100 — of long-settled mail. tasks.idempotency_key only holds while a card does: a dismissal writes no card to collide with, and a pruned, deleted or converted card releases its key. Together those two turned one label edit into ~100 duplicate cards.

Three rules make it work:

• Every TERMINAL verdict is recorded — actionable and dismissed. The dismissal is the point: it produces no card, so without a row it was invisible to every other guard and got re-judged by a non-deterministic decider on each re-read.
• An UNJUDGED item is never recorded. A degraded decider or a thrown call leaves the item on neither output, unlabelled at the source, and retried next run — recording it would make that retry impossible. Same reason collect_invoices does not record an ambiguous message it had no tiebreak for: freezing a non-decision would mean connecting a channel later never revisits it.
• A recognised item is REPLAYED, not re-decided — onto the output it earned the first time, so a downstream dispatch never acts on a verdict nobody made, and it still reaches tag, which is how the source exclusion re-converges on the label the user just changed to.

It back-fills itself rather than by migration: an existing card found by the idempotency check is written to the ledger as it is skipped, so the corpus is healed by the runs that touch it. Cards predate the ledger and nothing records which workflow imported them, so a migration could only have guessed.

workflow_id is in the key deliberately. Two workflows may legitimately read one mailbox — the triage flow and the invoice collector both do — and an org-wide key would let whichever ran first starve the other. This does not replace tasks.idempotency_key, which stays user-scoped and is stricter in its own direction (one card per user per message, across every workflow); re-scoping that key to org+workflow would permit cross-workflow duplicates that are impossible today.

The exclusion and the tag that satisfies it are a pair. read_gmail’s excludeLabel and the downstream tag’s label are independent free-text fields, and setting them differently leaves a flow that never converges: every run re-fetches the same page, hands it to a node that has already handled it, and applies a label the query does not exclude. The ledger stops that from creating duplicates but cannot make the query converge. findGmailLabelMismatches (domain) reports the mismatch — on the canvas as you edit, and as a drift-lock over the shipped templates. It is reported and never auto-corrected: rewriting one node because another changed is an edit the user did not make, on a field only they know the right value for.

Calendar reconciliation — how a deleted event reaches the board

calendar_to_tasks is the one ingest node that retires what it created. Everything else in the tree is create-only plus an idempotency skip; this one keeps a card in step with a source that can change under it.

A windowed read cannot observe a deletion. events.list over [now, now+N) simply stops returning a deleted event, which is indistinguishable from one that moved out of the window, and updatedMin never returns tombstones at all. Google’s answer is syncToken: the first call mints one, every later call returns only what changed since — deletions included, as items with status: 'cancelled'. That is the primary read, and edits arrive on the same delta.

Push notifications (events.watch) were considered and deferred. The callback carries no payload — it says “something changed” and you must run the incremental sync anyway — and it additionally needs a Search-Console-verified receiving domain plus channel renewal bookkeeping. It is a latency optimisation on top of the sync, never a substitute for it.

Three cursors, because one read cannot do everything:

• ingestion_sync_cursors holds the per-calendar sync token. A separate table from ingestion_watermarks, not a column on it: that table’s contract is monotonicity (GREATEST on conflict), a token is an opaque string with no ordering, and — unlike a watermark — a token must be clearable, since Google answers 410 GONE once it ages out.
• The :farEdge watermark survives, because a sync token has one blind spot: an event that never changed but scrolled into the window is in no delta, and the initial sync saw it while it was still outside. A warm run therefore also reads [lastFarEdge, now+N) unfiltered.
• The window itself is applied in the step. A sync token may not be paired with timeMin/timeMax, so the delta is unbounded in time; creates are filtered client-side. Edits and deletions are not, because they only ever touch a card that already exists.

Four rules govern the writes:

• Only a todo card is ever touched. Once the user moves a card, it is theirs — Google changes are counted (skippedUserOwned) and ignored. One rule covering edits and deletions alike, and the reason the step needs card ROWS (findByIdempotencyKeys), not merely the set of keys that exist.
• A tombstone is honoured only from the calendar the card records. A meeting on two calendars is deduped to ONE card — same Google event id, and the partial-unique index on idempotency_key permits only one — so the calendar that lost that race must not retire it. metadata.calendar carries the owner. Cards created before that existed carry none; their tombstones are trusted, and a later run re-creating the card under the surviving calendar heals the rare wrong call.
• An unchanged event writes nothing. Not an optimisation: a needless write bumps updated_at on every calendar card every run, and that is the clock the board’s retention prune reads.
• Absence is evidence only after a full sync. A deletion that happened while a token was stale produces no tombstone — the resync just omits the event — so a full sync also sweeps the window for cards whose events are gone. The sweep is scoped to the calendars that actually full-synced this run, since an incremental delta legitimately omits every unchanged event; sweeping on one would archive the board. A card naming no calendar is swept only when every calendar full-synced.

Cursors and tokens advance only after the reads and the writes succeed, so a failed write leaves the delta re-readable rather than persisting a token that would skip it.

onDeleted (step config) chooses archive — the default, which also tags the card calendar:cancelled — or delete.

Not every calendar row is a meeting, and the title cannot tell you which. Google mixes working locations, out-of-office blocks, focus time, birthdays and Gmail-derived entries into the same events.list result. A working-location row in particular has an id, an all-day start and a summary that is the location itself — “Home”, “Office”, a city name — so it lands on the board looking exactly like an event, and filtering it by title means matching user text that is localised and freely collides with a real meeting. The deterministic discriminator is the API’s own eventType field (default · outOfOffice · focusTime · workingLocation · fromGmail · birthday), which CalendarEvent now carries verbatim: platform-ingestion reports the kind and filters nothing, because which kinds are worth a card is the consumer’s policy.

ignoreEventTypes (step config, a multi-enum — checkboxes on the node form) is that policy. default is not offered, since ignoring ordinary events would silently switch the node off. An absent value takes the default ['workingLocation'] — those are exactly the nodes saved before the field existed, and the rule was added for them — while an explicitly emptied list means “ignore nothing”; collapsing the two would make the field impossible to switch off. A preset’s ${input:…} substitution arrives as a comma/newline-separated string and is accepted as one.

A card made from such an entry before it was ignored is retired on the next run and tagged calendar:ignored, counted as retiredIgnored. That is deliberate rather than left to the absence sweep above: the sweep would eventually notice (these events no longer enter liveEventIds), but it fires only on a full sync — weeks late and unpredictably — and would tag the card calendar:cancelled, which is untrue; nothing was cancelled. The todo-only rule still applies, so a card the user has already moved is counted as skippedUserOwned and left alone.

The provenance backfill has two halves, because SQL cannot supply the second one. Cards created before any of this recorded their event id in exactly one place: the idempotency key manual:<user_id>:<event_id>. Migration 20260901130000_backfill_calendar_card_provenance parses that back out into external_id and metadata.calendar.eventId. It cannot recover the CALENDAR id — that was never persisted, and a card’s calendar:<name> tag is a human label, not Google’s opaque id. So the step adopts the calendar id from the first live event it sees for that card, which the first run after deploy does wholesale: no sync tokens exist yet, so every calendar full-syncs and every card whose event is still live is adopted in one pass. Cards whose event is already gone are swept on that same run instead. Until a card is adopted it has no calendar to compare against, and a tombstone naming it is trusted from anywhere — the old behaviour, now confined to rows not yet seen.

Two details in that migration are load-bearing, both verified on a throwaway database. It matches the key prefix with starts_with, not LIKE: a better-auth id is a mixed-case nanoid that may contain _, which LIKE reads as a single-character wildcard — on real seeded rows the LIKE form matched a different user’s key and would have stamped their event id onto the card. And it skips rows whose metadata is not a JSON object, because jsonb_set does not ignore a scalar, it raises cannot set path in scalar and would abort the whole migration over one malformed row. It leaves updated_at alone throughout — a backfill is not the user touching their card.

Runs — what a run leaves behind

workflow_runs + workflow_node_runs (Drizzle, apps/api/workflow/adapters/). Per node: status, skipReason, itemsIn/itemsOut, per-output counts (how a condition split), message, error, incomplete, usage, condition signalSamples, and a hard-capped output sample — at most 3 items AND 8 KB serialized, whichever bites first (capSample), truncation labelled. The cap is not a nicety: persisting whole contexts is what once put 631 KB of scraped bytes into one jsonb row and failed writes on NUL characters. The sample doubles as the field picker’s second source: with one item shape there is no declared schema for everything a node emits, so the designer offers declared emits ∪ last-run keys, in a combobox that still accepts a typed path — both sources only ever describe.

A run exists from the moment it is asked for, not from the moment it ends. It moves queued → running → completed/failed/cancelled, and every one of those is a write to the same row (the header upsert is what makes that one row). This is not bookkeeping: the queue is drained by a single serialized worker, so a manual fire can wait minutes behind the cron backlog, and before this nothing was written until the run FINISHED — for the whole wait the run view had nothing to show and ?run=latest resolved to the PREVIOUS run. An old timestamp sitting still reads as a dead button, and it got clicked again: one workflow collected five stacked manual runs in four minutes.

Node rows follow the same rule. A node is written running before its executor is called and overwritten with its outcome after, so the record always names the step the run is inside — the answer to “where has it got to”, which for an LLM or a scrape is the only thing moving. Progress writes are swallowed on failure: a courtesy write must never abort a run that is doing chargeable work.

Two fields exist for the person reading a failure. trigger (manual/schedule/event) says whether the cron did this or you did — a column that existed since the table did and went unwritten until it became load-bearing for queue position. And a failed node may carry a fix ({ label, href }), declared by the node that knows the cause, rendered as a link under the message; see “Errors that name a remedy” below.

The run list is deliberately lean (headers only); opening a run is its own request (GET /:id/runs/:runId), because node rows carry the samples. A queued run additionally carries queuePosition — how many runs will be picked up first — computed only for that state and never stored, because it is true only at the instant it is read.

Templates — the ex-presets as graphs

WORKFLOW_TEMPLATES (packages/platform-domain/src/workflows/entities/WorkflowTemplates.ts): the ex-presets whose steps are all native — inbox-links-to-board, newsletters-to-social-drafts, gmail-triage, collect-invoices, daily-tech-digest, research-agent — plus graphs with no preset ancestor: the AI-team pair, clean-ai-slop-drafts, tech-autopilot (3 Show HN + 2 GitHub stories scheduled to X at five times a day, unreviewed) and publish-scheduled-posts (*/15 → publish → post_comment, the Publish-now graph on a clock). The second is what makes the first publish: nothing else sweeps a post whose dueDate is in the future. A catalog in code, not seeded rows — rows need an owner, a migration and an update story; a catalog versions with the nodes it names. “Seeded” happens at clone time: GET /api/workflows/templates serves the catalog filtered to what the deployment registered (same gating as the palette), and “Use template” on /workflows is the ordinary create route handed the template’s graph. Two translation rules: ${input:x} placeholders became baked params (personal values left blank for the inspector), and the side channels became drawn edges (declined/dismissed/rejected → tag). Drift is locked from both layers: a domain test resolves every graph and checks each edge against real outputs; an API test builds the full registry and asserts every template type registers.

HTTP surface and SDK

Mounted at /api/workflows (createWorkflowsApp); static paths registered before /:id:

Method	Path
GET	/nodes	registered descriptors — the palette
GET	/event-types	event catalog + declared payload fields
GET	/templates	clone-able starting points, registry-gated
POST / GET / GET /:id / PATCH /:id / DELETE /:id		CRUD; PATCH validates graph + trigger presence only when active: true
GET	/:id/runs	run headers, newest first (limit 25)
GET	/:id/runs/latest	most recent completed run, nodes included — the field picker’s sample source
GET	/:id/runs/:runId	one run with node rows — what the run view opens

SDK: WorkflowClient (platformClient.workflows.*) — list/getById/create/update/delete, listNodes, listEventTypes, listTemplates, listRuns, latestRun, getRun. Types come from the domain barrel; the SDK redefines nothing.

The canvas (apps/dashboard/src/routes/workflows*, components/workflow/)

React Flow supplies the generic hard parts (viewport, edge routing, handle hit-testing); domain stays local. Connection legality is structural only — a cycle-closing edge refuses to attach while being dragged. Selected node, viewed run, and expanded run-node all live in search params (?node=, ?run=), so any state is a URL. Saving is explicit. The NodeInspector renders three tiers: bespoke editors (condition rows on if only, the event picker, read_gmail’s nested filter — see components/workflow/nodeParams/), the declared inputs schema (with format: 'skill' → the real SkillPicker and format: 'connector' → a channel pin or destination picker), and an “Advanced — raw params” JSON disclosure — a node accepts keys no schema declares, and templates bake them. A dropped node arrives pre-filled from declared defaults (defaultsFromSchema), not {}. The run view (Runs button) lists runs, shows per-node rows with per-output splits, skip reasons in words, condition signals and the capped sample; canvas nodes wear count chips while a run is viewed.

A skill param stores the skill’s ID, not its slug. skills is unique on (user_id, slug) and resolveSkillSteering looks a ref up owner-scoped (findById first, then findBySlug), so a stored slug names whichever skill the WORKFLOW’S OWNER happens to have under it — two accounts each owning urls-to-links gave two live workflows the same ref and two different rubrics, with nothing in the graph, the logs or the run view saying which. An id (skill_<hex>) is global, so it names one row wherever it is read. Slug refs still resolve (old graphs, and ${skill:<id|slug>} in a step config), and 20260829170000_skill_refs_to_ids rewrote the stored ones to the id each owner already resolved — a slug naming nothing for that owner was left alone rather than guessed at. Two consequences worth keeping: a template must bake no skill ref (WorkflowTemplates.test.ts enforces it — “Use template” copies params verbatim, so a per-owner ref would follow the clone to someone else’s rubric or to none), and every skillable node resolves through the ONE shared resolver, which is what makes the @<version> pin and the kind guard apply to all three rather than to whichever node last had them written out.

The run transport — how a drawn workflow actually fires

Every fire path lands on the dedicated workflow-run pg-boss queue (mirroring webhook-delivery) and executes through one seam, executeWorkflowRun: load the workflow fresh (a graph edited between a cron registration and its tick runs the current graph, never a payload snapshot — the drift job_schedules templates suffered from), guard, runWorkflow, persist the run. The worker runs batchSize: 1 — serialization is a property here, not a limitation (shared services, a credits balance, source cursors).

Sends carry retryLimit: 0, passed explicitly. A run is not blindly idempotent (LLM spend is real money out) and failures belong in the run history where the run view explains them — but for a long time this doc and the worker both said “no retry” while setting nothing, and pg-boss defaults retry_limit to 2. Saying nothing bought the opposite of what the sentence claimed: a run that hung past the 900 s expiry came back twice to hang again, so one stuck node could hold the queue for 45 minutes and bill three times for the work it did first. Alongside it, each node runs under a deadline derived from that same expiry rather than picked (NODE_DEADLINE_MS, one minute inside it) — so it can only ever end runs pg-boss was already killing, one minute earlier and with a record naming the node that stopped. The losing promise cannot be cancelled, so what it reliably frees is the QUEUE, not the node.

Stopping a run

Nothing on the platform can be killed: a run is one promise inside the API process, the model call inside it is a fetch, and a JS promise cannot be interrupted. So a stop is cooperative, in the same shape the coding-card cancel uses (settle the row, let the worker discover it):

• POST /api/workflows/:id/runs/:runId/cancel (SDK cancelRun, the Stop button beside Run) is one conditional write — queued | running → cancelled, with the reason as error and finishedAt now. 409 when there was nothing to stop: the run finished on its own first, and a cancelled row over a completed one would be a lie. A card-level stop (POST /api/tasks/:id/process/cancel, the Stop beside Re-enrich) does the same write against the internal chain’s run first, then files the card as failed with “Stopped from the board.” — in that order, so the closer cannot overwrite the card afterwards.
• The executor asks between nodes (checkCancelled, one primary-key read via peekStatus) and polls while a node works (CANCEL_POLL_MS, 3 s), racing the poll against the node the same way the deadline is raced. Winning aborts an AbortController whose signal rides NodeRunContext.signal → the AI gateway binding (ResolveAiInput.signal, stamped on every chat/reason the binding makes) → LlmOptions.signal / AgentRequest.signal → the Anthropic, Cline and Vercel adapters. A model call therefore returns within one round-trip of the click, not at the 14-minute deadline. What the signal does NOT reach: a scraper’s own timeout, a CDP session — those finish late, never wrong, because the engine records the node as skipped-cancelled regardless.
• A queued run stopped before pick-up is refused by executeWorkflowRun on the same peek, and the pg-boss job completes without executing. A stopped run runs no compensation (whoever stopped it already wrote what the subject should say) and commits no watermarks (the work is unfinished and must be re-read). Every node the loop never reached is recorded skipped-cancelled so the run view shows the whole graph.
• save never downgrades a cancelled row (a CASE on the upsert for status, error, finished_at): the executor’s progress writes are already in flight when a stop lands, and the last of them must not put the run back to running — or close it as completed over the reason. Node rows still replace, so the view shows where it stopped.

Order within the queue is not first-come: a manual fire is sent with a pg-boss priority so a person’s click is picked up ahead of a cron backlog. Ordering only — serialization is untouched. The trade is deliberate and worth stating: a cron tick can now wait behind an ad-hoc click. Crons are not latency-sensitive and already skip when they overlap; a person staring at a button is.

Without pg-boss the enqueuer degrades to inline next-tick execution.

No transport can enqueue an invisible run. withQueuedRun wraps the dispatch rather than each call site, so the route, the event dispatcher and the internal ad-hoc chains all write the queued row before the message goes on the queue. A pg-boss CRON is the one fire that cannot go through it — pg-boss creates that job itself from a payload stored at registration — and it is also the one nobody is watching a screen for; it becomes visible when the worker picks it up. Do not “fix” that asymmetry by putting a runId in the cron payload: the payload is replayed every tick, so all of that workflow’s runs would upsert one row.

Per trigger kind (buildWorkflows’ runtime block wires all three at boot):

• Event — WorkflowTriggerDispatcher subscribes to the domain event bus; its store re-derives bindings from each active graph via findActiveByEventType + resolveWorkflowTriggers, so the graph stays the only binding. The event’s payload seeds the trigger node.
• Schedule — BossWorkflowScheduler registers one cron per trigger.schedule node, keyed wf.<workflowId>.<nodeId> on the shared queue (the same key mechanism PgBossJobRunner uses). PATCH /:id re-syncs on every update (activate registers, deactivate/edit re-derives), DELETE sweeps, and boot reconcile diffs the wf.-prefixed keys against listActiveScheduled() — only our prefix is touched. Guards mirror the old fire path: a queued tick on a deactivated workflow is skipped, and skip-if-previous-still-running holds for schedule fires via runs.hasRunning — age-bounded, and that bound is load-bearing: a worker killed mid-run leaves a row claiming to be in flight forever, and an unbounded check would then skip every cron tick of that workflow until someone noticed by hand. Rows older than the queue’s own expiry are leftovers, not runs; boot sweeps them to failed with a sentence saying the server restarted.
• Manual — POST /api/workflows/:id/run (SDK run(), the canvas Run button): the documented bypass, exactly as POST /schedules/:name/run was — no active requirement (testing a drawn workflow is the point), no overlap skip, 202 + the run view. Seeds every trigger node with one { firedAt } item; an unseeded trigger emits nothing and its branch skips quietly. The 202 carries the runId, so the canvas opens that run rather than asking for the latest. A second click while one is still pending returns the SAME run (alreadyPending) instead of stacking a duplicate — not the overlap policy in disguise (that one is about two SCHEDULED fires racing an export target), but about a person who could not see the first click land.

Metering adapts the jobs credits meter per node: isMetered reads the registry descriptor’s mode === 'ai', charges land per node under the run id. Watermarks commit through the shared store, scoped by the run’s own user.

Org scope. A run’s orgId is the message’s, else the workflow row’s (stamped from the session’s active org at POST / creation), else resolved at fire time from the owner via resolveOrgId → resolveActiveOrg — the same run-time resolution processJob does on the job engine. The fallback is load-bearing: rows created before the stamp and every internal-chain message carry no org, and without it org-guarded nodes (best_of_tech, triage, collect_invoices, the step-backed sinks) halt while org-scoped connector lookups quietly resolve to nothing.

One-off runs — the ad-hoc chains

A one-off carries its graph in the message (WorkflowRunMessage.graph) instead of pointing at a workflows row. It saves under a synthetic internal:<kind>:<id> workflow id — the column has no FK, by design — and never appears on the canvas, because these chains are per-subject: a saved workflow would be either wrong or one-per-card. apps/api/workflow/internalChains.ts builds them.

Chain	Id	Nodes
Link enrichment	internal:link-enrichment:<artifactId>	read_url → enrich → update_link
Contact enrichment	internal:contact-enrichment:<taskId>	find_linkedin_profile → enrich_contact → update_contact
Speech	internal:tts:<taskId>	tts
Publish now	internal:publish-now:<channel>	publish → post_comment (post), publish alone (email)
Run agent	internal:implement:<taskId>	implement_task
Zen prompt	internal:zen:<promptId>	agent, tool-less
Suggest next step (POST /api/tasks/:id/next-action, CRM cards only)	internal:next-action:<taskId>	agent (tool-less, one step, over the card + its last 10 timeline entries rendered by renderNextActionContext) → log_activity (runOnFailure, so an unreachable model still leaves a line on the timeline saying why)

They drain from a SECOND queue, workflow-run-adhoc, with its own batchSize: 1 worker. Both queues drain one run at a time; the split is what stops a click queueing behind a cron. MANUAL_RUN_PRIORITY only reorders the pending set — it cannot preempt a node already running, which may hold its queue for the full 14-minute deadline, while the publish watcher gives up after 60 seconds. createBossWorkflowRunEnqueuer picks the queue from the run’s id, so no caller can put a one-off on the wrong one, and it throws if a message’s graph and id disagree.

countAhead (the run view’s queue position) is scoped to the run’s own queue for the same reason: counting the other queue’s backlog would measure the wrong wait.

No overlap guard applies to an ad-hoc run. Two Publish-now clicks both run; what keeps them from double-publishing is PublishNode’s atomic reviewed → posting claim, one row at a time. Do not add a chain-level lock — it would turn a safe overlap into a refused click.

Where the outcome lands. Link, contact, speech, Publish now and Run agent all report on the CARD; only the zen prompt reads its answer back out of the run, from the agent node’s output sample. That is why the zen node raises sampleMaxBytes to 64 KB: capSample does not truncate an item that alone exceeds the cap, it DROPS it, so an oversized answer would come back as an empty sample and the surface would show nothing. The field is stripped when a graph is saved from the canvas — a saved workflow able to widen its own samples could write unbounded rows into workflow_node_runs.

Errors that name a remedy

A run’s failure used to name a condition in the code’s own vocabulary — best_of_tech requires an organisation scope — and leave the reader nowhere to go. A halt may now carry fix: { label, href } (NodeFix), which runWorkflow copies onto the node record and the run view renders as a link under the message.

The rule that makes it safe: the node that knows the cause declares the cure. Nothing downstream pattern-matches a sentence to decide what to offer. That coupling has already cost us once outside this subsystem — SocialTokenManager.notConnectedMessage ends with an instruction and lib/publish.ts keeps a SELF_EXPLANATORY regex matching that literal to suppress its own, so a reworded sentence in one app printed two contradicting instructions in the other.

Three things that are easy to get wrong here, each learned from production:

• Fix the message where it actually fires. Three org-scope halts were reworded by hand first; prod then showed that of 34 failed runs, 32 were this guard across five node types, and the three chosen accounted for three of them while capture_comments — responsible for 26 — still said requires an orgId. The sentence now lives once (ORG_SCOPE_MESSAGE + ORG_SCOPE_FIX), so improving it improves all seven. Two spellings of one condition (requires an orgId vs requires an organisation scope) were the tell that per-site wording drifts.
• The self-sourcing sinks say it through their base. capture_comments, youtube_to_tasks, calendar_to_tasks, social_audience and tts extend SelfSourcingNode, whose fail(message, fix) is the only way they report a failure — so a remedy cannot be dropped by forgetting to carry it. SelfSourcingNode.test.ts pins that.
• Only offer a page that can help. A dispatch failure points at /channels only when the destination HAS one — gated on isInternalScheme, since dashboard/sales/link write to the platform’s own tables. Prod hit exactly the other case: Target 'dashboard' failed on a task INSERT, where no amount of reconnecting helps. An unregistered node type is given no fix at all, deliberately — it is a deployment gap, and inventing a link is worse than saying plainly that it is not on the user.

A failure must report what the database actually said. describeError flattens a DrizzleQueryError — whose own message is the generic “Failed query: …” with the reason on cause — and every node catch site goes through it. Five had drifted back to error.message alone, which is what left a prod failure saying nothing but the SQL echoed back and turned a one-line diagnosis into an investigation. Grep for error instanceof Error ? error.message before adding a catch.

Text reaching the database is sanitised at the write boundary. A JS string is UTF-16 and can hold half an astral character; a title cut to a length limit mid-emoji leaves a lone surrogate, Bun’s driver writes 0xef 0xbf, and Postgres rejects the statement with 22021 invalid byte sequence — losing the whole batch over one card. toStorableText (applied in DrizzleTaskRepository) repairs unpaired surrogates and drops NULs. It belongs at the boundary, not per producer: text arrives from scrapers, LLMs, Gmail, Notion, YouTube and whatever is added next, and there is one database.

The schedules migration — done, and deleted

migrateJobSchedulesToWorkflows ran at boot after the cron reconcile and translated each job_schedules row into a graph: the RESOLVED chain (preset expanded, ${input:x} substituted from the stored inputs) became a workflow created active under the schedule’s own slug, its cron synced, and the old row then disabled through the same usecase the pause button used — so exactly one engine fired each automation from that boot on. The slug was the idempotency key.

It has finished and the code is gone. Every row migrated (prod: 14 rows, none enabled, 11 of them preset-bearing and all disabled), and the preset expansion it depended on no longer exists, so a second pass could only ever have been a no-op holding a dead concept up. What survives in apps/api/workflow/systemWorkflows.ts is ensureSystemWorkflows, which never touched presets: it seeds the system automation (recurring invoicing) on a fresh environment — one that never had the job_schedules row to convert — and re-syncs its cron on every boot.

The disabled rows are left in place. Nothing reads them; dropping the table is a separate change.

Part 2 — Publishing, and the boundary

The publish worker sees only ONE of a post’s two arms

A post row carries a destination strategy in Task.destination.scheme, and it is the whole difference between a post Studio publishes and one the user publishes themselves:

arm	destination.scheme	who posts it	needs a connector
automated	automated	the publish worker, through the network’s API	yes
manual	manual	the USER, from the network’s own composer, prefilled	no

PublishNode.findDue selects on destinationScheme = 'automated' — an exact match in both findDueForPublishing and findAwaitingComment. So a manual row is unreachable by the publisher structurally: there is no opt-out flag to invert and no branch to forget, which is why the manual arm is expressed as this field rather than as a boolean beside it. apps/api/publishing/ contains no manual-arm code at all, and PublishNode.test.ts asserts that by publishing an automated row and a manual twin in one pass.

The manual arm is why the prefilled-composer mode exists: it is the only mode available on a network whose write API we may not use for this (X — the Developer Policy’s Pay to engage clause governs use of the API, so a composer the user submits themselves is outside it) or cannot get approved for (Meta Advanced Access, LinkedIn Community Management). It carries no ToS exposure and stores no credential.

Two statuses belong to it, and selectableStatuses(entity, arm) keeps each arm out of the other’s:

• handed_off — the composer was opened with this text. Nobody has confirmed it went out and nobody can; there is no receipt to poll for.
• posted_by_hand — terminal success as asserted by the operator.

posted_by_hand is permanently distinct from posted, and must stay so: posted means an API accepted the post and handed back an id, this means a person said so. Anything that reconciles, audits or reports on publishing has to be able to tell them apart. Conversely reviewed — the publish TRIGGER — is refused on a manual row, where it would sit forever meaning nothing while reading on the card exactly like a row about to go out.

The transport is composerUrlFor (packages/platform-domain/src/tasks/destinations/composerUrl.ts), which returns the deep link and the list of fields it cannot carry. No composer accepts media and most accept only a body, so a builder returning a bare URL would silently drop an attached image; a URL over MAX_COMPOSER_URL_LENGTH returns no link at all rather than one the platform would truncate. These are undocumented third-party endpoints — verify them against the live composer before trusting the table in that file.

Monitoring

GET /api/workflows/queue-stats — admin-only (ADMIN_EMAILS), live depth + 24 h rollup per queue via IQueueStats (the runtime-managed pgboss schema is read-only — never add a drizzle migration for it). It reports on workflow-run, workflow-run-adhoc and webhook-delivery. It used to live at GET /api/jobs/queue-stats and, while there, its allowlist named the job queue and omitted both run queues — so the card reported on the queue that was about to be deleted and nothing about the ones carrying work.

/api/health no longer reports systemSchedulesHealthy. It had become a hardcoded true once system automations were workflows seeded by ensureSystemWorkflows, so it answered “healthy” whatever the truth was; a monitor keyword-matching on it was being reassured by a constant.

Runs are read through the run view.

Domain boundary

Unchanged and enforced twice: packages/platform-domain has only zod as a runtime dep, and its eslint config blocks drizzle-orm, pg-boss, hono, Node built-ins and any apps/* import. The workflow entities are additionally browser-safe — the dashboard imports the barrel directly.

Part 3 — What the Job engine left behind

The deletion is complete. apps/api/jobs/, packages/platform-domain/src/jobs/’s job-specific half, the SDK JobClient, the dashboard’s job surfaces, the CLI --jobs flag and both tables (jobs, job_schedules) are gone. The three surfaces that rode it — Publish now, Run agent, zen prompts — enqueue ad-hoc workflow runs.

What survives from that module, and why:

• apps/api/work/ — the wiring hub both engines always shared, renamed from buildJobs. One AI gateway, one connector registry, one watermark/cursor store, one export-target map, the pg-boss instance and the sink node instances. It was never an engine.
• SINK_INPUTS (the eight sink input schemas) and InputFieldSchema, which moved to common/ because two domain modules import it across what used to be the jobs/ boundary.
• IQueueStats, moved to workflows/ports/ — queue health was never job-specific.

Phase 2 finished the job. The nine sinks were the last classes written against the old contract: each took a WorkflowContext and returned a StepResult, and a StepBackedNode wrapper translated that to INode at run time. They now extend SelfSourcingNode (workflows/ports/SelfSourcingNode.ts) — run(params, ctx) returning a NodeResult, with three helpers stating the only three outcomes a sink has: done(message, summary) (ran, and the summary rides on the one item it emits on main), stop(message) (nothing to do — a green halt, never with a fix), fail(message, fix?). With nothing left implementing it, the whole packages/platform-domain/src/jobs/ tree was deleted: IWorkflowStep, BaseWorkflowStep, WorkflowContext, JobStep, JobStepCatalog and JobSteering had no consumer outside it. The nine live in <vertical>/nodes/*Node.ts.

Two rules survive that deletion and are easy to lose:

• A sink must emit that one summary item. An empty output makes the executor SKIP everything downstream, and publish → post_comment has to sequence.
• Spend is reported through billableNodeUsage, not NodeResult.usage directly. The executor debits every entry it is handed; a TTS card that fails to render reports zero tokens and zero cost, and passing that through opens a 0-credit ledger debit per failed card.

Ad-hoc runs are the replacement pattern, and they predate the deletion: link enrichment, contact enrichment and TTS already ran this way. A one-off carries its graph in the message, saves under a synthetic internal:<kind>:<id> workflow id (the column has no FK by design), never appears on the canvas, and drains from workflow-run-adhoc so a click never queues behind a cron.