The Deploy Observer: Outcome, Non-Outcome, Not-Yet

ELI5

Railway and Vercel deploy Traceo straight from git, so nothing in GitHub knew whether a merge actually reached production. Ten merges once sat undeployed for almost two weeks under a green board. The deploy observer is a daily job that reads Railway and asks one question: is the code running in production the code on main? It is careful about three answers: yes, no, and “not finished yet”.

Technical Deep Dive

Source: traceo-cat/.github/workflows/deploy-observability.yml (origin/main 7d647ab, whole file), engine/railway.toml:16, traceo_mcp_server/railway.toml; decisions 0070, 0072, 0073 (accepted), 0131 (proposed); read 2026-10-25

Why it exists

The deploy-backend.yml / deploy-web.yml workflows were deleted on 2026-07-11 as redundant, which was correct because the hosts deploy from their own git integrations. From that date, no GitHub job could fail on deployability. The workflow header records the cost: ten merges between 2026-08-16 and 2026-08-17 did not reach production, and a human found out by reading the Railway API thirteen days later.

Triggers

Three categories (0070)

Unknown counts as an outcome on purpose, so the check fails closed. Classifying an unexpected status as pending would resolve it to a pass.

stateDiagram-v2
  [*] --> Read : daily cron or dispatch
  Read --> Fail : list unreadable, unparsable or empty
  Read --> Classify
  Classify --> NonOutcome : SKIPPED or REMOVED (ignored)
  NonOutcome --> Classify
  Classify --> NotYet : BUILDING, DEPLOYING, QUEUED, ...
  Classify --> Outcome : anything else
  Outcome --> Fail : not SUCCESS
  Outcome --> ShaCheck : SUCCESS
  ShaCheck --> Pass : deployed sha = main, or no changed path builds this service
  ShaCheck --> Fail : main moved and changed paths build this service
  NotYet --> Age
  Age --> Fail : pending deploy older than MAX_PENDING_AGE_HOURS (6)
  Age --> Recheck : younger
  Recheck --> Classify : after RECHECK_WAIT (90 s)

Bounding “in flight” (0072, 0073)

• 0072: a deploy still in flight past one cron interval is stuck, and stuck is a failure. One re-read is not a bound.
• 0073: the bound measures the age of the pending deploy, not the age of the newest outcome. After a quiet week the newest outcome is always old, so measuring it would flag a deploy that started two minutes ago.

MAX_PENDING_AGE_HOURS defaults to 6. It is an age bound, not a cadence, and 0131 left it unchanged when the schedule went daily.

Which commits matter to which service

Build-path sets come from each service’s railway.toml. traceo:engine declares watchPatterns = ["engine/**", "traceo_mcp_server/**"]. traceo:mcp declares none, which means every path builds it. The workflow handles “empty pattern set = all paths” explicitly. A service may lag main without failing only when no commit since its deployed sha touched its build paths.

flowchart LR
  M["origin/main sha"] --> D{"deployed sha = main?"}
  D -- yes --> P["pass"]
  D -- no --> C["diff deployed..main"]
  C --> W{"any changed path in this service's build set?"}
  W -- "no (engine only)" --> P
  W -- yes --> F["FAIL: lists the changed paths"]
  F --> PG["Page #pipeline via DISCORD_DEPLOY_WEBHOOK"]

Anti-vacuity

Per 0042 the check fails when it inspects nothing. An unset RAILWAY_TOKEN, an unreadable or empty deploy list, or a list with no outcome at all each fail. Every run writes observer-verdicts.txt with one verdict row per service, so a healthy run and an unwired one leave different evidence. If DISCORD_DEPLOY_WEBHOOK is unset, the paging step itself fails.

Key Terms

• Non-outcome → a deploy status that says nothing about the service (SKIPPED, REMOVED). It is skipped over, not counted.
• Not-yet-outcome → in flight. It is re-read once and bounded by the pending deploy’s age.
• Fail-closed classification → an unrecognised status is treated as a terminal outcome, so it can fail the check.
• Drill → a dispatch input that forces a synthetic failure to prove the pager reaches Discord.

Q&A

Q: Why does the observer not run on push? A: Right after a merge the deploy has not finished, so the question cannot be answered yet. A push-triggered pass would be the default answer at exactly the moment the truth is unknown (0070 §3).

Q: Which Railway statuses are non-outcomes, and why does an unknown status count as an outcome? A: SKIPPED and REMOVED. An unknown status is treated as an outcome so it fails rather than waiting forever as “pending”. CRASHED is the concrete case that must not read as “still working”.

Q: What does the staleness bound measure, and what is its default? A: The age of the pending (in-flight) deploy, not the age of the newest outcome (0073). The default is 6 hours (MAX_PENDING_AGE_HOURS).

Q: What does the observer do when it reads an empty deploy list? A: It fails (“deploy list is empty — inspected nothing”). An observer that saw nothing must never report success.

Examples

You merge a docs-only change under web/apps/landing/. The next morning the engine’s deployed sha is behind main, but none of the changed paths match engine/** or traceo_mcp_server/**, so the engine passes. traceo:mcp has no watch patterns, so it must have deployed the new main. If Railway marked that deploy SKIPPED, the observer looks past it to the newest real outcome.