CRUMB a card from devarno-cloud

Docs Are Not the Deployment: The Corroborated False Description

traceo intermediate 6 min read

ELI5

Three notes on the fridge all say “the milk is fresh”. Each note-writer read one of the other notes. Nobody opened the fridge. Traceo has been burned by exactly this, more than once: a CI comment, a decision and a job list all agreed that the tenancy test used the safe database role, and none of them was the test. The rule that came out of it is short. To know what a machine does, look at the machine.

Technical Deep Dive

Source: traceo-cat/traceo_mcp_server/docs/decisions/0061-name-the-corroborated-false-description-anti-shape.md (accepted), 0047 (cited), 0042, 0075; traceo_mcp_server/traceo_mcp_server/routes/health.py:1-9; traceo_mcp_server/railway.toml:11; live probes of mcp.traceo.cat and engine.traceo.cat; traceo-cat origin/main 7d647ab, read 2026-10-25

The anti-shape (0061)

0047 named a family of dormant controls: absent, blind, dormant. 0061 adds a fourth variant: a description that was never true, held in place because artifacts corroborate each other, none of which is the thing described.

The original instance: a step in ci.yml said it ran the live-Postgres RLS suite “which exercises the non-bypass traceo_app role”. The job’s DATABASE_URL named the superuser. RLSContext.set_context never issued SET ROLE. traceo_app was NOLOGIN, and a grep found the role only in comments. Three artifacts agreed: the comment, decision 0025 §6, and the job list showing Integration Tests. The claim survived several audits and four verification rounds. What exposed it was not a document but the mechanism itself: two DID NOT RAISE failures the first time it was allowed to run.

The counter is a rule about warrant. A statement that a mechanism does something is discharged only by an artifact the mechanism produces, one that would fail if the claim were false. Documents agreeing with each other is not evidence about the mechanism. Re-checking against the other documents actively confirms the error.

0061 also bounds itself. It does not claim production was exposed. On 2026-08-17 the deployed database had traceo_app with bypassrls=false and RLS forced. The claim and the fact were independent, and their agreement was luck.

flowchart LR
  C1["CI comment: suite exercises traceo_app"] --- C2["0025 §6: live-RLS gate is the tenancy check"]
  C2 --- C3["job list: Integration Tests runs test_rls.py"]
  C3 --- C1
  C1 -.->|"never consulted"| M["the job's DATABASE_URL: superuser, no SET ROLE"]
  M -->|"first honest run"| T["DID NOT RAISE"]

Same shape, elsewhere in Traceo

DeclaredProduced (what the deployed process does)Card
app.py installs metrics, logging and a 100/min rate limiterRailway runs server.py, which never imports app.py. The live MCP host returns no x-ratelimit-* headers and empty http_requests_* seriestraceo-006, traceo-011
/metrics listed in server.py’s auth exclusions and named by 0075it was 404 on the deployed host until moved onto a router (routes/health.py docstring)traceo-011
docs-invariants treated as an existing check21 runs, 21 failures, never green, its checks aimed at a tree shape this repo lacks (0042)traceo-019
earlier crumb cards: engine posts a completion webhook, ingestion writes requirements rowsno webhook code exists, and ingestion writes YAML to the container’s temp disktraceo-007
”RLS is a filter, not a boundary, until G-19-1 closes”, repeated across M-22 recordsG-19-1 closed on 2026-09-02 (113 leaked → 0). The stale claim was copied from projections (finding 20261010-0857)traceo-004

The last row runs the other way: a claim that was true and went stale, kept alive by copying.

quadrantChart
  title Where a claim gets its warrant
  x-axis Other documents --> The mechanism itself
  y-axis Cannot fail --> Can fail if false
  quadrant-1 Evidence
  quadrant-2 Fragile test
  quadrant-3 Corroboration
  quadrant-4 Unasserted observation
  CI comment: [0.15, 0.1]
  Decision citing comment: [0.25, 0.2]
  Card citing README: [0.2, 0.15]
  Identity row printed by job: [0.85, 0.9]
  Live header probe: [0.9, 0.8]
  Railway deploy list read: [0.8, 0.85]

Rules that fell out

  • 0042, anti-vacuity: every check fails when it inspects nothing. An empty glob or deploy list is a failure, not a pass.
  • Print the identity you assert on: test_rls_privilege_drop.py prints traceo_app f f into the job log (traceo-004).
  • Cite at the deployed config: start from railway.toml’s startCommand, not the README, when asking “does production do X?” (traceo-001).

Key Terms

  • Corroborated false description → a never-true claim kept in place by documents agreeing with each other (0061).
  • Dormant control → a correctly specified check that never runs or cannot fail (0047).
  • Warrant → what a claim is allowed to rest on. For mechanism claims, only an artifact the mechanism produces.
  • Anti-vacuity → a validator that observes nothing must never report success (0042).

Q&A

Q: What is a corroborated false description, and why does comparing documents fail to catch it? A: It is a claim that was never true, held up by several artifacts citing each other. Comparing documents is the standard audit move, and here every comparison confirms the claim. The agreement is what makes it durable.

Q: Where may a claim that a mechanism does something get its warrant, under 0061? A: Only from an artifact the mechanism itself produces that would fail if the claim were false: a printed identity row, a live response header, a deploy-list read. Comments and decisions do not count, however many there are.

Q: Name two Traceo mechanisms that existed in code but not in the deployed process. A: The MCP rate limiter and MetricsMiddleware, which are both wired in app.py, which Railway never runs. The /metrics route was a third until it moved to routes/health.py.

Examples

A teammate says “the MCP server is rate limited, it’s in rate_limit.py and config.py defaults to 100/minute”. Don’t argue from more documents. Run curl -sD - https://mcp.traceo.cat/api/requirements -o /dev/null | grep -i ratelimit and look. On 2026-10-25 it printed nothing.

neighbors on the map