Observability State & Gaps (as served, 2026-10)

ELI5

Both services now have a /metrics window. Through the MCP window you can see the tenant-lookup counters and little else, because the machinery that counts web requests is bolted onto a file production does not run. The engine’s window counts every request, but it hands the numbers over gift-wrapped as JSON, which a Prometheus scraper cannot read, and two of its dials stick at their first reading.

Technical Deep Dive

Source: traceo-cat/traceo_mcp_server/traceo_mcp_server/routes/health.py:1-45, middleware/metrics.py:30-90, auth/middleware.py:43-63, app.py:85-91, server.py:88-160, engine/src/engine/server.py:142-168 and :360-410, engine/src/engine/middleware/logging.py:1-25, traceo_mcp_server/traceo_mcp_server/logging_config.py:1-40, .github/workflows/deploy-observability.yml (origin/main); decision 0075; live scrapes of mcp.traceo.cat/metrics and engine.traceo.cat/metrics (twice) on 2026-10-25; traceo-cat origin/main 7d647ab

What each /metrics serves (live, 2026-10-25)

Three defects the v1.0 card could not have listed:

1. MCP request metrics never increment. server.py’s combined app installs only AuthMiddleware. MetricsMiddleware, SentryContextMiddleware and LoggingMiddleware are added only in app.py, which is not deployed.
2. The engine’s /metrics is not scrapeable as Prometheus text. The handler returns the string from generate_latest(...).decode() without PlainTextResponse, so FastAPI JSON-encodes it. A Prometheus scraper expects plain text.
3. The engine gauges stick. Each scrape tries to register Gauge(...) again, and the except ValueError: pass on the duplicate skips the .set(). Two scrapes 15 s apart both read traceo_engine_uptime_seconds 760.69, while /health moved from 775.7 to 775.9.

At the 2026-10-25 scrape, the MCP tenant-resolve counters read ok 0, fail 0, with a creation timestamp moments earlier, so the process had just restarted. A zero there means “no authenticated tenant request since boot”. It is not a fault on its own.

flowchart LR
  subgraph mcp["MCP as deployed (server.py)"]
    A1["AuthMiddleware"] -->|"record_tenant_resolve"| C1["traceo_tenant_resolve_*"]
    MM["MetricsMiddleware"]:::dead
    APP["app.py only"]:::dead --> MM
  end
  subgraph eng["Engine"]
    M2["MetricsMiddleware"] --> H2["http_requests_* (live)"]
    G2["uptime / jobs gauges"] -->|"set skipped after first scrape"| F2["frozen"]
  end
  C1 --> P["GET /metrics text/plain"]
  H2 --> J["GET /metrics application/json string"]
  G2 --> J
  classDef dead stroke-dasharray: 5 5

Logging and correlation IDs

The v1.0 card said there were no correlation IDs anywhere. The engine has them. The MCP server has the code but does not run it in production.

Deploy observability

The deploy path is watched by deploy-observability.yml, which checks that both Railway services deployed origin/main. It runs daily since 0131 (proposed) and is covered in traceo-018.

mindmap
  root((Traceo observability))
    MCP
      /metrics text OK
      tenant resolve counters OK
      http_requests empty
      no correlation middleware
    Engine
      /metrics served as JSON
      http_requests live
      uptime and jobs gauges frozen
      correlation IDs live
    Deploy
      daily Railway observer
      verdict artifact every run
    Errors
      Sentry SDK both tiers
      shared exception handlers on combined app

Key Terms

• Tenant-resolve counters → traceo_tenant_resolve_total{result} and traceo_tenant_resolve_consecutive_failures, mirrored from AuthMiddleware’s module globals onto Prometheus (0075).
• Exposition format → the plain-text format Prometheus scrapes. A JSON-quoted copy of it is not the same thing.
• Correlation ID → a per-request identifier carried in contextvars, so log lines and Sentry events for one request can be joined.

Q&A

Q: Which deployed service serves /metrics, and what does each one actually emit? A: Both. MCP serves Prometheus text with live tenant-resolve counters and empty HTTP request series. The engine serves populated HTTP request series plus uptime and job gauges, but as a JSON string, and the gauges freeze after the first scrape.

Q: Why are the MCP server’s http_requests_ series empty?* A: The metrics are registered in the shared registry, but MetricsMiddleware is only installed in app.py. Production runs server.py, whose combined app adds only AuthMiddleware.

Q: What can Prometheus not do with the engine’s /metrics response? A: Parse it as exposition text. The handler returns a bare str, so FastAPI serves it as application/json, a quoted string with escaped newlines. Wrapping it in PlainTextResponse (as the MCP route does) would fix it.

Q: Which tier has correlation-ID middleware in production? A: The engine (LoggingMiddleware). The MCP server’s equivalent sits only in the undeployed app.py.

Examples

You write an alert on rate(http_requests_failed_total{job="traceo-mcp"}[5m]) > 0. It never fires, because the series has no samples on the deployed MCP server. Alert on the tenant-resolve counters there instead, for example traceo_tenant_resolve_consecutive_failures > 0. Before pointing a scraper at the engine, fix its /metrics to return plain text.