CRUMB a card from devarno-cloud

Engine Observability: /health, /metrics, DSN and Grafana Cloud

sparki intermediate 5 min read

ELI5

The engine has two windows. /health is the front window: anyone can look and see whether builds are getting picked up. /metrics is a back window with a lock: only the estate’s monitor (DSN), holding the right key, may look in. A third, optional pipe can push the same numbers to Grafana Cloud. The big observatory drawn in older plans (storm, ELK, Istio, Kubernetes) was never built for sparki.

Technical Deep Dive

Source: sparki-tools/api-engine pkg/api/metrics.go:12-46, pkg/api/server.go:174-189,231-236,466-517, pkg/api/auth.go:42-55, cmd/api/main.go:124-142, pkg/bootstrap/config.go:80-84, docs/security/build-isolation.md:52-56 @0b0309f; devarno-cloud/deep-space-network deploy/prometheus/prometheus.yml:252-268 @fa18c98; live probe of api.sparki.tools/health and /metrics, read 2026-10-25.

Signal surfaces that exist

SurfaceGuardContent
GET /healthpublicstatus (healthy or degraded), workers.state, workers.last_claim_at, builds.oldest_pending_seconds; always HTTP 200
GET /readypublicdependency check via common-go-lib HealthManager; 503 only when unhealthy
GET /metricsproduction: mounted only with METRICS_SCRAPE_TOKEN of 32+ chars, then Authorization: Bearer compared in constant time over sha256 digestsPrometheus exposition (promhttp default registry, request metrics middleware)
Grafana Cloud remote writeGRAFANA_CLOUD_ENABLED and endpoint setpushes the same metrics with labels service=sparki-api, environment
structured logszap JSONX-Request-ID correlation id on every line; worker audit lines on every claim and completion
flowchart LR
  E["api-engine"] -- "GET /metrics, Bearer token" --> DSN["DSN Prometheus job sparki-engine"]
  E -. "optional remote write" .-> GC["Grafana Cloud"]
  E -- "GET /health" --> SITES["DSN sites probe of api.sparki.tools"]
  E -- "zap JSON logs" --> RL["Railway log stream"]
  DSN --> DUR["durable-1 remote_write tee"]

What /health means

stateDiagram-v2
  [*] --> healthy
  healthy --> degraded: worker routes not ok in production
  healthy --> degraded: oldest pending build older than 5 min
  degraded --> healthy: worker configured and queue drains

workers.state is one of ok, unconfigured (no token digests), invalid (malformed worker env) or no_database. Degraded is reported only when ENVIRONMENT is production or prod; the status code stays 200 so liveness probes never restart the API (server.go:466-494).

State on 2026-10-25

  • /health returned {"status":"degraded","workers":{"state":"unconfigured"},"builds":{"oldest_pending_seconds":0}}.
  • /metrics returned 404: the production engine is not armed with METRICS_SCRAPE_TOKEN, so the route is not mounted and the DSN sparki-engine job reads up == 0. The DSN config says so itself and routes outages through the sites probe instead.
  • Whether Grafana Cloud remote write is enabled on the Railway service: UNVERIFIED (env not read).

Not deployed

observability-storm (no deploy config, last main 2026-02-24), ELK/Loki pipelines, Istio, OTel tracing across services, and Kubernetes manifests in the api-engine repo (k8s/, manifests/) are not on the deployed path. The old card’s trace boundaries for detect, generate and deploy described operations that do not run.

Key Terms

  • METRICS_SCRAPE_TOKEN → the bearer DSN presents; unset in production means no /metrics route at all.
  • sparki-engine → the DSN Prometheus job that scrapes https://api.sparki.tools/metrics every 30 s.
  • degraded → /health status for an unworkable build queue; never a non-200.
  • worker audit → structured log line with event, token_id, build_id, workspace_id.

Q&A

Q: Under what conditions is GET /metrics mounted in production, and how is it guarded? A: Only when METRICS_SCRAPE_TOKEN is set and at least 32 characters. The route then requires Authorization: Bearer <token>, checked by comparing sha256 digests in constant time; a wrong token is 401 and an unarmed engine is 404.

Q: What makes /health report degraded, and why does it still return 200? A: In production, worker routes that are not ok, or a pending build older than 5 minutes. It stays 200 because /health is the liveness probe and a degraded queue must not restart the API.

Q: Which parts of the old storm strategy are deployed? A: None of the storm stack. What runs is the engine’s own Prometheus exposition (when armed), the optional Grafana Cloud push, /health, and JSON logs.

Examples

After an operator sets METRICS_SCRAPE_TOKEN on Railway and writes the same value to DSN’s /etc/prometheus/secrets/sparki-metrics-token, the scrape flips from 404 to 200 and up{job="sparki-engine"} goes to 1. Prove the token by hand first, as the DSN config comment asks, because a 401 and a 404 look identical in up.

neighbors on the map