Engine Metrics Catalogue: Request, Pellet and Supply-Chain Series

ELI5

The engine keeps two kinds of tally. Request and pellet tallies live in the running server’s memory. Security tallies are made by the command-line tool, which exits after every run, so it writes them into a small shared notebook on disk; the server reads that notebook whenever someone asks for metrics. If the notebook cannot be opened, the server raises a flag instead of pretending everything is zero.

Technical Deep Dive

Source: nestr-tools/engine main.go:49-60,137-160, internal/adapters/metrics.go:28-92, internal/pellets/metricstore/store.go:70-100, internal/pellets/metricstore/collector.go:28-110, internal/pellets/supplymetrics/metrics.go:36-39, internal/server/rest.go:135,600-602 @e9180e4; nestr-tools docs/decisions/2026-06-24-c3-supply-metrics-sqlite-not-pushgateway.md (closed) @b8f0585; read 2026-10-25.

Two families

classDiagram
  class InProcess_nestr {
    Counter operations_total
    Summary request_duration_seconds
    CounterVec requests_total
    Counter cache_hits_total
    Counter cache_misses_total
    Histogram pellet_compress_duration_seconds
    Histogram pellet_extract_duration_seconds
    Counter pellet_compress_total
    Counter pellet_extract_total
    Gauge cache_hit_ratio
    Gauge cache_size_bytes
  }
  class Persisted_metricstore {
    pellets_installed_by_slsa_level by slsa_level
    pellets_installed_by_signer_type by signer_type
    hook_invocations_total by capability, decision
    sandbox_denials_total by syscall
    nestr_diff_signal_triggered_total by signal_id, severity
    nestr_diff_evaluations_total
    nestr_diff_baseline_missing
    nestr_metricstore_unavailable
  }

• In-process nestr_* series are registered on the engine’s own registry (adapters/metrics.go). request_duration_seconds is still a Summary (quantiles 0.5, 0.95, 0.99), so it cannot be aggregated across replicas; compress and extract histograms use exponential buckets from 0.1 s, factor 2, count 10.
• Supply-chain and diff series are written by CLI verbs (install, verify, diff, audit, hook) into SQLite at <NESTR_CACHE_DIR or ./.pellets/cache>/supply-metrics.db and read back by a collector registered in nestr serve (main.go:137-160).

flowchart LR
  CLI["nestr install, verify, diff, hook"] --> SM["supplymetrics callbacks"]
  SM --> DB[("supply-metrics.db SQLite")]
  SERVE["nestr serve /metrics"] --> COL["metricstore.Collector at scrape time"]
  COL --> DB
  COL -- "store unreadable" --> FLAG["nestr_metricstore_unavailable = 1, one WARN"]

Why not Pushgateway

Decision C3 (2026-06-24, closed) chose a local SQLite store over a Pushgateway, which would add an operated component and keep stale series after a crash, and over remote write from the Railway engine into the cluster, which would open a new trust seam. In-process counters were replaced, not dual-written.

Caveats

• The store is shared by file path, so only CLI runs on the same filesystem as nestr serve feed it; in serve the callbacks are wired but “simply unused” (main.go:56-57). Installs on customers’ machines never reach the hosted engine’s /metrics.
• orchestrator_* series in internal/pkg/metrics.go are imported only by tests; the engine does not expose them.
• /metrics has no auth guard (rest.go:600-602).
• cache_hits_total and cache_misses_total exist, but the stats endpoint does not report a hit rate (nestr-004).

Key Terms

• metricstore → SQLite-backed counter store shared across processes by path.
• supplymetrics → nil-able callbacks the offline verify packages call, keeping PEL-SIG-022’s no-network guarantee.
• nestr_metricstore_unavailable → I-7 gauge distinguishing a broken store from a real zero.

Q&A

Q: How do counters incremented by short-lived CLI runs reach the serve process’s /metrics? A: The CLI persists each increment to supply-metrics.db in the cache directory; nestr serve registers a collector that reads that SQLite store at every scrape.

Q: How do you tell a genuine zero on a supply-chain panel from a broken store? A: Check nestr_metricstore_unavailable. When serve cannot open or read the store it sets that gauge to 1 and logs one WARN, so a degraded scrape is structurally different from an all-clear zero.

Q: Are the orchestrator_ series exposed by the engine?* A: No. They are defined in internal/pkg/metrics.go, which only test files import.

Examples

p99 compress time over five minutes: histogram_quantile(0.99, sum(rate(nestr_pellet_compress_duration_seconds_bucket[5m])) by (le)). Installs by SLSA level: sum by (slsa_level) (pellets_installed_by_slsa_level), read alongside nestr_metricstore_unavailable before trusting a flat line.