Perch Custom Services Roster

Status: pre-production. Perch is pre-production (nestr-tools CLAUDE.md:40 @b8f0585). None of these four services has a Helm template; each exists only as source plus a raw manifest under perch/k8s/ (nestr-tools/perch @f9ec790, read 2026-10-25), and each was last changed 2026-06-03.

ELI5

Perch ships four little assistants on top of vanilla Prometheus: an accountant (cost-monitor), a coach with a clipboard (slo-tracker), a security guard at the deploy door (policy-enforcer), and a detective with a magnifying glass linking traces to logs (trace-correlator).

Technical Deep Dive

Source: nestr-tools/perch cost-monitor/main.go, slo-tracker/main.go:21-28,58, policy-enforcer/main.go:20-55,114-133, trace-correlator/main.go, k8s/{cost-monitor,slo-tracker,policy-enforcer,trace-correlator}/, helm/templates/ (no template for any of them), helm/values.yaml:33-36 @f9ec790; nestr-tools CLAUDE.md:86-101 (I-5) @b8f0585; read 2026-10-25. Endpoints and defaults re-verified against source; deployment status corrected. I-5 requires Perch to ship PEL-DIFF, PEL-REP and PEL-LOG panels before generic golden-signal dashboards, and the Helm chart’s only bundled dashboards are pellets-supply-chain.json and pellets-transparency-log.json; these four generic services sit outside that priority.

flowchart TB
    P[Prometheus]
    L[Loki]
    J[Jaeger]
    GH[GitHub PR]
    SL[Slack / PagerDuty]

    subgraph Custom["Perch custom services (Go)"]
        CM[cost-monitor :8080]
        ST[slo-tracker :8080]
        PE[policy-enforcer :8080]
        TC[trace-correlator :8080]
    end

    P --> CM
    P --> ST
    P --> PE
    ST -->|error budget| PE
    GH -->|/webhook/github| PE
    PE --> SL
    L --> TC
    J --> TC
    P --> TC

cost-monitor (perch/cost-monitor/main.go)

• Polls Prometheus on UpdateInterval (default 5 m).
• Endpoints: GET /api/v1/costs, GET /api/v1/capacity, GET /api/v1/recommendations.
• Inputs: resource metrics (CPU, memory, storage), CloudProvider, Region for unit pricing.
• Outputs: per-service cost, capacity utilisation, rightsizing recommendations as JSON.

slo-tracker (perch/slo-tracker/main.go)

• Update interval default 60 s.
• Endpoints: GET /api/v1/slo/status?service=X, GET /api/v1/error-budget?service=X&window=30d.
• Re-emits its own SLO metrics into Prometheus so dashboards and alerts can subscribe; downstream of cost-monitor for spend-per-9 calculations.

policy-enforcer (perch/policy-enforcer/main.go)

• Three modes: audit, warn, enforce.
• Endpoints: GET /api/v1/policy/check?service=X, GET /api/v1/policy/status, POST /webhook/github.
• The GitHub webhook reads the slo-tracker’s error budget for the service named after the repository and answers the delivery with 403 when the budget is exhausted; it fails open when slo-tracker is unreachable (see Examples).
• Slack notification on enforcement is described in the service’s design; not verified in this revision.

trace-correlator (perch/trace-correlator/main.go)

• Bridges Jaeger ↔ Loki ↔ Prometheus by trace ID and request ID.
• Endpoints: GET /api/v1/trace/logs?trace_id=X&service=Y, GET /api/v1/logs/trace?request_id=X, GET /api/v1/correlate?id=X&start_time=…&end_time=….
• Default time window: last hour when not specified — important to remember when correlating an old incident.

Operator Decision Table

Key Terms

• Error budget → 1 - SLO minus consumed unreliability over a window; policy-enforcer gates on this.
• Rightsizing → cost-monitor’s recommendation to scale resources up or down based on observed utilisation.
• Audit / warn / enforce → policy-enforcer modes; only enforce blocks merges, the other two are observe-only.

Q&A

Q: Which service correlates a trace ID to log lines? A: trace-correlator: GET /api/v1/trace/logs?trace_id=...&service=..., plus /api/v1/logs/trace?request_id=... and /api/v1/correlate.

Q: What signal does the policy-enforcer GitHub webhook check? A: The touched service’s error budget, read from slo-tracker (-slo-tracker-url, default http://slo-tracker.perch-system.svc.cluster.local:8080) before allowing the deployment (policy-enforcer/main.go:114-133). Its default mode is warn.

Q: What is the SLO-tracker’s default update interval? A: 60 seconds (-update-interval, slo-tracker/main.go:28).

Q: What happens if the slo-tracker is down when policy-enforcer is asked? A: It fails open. CheckPolicy starts from Allowed: true and, when the error-budget fetch or parse fails, only records the reason (enforcer.go:113-136); the merge is not blocked. Whether a Slack message is sent on that path was not verified.

Q: Does cost-monitor write back to Prometheus? A: Recommendations are exposed only over its own JSON API; cost time-series themselves are computed on the fly from PromQL queries.

Q: Why split slo-tracker and policy-enforcer instead of one service? A: SLO computation is read-mostly and cacheable; policy enforcement carries a write surface (the webhook, and Slack by design). Separating them keeps blast radius small when the gating logic changes.

Examples

As coded (never observed running), a pull_request webhook with action opened or synchronize makes policy-enforcer map the repository name to a service, fetch error-budget?service=<repo>&window=30d from slo-tracker, and answer the webhook itself with 403 {allowed: false, reason, policy} when the budget is exhausted (main.go:114-150). It does not post a GitHub status check from this handler; a 403 to a webhook delivery blocks nothing on GitHub by itself.