Three-Component Architecture: Engine, Web, Perch

ELI5

Nestr is a kitchen with three stations: Engine is the freezer that vacuum-seals leftovers (Pellets), Web is the touchscreen on the fridge that shows what’s inside, and Perch is the security camera system watching every shelf and beeping when something spoils.

Technical Deep Dive

The repo splits into three top-level directories, each a distinct deployable unit:

Component Boundaries

flowchart LR
    subgraph Engine[engine/ — Go]
        ECLI[CLI: nestr serve]
        EREST[REST :8080]
        EWS[WebSocket /ws]
        EMETRICS["/metrics"]
    end
    subgraph Web[web/ — Next.js]
        WUI[Dashboard]
        WAPI[axios client]
    end
    subgraph Perch[perch/ — Kubernetes]
        PROM[Prometheus]
        BRIDGE[Bridge]
        CUSTOM[cost-monitor<br/>slo-tracker<br/>policy-enforcer<br/>trace-correlator]
        GRAF[Grafana]
    end
    WAPI -->|JWT Bearer| EREST
    WUI -.->|ws://| EWS
    PROM -->|scrape /metrics| EMETRICS
    BRIDGE -->|federation push| PROM
    CUSTOM --> PROM
    GRAF --> PROM

Each component is independently testable: engine/ ships a Makefile and Go test suite; web/ runs Vitest + Playwright against localhost:5173; perch/ deploys via helm install perch ./helm -n monitoring.

Key Terms

• Pellet → a zstd-compressed tarball of a workspace dependency tree, owned by Engine.
• Bridge → a Perch service that scrapes the Engine /metrics endpoint and federates to a remote Folio collector.
• perch-system → the Kubernetes namespace into which the Perch helm chart installs Prometheus, Grafana, and the custom Go services.

Q&A

Q: Where is the Pellet binary format defined — engine, web, or perch? A: engine/internal/pellets/pellets.go. Web only consumes Pellet JSON via the REST client; Perch never touches Pellets directly.

Q: How does Perch know about the Engine? A: The Bridge service is configured with NestrMetricsURL (default points at the Engine’s :9090/metrics) and pulls every 30 seconds.

Q: Can the Web dashboard run without Perch? A: Yes. Web talks directly to Engine REST. Perch is only required for cross-cluster federation, dashboards, and SLO/cost tooling.

Examples

A pellet compression event flow: dev hits “Compress” in Web → axios POST /api/pellets/compress to Engine → Engine writes .pellet + .meta.json, increments nestr_pellet_compress_total, broadcasts pellet:compressed over /ws → Web invalidates TanStack Query keys → next Prometheus scrape (15s) is picked up by Perch Bridge and federated outward.