CRUMB a card from devarno-cloud

Engine Packages: The Shipped Path vs the Subsystems Tree

sparki intermediate 6 min read

ELI5

The api-engine repo has two kitchens. One is the line that serves customers: projects, pipelines, builds, billing. The other is a back room full of prepared stations (scan, score, loco, forgery, bind, link) that were built but never connected to the serving line. Both are in the same building, so it is easy to mistake one for the other.

Technical Deep Dive

Source: sparki-tools/api-engine pkg/api/server.go:3-32,132-134,309-353, pkg/api/handlers/projects.go:375-387, pkg/api/handlers/pipelines.go:76-87, pkg/bootstrap/bootstrap.go:8-12 @0b0309f; import graph read with git grep "api-engine/internal/\|api-engine/subsystems" over cmd/, pkg/, internal/, read 2026-10-25.

PackageRole on the deployed path
pkg/api, pkg/api/handlers, pkg/api/middlewareFiber server, routes, auth shim, tier validator, worker API
internal/detection (+ detectors, types)constructed at startup, but unreachable: POST /api/v1/projects/:id/scan answers 501 not_available (projects.go:375-387)
internal/pipelinethe stored flat-step config shape; its generator needs cached detection results, which nothing can now produce
pkg/billing, pkg/billing/tollplan ladder, entitlement cache, build admission, TOLL client
pkg/websocketper-build log/status fan-out
pkg/workerapiwire contract with the build worker
pkg/database, pkg/cache, pkg/tier, pkg/demo, pkg/workspacesPostgres, Redis, tier config, demo reset, /me
flowchart TD
  MAIN["cmd/api"] --> API["pkg/api"]
  MAIN --> BOOT["pkg/bootstrap"]
  API --> H["pkg/api/handlers"]
  API --> DET["internal/detection"]
  API --> PIPE["internal/pipeline"]
  API --> BILL["pkg/billing"]
  API --> WS["pkg/websocket"]
  H --> WAPI["pkg/workerapi"]
  BOOT --> DB["pkg/database"]
  BOOT --> CACHE["pkg/cache"]
  BOOT --> TIER["pkg/tier"]

subsystems/ on main holds bind, diagnose, forgery, integration, link, loco, run, runimpl, scan, score (last touched 2026-10-18). They are imported only by internal/cmd, internal/integration and internal/tui (the root main.go CLI), never by cmd/api or cmd/build-worker. docs/security/build-isolation.md:58-60 adds a go list -deps test that fails if cmd/build-worker links any subsystems/* package.

flowchart LR
  CLI["root main.go CLI"] --> IC["internal/cmd"]
  IC --> SUB["subsystems: scan, score, loco, forgery, bind, link, run"]
  APIBIN["cmd/api"] -. "no import" .-> SUB
  WBIN["cmd/build-worker"] -. "forbidden by go list -deps test" .-> SUB
SubsystemStatus on 2026-10-25
locodesigned, not shipped (sparki-008, sparki-009)
scandesigned, not shipped: neither subsystems/scan nor the linked internal/detection engine is reachable from a route (sparki-010)
scoredesigned, not shipped (sparki-011)
forgery, bind, link, run, runimpl, diagnose, integrationCLI-side only; no API route

Key Terms

  • deployed path → everything reachable by imports from cmd/api or cmd/build-worker.
  • subsystem → a Go package under subsystems/; on main today, CLI-only.
  • internal/detection → the shipped detection engine (languages, frameworks, package managers, build tools).

Q&A

Q: Which packages does cmd/api actually link, and which subsystems does it not? A: It links pkg/api (and handlers/middleware), pkg/bootstrap, internal/detection, internal/pipeline, pkg/billing, pkg/websocket, pkg/workerapi and the storage packages. It links none of the ten subsystems/* packages.

Q: Is project detection reachable on the deployed path? A: No. internal/detection is linked and constructed in NewServer (server.go:133), but its only route, POST /api/v1/projects/:id/scan, returns 501 not_available without reading the request (projects.go:375-387): detection used to run on a client-chosen path on the API host’s filesystem. Pipeline auto-generation reads cached detection results and so cannot succeed either (pipelines.go:76-87).

Q: How would you prove a subsystem is not on the deployed path? A: Grep the import paths from cmd/api down. pkg/api imports only internal/detection and internal/pipeline from internal/, and nothing imports subsystems/ except the CLI packages. For the worker, the go list -deps test enforces it.

Examples

A teammate proposes surfacing the quality score in the dashboard. The score types exist in subsystems/score/types.go, but no route serves them and the engine binary does not link the package. The work is a new handler plus wiring, and customer copy must not mention a score until it ships (launch decision §6).

neighbors on the map