Build Lifecycle: Pending Row to Remote Worker to Final Status

ELI5

A build is a ticket on a board in the office (Postgres). A worker in a separate, sealed workshop phones in, takes the oldest ticket, and must call back every half minute to say it is still working. If it stops calling, the office tears up the ticket as “worker lost”. Every ticket also has a hard deadline, set by the shorter of thirty minutes and the build minutes the customer has left.

Technical Deep Dive

Source: sparki-tools/api-engine pkg/api/handlers/worker.go:25-42,121-210,251-433,528-790, pkg/api/worker_routes.go:35-116, pkg/workerapi/workerapi.go:20-51, pkg/api/server.go:466-497, docs/security/build-isolation.md:62-96 @0b0309f, read 2026-10-25. The previous revision of this card described an in-process goroutine pool fed by RabbitMQ; that design is not on the deployed path.

States

stateDiagram-v2
  [*] --> pending: POST /api/v1/builds admitted
  pending --> running: claim, lease 60s
  pending --> failed: unrunnable at claim
  running --> running: heartbeat renews lease
  running --> success: complete
  running --> failed: complete or reaper
  running --> timed_out: complete
  running --> cancelled: complete
  success --> [*]
  failed --> [*]
  timed_out --> [*]
  cancelled --> [*]

“Unrunnable at claim” covers no pipeline, a non-https repository, a bad ref, or exhausted build minutes; such builds are failed without being started and are charged nothing (worker.go:166-171).

One build, end to end

sequenceDiagram
  autonumber
  participant U as web-app
  participant E as api-engine
  participant DB as Postgres
  participant K as build-worker
  U->>E: POST /api/v1/builds
  E->>E: admission via billing enforcer
  E->>DB: insert build status pending
  K->>E: POST /internal/worker/claim
  E->>DB: lock oldest pending, FOR UPDATE SKIP LOCKED
  E->>DB: set running, claimed_by, lease_expires_at, max_duration_seconds
  E-->>K: job with steps, env, deadline
  loop every step
    K->>E: POST logs batch
    E->>DB: insert build_logs once per seq
    K->>E: POST heartbeat
  end
  K->>E: POST complete with final status
  E->>DB: set final status, duration_ms

Hard numbers

Fairness and failure

• The claim first peeks the oldest pending build’s workspace, reads that workspace’s budget, then locks only that workspace’s oldest build. If a workspace’s plan cannot be read it is skipped and the next one tried, so one tenant’s billing trouble does not stall the queue (worker.go:251-298).
• Every per-build call must come from the token that claimed the build; others get 404, and a heartbeat for a build no longer running gets 409 so the worker stops (build-isolation.md:67).
• The reaper fails running builds whose lease lapsed more than 2 minutes ago (worker lost, charged up to the last lease expiry) and builds still running 2 minutes past their deadline (worker.go:191-210).

Key Terms

• lease → lease_expires_at on the build row; renewed by heartbeats.
• claimed_by → the worker token id (first 8 hex of its sha256 digest).
• deadline → the time by which the build must finish; enforced by the worker and backstopped by the reaper.
• worker lost → the error_message written when a lease lapses.

Q&A

Q: What is the build queue, and how does a worker take a build from it? A: The queue is the set of builds rows with status pending. A worker calls POST /internal/worker/claim; the API locks the oldest eligible row with FOR UPDATE SKIP LOCKED, flips it to running, records the token id and a 60-second lease, and returns the job.

Q: What bounds a build’s run time, and what happens when the bound is hit? A: The deadline, which is the earlier of 30 minutes and the workspace’s remaining build minutes. The worker kills the step container when it fires and reports timed_out; if the worker ignores it, the reaper fails the build 2 minutes after the deadline.

Q: What makes the API fail a running build without the worker reporting? A: The lease reaper, every 30 seconds: a lease that expired more than 2 minutes ago fails the build as worker lost, and a build still heartbeating 2 minutes past its deadline is failed as overdue.

Examples

A worker host reboots mid-build. Heartbeats stop, the lease expires 60 seconds after the last one, and within the next 2.5 minutes the reaper marks the build failed with worker lost; its minutes are charged only up to the last lease expiry. When the worker restarts, its startup reaper kills any step containers left behind (sparki-019).