Where Traceo CI Runs: Hosted, Hermes Fleet, Hosted Nightly

ELI5

Traceo’s tests have moved house twice. First they ran on GitHub’s free machines, until the free minutes ran out and every job quietly failed. Then they moved to the estate’s own server, Hermes eu-1. That server is being retired, so they are moving back to GitHub’s machines. There are only enough free minutes for a nightly run, so a PR now gets tested only if someone presses “run”.

Technical Deep Dive

Source: traceo-cat/traceo_mcp_server/docs/decisions/0090 (2026-09-07), 0096 (2026-09-25, accepted), 0110 (2026-10-16), 0131 (2026-10-24, proposed, human countersign PENDING) at origin/main 7d647ab; .github/workflows/{ci,nightly-gates,ci-predicate,deploy-observability}.yml at origin/main; proposal branch ci/eu1-retire-hosted-nightly (worktree traceo-wt-eu1-retire); read 2026-10-25

Timeline

timeline
  title Traceo CI host
  2026-09-07 : 0090 merge predicate over named jobs, a skipped job asserted nothing
  2026-09-11 : hosted pool exhausted (Free plan, 2000 of 2000 min, $0 limit), every ubuntu-latest job fails with no runner
  2026-09-25 : 0096 every job moves to self-hosted ci-shared (Hermes eu-1)
  2026-10-16 : 0110 CIPREDICATE v2 evaluated by machine on every CI run, posted as a commit status, still not a gate
  2026-10-24 : 0131 proposed, eu-1 retires with no successor, back to ubuntu-latest with a nightly batch

Why 0096 moved to the fleet

From 2026-09-11T18:02Z every runs-on: ubuntu-latest job ended conclusion: failure with steps: [] and runnerName: none (run 36096536663). The org was on the Free plan with a $0 spending limit and had used its 2000 minutes. 0090 evaluates the merge predicate over nine jobs by name, so a job with no runner is a non-outcome that can never satisfy it. 0096 measured the fleet’s capabilities rather than assuming them. services: and podman worked. actions/setup-python did not, and neither psql nor poetry was installed on the ephemeral runners. It moved every job to [self-hosted, linux, arch, x64, ci-shared].

What 0131 proposes, and what is already merged

Hermes eu-1 retires with no successor. Hermes’ retire gate counted all 17 traceo-cat jobs as offenders. Hosted per-push and per-PR CI would cost about 6,424 min/mo (292 runs × 22 min) against a 1,500 min/mo cap, so 0131 proposes:

The projected spend is 1,115 min/mo, which leaves room for about 16 hand pre-merge runs a month. The workflow changes are on main (#385, 7d647ab), but the decision’s §7 human countersign is still PENDING. Read it as proposed policy whose mechanism has already merged, not as ratified.

flowchart TD
  PR["PR opened"] --> CHEAP["conflict-markers, docs-invariants, landing-links run"]
  PR --> Q{"someone dispatches Traceo CI on the branch?"}
  Q -- yes --> RUN["ci.yml on branch head (rebase first: it does not test the merge ref)"] --> PRED["ci-predicate posts CIPREDICATE v2 status"]
  Q -- no --> MERGE["merge proceeds unchecked by ci.yml"]
  PRED --> MERGE
  MERGE --> NIGHT["nightly-gates 03:27: main moved, so the full batch runs"]
  NIGHT --> RED{"red?"}
  RED -- yes --> FIX["fix forward, one red costs one batch"]

What never changed: CI is not a merge gate

GET /branches/main/protection answers 403 on this plan (0045 F-2-2), so no check has ever been required. 0110’s machine predicate is posted as a commit status and is explicitly still not a gate. Merges are judged by the PR-body record (0090 §5) and that status. 0131 states its main cost plainly: a PR nobody dispatches merges without ci.yml and is caught by that night’s batch at the latest.

Key Terms

• Non-outcome → a job that never ran, such as one with no runner. 0090 rules that it asserted nothing.
• ci-shared → the Hermes self-hosted runner label (rootless podman on Arch).
• Nightly batch → nightly-gates.yml, which runs CI once a night only if main moved.
• CIPREDICATE v2 → the machine-evaluated merge predicate, posted as a commit status. Advisory.

Q&A

Q: Why did CI move onto the Hermes fleet in September 2026? A: The hosted pool had no minutes left from 2026-09-11 (Free plan, $0 limit), so every ubuntu-latest job failed with no runner assigned. Those non-outcomes could never satisfy 0090’s predicate. The fleet was alive and measured capable.

Q: Under 0131, when does Traceo CI run, and what does that cost a PR? A: Nightly at 03:27 UTC if main moved, and by hand (workflow_dispatch) on a PR branch. A PR gets no automatic ci.yml run, and a hand run tests the branch head rather than the merge ref.

Q: Is any Traceo CI job a required check? A: No. Branch protection returns 403 on the Free plan, so nothing can be required. CI has always been evidence, never a gate.

Q: Is 0131 in force? A: Its status is proposed and its human countersign is pending. Its workflow changes merged to main in #385, so the nightly schedule is what actually runs, but the policy is unratified.

Examples

You open a PR touching traceo_mcp_server/. Only the cheap PR workflows run. Rebase on main, run gh workflow run ci.yml --ref <your-branch>, wait for ci-predicate to post its status, and cite the run ID in the PR body per 0090 §5. If you skip this, the 03:27 nightly batch tests your change after it has merged.