Engine CSV Ingestion Jobs: In-Memory, YAML-Out, No Webhook

ELI5

You drop a stack of forms at the engine’s desk and get a ticket. A clerk processes them in the back room while you keep asking about your ticket. The surprise is where the forms go. The clerk files them as YAML files in a temp folder on the engine’s own disk, not in the database the web app reads, and nobody phones the MCP server to say they are done.

Technical Deep Dive

Source: traceo-cat/engine/src/engine/server.py:229-358, engine/src/engine/ingestion/jobs.py:30-37, :97-102, :143-215, engine/src/engine/ingestion/yaml_writer.py:12-24, migrations/008_reconcile_schema.sql:121-124 (job_status enum); web/apps/client/src/services/engineClient/client.ts; live engine /health; traceo-cat origin/main 7d647ab, read 2026-10-25

Endpoints (engine :8001)

What process_job does

1. Writes job.content to a temp file with a .csv suffix.
2. load_requirements_from_csv sets total_rows.
3. run_csv_ingestion_job_with_report(..., requirements_root, dry_run=False) validates each row and writes requirements/<ID>/definition.yml via write_requirement_definition.
4. Copies per-row issues into JobIssueRecords, sets processed_rows = total_rows, then completed (or failed with an internal_error issue).

AppState builds JobManager() with no root, so the root is tempfile.gettempdir()/traceo-ingestion inside the engine container. Nothing in this path opens a Postgres connection. The engine has a repositories/postgres.py with scoped_connection(workspace_id), but a search on 2026-10-25 found no caller of it from server.py or ingestion/. So a completed job leaves YAML on an ephemeral container disk and no rows in requirements.

.xlsx and .xls pass the extension check, but step 1 always parses the bytes as CSV. An Excel upload most likely fails during parsing. That is inferred from the code and was not executed (UNVERIFIED).

Job state machine

stateDiagram-v2
  [*] --> pending : POST /jobs/ingest (202)
  pending --> running : background task starts
  running --> completed : report collected
  running --> failed : any exception
  completed --> [*]
  failed --> [*]

JobStatus in jobs.py has pending, running, completed and failed. The database enum job_status in 008 is pending, processing, completed, failed, cancelled, a different vocabulary, used by the ingestion_jobs table. Only the uncalled engine/repositories/postgres.py references that table. There is no cancel. A restart drops every job, because JobManager is a dict behind a lock.

No webhook, and no live web caller

The v1.0 card said the engine posts a completion webhook to the MCP server. It does not. A search of engine/src/engine/ for webhook, httpx.post and requests.post returned nothing. The client learns the outcome only by polling GET /jobs/{id}.

The web client’s engineClient calls GET /runs, a route the engine does not serve. A search of web/ for jobs/ingest found no caller. The requirement import that users actually reach is the MCP tool import_requirements (traceo-008), which writes through the RLS-scoped repository.

sequenceDiagram
  participant C as Caller with JWT
  participant RL as RateLimit + Auth
  participant E as /jobs/ingest
  participant JM as JobManager (memory)
  participant D as Engine temp disk
  C->>RL: POST /jobs/ingest (file)
  RL->>E: authenticated
  E->>JM: create_job(job_id, content)
  E-->>C: 202 job_id, pending
  JM->>JM: running
  JM->>D: write requirements/ID/definition.yml per row
  JM->>JM: completed with issues
  loop poll
    C->>E: GET /jobs/job_id
    E-->>C: status, progress, issues
  end

Rust backend

engine/railway.toml builds with DISABLE_RUST=1, and the live /health on 2026-10-25 reported rust_backend: fallback. Production runs the pure-Python validators.

Key Terms

• JobManager → an in-memory, lock-guarded registry of jobs. It does not persist across restarts.
• definition.yml → the per-requirement YAML file the ingestion writes under requirements_root.
• requirements_root → tempfile.gettempdir()/traceo-ingestion by default, inside the container.
• JobIssueRecord → a validation finding with a severity and an optional row_number.

Q&A

Q: What HTTP status does /jobs/ingest return on accept? A: 202 Accepted, with {job_id, status: "pending"}. A missing filename or a disallowed extension gets 400.

Q: What are the four job states? A: pending, running, completed, failed (JobStatus in ingestion/jobs.py). The database enum’s processing/cancelled values belong to the ingestion_jobs table, which the live path never writes.

Q: Where does a processed row end up? A: In a YAML file, requirements/<ID>/definition.yml, under the engine container’s temp directory. Not in Postgres.

Q: How does the client know the job has finished? A: By polling GET /jobs/{job_id}. There is no webhook and no push.

Examples

You upload reqs.csv and the job reaches completed with zero issues, but the web app shows nothing new. That is expected on this path: the rows became YAML files on the engine’s disk, and the next redeploy erases them. To get requirements into a workspace, use the MCP import_requirements tool.