Pack and Extract: zstd Payload, BLAKE3 Checks, Read-Only Store

ELI5

Packing squeezes the files into one zstd-compressed bundle and stamps its fingerprint in the label. Unpacking is the reverse, but only after the fingerprint, the signatures and the builder’s receipt all check out, and the unpacked files are then locked read-only so nothing can quietly edit them.

Technical Deep Dive

Source: nestr-tools/engine internal/pellets/pellets.go:1-16,289-370,491-530,555-600, internal/pellets/cas.go:1-25, internal/server/rest.go:156-164 @e9180e4; nestr-tools docs/north-star/intent.md:45-49 @b8f0585; read 2026-10-25. The previous revision’s io.MultiWriter(file, sha256) pipeline and sha256 checksum no longer exist.

Pack

Store.Compress(name, sourcePath, arch, level) / Store.Pack(opts) walk the source into a tar, encode it with klauspost zstd into a buffer, put payload_blake3 and payload_size into the header (PEL-ART-002, PEL-ART-012), frame MAGIC + header + payload + footer, compute the PID, and write into the CAS directory.

Extract

sequenceDiagram
  participant C as caller
  participant S as Store.EnsureExtracted
  participant V as verifiers
  participant FS as CAS
  C->>S: EnsureExtracted(pid)
  S->>FS: extracted dir non-empty?
  alt already extracted and no below-L3 marker
    S-->>C: path, no work
  else
    S->>S: recompute PID and payload_blake3
    S->>V: signature, provenance, log, policy
    V-->>S: pass or refuse
    S->>FS: zstd decode, tar extract, no exec
    S->>FS: chmod files 0444, dirs 0555
    S-->>C: extracted path
  end

There is no unverified mode: a Store with no footer verifier returns ErrNoVerifier (PEL-SIG-020, PEL-SIG-021 spirit). A pellet accepted below SLSA L3 by an opted-out policy carries a marker file, and every later EnsureExtracted re-runs all gates on it under the current verifiers (N44, I-7).

Why ratio is not the point

flowchart LR
  R["better zstd ratio"] --> Q{"paired with a security PR?"}
  Q -- "no" --> DW["drift watch 1: pellets-as-compression"]
  Q -- "yes" --> OK["acceptable, cite PEL-* IDs per I-1"]

intent.md:45-49 names “pellets-as-compression” as drift watch 1: benchmarks improving while signature, sandbox and log work sits unenforced. The REST route POST /api/pellets/compress still exists on the hosted engine and is a Community-capped producer convenience (nestr-025); POST /api/pellets/extract runs the verification gate and is never capped.

Key Terms

• payload_blake3 → BLAKE3 of the zstd payload, stored in the header and rechecked before extract.
• EnsureExtracted → idempotent verify-then-extract into the CAS.
• provenance marker → file marking a pellet accepted below L3; forces re-verification.

Q&A

Q: Which zstd encoder preset does level 7 select? A: SpeedDefault (levels 5-9). 1-4 select SpeedFastest, 10 and above SpeedBestCompression.

Q: What does EnsureExtracted check before extracting, and in what order? A: It first short-circuits if the CAS extracted/ dir already has content and no below-L3 marker. Otherwise it recomputes the PID and payload_blake3, then runs the configured signature, provenance, log and policy verifiers, and only then decodes and extracts, finishing with a read-only chmod.

Q: Why is compression ratio not a success metric for nestr? A: Because the north star names optimising ratio without paired security work as drift watch 1. Speed and size are side effects of content addressing and zstd, not the claim.

Examples

Two installers request the same PID. The first verifies and extracts into store/blake3/ab/c1/<hex>/extracted/; the second finds the directory populated and returns immediately, and both workspaces get hardlinks into the same read-only tree.