Pellet Data Model

ELI5

A Pellet is a vacuum-sealed bag of code: the bag itself is a zstd-compressed tarball, and a sticky label on the outside (.meta.json) lists what’s inside, when it was packed, and the receipt of contents (sha256).

Technical Deep Dive

Defined in engine/internal/pellets/pellets.go. Each pellet on disk is a pair of files in the cache directory (default ./.pellets/cache):

{id}.pellet       # tar → zstd binary
{id}.meta.json    # sidecar with the Pellet struct

Class Diagram

classDiagram
    class Pellet {
        string ID
        string Name
        Architecture Architecture
        CompressionInfo Compression
        Checksums Checksums
        time.Time CreatedAt
        string SourcePath
    }
    class CompressionInfo {
        string Algorithm
        int Level
        int64 OriginalSize
        int64 CompressedSize
        float64 Ratio
    }
    class Checksums {
        string Deps
    }
    class Architecture {
        <<enum>>
        ArchFrontend
        ArchBackend
        ArchGo
        ArchRust
        ArchPython
    }
    Pellet --> CompressionInfo
    Pellet --> Checksums
    Pellet --> Architecture

Field Notes

The README markets the system as “architecture-aware”, but no auto-detection step is observable in the source — Architecture is supplied by the caller; the enum lists the values the codebase has names for.

Key Terms

• Sidecar → the .meta.json file written next to each .pellet binary, holding the JSON-marshalled Pellet struct.
• Ratio → compressed size divided by original size; smaller is better.
• Deps checksum → sha256 over the compressed byte stream, NOT over the original directory contents.

Q&A

Q: If the source tree changes but the Pellet is rebuilt, does Deps change? A: Yes — the hasher consumes the compressed output, which is downstream of tar walking the source. Identical inputs at the same zstd level produce identical bytes and identical hashes.

Q: What happens to a pellet whose .meta.json is deleted? A: Store.Get(id) returns an error; Store.List() globs *.meta.json so it disappears from listings. The .pellet binary becomes orphaned and is only reaped by Prune if its mtime falls past the cutoff.

Q: Are levels above 9 actually different from level 9? A: Yes. Compress selects SpeedBestCompression for levels 10–22, vs SpeedDefault (5–9) and SpeedFastest (1–4); the underlying encoder honours the numeric level within those bands.

Examples

Result of compressing a 120 MB node_modules at level 19: {ID:"1714946400123456789", Algorithm:"zstd", Level:19, OriginalSize:125829120, CompressedSize:18874368, Ratio:0.15, Checksums:{Deps:"a4c3..."}} — written as 1714946400123456789.pellet plus .meta.json.