In-Memory Registry Architecture

ELI5

The registries are like three specialised filing cabinets that live entirely in the computer’s memory (RAM). The sprite cabinet has a special index by name+version so you can’t accidentally file two identical cards. The council cabinet has an index by domain. The history cabinet has an index by chain so you can quickly find all past runs. Because they’re in memory, they’re very fast — but everything disappears if you restart the server.

Technical Deep Dive

Registry Overview

classDiagram
    class SpriteRegistry {
        +dict[UUID, Sprite] _sprites
        +dict[(str, str), UUID] _name_version_index
        +asyncio.Lock _lock
        +create(sprite_data, fingerprint) Sprite
        +get_by_id(uuid) Sprite|None
        +get_by_name(name) Sprite|None
        +update(uuid, data, fingerprint) Sprite
        +delete(uuid) void
        +list_all() Sprite[]
        +list_by_tag(tag) Sprite[]
        +count() int
        +verify_fingerprint(sprite, hash) bool
        +clear() void
    }
    class CouncilRegistry {
        +dict[UUID, Council] _councils
        +dict[str, UUID] _domain_index
        +SpriteRegistry _sprite_registry
        +asyncio.Lock _lock
        +create(domain, sprites, gates, chains, rules) Council
        +get_by_id(uuid) Council|None
        +get_by_domain(domain) Council|None
        +list_all() Council[]
        +count() int
        +delete(uuid) void
        +clear() void
    }
    class ExecutionHistoryRegistry {
        +dict[UUID, ExecutionHistory] _history
        +dict[UUID, UUID[]] _chain_index
        +asyncio.Lock _lock
        +create(history) ExecutionHistory
        +get_by_id(id) ExecutionHistory|None
        +list_by_chain(chain_id, limit, offset, status) tuple
        +count() int
        +clear() void
    }
    SpriteRegistry --> CouncilRegistry : referenced by

SpriteRegistry

Primary storage: dict[UUID, Sprite] — O(1) lookup by UUID Secondary index: dict[(name, version), UUID] — O(1) duplicate detection

class SpriteRegistry:
    def __init__(self):
        self._sprites: dict[UUID, Sprite] = {}
        self._name_version_index: dict[tuple[str, str], UUID] = {}
        self._lock = asyncio.Lock()

    async def create(self, sprite_data, fingerprint):
        async with self._lock:
            key = (sprite_data.name, sprite_data.version)
            if key in self._name_version_index:
                raise ConflictError(f"Sprite {key} already exists")
            sprite = Sprite(
                id=uuid4(),
                fingerprint=fingerprint,
                created=datetime.now(timezone.utc),
                **sprite_data.model_dump()
            )
            self._sprites[sprite.id] = sprite
            self._name_version_index[key] = sprite.id
            return sprite

Key operations:

• create() — Rejects duplicates, generates UUID and timestamp
• get_by_id() — Direct dict lookup
• get_by_name() — Linear scan (returns first match, not version-aware)
• update() — Validates version monotonicity, updates both stores
• delete() — Removes from both main dict and index
• list_by_tag(tag) — Searches metadata.tags in all sprites
• verify_fingerprint() — Case-insensitive hash comparison

CouncilRegistry

Primary storage: dict[UUID, Council] Secondary index: dict[str, UUID] — domain → council mapping

class CouncilRegistry:
    def __init__(self, sprite_registry: SpriteRegistry | None = None):
        self._councils: dict[UUID, Council] = {}
        self._domain_index: dict[str, UUID] = {}
        self._sprite_registry = sprite_registry
        self._lock = asyncio.Lock()

    async def create(self, domain, sprite_ids, gate_agent_ids, chains, rules):
        async with self._lock:
            if domain in self._domain_index:
                raise ConflictError(f"Domain '{domain}' already exists")
            # Cross-validation: all sprite IDs must exist
            for sprite_id in sprite_ids:
                if not self._sprite_registry.get_by_id(sprite_id):
                    raise NotFoundError(f"Sprite {sprite_id} not found")
            # ... validation continues

Key feature: Accepts an optional SpriteRegistry reference for cross-validation during council creation.

ExecutionHistoryRegistry

Primary storage: dict[UUID, ExecutionHistory] Secondary index: dict[UUID, list[UUID]] — chain_id → execution_ids

class ExecutionHistoryRegistry:
    async def list_by_chain(self, chain_id, limit=20, offset=0, status=None):
        async with self._lock:
            execution_ids = self._chain_index.get(chain_id, [])
            results = [
                self._history[eid] for eid in execution_ids
                if status is None or self._history[eid].status == status
            ]
            # Sort by completed_at DESC, then paginate
            results.sort(key=lambda x: x.completed_at or x.started_at, reverse=True)
            return results[offset:offset+limit], len(results)

Singleton Management

flowchart LR
    A["Module load"] --> B["_sprite_registry = None"]
    B --> C["get_sprite_registry()"]
    C -->|First call| D["Create SpriteRegistry()"]
    C -->|Subsequent| E["Return existing"]
    D --> F["Lazy singleton"]
    E --> F

All three registries use lazy singleton pattern:

• get_sprite_registry() — Creates on first call
• get_council_registry() — Links to SpriteRegistry on creation
• get_execution_history_registry() — Independent singleton
• initialize_registries() — Forces creation at startup
• reset_registries() — Clears all (testing only)

Thread Safety

All operations are wrapped with asyncio.Lock:

• Reads and writes are atomic within the lock context
• Prevents race conditions during concurrent sprite creation
• Serializable isolation (equivalent to ACID transactions)

Limitations:

• No persistence across restarts
• All data lost on crash
• Memory usage grows with sprite/council count
• No distributed locking (single-process only)

Key Terms

• In-memory registry → A thread-safe dictionary store using asyncio.Lock for concurrent access
• Secondary index → Additional lookup structure (e.g., name+version → UUID) for efficient queries
• Lazy singleton → Object created on first access, reused thereafter
• Cross-validation → CouncilRegistry verifying sprite IDs exist in SpriteRegistry before creation
• Case-insensitive comparison → Fingerprint hashes compared in lowercase to prevent encoding issues
• Serializable isolation → Lock-based concurrency ensuring operations appear to execute one at a time

Q&A

Q: Why in-memory instead of PostgreSQL? A: The in-memory registries enable rapid development and testing. PostgreSQL ORM models are defined in db_models.py and the async session factory is wired, but init_db() is a placeholder. Future versions will migrate to PostgreSQL persistence.

Q: What happens to data when the server restarts? A: All data is lost. This is acceptable for development but not production. The Docker Compose stack includes PostgreSQL for future persistence.

Q: How much memory do the registries use? A: Roughly proportional to the number of sprites × average sprite size. A typical sprite YAML is 1–5 KB. With 1,000 sprites, expect ~5 MB of memory usage.

Q: Can multiple iris-service instances share registries? A: No. Each process has its own singleton registries. For multi-instance deployments, PostgreSQL persistence or a shared cache (Redis) is required.

Q: How do I reset registries for testing? A: Call await reset_registries() from the test suite. This clears all three registries atomically.

Examples

The registries are like a doctor’s office filing system:

• SpriteRegistry = Patient records filed by medical ID number, with an alphabetical index by name+date of birth
• CouncilRegistry = Appointment schedules filed by clinic room number (domain), with a cross-reference checking that every scheduled patient actually exists
• ExecutionHistoryRegistry = Visit logs filed by appointment type (chain_id), so you can quickly pull up all past visits of a specific type
• asyncio.Lock = The “one person at a time” rule at the filing cabinet — prevents two nurses from grabbing the same folder simultaneously
• In-memory = The records are on the desk, not in the basement archive — fast to access, but if someone bumps the desk, papers scatter