System Architecture Overview

ELI5

Skyflow is like a restaurant where the front desk (Core API) takes orders, a Python-speaking sous-chef (Timeline) cooks the AT Protocol data, and a scorekeeper (Gamification) hands out XP. Instead of the cooks shouting at each other, every order is dropped into a NATS conveyor belt that the right station picks up. A loudspeaker (Realtime Gateway) tells diners as soon as their dish is ready.

Technical Deep Dive

Skyflow v2 is an event-driven, service-oriented stack with five backend services behind a Caddy gateway. Services never call each other directly for state mutations — they publish protobuf events on NATS JetStream and downstream consumers react.

Services

C4 Context

---
title: "Skyflow v2 — Container Context"
---
flowchart TD
  user(("<b>Skyflow User</b><br/>Bluesky account holder")):::person
  bsky["<b>AT Protocol / Bluesky</b><br/>External social graph + posts"]:::ext
  polar["<b>Polar.sh</b><br/>Subscription billing"]:::ext
  subgraph sf ["**Skyflow**"]
    caddy["<b>Caddy v2</b><br/><i>TLS + routing</i>"]:::container
    core["<b>core-api</b><br/><i>Go/Fiber</i>"]:::container
    tl["<b>timeline-service</b><br/><i>Python/gRPC</i>"]:::container
    gam["<b>gamification-service</b><br/><i>Go/gRPC</i>"]:::container
    rt["<b>realtime-gateway</b><br/><i>Go/SSE</i>"]:::container
    er["<b>event-router</b><br/><i>Go</i>"]:::container
    pg[("<b>Postgres 16</b><br/><i>Users, subs, XP, analyses</i>")]:::db
    rd[("<b>Redis 7</b><br/><i>Leaderboards, sessions, rate limits</i>")]:::db
    nats["<b>NATS JetStream</b><br/><i>Event bus</i>"]:::container
  end
  user -- "HTTPS + SSE" --> caddy
  caddy -- "/api/*, /webhooks/*" --> core
  caddy -- "/sse" --> rt
  core -- "pgx" --> pg
  core -- "sessions, rate" --> rd
  core -- "publish" --> nats
  core -- "checkout API" --> polar
  polar -- "webhooks" --> core
  tl -- "atproto SDK" --> bsky
  tl -- "consume + publish" --> nats
  gam -- "consume + publish" --> nats
  gam -- "ZADD leaderboards" --> rd
  rt -- "consume realtime.>" --> nats
  er -- "consume all" --> nats
  er -- "audit_log, dlq_events" --> pg


  classDef person fill:#1c1c24,stroke:#e85d3e,color:#f0ece6
  classDef system fill:#1c1c24,stroke:#d4a574,color:#f0ece6
  classDef ext fill:#141419,stroke:#8b7e74,color:#f0ece6,stroke-dasharray: 4 3
  classDef db fill:#1c1c24,stroke:#d4a574,color:#f0ece6
  classDef container fill:#1c1c24,stroke:#d4a574,color:#f0ece6

Request → Event Flow

flowchart LR
    C[Client] -->|HTTPS| G[Caddy]
    G -->|REST| API[core-api]
    API -->|publish events.timeline.requested| N((NATS))
    N --> TL[timeline-service]
    TL -->|publish events.timeline.completed| N
    N --> GAM[gamification-service]
    GAM -->|publish events.xp.earned| N
    N --> RT[realtime-gateway]
    RT -->|SSE event: xp_earned| C

Key Terms

• Service-oriented (not micro) → 5 services, each with distinct ownership; avoids the distributed-monolith trap
• Event-driven mutation → state changes are published as protobuf events; only reads use direct gRPC
• Caddy v2 → API gateway providing automatic TLS and path-based routing
• AT Protocol → Bluesky’s federated social protocol; the only external read source

Q&A

Q: Why is Timeline Service Python while everything else is Go? A: The official atproto SDK and the existing analysis logic (Flow/Clout/Kudos/Pulse) are Python; rewriting in Go was rejected as wasted effort.

Q: Which service receives Polar webhooks? A: core-api (POST /webhooks/polar). It verifies the HMAC signature, then re-publishes a corresponding events.subscription.* event onto NATS so the rest of the system reacts asynchronously.

Q: Why does Gamification not call Timeline directly to award XP? A: To keep services loosely coupled. Timeline publishes events.timeline.completed; Gamification consumes it. Adding a new XP source means subscribing to a new subject, not modifying Timeline.

Q: Where does the Real-Time Gateway live in the request path? A: It is a separate long-lived SSE endpoint at GET /sse (port 8003 behind Caddy). It does not sit inline with REST traffic — clients open it once and keep it open.

Examples

A pizza shop: the cashier (core-api) takes the order and drops a ticket onto a rail (NATS). The pizza chef (timeline) makes the pizza and clips a “ready” ticket back. The points clerk (gamification) sees the ticket and stamps loyalty points. The display board (realtime-gateway) shouts the customer’s name as soon as the points stamp is logged.