Bot Detection & Visitor Deduplication

ELI5

Search engines, chat-app previewers and scripts follow links too. The edge worker spots the obvious ones by their User-Agent and still sends them on, but does not report their click. For real people, two different “fingerprints” are computed, one at the edge and one at the API. Only the API’s one is stored, and it never resets: the same browser on the same network counts as one visitor for a link forever, not once per day.

Technical Deep Dive

Source: zoomies-edge src/middleware/bot-detect.ts:1-55, src/analytics/visitor-hash.ts:16-76, src/analytics/tracker.ts:16-77, src/index.ts:80, src/handlers/redirect.ts:163-171 (origin/main 5bdde59); purr-api internal/handler/redirect_handler.go:322-385, internal/service/analytics_service.go:70-160, 341-346, 408-417, internal/repository/link_repository.go:279-297 (bb6cfc2), read 2026-10-25.

Two bot filters

flowchart TD
    A[GET slug at the edge] --> B{"edge isBot: UA contains one of 35 patterns ?"}
    B -->|Yes| R1[Redirect, no track POST]
    B -->|No| T[Redirect, then POST /api/analytics/track]
    T --> C{"purr-api isBot: UA contains one of 8 patterns ?"}
    C -->|Yes| I1["insert click with is_bot = true, counters still incremented"]
    C -->|No| I2["insert click with is_bot = false"]

• Edge (bot-detect.ts:1-37): 35 lowercase substrings, matched with includes, including generic bot, crawl, spider, slurp, named crawlers, social unfurlers (facebookexternalhit, twitterbot, linkedinbot, whatsapp, telegrambot), monitors (uptimerobot, pingdom), headless browsers and HTTP clients (curl, wget, python-requests, go-http-client, httpx, scrapy, node-fetch, axios). No User-Agent means not a bot.
• purr-api (analytics_service.go:408-417): a shorter list of 8 (bot, crawl, spider, slurp, mediapartners, adsbot, googlebot, bingbot) that only sets the is_bot column. It does not stop IncrementClicks, RecordVisitor or IncrementTotalClicks.

Bots that pass the edge filter (for example a UA with none of the 35 substrings) are therefore counted in links.total_clicks and feed pawprintz.

Two visitor hashes, one stored

trackClickRequest parses visitor_id (redirect_handler.go:330) but HandleTrackClick never copies it into the ClickRequest, so the edge’s daily hash is discarded.

What “unique visitors” means today

RecordVisitor inserts (link_id, visitor_id) into link_visitors with ON CONFLICT DO NOTHING and bumps links.unique_visitors only on first sight (link_repository.go:279-297). With a dateless hash, that is lifetime-unique per link per IP+UA pair. The analytics queries’ count(DISTINCT visitor_id) or uniqExact(visitor_id) over a 7-day or 30-day window inherit the same definition.

The IP comes from CF-Connecting-IP, else X-Forwarded-For, at the edge (tracker.ts:25-27), and is passed through sanitizeClientIP in purr-api. purr-api has an anonymizeIP helper (analytics_service.go:419) that nothing calls; no IP column exists in link_clicks.

Key Terms

• isBot / isBotRequest — the edge User-Agent substring filter
• is_bot — purr-api’s stored flag; informational, does not gate counters
• visitor_id — in the database, sha256(ip+UA)[:16] computed by purr-api
• link_visitors — (link_id, visitor_id) set behind links.unique_visitors

Q&A

Q: How many bot patterns does zoomies-edge check? A: 35 substrings in BOT_PATTERNS (bot-detect.ts:1-37), case-insensitive via lowercasing.

Q: What three inputs go into the visitor hash? A: At the edge: IP, User-Agent and the UTC date. But the stored hash is purr-api’s, which uses only two inputs, IP and User-Agent, with no date.

Q: Do bot requests still get redirected? A: Yes. The edge filter only skips the tracking POST; the redirect, protection prompt or proxy still happen, so crawlers and link unfurlers work.

Q: Which visitor id actually lands in the database? A: purr-api’s generateVisitorHash, the first 16 hex characters of SHA-256 over IP and User-Agent. The edge’s visitor_id field is parsed and dropped.

Examples

A visitor clicks smo1.io/melty from a laptop on Monday and again on Tuesday from the same Wi-Fi and browser. links.total_clicks goes up by 2, links.unique_visitors by 1, and the Tuesday click is not a new unique visitor, contrary to what the edge’s daily hash was designed for. The same click from Slack’s unfurler (Slackbot) is caught by the edge’s bot substring and never reaches purr-api.