URL Shortener

System Design series #1 (EP22). Topic skill: .claude/agents/system-architect/skills/url-shortener/SKILL.md. The task is a TinyURL or bit.ly at scale, from requirement to operation.

The problem

The product fits in one sentence: take a long URL, return a short one, redirect to the original. Underneath sit most core system design themes: read-heavy load, unique key generation, cache, partitioning, consistency, abuse, analytics, multi-region, cost, and evolution.

There are two very different flows. Create is a moderate write: a link is created once. Resolve is a huge, latency-critical read: that link may be read millions of times. Design around redirect and bend everything else to it. Product and priority come first, technology second.

Requirements

Functional: long URL to unique short URL, resolve short to redirect, optional expiry, custom alias, basic analytics (click, timestamp, user agent, referer, approximate country), and disabling or banning malicious links.

Non-functional, in priority order: redirect availability first (a create failing for a few seconds is bad, a redirect failing kills the product), then redirect latency in tens of ms on cache hit, durability of every issued code, horizontal read scale, security and anti-abuse (the product turns into a phishing, malware, and spam vector fast), and observability with audit.

Staff-level extras: TTL links, logical delete with tombstone, optional dedup, preview page for suspicious links, rate limit per account, IP, and tenant, multi-tenant custom domains, and different SLAs for redirect and analytics.

Scale sizing

Assumptions: 100M new links per month, 3B redirects per month, read:write about 30:1, peak 5x average, 5-year retention.

Writes:  100M / 2.6M s  ~ 38 wps avg,   ~200 wps peak.   Trivial.
Reads:   3B   / 2.6M s  ~ 1157 rps avg, ~6k rps peak.    Comfortable.

Enterprise clients, a viral campaign, an event QR code, or global usage push reads to tens or hundreds of thousands per second, so design to grow even if v1 is small.

Storage is about 1 KB per record (code 8 to 10 B, long URL about 500 B, metadata 100 to 200 B). 6B records over 5 years is about 6 TB raw, 15 to 25 TB with indexes, replication, and backup. Hot data is small and total data is large, which calls for heavy caching in front of durable, partitionable storage.

Analytics is 3B events per month. Never keep a synchronous counter on the transactional DB; decouple it.

API

POST /v1/links   {long_url, custom_alias?, expires_at?, domain?, idempotency_key?}
              -> {short_url, code, created_at, expires_at}

GET /{code}   -> 301 if the mapping is immutable (lowest perceived latency on repeat)
                 302/307 if you want flexibility (avoids aggressive client caching when target
                 may change)

301 versus 302 is control versus efficiency. Clients and intermediaries cache a 301 hard, so repeats are instant but changing or revoking the link gets expensive. A 302 keeps control at the cost of a server round trip each time.

Three product questions change the architecture: can a link be edited after create, can an alias be reused after expiry, and does the same long URL get the same code. They drive idempotency, cache, and invalidation.

Data model

Two domains, kept apart. The transactional links table:

code PK, long_url, url_hash?, owner_id?, domain, created_at, expires_at?,
status(active|disabled|expired|banned), is_custom, redirect_type, metadata_json

Indexes: PK on code, (owner_id, created_at), expires_at if TTL cleanup runs often, url_hash if you dedup.

Analytics events go to a columnar store, data lake, or stream, asynchronously and never on the synchronous redirect path:

code, timestamp, ip_prefix or hashed IP, user_agent_hash, referer_domain, country, device_type

Short-code generation

Hash of the long URL, truncated, base62. Deterministic and easy to dedup, but truncation collides, the same input always yields the same code (bad when two users want distinct links for one landing page), and the output is predictable and enumerable.

Sequential ID, base62. Short and collision-free with a good generator, but predictable: it leaks your total volume and anyone can scrape it by incrementing.

Unique ID plus reversible obfuscation, base62 (recommended). Generate a 64-bit unique ID, pass it through a keyed bijection (a Feistel network or another bijective permutation), then base62-encode, with an optional fixed-length pad. The bijection keeps uniqueness and removes predictability: you cannot guess neighbors without the key.

A Feistel network (Horst Feistel, IBM, 1970s, the basis of DES) turns any function into an invertible permutation over a fixed bit width. For short codes it gives a 1:1, reversible, key-dependent scramble of the ID space. This is the standard "encrypt the counter" trick.

Base62 is [A-Za-z0-9]. Capacity is 62^7 ~ 3.5 trillion and 62^8 ~ 218 trillion. Seven characters fit most platforms; eight give operational slack. Custom aliases bypass this.

The staff answer: unique IDs allocated centrally by ranges (or decentralized with a uniqueness guarantee), a reversible bijection against predictability, base62 encoding, and a unique index in storage as the last line of defense.

Unique ID generation

Twitter's Snowflake (2010) is the canonical 64-bit scheme: about 41 bits timestamp, 10 bits machine, 12 bits sequence. Range allocation and Snowflake both work for a URL shortener.

Create flow

Validate the URL, check availability and policy if there is a custom alias, generate the code, persist to the transactional DB, write through to cache, return short_url.

URL validation is a security control. Accept only http and https. Block SSRF targets: 127.0.0.1, 169.254.169.254 (cloud metadata), RFC1918 private ranges, internal hostnames. Canonicalize, enforce a size limit, and handle punycode and suspicious characters. SSRF (Server-Side Request Forgery) is your validator fetching an attacker-supplied internal URL.

Idempotency: store the response per idempotency_key for a short window so a client retry after a timeout does not mint duplicate links.

Dedup defaults to no. Global dedup by long URL breaks per-campaign and per-tenant analytics, leaks privacy (a user learns someone already shortened that link), and blocks distinct links for the same landing page. If you want it, dedup only as an internal storage optimization, separating the logical link from the physical URL target.

Redirect flow

sequenceDiagram
  Client->>Edge: GET /abc123X
  Edge->>Cache: lookup code
  alt cache hit
    Cache-->>Edge: target
  else miss
    Edge->>DB: read (replica), validate status + expiry
    DB-->>Edge: target
    Edge->>Cache: populate (TTL)
  end
  Edge-)Analytics: emit click event (async)
  Edge-->>Client: 301/302 Location

Negative caching: when someone hammers random codes, every miss hits the DB in a miss storm. Cache the "nonexistent" result for 30 to 60 s.

TTL: if the mapping is immutable, TTL can be hours and invalidation nearly disappears. If links can be disabled or edited, pick a short TTL, event-based invalidation, or split layers: cache the destination hard and keep a fast blacklist layer for urgent blocks.

Cache and the read path

This is a cache-heavy read path. L1 is a local in-process cache for extreme hot keys (small, short TTL). L2 is a distributed cache (Redis) shared across instances. The DB is the source of truth.

A viral link concentrates load on one hot key. Single-flight (request coalescing) on miss means that when 1000 requests miss at once, one fetches from the DB and the rest wait on its result. Without it, a cold hot key causes a cache stampede (thundering herd, dog-piling) that can topple the DB.

Refresh-ahead refreshes a hot entry before it expires so it never goes cold under load. The probabilistic variant (Vattani et al., Optimal Probabilistic Cache Stampede Prevention, 2015) refreshes early with a probability that rises as expiry approaches. Make sure the hot value fits in L1 and limit concurrent refresh.

Storage choice

The workload is PK lookup by code, few relations, moderate write, very high read, strong durability. SQL or a persistent KV both fit. What matters is fast lookup by key, mature replication, reliable backup and restore, and team familiarity.

The pragmatic pick is Postgres with partitioning when needed, read replicas, and heavy cache in front. Move to a Dynamo or Cassandra style distributed KV only when scale forces it. The staff answer is the smallest system that holds the load with margin and evolves safely. Amazon Dynamo (DeCandia et al., 2007) is the reference for the distributed-KV end: consistent hashing, quorum reads and writes, eventual consistency.

Partitioning

Partition by code or its internal ID. Hash partitioning spreads load evenly and suits random lookup, but rebalancing is harder and there is no time locality. Consistent hashing (Karger et al., 1997) minimizes the keys that move when you add or remove a node.

Range partitioning by time or ID gives good archive, lifecycle, and locality, but a monotonic ID creates a hot newest shard. For redirect lookup prefer hash or a pseudo-random distribution over the obfuscated ID. Analytics partitions by time.

Consistency

Strong consistency is required for custom-alias uniqueness (unique index on (domain, code)), for persisting the link before the success response, and for critical admin status changes. Eventual consistency is fine for analytics, cross-region DR replication, and aggregated dashboards.

Read-after-write: a user who creates a link and clicks it at once expects it to work, even if a replica lags. Solve it with region-sticky reads for a few seconds, a write-through cache (the cleanest, since the redirect reads what the create wrote a moment earlier), or a fallback to primary when the replica lags.

Multi-region

Separate create and resolve. Resolve is read-dominant and cacheable, so push it to the edge and to multiple regions as a stateless regional service with a strong regional cache. Create can start as a single writer in a primary region, which keeps alias uniqueness and ID generation simple.

Evolve in three steps: one create region with many redirect regions on replica plus cache, then multi-region create with per-region ID ranges or namespaces, then active-active only if the business demands it. Avoid premature active-active.

Analytics off the redirect path

Redirect is path A and analytics is path B; never couple them tightly. Redirect responds fast, the click event goes to a queue or log, consumers aggregate counters per minute, hour, day, country, device, and referer, and dashboards query a separate analytical store.

If the queue dies, choose best-effort (drop analytics, keep redirect), a short local buffer with retry, or sampling under degradation. Redirect always wins. Retention tiers: raw 30 days, hourly aggregate 1 year, daily aggregate 5 years. For unique visitors at this volume, HyperLogLog (Flajolet et al., 2007) estimates cardinality in kilobytes instead of storing every ID.

Security and anti-abuse

Risks: phishing, malware distribution, spam, link enumeration, open-redirect abuse, SSRF during validation. Controls: rate limit per IP, token, tenant, and suspicious ASN, domain reputation at create time, safe-browsing checks (sync or async by risk), fast link disable, a preview interstitial for suspicious links, progressive friction and CAPTCHA, and stronger auth for high-volume accounts.

Anti-enumeration combines obfuscation, adequate code length, rate limiting on resolve, and scan-pattern monitoring. For privacy, minimize, truncate, or short-window-hash IPs.

Lifecycle and custom alias

Expiry: persist expires_at, validate at read time (an offline cleanup job alone lets an expired link survive in cache), evict on expiry, and run an async cleanup for archive or logical delete. Tombstone banned or removed links to prevent reuse and keep an audit trail.

Custom aliases need strong consistency (unique index on (domain, code)), reserved words (admin, login, api), per-tenant policy, and no collision with internal routes. They are a small share of traffic with high value, which justifies a stricter create flow.

Observability

Metrics: create and resolve QPS, resolve p50, p95, p99, cache hit ratio per layer, errors by class, not-found rate, banned and expired access rate, create-to-first-resolve propagation time, analytics throughput and lag. Logs: sampled access logs and audit logs for admin operations, structured with a correlation id. Tracing: full on create, sampled on the hot resolve path.

Alert on a cache hit ratio drop, a p99 rise, redirect errors over threshold, abnormal 404 growth (a scan), and analytics backlog growth.

Failure modes

Roadmap

MVP: one region, stateless API, Postgres primary plus replica, Redis cache, range ID generator, analytics on a queue, basic offline dashboard.

Medium scale: L1 local cache, hot-key control, DB partitioning or distributed storage, multi-region redirect, maturer analytics, layered reputation.

Global scale: multi-region create with per-region ranges, tested automatic failover, per-tenant custom domains, premium branded links with per-client SLA, edge compute for some redirects.

Common mistakes

Starting from technology instead of requirements. Letting analytics block the redirect path. Using a truncated hash without discussing collision and predictability. Ignoring security and abuse. Sharding early when relational plus cache still holds. Skipping cache invalidation, expiry, and read-after-write. Skipping operations: metrics, failover, degradation.

References

Martin Kleppmann, Designing Data-Intensive Applications, O'Reilly, 2017 (load parameters, consistency, replication, partitioning).

DeCandia et al., Dynamo: Amazon's Highly Available Key-value Store, SOSP, 2007 (distributed KV, consistent hashing, eventual consistency).

Karger et al., Consistent Hashing and Random Trees, STOC, 1997.

Twitter Engineering, Announcing Snowflake, 2010 (distributed unique IDs).

Horst Feistel, Cryptography and Computer Privacy, Scientific American, 1973 (Feistel networks).

Flajolet et al., HyperLogLog: the analysis of a near-optimal cardinality estimation algorithm, 2007.

Vattani, Chierichetti, Lowenstein, Optimal Probabilistic Cache Stampede Prevention, VLDB, 2015.

Michael Nygard, Release It!, 2nd ed., 2018 (circuit breaker, bulkhead, single-flight thinking).

OWASP, Server-Side Request Forgery Prevention Cheat Sheet.

This page mirrors URL Shortener in the wiki. Edit it there.