Twitter Fan-Out & Timeline Architecture

When a regular user tweets, a fan-out service takes the tweet ID and pushes it into each follower's home timeline — a Redis sorted set keyed by user ID, scored by Snowflake timestamp. For a user with 10K followers, this means 10K Redis ZADD operations per tweet. Redis sorted sets keep the timeline naturally ordered, so reads are a simple ZREVRANGE call returning the latest N tweet IDs. This approach trades significant write amplification for constant-time, sub-5ms reads on the hot path.

When Katy Perry tweets to 100M followers, pushing to 100M Redis sorted sets would take minutes and overwhelm the entire cache cluster. Instead, tweets from accounts exceeding a follower threshold (~500K) are excluded from write-time fan-out. At read time, the user's precomputed timeline is merged with fresh tweets from any followed celebrity accounts fetched on demand. This mixed approach keeps write latency bounded at the cost of slightly more complex read-path logic and marginally higher tail latency.

Twitter's Snowflake generates 64-bit unique IDs composed of 41 bits for timestamp, 10 bits for machine ID, and 12 bits for sequence number. These IDs are sortable by creation time without any database lookup, which means Redis sorted sets can use the raw ID as the score for chronological ordering. Each Snowflake worker generates up to 4,096 IDs per millisecond. This eliminates the need for a centralized auto-increment counter — a critical single point of failure at Twitter's write volume.

Each user's home timeline is a Redis sorted set capped at roughly 800 tweet IDs. At read time, the client fetches the latest 20–50 tweet IDs via ZREVRANGE, then batch-fetches actual tweet content from a separate tweet object cache in a single multi-get. Keeping timelines in-memory means the common-case read latency is under 5ms. The trade-off is memory: 800 IDs × hundreds of millions of active users requires a massive Redis fleet — Twitter operated one of the largest Redis deployments in the world.

Twitter built FlockDB, a distributed graph database optimized for adjacency-list queries like 'who follows this user.' It stores edges (follower → followee) sharded by source node ID across MySQL backends, with a graph-aware query layer supporting set operations like intersection (mutual followers) and difference. When a tweet triggers fan-out, the fan-out service queries FlockDB to retrieve the full follower list. FlockDB is optimized for high read throughput on large adjacency lists — critical when a single user can have millions of followers.