YouTube Video Processing Pipeline

When a creator uploads a video, the client splits it into chunks and uploads them in parallel via resumable upload APIs. If the connection drops, only the missing chunks need to be retransmitted. Raw chunks are stored in Google's Colossus distributed filesystem (successor to GFS). Each upload gets a unique blob ID, and metadata (title, description, creator) is written to a separate metadata store. This decoupling of content and metadata allows the processing pipeline to begin before the upload is even complete — chunks can be transcoded as they arrive.

Transcoding is embarrassingly parallel — a video is split into GOP-aligned segments (Groups of Pictures, typically 2–5 seconds), and each segment is independently encoded across a fleet of transcoding workers. Each segment is encoded into multiple codec/resolution/bitrate combinations: VP9, H.264, and AV1 at resolutions from 144p to 4K HDR. AV1 provides ~30% better compression than VP9 at the same visual quality but requires ~10× more compute. A DAG-based task scheduler manages dependencies — thumbnail generation and Content ID can run in parallel with transcoding.

Content ID compares every uploaded video against a reference database of millions of copyrighted files provided by rights holders. The system generates audio and video fingerprints — perceptual hashes that are robust to re-encoding, cropping, and speed changes. Fingerprints are compared against the reference database using approximate nearest-neighbor search. A match triggers the rights holder's policy: block the video, monetize it with ads, or track viewership statistics. Content ID runs in parallel with transcoding to avoid adding latency to the processing pipeline.

YouTube uses MPEG-DASH and HLS for adaptive bitrate streaming. Each video is available in multiple renditions (resolution × bitrate × codec), and the player dynamically switches between them based on real-time bandwidth estimation. The manifest file lists all available renditions and their segment URLs. Segments are typically 2–5 seconds long — short enough to adapt quickly to bandwidth changes, long enough to maintain compression efficiency. The player maintains a buffer of 10–30 seconds, fetching segments progressively and switching quality at segment boundaries without visible artifacts.

Rather than waiting for all renditions to complete before publishing, YouTube incrementally pushes completed renditions to its CDN. The lowest-resolution version is often available within a minute of upload, while 4K HDR may take several more minutes. Google's global CDN (with edge caches in ISPs similar to Netflix's Open Connect) serves the video segments. Popular videos are cached at edge locations worldwide; long-tail content is served from regional origin servers. Cache admission policies balance storage cost against hit rate, with ML models predicting which newly uploaded videos will go viral.