LLM Serving Infrastructure

A 70B parameter model in FP16 requires ~140GB of memory, far exceeding a single A100's 80GB capacity. Tensor parallelism (TP) splits individual layers across GPUs — each GPU holds a slice of every weight matrix and computes a portion of each layer's output. The partial results are combined via all-reduce communication after each layer. TP requires high-bandwidth interconnect (NVLink at 900GB/s) because every layer involves a synchronization point. Pipeline parallelism (PP) splits the model by layers — GPU 0 gets layers 0–19, GPU 1 gets layers 20–39, etc. PP has less communication overhead (only activation tensors between stages) but introduces pipeline bubbles where GPUs wait for earlier stages. In practice, production systems use TP within a node (2–8 GPUs connected via NVLink) and PP across nodes (connected via InfiniBand). vLLM and TensorRT-LLM support both strategies with automatic configuration based on model size and available hardware.

Traditional static batching waits for a batch of requests to accumulate, processes them together, and returns all results simultaneously. This creates two problems: short requests must wait for the longest request in the batch to finish (head-of-line blocking), and the GPU sits idle between batches. Continuous batching (also called iteration-level scheduling, pioneered by Orca) processes requests at the granularity of individual decoding steps. When a request finishes generating (hits EOS or max length), a new request immediately takes its slot in the batch — no GPU cycles are wasted. vLLM's scheduler maintains a priority queue of pending requests, dynamically adjusting the batch size every iteration. This increases throughput by 2–4× compared to static batching at the same latency target. The key challenge is memory management: each active request maintains a KV cache that grows with sequence length, so the scheduler must balance batch size against available GPU memory.

Quantization reduces model weights from 16-bit floating point to lower precision (8-bit, 4-bit, or even 2-bit), dramatically reducing memory footprint and enabling faster inference. GPTQ (post-training quantization) uses calibration data to find optimal quantized weights that minimize output error — it processes one layer at a time, solving a layer-wise reconstruction problem. AWQ (Activation-Aware Weight Quantization) observes that only 1% of weights are critical for model quality (those corresponding to large-magnitude activations) and preserves these at higher precision. GPTQ and AWQ at 4-bit reduce memory by 75% with minimal quality degradation (typically <1% on benchmarks). FP8 quantization is emerging on H100 GPUs with native hardware support, offering 2× memory reduction with virtually no quality loss. The inference speed improvement from quantization is primarily memory-bandwidth-bound: LLM decoding is limited by how fast weights can be loaded from HBM to compute cores, so smaller weights mean proportionally faster inference.

Triton Inference Server is NVIDIA's production inference platform that handles the operational complexity of serving multiple models simultaneously. It supports all major frameworks (PyTorch, TensorFlow, TensorRT, ONNX, vLLM) behind a unified gRPC/HTTP API. Key features: dynamic batching aggregates individual requests into GPU-efficient batches with configurable maximum latency, model ensembles chain multiple models in a single request (e.g., tokenizer → LLM → post-processor), model versioning enables A/B testing and canary deployments, and concurrent model execution shares a single GPU across multiple models using CUDA MPS (Multi-Process Service). For LLM serving specifically, Triton integrates with TensorRT-LLM for optimized inference kernels and supports in-flight batching (continuous batching). Resource management is critical: Triton's rate limiter prevents GPU OOM by tracking memory usage per model instance and queuing requests when capacity is exceeded.

GPU instances cost $1–$30/hour depending on type (A10G, A100, H100), making autoscaling essential for cost management. Kubernetes Horizontal Pod Autoscaler (HPA) scales LLM serving pods based on custom metrics: GPU utilization, request queue depth, tokens-per-second throughput, or p99 latency. The fundamental challenge is cold start time — loading a 70B model takes 30–120 seconds, during which the new pod cannot serve requests. Mitigation strategies include: pre-warming pools (keeping standby pods with models loaded), model caching on NVMe (loading from local SSD is 5–10× faster than network storage), and predictive scaling (using historical traffic patterns to scale proactively). Spot/preemptible GPU instances reduce costs by 60–70% but require graceful handling of interruptions — request draining, checkpoint saving, and automatic migration to on-demand instances. Multi-tier serving routes simple queries to smaller/cheaper models (7B on A10G at $1/hour) and complex queries to larger models (70B on A100 at $10/hour), optimizing the cost-quality tradeoff per request.