LinkedInBook a call
All articles

AI & Machine Learning

vLLM and Parallelized Inference: Scaling LLM Serving to Production

10 January 202518 min readBy Bayseian Engineering

Deep dive into vLLM architecture, continuous batching, PagedAttention, tensor parallelism, and advanced techniques for serving large language models at scale with optimal throughput and latency.

Introduction: The LLM Serving Challenge

Serving large language models (LLMs) in production presents unique challenges:

  1. 1.Memory bottlenecks: a single 70B model at FP16 needs ~140GB just for weights, before you've served a single request or allocated any KV cache.
  2. 2.KV cache management: attention needs a key-value cache per active sequence, and it grows with every token generated, so naive allocation wastes most of your GPU memory to fragmentation.
  3. 3.Throughput optimization: latency and throughput trade off directly; batching more requests together raises throughput but makes each individual request wait longer.
  4. 4.Dynamic batching: requests arrive with wildly different prompt and output lengths, so a fixed-shape batch either pads (wastes compute) or blocks (wastes time).

vLLM addresses these challenges through PagedAttention, continuous batching, and efficient memory management. These techniques were novel when vLLM introduced them and are now the baseline every serious serving stack is measured against. vLLM has since become the de-facto standard open-source inference engine, with alternatives like SGLang (whose RadixAttention excels at heavily shared-prefix workloads) and NVIDIA's TensorRT-LLM competing at the margins. This article explores vLLM's architecture and parallelization strategies for production-scale LLM serving, and the concepts transfer to whichever engine you run.

The Memory Challenge: Understanding KV Caches

KV Cache Basics:

  • •Key vectors: [batch_size, num_heads, seq_len, head_dim]
  • •Value vectors: [batch_size, num_heads, seq_len, head_dim]
  • •num_layers = 80
  • •num_heads = 64
  • •head_dim = 128
  • •dtype = float16 (2 bytes)

Memory per token = 2 80 64 128 2 bytes = 2.6 MB

For a 2048 token sequence: 2.6 MB * 2048 ≈ 5.3 GB!

Traditional serving wastes 60-80% of memory due to fragmentation.

Quantization fits the weights on the GPU, but the fragmented, unpredictable KV cache still wastes most of what is left.

PagedAttention: Virtual Memory for LLMs

How PagedAttention Works:

  1. 1.Divide KV cache into fixed-size blocks (e.g., 16 tokens)
  2. 2.Store each block in non-contiguous memory
  3. 3.Maintain block table mapping logical to physical blocks
  4. 4.Compute attention using block-level addressing
  • •2-4x better memory utilization: no more pre-allocating a worst-case contiguous block per sequence and wasting most of it.
  • •Support for longer sequences: blocks can be added on demand instead of requiring a contiguous region sized for the maximum possible length.
  • •Higher batch sizes: memory saved from eliminating fragmentation goes straight into serving more concurrent requests.
  • •Prefix caching for common prompts: shared prefixes (system prompts, few-shot examples) map to the same physical blocks instead of being recomputed and re-stored per request.

Fixed-size blocks plus a block table: non-contiguous allocation, near-zero fragmentation, prefix sharing, dynamic growth.

vLLM Architecture

Python
# vLLM Server Architecture

from vllm import LLM, SamplingParams
from vllm.engine.arg_utils import AsyncEngineArgs
from vllm.engine.async_llm_engine import AsyncLLMEngine

class vLLMServer:
    def __init__(self, model_name: str, tensor_parallel_size: int = 1):
        engine_args = AsyncEngineArgs(
            model=model_name,
            tensor_parallel_size=tensor_parallel_size,
            # PagedAttention configuration
            block_size=16,  # Tokens per block
            max_num_batched_tokens=8192,
            max_num_seqs=256,  # Max concurrent requests
            # Memory management
            gpu_memory_utilization=0.95,  # Use 95% of GPU memory
            swap_space=4,  # GB of CPU swap space
            # Performance
            enable_prefix_caching=True,
            disable_log_stats=False
        )
        
        self.engine = AsyncLLMEngine.from_engine_args(engine_args)
        
    async def generate(
        self,
        prompt: str,
        max_tokens: int = 512,
        temperature: float = 0.7
    ):
        sampling_params = SamplingParams(
            temperature=temperature,
            max_tokens=max_tokens,
            top_p=0.95
        )
        
        # Continuous batching handles request scheduling
        result_generator = self.engine.generate(
            prompt,
            sampling_params,
            request_id=generate_request_id()
        )
        
        async for output in result_generator:
            yield output

# Key features:
# - PagedAttention for memory efficiency
# - Continuous batching for throughput
# - Prefix caching for common prompts
# - Automatic request scheduling

Continuous Batching

Continuous Batching Benefits:

  1. 1.Higher GPU Utilization: a finished request's slot is refilled immediately instead of sitting idle until the whole batch clears.
  2. 2.Better Throughput: 2-3x improvement over static batching, since the GPU is doing useful work in every slot on every step.
  3. 3.Lower Latency: a 20-token request no longer sits behind a 2000-token one just because they started in the same batch.
  4. 4.Automatic Scheduling: the engine handles admission and eviction per token, so you don't hand-tune batch windows or timeouts.
  • •Track per-request generation state: each sequence's position and KV cache blocks are tracked independently, not as part of a fixed batch tensor.
  • •Add new requests to batch when slots free: admission happens continuously, not on a fixed batch boundary.
  • •Remove completed requests immediately: freed slots and blocks go back to the pool the moment a sequence hits its stop condition.
  • •Recompute attention only for active sequences: padding and wasted compute on finished/empty slots disappear.

Slots are refilled per token, not per batch, so short requests never wait for long ones.

Tensor Parallelism

Tensor Parallelism Strategies:

  1. 1.Attention Head Parallelism: splits attention heads across GPUs so each device computes a slice of every layer in lockstep, requiring an all-reduce per layer but keeping latency low.
  2. 2.Layer-wise Parallelism: different layers live on different GPUs (pipeline stages), trading latency for simpler communication since GPUs don't need to sync mid-layer.
  3. 3.Pipeline Parallelism: micro-batches flow through the GPU pipeline stage by stage, keeping every GPU busy instead of idling while stage 1 waits on stage 4.
  • •Model too large for single GPU (>40GB): tensor parallelism is a memory-capacity decision first, a latency decision second.
  • •Need lower per-request latency: splitting compute across GPUs speeds up each individual request, unlike data parallelism which only adds throughput.
  • •Have high-bandwidth GPU interconnect (NVLink, InfiniBand): the all-reduce after every layer is chatty; without fast interconnect it becomes the bottleneck and erases the latency win.

Each GPU processes a different slice of attention heads in parallel; NCCL stitches the results back together.

Production vLLM Deployment

Python
# production_vllm_server.py
import asyncio
from vllm import AsyncLLMEngine, AsyncEngineArgs, SamplingParams
from prometheus_client import Counter, Histogram, Gauge
import logging

# Metrics
REQUEST_COUNT = Counter('vllm_requests_total', 'Total requests')
REQUEST_DURATION = Histogram('vllm_request_duration_seconds', 'Request duration')
GPU_MEMORY = Gauge('vllm_gpu_memory_used_bytes', 'GPU memory used')
ACTIVE_REQUESTS = Gauge('vllm_active_requests', 'Active requests')

class ProductionvLLMServer:
    def __init__(self):
        self.engine_args = AsyncEngineArgs(
            model="meta-llama/Llama-2-70b-chat-hf",
            tensor_parallel_size=4,  # 4x A100 80GB
            dtype="float16",
            # PagedAttention config
            block_size=16,
            max_num_batched_tokens=16384,
            max_num_seqs=512,
            gpu_memory_utilization=0.95,
            # Performance optimizations
            enable_prefix_caching=True,
            enable_chunked_prefill=True,
            # Reliability
            disable_log_stats=False,
            max_log_len=100,
        )
        
        self.engine = AsyncLLMEngine.from_engine_args(self.engine_args)
        self.logger = logging.getLogger(__name__)
        
        # Start monitoring
        asyncio.create_task(self._monitor_resources())
    
    async def generate(
        self,
        prompt: str,
        max_tokens: int = 512,
        temperature: float = 0.7,
        request_id: str = None
    ):
        REQUEST_COUNT.inc()
        ACTIVE_REQUESTS.inc()
        
        try:
            sampling_params = SamplingParams(
                temperature=temperature,
                max_tokens=max_tokens,
                top_p=0.95,
                frequency_penalty=0.1,
                presence_penalty=0.1,
                stop=["</s>", "User:", "Assistant:"]
            )
            
            with REQUEST_DURATION.time():
                result_generator = self.engine.generate(
                    prompt,
                    sampling_params,
                    request_id=request_id or self._generate_id()
                )
                
                async for output in result_generator:
                    # Stream tokens back to client
                    yield {
                        'text': output.outputs[0].text,
                        'finished': output.finished,
                        'tokens': len(output.outputs[0].token_ids)
                    }
                    
        except Exception as e:
            self.logger.error(f"Generation error: {e}", exc_info=True)
            raise
        finally:
            ACTIVE_REQUESTS.dec()
    
    async def _monitor_resources(self):
        """Monitor GPU memory and engine stats"""
        while True:
            try:
                stats = await self.engine.get_model_config()
                # Update Prometheus metrics
                GPU_MEMORY.set(stats.gpu_memory_used)
                
                self.logger.info(
                    f"vLLM Stats - Active: {stats.num_active_seqs}, "
                    f"GPU Memory: {round(stats.gpu_memory_used / 1e9, 2)}GB"
                )
            except Exception as e:
                self.logger.error(f"Monitoring error: {e}")
            
            await asyncio.sleep(10)
    
    async def health_check(self) -> dict:
        """Health check endpoint"""
        try:
            stats = await self.engine.get_model_config()
            return {
                'status': 'healthy',
                'model': self.engine_args.model,
                'gpu_memory_utilization': stats.gpu_memory_used / stats.gpu_memory_total,
                'active_requests': stats.num_active_seqs
            }
        except Exception as e:
            return {'status': 'unhealthy', 'error': str(e)}

# FastAPI Integration
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel

app = FastAPI()
server = ProductionvLLMServer()

class GenerateRequest(BaseModel):
    prompt: str
    max_tokens: int = 512
    temperature: float = 0.7

@app.post("/v1/completions")
async def generate(request: GenerateRequest):
    try:
        tokens = []
        async for output in server.generate(
            request.prompt,
            request.max_tokens,
            request.temperature
        ):
            tokens.append(output)
        return {"completion": tokens[-1]['text']}
    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))

@app.get("/health")
async def health():
    return await server.health_check()

Performance Optimization Techniques

  • •Cache common prompt prefixes (e.g., system prompts): the same tokens re-sent on every request no longer need to be re-processed.
  • •Share KV cache blocks across requests: PagedAttention's block table lets identical prefixes physically share memory, not just skip compute.
  • •Reduces computation for repeated prefixes: the win scales with how much of each prompt is shared boilerplate vs. unique content.
  • •2-5x speedup for chat applications, where a long system prompt dominates the token count of most requests.
  • •Split long prompts into chunks: a 10K-token prompt no longer has to be processed as one giant blocking step.
  • •Overlap prefill with generation: decode steps for other requests can interleave with a long prompt's prefill instead of waiting behind it.
  • •Reduces time-to-first-token (TTFT), the metric users actually perceive as "responsiveness."
  • •INT8/INT4 quantization for weights: trades numeric precision for memory headroom.
  • •Reduces memory by 2-4x: the difference between fitting a 70B model on one GPU vs. needing four.
  • •Minimal quality loss with proper calibration: done carelessly, quantization silently degrades output quality; calibration data and method choice matter.
  • •AWQ, GPTQ, SmoothQuant methods: each makes different accuracy/speed trade-offs; benchmark on your own eval set before committing.
  • •Fused attention kernel: computes attention without materializing the full N×N attention matrix in GPU memory.
  • •2-3x faster attention computation: fewer memory reads/writes, since intermediate results never round-trip through HBM.
  • •Lower memory footprint: attention memory scales linearly instead of quadratically with sequence length.
  • •Required for long contexts (>4K tokens): beyond this, naive attention becomes the memory bottleneck, not the model weights.
  • •Use small model to predict tokens: a cheap draft model proposes several tokens ahead instead of the large model generating one at a time.
  • •Verify with large model in parallel: the large model checks all draft tokens in a single forward pass, which is nearly as cheap as generating one token.
  • •2-3x speedup for compatible prompts: the gain depends on how often the draft model's guesses match what the large model would have generated.

Scaling Strategies

Scaling Patterns:

  1. 1.Horizontal Scaling: add more vLLM instances behind a load balancer; the straightforward lever once a single instance is saturated.
  2. 2.Model Parallelism: tensor/pipeline parallelism for models too large to fit (or run fast enough) on one GPU, at the cost of inter-GPU communication overhead.
  3. 3.Mixed Models: route easy requests to a smaller, cheaper model and reserve the large model for the traffic that actually needs it.
  4. 4.Request Routing: route by complexity and latency requirements rather than sending everything to the same instance pool.
  • •4x A100 80GB: ~$10-15/hour on AWS/GCP. The baseline hardware cost for a 70B model at tensor-parallel 4.
  • •Throughput: ~100-200 concurrent requests, per instance, before you need to scale horizontally.
  • •Cost per 1M tokens: $0.50-1.00 (vs $20-60 for APIs). The self-hosting economics only work once utilization is high enough to justify the fixed hourly GPU cost.
  • •Break-even: 20M+ tokens/month. Below this volume, API pricing usually beats running your own GPUs once you count ops overhead.

Each instance handles ~100-200 concurrent requests; add instances behind the balancer to scale out.

Monitoring and Observability

PROMETHEUS
# Prometheus metrics for vLLM

# Request metrics
vllm_requests_total{model="llama-2-70b",status="success"} 15234
vllm_requests_total{model="llama-2-70b",status="error"} 12
vllm_request_duration_seconds_bucket{le="1.0"} 8500
vllm_request_duration_seconds_bucket{le="5.0"} 14800

# Resource metrics
vllm_gpu_memory_used_bytes{gpu="0"} 75000000000
vllm_gpu_memory_used_bytes{gpu="1"} 74500000000
vllm_active_requests 42
vllm_queue_size 8

# KV Cache metrics
vllm_kv_cache_usage_ratio 0.87
vllm_kv_cache_blocks_used 4250
vllm_kv_cache_blocks_total 5000

# Throughput metrics
vllm_tokens_generated_total 523847
vllm_tokens_per_second 1247.3

# Grafana Dashboard Queries:
# - Request latency p50, p95, p99
# - Throughput (tokens/sec, requests/sec)
# - GPU utilization and memory
# - Queue depth and wait times
# - Error rates by type

Key Takeaways

  1. 1.PagedAttention enables 2-4x better memory utilization by eliminating the fragmentation that comes from pre-allocating contiguous KV cache per request
  1. 2.Continuous Batching provides 2-3x higher throughput than static batching by refilling slots per token instead of waiting for the whole batch to finish
  1. 3.Tensor Parallelism allows serving models too large for a single GPU, at the cost of needing high-bandwidth interconnect between devices
  1. 4.vLLM combines these techniques for production-ready LLM serving: the default choice unless you have a specific reason (shared-prefix-heavy workloads, NVIDIA-specific tooling) to look elsewhere
  1. 5.Cost Efficiency: Self-hosted vLLM can be 20-40x cheaper than API calls at scale, but that math only holds once utilization is high enough to justify dedicated GPU cost
  1. 6.Performance:
  1. 7.When to Use vLLM:
  1. 8.Alternatives:

vLLM has become the standard for high-performance LLM serving, offering the best combination of throughput, latency, and cost efficiency for production deployments.

## Conclusion vLLM has earned its place as the default open-source serving engine by combining PagedAttention and continuous batching into exceptional performance and cost efficiency. For organizations running high-throughput inference workloads, vLLM can provide 2-4x better memory utilization and 2-3x higher throughput compared to naive serving approaches. Its v1 engine rewrite has kept it competitive as alternatives like SGLang push on specific workload shapes. The key advantages that make vLLM production-ready: - Memory efficiency through PagedAttention enables serving larger models or more concurrent requests on the same hardware - Throughput optimization via continuous batching maximizes GPU utilization and reduces queueing delays - Flexible parallelism strategies (tensor and pipeline) allow scaling to any model size - Cost savings of 20-40x compared to API-based solutions at scale At Bayseian, we've deployed vLLM for clients serving billions of tokens monthly, achieving sub-second latency at a fraction of the cost of commercial APIs. The framework has held up in production environments running 24/7 inference workloads. Whether you're building a chatbot, content generation system, or complex AI pipeline, vLLM provides the performance and efficiency needed for production deployment. Start with a single-GPU setup for prototyping, then scale horizontally or with tensor parallelism as your throughput requirements grow. Ready to deploy high-performance LLM inference? Contact us at contact@bayseian.com to discuss your requirements.

vLLMLLMInferenceParallelizationPerformanceProduction

Related Articles

AI & Machine Learning

AI Content at Scale Tends Toward the Average. Here Is Why, and What to Do About It.

On signal architecture, ADK skill design, and the engineering behind content that survives compression.

AI & Machine Learning

Why Your AI Coding Agent Keeps Going Off-Script, And How to Fix It

How spec-driven development turns unpredictable AI agents into reliable software factories through deterministic orchestration, bounded execution, and automated evaluation.

Working on something like this?

No pitch, just a practical conversation with the team that builds and operates these systems in production.

Start a conversation