CUDA Out of Memory in PyTorch and vLLM: Complete Step-by-Step Fix and Prevention Guide

If you encounter torch.cuda.OutOfMemoryError: CUDA out of memory during deep learning inference or fine-tuning in PyTorch and vLLM, the immediate solution is setting the environment variable export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to eliminate virtual memory fragmentation, while initializing vLLM with gpu_memory_utilization=0.90 and --max-model-len 4096. For iterative batch training, call torch.cuda.empty_cache() between forward passes and wrap generation loops inside torch.inference_mode(). This instantly releases unreserved tensor caching blocks and stops PyTorch allocation crashes without sacrificing throughput.

Why CUDA Out of Memory (OOM) Errors Occur

PyTorch uses a caching memory allocator to avoid continuous, expensive CUDA API allocations via cudaMalloc. While this dramatically accelerates forward and backward tensor operations, it causes severe virtual address space fragmentation. When a requested tensor block cannot find a contiguous block of free VRAM—even if the total free memory exceeds the tensor size—PyTorch panics and throws an unhandled OutOfMemoryError.

Step 1: Enable Expandable Segments to Eliminate Fragmentation

In PyTorch 2.1+, the expandable segments backend allows the allocator to allocate chunks without requiring contiguous virtual memory mappings. Add this configuration to your shell startup script or launch wrapper:

# Bash / Zsh Environment Configuration
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True,max_split_size_mb:128

# Python Native In-Code Configuration (Must run BEFORE torch is imported)
import os
os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "expandable_segments:True"
import torch

Step 2: Clamp vLLM GPU Memory Utilization and Block Sizing

By default, vLLM attempts to allocate 90% to 95% of available GPU VRAM for the KV cache. When running alongside tokenizers, FastAPI backends, or embedding models, this triggers immediate OOM kernel faults. Configure strict KV allocation ceilings:

# Starting vLLM OpenAI-Compatible API Server with Safe KV Limits
vllm serve meta-llama/Llama-3.1-8B-Instruct 
    --gpu-memory-utilization 0.88 
    --max-model-len 4096 
    --max-num-seqs 128 
    --block-size 16 
    --swap-space 16 
    --tensor-parallel-size 1

For an in-depth architectural breakdown of how KV-cache paging functions, see our technical guide on vLLM PagedAttention Optimization.

Step 3: Implement Safe Batch Inference with torch.inference_mode

Replacing torch.no_grad() with torch.inference_mode() disables view tracking and version counters, reducing tensor allocation overhead by up to 18%:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

device = "cuda" if torch.cuda.is_available() else "cpu"
model = AutoModelForCausalLM.from_pretrained(
    "mistralai/Mistral-7B-v0.1",
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1")

@torch.inference_mode()
def generate_safe(prompt: str):
    inputs = tokenizer(prompt, return_tensors="pt").to(device)
    outputs = model.generate(**inputs, max_new_tokens=256)
    return tokenizer.decode(outputs[0], skip_special_tokens=True)

Step 4: Resolve Containerized GPU Allocations and Kubernetes OOMKilled

If your PyTorch training scripts terminate unexpectedly inside Docker or Kubernetes without a stack trace, the process was killed by the OS kernel rather than Python. Read our companion walkthrough on Docker Exit Code 137 in Kubernetes and Deep Learning to configure IPC shared memory volumes properly.

Step 5: Architectural Comparison for Agentic Memory Demands

When chaining multiple model calls inside autonomous agent frameworks, context window explosion often overwhelms GPU memory. Review our comprehensive evaluation of LangGraph vs AutoGen vs CrewAI to implement state checkpointing and minimize unnecessary token accumulation.