Kubernetes aur Deep Learning mein Docker Exit Code 137: OOMKilled Containers ka Mukammal Step-by-Step Hal

Kubernetes ya containerized AI pipelines mein Docker Exit Code 137 (Linux fatal signal 9 SIGKILL: 128 + 9 = 137) ko resolve karne ke liye zaroori hai ke aap apni pod specification mein foran container memory limits ko barhayein (resources.limits.memory) ya phir PyTorch DataLoader workers ke liye IPC shared memory volume (/dev/shm) mount karein. Jab Linux cgroups v2 kernel ko pata chalta hai ke process ki memory usage hard cgroup limit se barh gayi hai, toh yeh PID 1 ko khatam karne ke liye OOM Killer ko bhejta hai. In-memory volume ya --ipc=host ke zariye /dev/shm ko expand karne se PyTorch DataLoader crashes foran ruk jatay hain.

Containerized AI Workloads Mein Exit Code 137 Kiun Ata Hai

Exit code 137 tab hota hai jab kisi containerized process ko SIGKILL ke zariye terminate kar diya jata hai. Yeh aksar Linux kernel ke Out-Of-Memory (OOM) killer ki wajah se hota hai jo memory cgroup limits ko enforce karta hai. Machine learning workloads mein, PyTorch multi-process DataLoader(num_workers > 0) tensor batches ko /dev/shm (shared memory) ke through pass karta hai. Docker containers mein by default sirf 64MB ki choti si shared memory allocation hoti hai, jis ki wajah se workers kernel limits ko hit kar jatay hain aur foran code 137 trigger ho jata hai.

Step 1: dmesg aur kubectl Ke Zariye Termination Ki Wajah Talash Karein

Yeh verify karne ke liye ke crash system-level OOM ki wajah se hua hai ya container cgroup limits ki wajah se, yeh command chalayein:

# Inspect host kernel ring buffer for OOM events
dmesg -T | grep -E -i "killed process|oom_reaper|out of memory"

# For Kubernetes Pod inspection
kubectl describe pod ai-inference-worker-pod -n production | grep -A 8 "Last State"
# Look for: Exit Code: 137, Reason: OOMKilled

Step 2: Docker Aur Compose Mein Shared Memory (/dev/shm) Ko Barhayein

Apne Docker CLI command ya compose file mein default 64MB shared memory limit ko override karein:

# Docker CLI: Set shared memory size to 16GB
docker run --gpus all --shm-size=16g -it pytorch-training:latest

# Docker Compose Configuration
services:
  training-node:
    image: pytorch-training:latest
    shm_size: '16gb'
    deploy:
      resources:
        limits:
          memory: 32G

Step 3: emptyDir Ke Sath Kubernetes Shared Memory Configure Karein

Kubernetes mein koi native shmSize parameter nahi hota. Is ke bajaye, apni Pod spec mein aik in-memory emptyDir volume ko /dev/shm par mount karein:

apiVersion: v1
kind: Pod
metadata:
  name: pytorch-worker-cluster
spec:
  containers:
  - name: trainer
    image: nvcr.io/nvidia/pytorch:26.02-py3
    resources:
      requests:
        memory: "28Gi"
      limits:
        memory: "32Gi"
    volumeMounts:
    - mountPath: /dev/shm
      name: dshm
  volumes:
  - name: dshm
    emptyDir:
      medium: Memory
      sizeLimit: "16Gi"

Agar container memory pressure ke sath sath aap ke workload ko VRAM memory exhaustion ka samna bhi hai, toh hamari troubleshooting guide parhein CUDA Out of Memory in PyTorch and vLLM.

Step 4: Background Webhook Workers Ko OOM Spikes Se Bachana

Event-driven microservices jo webhook payloads ko process karti hain, load barhne par aksar unlimited subprocesses bana leti hain, jis se container OOM crashes ho jatay hain. Backpressure queuing ko implement karne ke liye hamara production reference check karein Production Webhook Pipelines with Node.js and Stripe.

Step 5: vLLM Memory Allocations Ki Tuning

Baray پیمانے (large-scale) par model inference ke liye, containers ke andar memory limits ko host system RAM aur GPU VRAM dono ke sath balance karna parta hai. Hamari mukamal guide mein seekhein ke physical memory pages ki tuning kaise ki jati hai vLLM PagedAttention Optimization.