Docker Exit Code 137 in Kubernetes and Deep Learning: Complete Step-by-Step Fix for OOMKilled Containers

Resolving Docker Exit Code 137 (Linux fatal signal 9 SIGKILL: 128 + 9 = 137) in Kubernetes or containerized AI pipelines requires immediately increasing container memory limits in your pod specification (resources.limits.memory) or mounting an IPC shared memory volume (/dev/shm) for PyTorch DataLoader workers. When the Linux cgroups v2 kernel detects that process memory usage exceeds the hard cgroup limit, it dispatches the OOM Killer to terminate PID 1. Expanding /dev/shm via an in-memory volume or --ipc=host stops PyTorch DataLoader crashes instantly.

What Triggers Exit Code 137 in Containerized AI Workloads

Exit code 137 occurs when a containerized process is terminated by SIGKILL. This is almost exclusively triggered by the Linux kernel Out-Of-Memory (OOM) killer enforcing memory cgroup limits. In machine learning workloads, PyTorch multi-process DataLoader(num_workers > 0) passes tensor batches through /dev/shm (shared memory). Docker containers default to a tiny 64MB shared memory allocation, causing workers to hit kernel limits and instantly trigger code 137.

Step 1: Diagnose the Termination Reason with dmesg and kubectl

Verify whether the crash was caused by system-level OOM or container cgroup limits by running:

# Inspect host kernel ring buffer for OOM events
dmesg -T | grep -E -i "killed process|oom_reaper|out of memory"

# For Kubernetes Pod inspection
kubectl describe pod ai-inference-worker-pod -n production | grep -A 8 "Last State"
# Look for: Exit Code: 137, Reason: OOMKilled

Step 2: Expand Shared Memory (/dev/shm) in Docker and Compose

Override the default 64MB shared memory limit in your Docker CLI invocation or compose file:

# Docker CLI: Set shared memory size to 16GB
docker run --gpus all --shm-size=16g -it pytorch-training:latest

# Docker Compose Configuration
services:
  training-node:
    image: pytorch-training:latest
    shm_size: '16gb'
    deploy:
      resources:
        limits:
          memory: 32G

Step 3: Configure Kubernetes Shared Memory with emptyDir

Kubernetes does not have a native shmSize parameter. Instead, mount an in-memory emptyDir volume to /dev/shm in your Pod spec:

apiVersion: v1
kind: Pod
metadata:
  name: pytorch-worker-cluster
spec:
  containers:
  - name: trainer
    image: nvcr.io/nvidia/pytorch:26.02-py3
    resources:
      requests:
        memory: "28Gi"
      limits:
        memory: "32Gi"
    volumeMounts:
    - mountPath: /dev/shm
      name: dshm
  volumes:
  - name: dshm
    emptyDir:
      medium: Memory
      sizeLimit: "16Gi"

If your workload experiences VRAM memory exhaustion alongside container memory pressure, consult our troubleshooting guide on CUDA Out of Memory in PyTorch and vLLM.

Step 4: Hardening Background Webhook Workers Against OOM Spikes

Event-driven microservices that process webhook payloads often spawn unbounded subprocesses under load, triggering container OOM crashes. Check our production reference for Production Webhook Pipelines with Node.js and Stripe to implement backpressure queuing.

Step 5: Tuning vLLM Memory Allocations

For large-scale model inference, memory limits inside containers must balance host system RAM with GPU VRAM. Learn how to tune physical memory pages in our complete guide on vLLM PagedAttention Optimization.