LLM Memory vs Compute: Roofline Bottlenecks & OS Failures

When the GPU Locks Up at Midnight

A production LLM cluster fails in two fundamentally different ways, and confusing them will send you down the wrong debugging rabbit hole for hours.

In the first failure mode, an inference node running a 70B parameter model suddenly crashes with an Out Of Memory (OOM) error. The GPU memory was completely full, yet the tensor cores were barely warm. In the second failure mode, a batch of requests with 8k prompt tokens causes the node to hang indefinitely. nvidia-smi reports 100% compute utilization, processes enter uninterruptible sleep (D-state), kernel watchdogs trigger, and even kill -9 fails to terminate the worker process.

These two failure modes stem from the physical divide between memory (the spatial capacity and bandwidth required to hold and move bits) and computation (the temporal execution of arithmetic floating-point operations).

Understanding where large language model workloads sit on this divide is the difference between blindly throwing hardware at latency spikes and actually resolving architectural bottlenecks.

The Physics of Space vs. Time

At the hardware level, every accelerator is constrained by two physical actions: transforming data and transporting data.

Computation (The Temporal Axis)

Computation is transformative. Modern accelerators pack tens of thousands of arithmetic logic units (ALUs) and specialized Tensor Cores into arrays of Streaming Multiprocessors (SMs).

The raw computational throughput of a chip is measured in floating-point operations per second (FLOP/s):

$\text{Peak FLOP/s}=N_{\text{cores}}\times f_{\text{clock}}\times \text{Ops per cycle}$

When an operation is compute-bound, the ALUs are fully saturated at their maximum clock frequency. The limiting factor is purely temporal: mathematical transformations require a deterministic number of clock cycles to execute.

If an operation requires $10^{14}$ operations and the GPU delivers $10^{14}$ FLOP/s, execution cannot take less than 1.0 second regardless of how fast memory is delivered.

Memory (The Spatial Axis)

Memory is conservative and stateful. Modern AI accelerators rely on High Bandwidth Memory (HBM3/HBM3e) stacked on an interposer silicon substrate alongside on-chip SRAM caches (L1/L2).

Memory performance is split into two non-interchangeable metrics:

1. Capacity (GB): The total physical volume of states that can be retained simultaneously.
2. Bandwidth (TB/s): The maximum rate at which bytes can be transferred across physical pins and buses from main DRAM into on-chip registers.

$\text{Transfer Time (s)}=\frac{\text{Bytes Read}+\text{Bytes Written}}{\text{Memory Bandwidth (Bytes/s)}}$

A memory-bound operation occurs when ALUs sit idle waiting for data to traverse the interconnect bus. This condition—memory bandwidth starvation—dominates modern generative workloads because compute capabilities have historically scaled at $\sim 3.1\times$ per generation, while memory bandwidth has only grown at $\sim 1.4\times$ per generation.

The Roofline Model: Finding the Machine's Ridge Point

The Roofline model provides a quantitative framework for determining whether a specific kernel will be throttled by the memory bus or saturated on the compute cores.

Mathematical Foundations

Two terms govern the Roofline model:

1. Arithmetic Intensity ($I$): An intrinsic property of the algorithm, defined as the ratio of floating-point operations performed to the total bytes moved across main memory:

$I=\frac{\text{Total Operations (FLOPs)}}{\text{Total Memory Traffic (Bytes)}}$

2. Ridge Point ($I_{\text{ridge}}$): An intrinsic property of the hardware accelerator, defined as the minimum arithmetic intensity needed to achieve peak theoretical compute throughput:

$I_{\text{ridge}}=\frac{\text{Peak Compute Throughput (FLOP/s)}}{\text{Peak Memory Bandwidth (Bytes/s)}}$

The attainable performance $P$ is bounded by the minimum of the two constraints:

$P=\min \left(\text{Peak Compute Throughput},\text{Peak Memory Bandwidth}\times I\right)$

• If $I<I_{\text{ridge}}$ (Memory-Bound): The workload is pinned to the slanted roofline. Shaving FLOPs from your algorithm yields exactly zero speedup; the only way to reduce latency is to shrink the byte footprint or increase memory bandwidth.
• If $I\ge I_{\text{ridge}}$ (Compute-Bound): The workload sits on the flat plateau. Memory bandwidth is sufficient, and the execution units run at full saturation.

Modern Accelerator Ridge Points

Comparing modern hardware accelerators demonstrates how dramatically the balance has shifted toward extreme compute density:

On an H100 SXM, an algorithm must execute at least 295 floating-point operations for every single byte loaded from HBM to avoid leaving tensor cores idle.

Prefill vs. Decode: Two Opposing Workloads in One Model

An autoregressive transformer executes two distinct computational phases that sit on opposite ends of the Roofline spectrum.

1. The Prefill Phase (Compute-Bound)

During prefill, the user provides a prompt sequence of length $S$. The model evaluates all tokens in parallel to generate the prompt's key-value activations:

For a linear projection layer with weight matrix $W\in {\mathbb{R}}^{d_{\text{in}}\times d_{\text{out}}}$, processing batch size $B$ and sequence length $S$:

$\text{FLOPs}=2\cdot B\cdot S\cdot d_{\text{in}}\cdot d_{\text{out}}$

$\text{Bytes Read}=2\cdot d_{\text{in}}\cdot d_{\text{out}}+2\cdot B\cdot S\cdot d_{\text{in}}(\text{at FP16 precision})$

When $S$ is large (e.g., $S=2048$, $B=4$), the weight matrix is loaded once and reused across $2048\times 4=8192$ token vectors. The arithmetic intensity simplifies to:

$I_{\text{prefill}}\approx \frac{2\cdot B\cdot S\cdot d_{\text{in}}\cdot d_{\text{out}}}{2\cdot d_{\text{in}}\cdot d_{\text{out}}}=B\cdot S$

For an H100 running an 8k prompt, $I_{\text{prefill}}>2000\text{ FLOP/byte}\gg I_{\text{ridge}}(295)$. The prefill phase saturates the SMs, generating high thermal load. The core performance metric is Time to First Token (TTFT).

2. The Decode Phase (Memory-Bound)

During decoding, tokens are emitted autoregressively one by one ($S=1$). To produce a single token, the engine must stream every parameter of the neural network from HBM into the compute cores:

$I_{\text{decode}}\approx \frac{2\cdot B\cdot 1\cdot d_{\text{in}}\cdot d_{\text{out}}}{2\cdot d_{\text{in}}\cdot d_{\text{out}}}=B\text{ FLOP/byte}$

At batch size $B=1$, the arithmetic intensity is $\approx 1\text{ FLOP/byte}$.

Furthermore, self-attention requires loading the entire historical KV cache for every single token step. Because each sequence in the batch maintains a distinct KV history, batching does not amortize KV reads:

$I_{\text{attention-decode}}\approx \frac{2\cdot B\cdot d_{\text{model}}\cdot \text{Context}}{4\cdot B\cdot d_{\text{model}}\cdot \text{Context}}=0.5\text{ FLOP/byte}$

The Math of Bandwidth Bottlenecks

Consider serving a Llama-3-8B model (15.0 GB weights at FP16) on an NVIDIA H100 (3.35 TB/s bandwidth):

• Single Request ($B=1$, 2048 context):
  • Weights loaded: $15.0\text{ GB}$
  • KV Cache loaded: $0.27\text{ GB}$
  • Minimum step time:

${\text{TPOT}}_{\text{min}}=\frac{15.0\text{ GB}+0.27\text{ GB}}{3350\text{ GB/s}}=4.56\text{ ms}⟹219\text{ tokens/sec}$

• Batched Request ($B=64$, 2048 context):
  • Weights loaded: $15.0\text{ GB}$ (loaded once for the whole batch)
  • KV Cache loaded: $64\times 0.27\text{ GB}=17.28\text{ GB}$
  • Minimum step time:

${\text{TPOT}}_{\text{min}}=\frac{15.0\text{ GB}+17.28\text{ GB}}{3350\text{ GB/s}}=9.63\text{ ms}$

While overall throughput jumps from 219 to $64/0.00963=6645\text{ tokens/sec}$, the latency per step doubled purely because the memory bus was overwhelmed transferring 17.28 GB of KV cache per forward pass.

Operating System Diagnostics: Decoding Kernel & Driver Faults

When resource demands violate physical or logical hardware limits, failures bubble up through the NVIDIA Resource Manager (NVRM) kernel module into the Linux kernel ring buffer.

+----------------------------------------------------------------------------+
| Linux Kernel Ring Buffer (dmesg / /var/log/syslog)                         |
|                                                                            |
| [14201.84] NVRM: Xid (PCI:0000:01:00): 31, pid=48210, MMU Page Fault:    |
|            PTE 0x00000000 Address 0x7fa2b0000000                          |
| [14322.10] NVRM: Xid (PCI:0000:01:00): 43, pid=49112, GPU Stopped        |
|            Processing (Watchdog Timeout)                                   |
| [14450.00] INFO: task vllm_worker:49112 blocked for more than 120 seconds. |
+----------------------------------------------------------------------------+

System administrators can inspect these events directly:

# View recent GPU kernel events
sudo dmesg -T | grep -E "NVRM|Xid|blocked for more than"

# Extract structured Xid events from systemd journal
journalctl -k --grep="Xid" --no-pager -n 50

Diagnosing Memory Failures (Spatial Domain)

Memory failures represent addressing, allocation, or hardware integrity errors.

1. User-Space Allocator Fragmentation (PyTorch OOM)

The PyTorch CUDA Caching Allocator avoids frequent cudaMalloc calls by maintaining a cached memory pool divided into two buckets:

• small_blocks: Allocations $\le 1\text{ MiB}$
• large_blocks: Allocations $>1\text{ MiB}$

If an application alternates between allocating 2 MiB tensors and 512 KiB tensors, free memory becomes trapped in non-contiguous fragments. When an 8 MiB allocation arrives, the allocator fails to find a contiguous block, triggers cudaErrorMemoryAllocation, flushes unallocated cached blocks, and throws:

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 8.00 GiB. 
GPU 0 has 79.15 GiB total capacity; 68.20 GiB already allocated; 
1.20 GiB free; 9.75 GiB reserved in cache by PyTorch (fragmentation).

Because user-space OOMs terminate cleanly before hitting the kernel MMU, no NVRM Xid codes are emitted.

2. GPU Page Table Faults: Xid 31

When memory addressing errors bypass user-space bounds, the GPU Memory Management Unit (MMU) halts the context:

NVRM: Xid (PCI:0000:07:00): 31, pid=18921, name=python3, MMU Page Fault: Engine=GRAPHICS, Fault_Type=PTE_MISS, Virtual_Address=0x7f8840000000

Root Cause: The GPU MMU attempted to translate a virtual address whose Page Table Entry (PTE) was empty or invalid. This is the GPU equivalent of a CPU segmentation fault (SIGSEGV). It is caused by:

• Use-after-free conditions in custom CUDA kernels.
• Multi-process concurrency bugs where two runtimes share unified virtual memory without proper synchronization.
• Corrupted host-to-device IPC pointers.

Remediation: Run the application through compute-sanitizer (cuda-memcheck) to isolate invalid pointer dereferences.

3. Physical Bit-Flips: Xid 48, Xid 63, Xid 94/95

• Xid 94 / 95 (Contained vs. Uncontained Single-Bit ECC): The GPU hardware detects a single bit flip in HBM and dynamically corrects it via Error-Correcting Code (ECC). The driver logs an informational Xid 94. If repeated errors hit the same physical bank, the driver retires the memory page (Xid 63: ECC Page Retirement).
• Xid 48 (Double-Bit ECC Error): Two bits flip in the same memory word. ECC can detect the corruption but cannot correct it:

NVRM: Xid (PCI:0000:07:00): 48, pid=18921, Double-bit ECC error detected on memory bank 3

Remediation: Fatal. The operating system kills the process to prevent gradient or weight corruption. A GPU hardware reset (nvidia-smi --gpu-reset) or full node reboot is mandatory. If Xid 48 persists after rebooting, initiate a hardware RMA.

Diagnosing Compute Failures (Temporal & Logic Domain)

Compute failures occur when kernels violate execution rules, trigger hardware exceptions, or run longer than the operating system's scheduler permits.

1. Execution Exceptions: Xid 13 and Xid 32

• Xid 13 (Graphics Engine Exception): An SM encounters an illegal instruction, warp arithmetic exception, or division by zero. The driver terminates the faulting kernel context while leaving other SMs operational.
• Xid 32 (Invalid Push Buffer): The GPU command processor reads a corrupted command stream from the host CPU push buffer. Typically caused by driver version mismatches or PCIe link degradation.

2. Watchdog Timeouts: Xid 43

Operating systems enforce execution deadlines on GPU commands to prevent display and kernel starvation:

NVRM: Xid (PCI:0000:03:00): 43, pid=31405, GPU Stopped Processing

Root Cause: A long-running compute kernel (such as an un-chunked 32k prompt prefill) occupied the GPU continuously without yielding to the scheduler.

Remediation: Split large matrix operations into smaller tiles, enable chunked prefill, or increase kernel timeout thresholds.

3. Cascading Dropouts: Xid 45 and Firmware Crashes (Xid 61/62)

• Xid 45 (Preemptive GPU Removal): If a compute workload triggers repeated timeouts or synchronization between host and device breaks down completely, the kernel driver detaches the GPU from the PCIe bus. The card remains physically seated, but nvidia-smi reports Unable to determine the device handle.
• Xid 61 / 62 (GSP Firmware Breakpoint / Crash): Modern architectures offload scheduling logic to an on-die RISC-V microcontroller called the GPU System Processor (GSP). If the GSP firmware crashes under heavy scheduling contention, it logs Xid 62.

4. The Dreaded D-State (TASK_UNINTERRUPTIBLE) Lockup

The most severe compute failure occurs when a worker process becomes trapped in Linux D-state (TASK_UNINTERRUPTIBLE):

When an inference runtime submits work to the GPU via an ioctl system call, the process transitions to sleep while holding critical kernel mutexes (dma_resv).

If the GPU encounters an internal hardware hang or GSP RPC timeout (Xid 119/120), the expected hardware interrupt never arrives. The process remains permanently in D-state.

Why kill -9 fails: Linux forbids delivering asynchronous signals to processes in TASK_UNINTERRUPTIBLE because killing a process holding internal kernel locks would corrupt the operating system's internal data structures.

After two minutes of zero response, the Linux kernel watchdog floods dmesg:

INFO: task python3:49112 blocked for more than 120 seconds.
      Not tainted 5.15.0-101-generic #111-Ubuntu
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Remediation: A graceful software kill is impossible. Trigger a clean kernel reboot via Magic SysRq (REISUB) or execute an IPMI baseboard hardware power cycle:

# Emergency sync and restart if terminal is responsive
echo 1 | sudo tee /proc/sys/kernel/sysrq
echo b | sudo tee /proc/sysrq-trigger

Field Guide: Diagnostic Decision Matrix

Divergent Engineering: Solving Memory vs. Compute

Because memory and compute bottlenecks originate from different physical properties, mitigating them requires opposing strategies:

1. Mitigating Memory Bottlenecks (Spatial Re-architecture)

vLLM Preemption: Recomputation Beats Swapping

Inference engines like vLLM pre-allocate a fixed proportion of GPU memory for KV cache blocks via gpu_memory_utilization = 0.9. When incoming requests exceed physical HBM capacity, the scheduler initiates preemption:

In modern systems, Recompute Mode is almost always faster than Swap Mode. The compute units are so fast that regenerating attention states at thousands of TFLOP/s takes less wall-clock time than transporting gigabytes of raw KV tensors across the PCIe bus.

PyTorch Allocator Virtual Memory Paging (cuMemMap)

To eliminate physical memory fragmentation without incurring cudaMalloc synchronization penalties, configure PyTorch to use expandable segments:

export PYTORCH_CUDA_ALLOC_CONF="expandable_segments:True,roundup_power2_divisions:16"

Instead of managing separate fixed segments, PyTorch reserves a single massive contiguous virtual address range. As new memory is needed, it allocates physical chunks and maps them directly into the contiguous virtual space, completely eliminating the segment boundary fragmentation that triggers spurious OOMs.

2. Mitigating Compute Bottlenecks (Temporal Load Balancing)

Chunked Prefill: Bridging the Roofline Gap

Executing pure prefills creates huge compute spikes, while executing pure decodes leaves tensor cores starved. Chunked Prefill co-locates both workloads in the same forward pass:

By capping the prefill budget per iteration (e.g., max_num_batched_tokens = 512), long prompts are sliced into uniform blocks and evaluated alongside active decode tokens. This shifts the collective arithmetic intensity directly toward the Ridge Point, maximizing GPU utilization while preventing compute watchdog timeouts (Xid 43).

In vLLM, enable this with:

vllm serve meta-llama/Meta-Llama-3-70B-Instruct \
  --enable-chunked-prefill \
  --max-num-batched-tokens 512 \
  --gpu-memory-utilization 0.90

Parallelism: Sharding ALUs vs. Layers

• Tensor Parallelism (TP): Shards individual weight matrices across multiple GPUs (e.g. Megatron-style column-parallel and row-parallel GEMMs). Each GPU computes $1/N$-th of the FLOPs simultaneously, directly reducing the latency burden on ALUs.
• Pipeline Parallelism (PP): Shards sequential layers across different GPUs. This distributes both the memory capacity footprint and compute burden across physical devices.

OS Kernel and Driver Watchdog Tuning

For high-latency distributed workloads utilizing NCCL over InfiniBand or RoCE, prevent premature kernel panics and hung task alerts:

# Increase kernel hung task timeout from 120s to 300s
sudo sysctl -w kernel.hung_task_timeout_secs=300

# Prevent NCCL communication deadlocks from hanging silently
export TORCH_NCCL_ASYNC_ERROR_HANDLING=1
export NCCL_BUFFSIZE=4194304

# If GSP firmware instability causes Xid 119/120 hangs on specific kernels,
# disable GSP offload and revert scheduling to host CPU:
echo "options nvidia NVreg_EnableGpuFirmware=0" | sudo tee /etc/modprobe.d/nvidia-gsp.conf
sudo update-initramfs -u

The Next Architectural Shift

The tension between memory and compute has defined AI infrastructure for a decade, forcing software engineers to build increasingly intricate abstractions—virtual memory pagers, chunked schedulers, speculative verification heads, and custom allocator pools.

Yet the landscape is shifting. Disaggregated serving architectures—where dedicated prefill nodes (compute-optimized clusters) stream intermediate KV activations over ultra-low-latency interconnects to dedicated decode nodes (memory-bandwidth-optimized clusters)—are beginning to separate these workloads physically rather than multiplexing them on single chips. At the same time, alternative model architectures like state-space models (Mamba) and linear attention mechanisms attempt to eliminate the KV cache entirely, replacing $O(N)$ memory growth with constant-size hidden states.

If state size ceases to scale with context length, the decode phase will no longer be trapped at $0.5\text{ FLOP/byte}$. Until then, every production serving stack remains an ongoing balancing act across the Roofline boundary—managing spatial cache allocations on one side and temporal arithmetic deadlines on the other.

References

• Williams, S., Waterman, A., & Patterson, D. (2009). Roofline: An Insightful Visual Performance Model for Multicore Architectures. Communications of the ACM, 52(4), 65-76.
• Kwon, W., Li, Z., Zhuang, S., et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. Proceedings of the 29th ACM Symposium on Operating Systems Principles (SOSP).
• Agrawal, A., Kedia, N., Panwar, A., et al. (2023). Sarathi: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills. arXiv:2303.01469.
• Patel, P., Choukse, E., Zhang, C., et al. (2024). Splitwise: Efficient Generative LLM Serving Using Disaggregated Context and Generation Phases. ISCA 2024.
• Sarkar, A. (2024). The Complete NVIDIA Xid Error Field Guide.
• NVIDIA Corporation. (2024). NVIDIA GPU Troubleshooting: NVIDIA System Management Interface (nvidia-smi) & Xid Errors Documentation.
• PyTorch Engineering Team. (2024). CUDA Caching Allocator Design and Virtual Memory Integration. PyTorch Documentation.
• Yu, G.-I., Jeong, J. S., Kim, G.-W., et al. (2022). Orca: A Distributed Serving System for Transformer-Based Generative Models. OSDI.