Comprehensive LLM and LLMops Interview Question Bank

This document provides a structured set of technical interview questions designed to evaluate candidates across four progression levels: Foundational Architecture, Intermediate Internals & Fine-Tuning, Advanced Distributed Systems & Alignment, and Production LLMOps & System Design. In accordance with your criteria, topics covering Retrieval-Augmented Generation (RAG) and autonomous AI agents have been completely excluded.

Level 1: Basic & Foundational Architecture

1. Tokenization Mechanics

• Question: What is Byte-Pair Encoding (BPE), and why are subword tokenizers preferred over pure word-level or character-level tokenization in Large Language Models?
• Expected Answer / Key Evaluation Points:

• How BPE Works: BPE starts at the individual byte/character level and iteratively merges the most frequent adjacent pairs of tokens until a target vocabulary size ![][image1] is reached.
• Trade-off Analysis: Word-level tokenization suffers from massive vocabulary sizes and out-of-vocabulary (OOV) tokens. Character-level tokenization eliminates OOV tokens but drastically expands sequence length ![][image2], escalating self-attention computational cost, which scales quadratically as ![][image3].
• Subword Benefit: Subword tokenization balances vocabulary size and sequence length while maintaining the ability to break rare words down into meaningful subword roots or bytes.

2. Transformer Self-Attention Fundamentals

• Question: How does the Self-Attention mechanism compute token interactions, and why did the Transformer architecture replace Recurrent Neural Networks (RNNs) in natural language processing?
• Expected Answer / Key Evaluation Points:

• Mathematical Formulation: Each input token vector is projected into Query (![][image4]), Key (![][image5]), and Value (![][image1]) representations using learned projection matrices. The attention output is computed as:
  ![][image6]
• RNN Limitations: RNNs process tokens sequentially, creating an ![][image7] sequential dependency bottleneck during training that prevents hardware parallelization across GPU cores.
• Transformer Advantage: Transformers perform matrix multiplications across all tokens in parallel during forward/backward passes, dramatically improving training throughput on accelerator hardware.

3. Positional Encodings

• Question: What are Rotary Position Embeddings (RoPE), and why do modern LLMs favor relative positional encodings like RoPE over absolute sinusoidal encodings?
• Expected Answer / Key Evaluation Points:

• Mechanism: RoPE rotates Query and Key vector pairs in two-dimensional sub-spaces using a positional rotation matrix prior to computing inner products.
• Property: The inner product between query ![][image8] (at position ![][image9]) and key ![][image10] (at position ![][image11]) depends purely on their relative distance ![][image12], decaying naturally as sequence distance increases:
  ![][image13]
• Advantage: Unlike fixed absolute encodings, relative encodings like RoPE preserve inner product norm invariance and allow for context length extrapolation techniques (e.g., YaRN, Position Interpolation) during inference.

Level 2: Intermediate Internals, Decoding, & Parameter-Efficient Fine-Tuning

4. Attention Variants and KV Cache Footprint

• Question: Compare Multi-Head Attention (MHA), Multi-Query Attention (MQA), and Grouped-Query Attention (GQA). How do they affect Key-Value (KV) cache size during inference?
• Expected Answer / Key Evaluation Points:

• MHA: Every query head ![][image14] has a dedicated key head ![][image15] and value head ![][image16] (![][image17]). This consumes high HBM bandwidth to load the KV cache during autoregressive generation.
• MQA: All query heads share a single key head and value head (![][image18]), drastically reducing KV cache memory by ![][image19], but risking model quality degradation.
• GQA: Query heads are divided into ![][image20] groups, with each group sharing one KV head pair. This provides near-MHA quality while reducing KV cache memory footprint by ![][image21], permitting larger operational batch sizes.

5. Low-Rank Adaptation (LoRA) Mechanics

• Question: How does Low-Rank Adaptation (LoRA) work mathematically? Why are Matrix ![][image22] and Matrix ![][image23] initialized differently at the start of training?
• Expected Answer / Key Evaluation Points:

• Formulation: For a frozen pre-trained weight ![][image24], LoRA introduces a low-rank parameter update ![][image25], where ![][image26], ![][image27], and rank ![][image28]:
  ![][image29]
• Initialization: Matrix ![][image22] is initialized using a Gaussian distribution ![][image30], while Matrix ![][image23] is initialized to zero (![][image31]).
• Reasoning: Initializing ![][image31] ensures that ![][image32] at step ![][image33]. This guarantees that no initial noise or model output perturbation is introduced prior to gradient updates.

6. Quantization and PEFT (QLoRA)

• Question: What is QLoRA, and how does it combine quantization with low-rank adapters to enable fine-tuning on limited hardware?
• Expected Answer / Key Evaluation Points:

• Mechanism: QLoRA quantizes the base frozen model parameters to a 4-bit NormalFloat (NF4) data type while keeping the trainable LoRA adapter weights in 16-bit float (BF16/FP16).
• On-the-fly Dequantization: During the forward and backward passes, the 4-bit base weights are dynamically dequantized to 16-bit floating point representations to compute matrix multiplications alongside the adapters.
• Memory Impact: This dramatically reduces GPU static memory requirements (e.g., enabling a 65B model to be fine-tuned on a single 48GB GPU) without notable performance loss.

7. Generation Sampling Parameters

• Question: Explain the roles of Temperature, Top-K, and Top-P (Nucleus) sampling during text decoding.
• Expected Answer / Key Evaluation Points:

• Temperature: Scales the unnormalized output logits ![][image34] by ![][image35] before applying softmax. ![][image36] sharpens the distribution (making output more deterministic), while ![][image37] flattens the distribution (increasing diversity).
• Top-K: Restricts candidate selection to the ![][image5] highest-probability tokens, zeroing out probabilities for all others.
• Top-P (Nucleus): Dynamically selects the smallest set of top tokens whose cumulative probability sum exceeds threshold ![][image38] (e.g., ![][image39]). This adapts the candidate pool size based on model confidence.

Level 3: Advanced Alignment & Distributed Training

8. Direct Preference Optimization (DPO) vs. RLHF

• Question: What is Direct Preference Optimization (DPO), and how does its loss formulation eliminate the need for a separate reward model and PPO reinforcement learning loop?
• Expected Answer / Key Evaluation Points:

• Derivation Core: DPO analytical derivation shows that the optimal policy ![][image40] under the KL-constrained reward maximization objective can be expressed directly in terms of the reward function ![][image41]:
  ![][image42]
• Substitution: Substituting this implicit reward directly into the Bradley-Terry preference model yields the closed-form DPO loss over pairwise data ![][image43]:
  ![][image44]
• Operational Benefit: Bypasses PPO sampling rollouts, value network training, and instability, optimizing alignment directly via binary cross-entropy on static preference datasets.

9. Distributed Parallelism Strategies

• Question: Contrast Tensor Parallelism (TP) and Pipeline Parallelism (PP). Why is Tensor Parallelism typically restricted to intra-node execution, while Pipeline Parallelism can span inter-node networks?
• Expected Answer / Key Evaluation Points:

• Tensor Parallelism (TP): Shards individual layer matrices (e.g., Column/Row parallel matrix multiplies in attention layers) across GPUs. Requires All-Reduce communication collectives twice per transformer block. Because All-Reduce is extremely sensitive to latency, TP requires intra-node NVLink interconnects (![][image45]).
• Pipeline Parallelism (PP): Shards sequential blocks of layers across GPUs. Uses Point-to-Point (P2P) communication only at stage boundaries to forward activations and backward gradients. Because P2P messages are less frequent, PP tolerates lower interconnect bandwidths (e.g., InfiniBand/Ethernet across nodes).

10. DeepSpeed ZeRO Memory Sharding

• Question: Explain the three stages of the Zero Redundancy Optimizer (ZeRO) in distributed training.
• Expected Answer / Key Evaluation Points:

• Baseline Memory Setup: In mixed-precision training (FP16/BF16) with Adam, memory is divided into Model Weights (![][image46] bytes), Gradients (![][image46] bytes), and Adam Optimizer States (![][image47] bytes).
• ZeRO Stage 1: Optimizer states are sharded across ![][image48] data-parallel processes. Memory scales to ![][image49] bytes.
• ZeRO Stage 2: Both optimizer states and gradients are sharded across ![][image48] processes. Memory scales to ![][image50] bytes.
• ZeRO Stage 3: Model weights, gradients, and optimizer states are all sharded across processes. Parameters are gathered on-the-fly via All-Gather during execution and dropped immediately afterward, scaling memory to ![][image51] bytes.

Level 4: Expert Production LLMOps & Serving System Design

11. Batching Paradigms in LLM Serving

• Question: Explain Continuous (Iteration-Level) Batching and compare it to Static and Dynamic Batching in high-throughput LLM serving engines.
• Expected Answer / Key Evaluation Points:

• Static/Dynamic Batching: Batches requests at the request sequence level. Processing is locked until the longest sequence in the batch emits its end-of-sequence token, causing early-finished requests to sit idle.
• Continuous Batching: Schedules batching at the iteration step level. As soon as a request completes, it is evicted from the engine batch, and a waiting request's prefill phase is immediately scheduled into the freed batch slot on the very next token generation step.

12. PagedAttention Mechanics

• Question: How does PagedAttention (as implemented in vLLM) address GPU memory fragmentation during autoregressive generation?
• Expected Answer / Key Evaluation Points:

• Problem: Traditional allocation reserves contiguous blocks of HBM for the KV cache based on maximum sequence length ![][image52], causing severe internal and external memory fragmentation (wasting up to ![][image53] VRAM).
• PagedAttention Solution: Virtualizes the KV cache by partitioning token key/value states into fixed-size physical block pages (e.g., 16 tokens per block) allocated dynamically in non-contiguous HBM pages.
• Impact: Eliminates internal fragmentation, reduces external fragmentation to the final unfilled page of a sequence, and frees up VRAM to increase concurrency and serving throughput.

13. Production Observability and Metrics

• Question: What is the operational distinction between Time-To-First-Token (TTFT) and Time-Per-Output-Token (TPOT), and how do you optimize each phase in production?
• Expected Answer / Key Evaluation Points:

• TTFT (Prefill Phase): Measures the latency required to process the initial prompt tokens. The prefill phase is compute-bound (heavy parallel GEMM matrix multiplications). Optimization involves chunked prefills or offloading prefill tasks to compute-optimized GPU nodes.
• TPOT (Decode Phase): Measures the latency required to generate each sequential output token. The decode phase is memory-bandwidth bound (fetching large KV cache blocks for single token generations). Optimization involves KV cache quantization (INT8/FP8), Grouped-Query Attention (GQA), and high HBM-bandwidth hardware.

14. System Design Case Study: High-Concurrency LLM Inference Cluster

• Question: Walk through the end-to-end system design for serving a 70-billion parameter model in production to handle 10,000 concurrent user streams with a target P99 TTFT ![][image54] and TPOT ![][image55].
• Expected Answer / Key Evaluation Points:

• Quantization & Memory Sizing: FP16 weights require ![][image56] VRAM. Quantizing weights to INT8 or FP8 reduces static model memory to ![][image57], allowing a model instance to fit on two 80GB GPUs or four 40GB GPUs.
• Disaggregated Architecture: Disaggregate prefill nodes (compute-heavy) from decode nodes (memory-bandwidth heavy) to prevent long prompt processing from stalling active output token generation.
• Parallelism Scheme: Apply Tensor Parallelism (![][image58]) across NVLink within a node to split attention heads, combined with Pipeline Parallelism (![][image59] or ![][image60]) across nodes.
• Serving Infrastructure: Deploy vLLM with PagedAttention and continuous batching enabled. Enable FP8 KV cache quantization to compress memory footprint by ![][image61].
• Decoding Acceleration: Implement Speculative Decoding using a lightweight draft model (e.g., Llama-8B) to propose candidate tokens verified in parallel by the 70B model, driving inter-token decode latency well below the target limit.

Summary Evaluation Rubric for Interviewers