LLM training and fine-tuning interview questions and answers

What are the stages of training a modern LLM?

Pretraining on trillions of tokens teaches language and knowledge by next-token prediction. Many labs then do mid-training on higher-quality or longer-context data. Supervised fine-tuning teaches the model to follow instructions in a chat format, and preference or reinforcement learning tunes it towards helpful, safe and correct answers.

What loss is used to train a language model?

Cross-entropy on the next token. At each position the model outputs a probability for every token in the vocabulary, and the loss is the negative log probability it gave to the actual next token, averaged over positions. Lower loss means the model predicts the text better.

What is perplexity?

Perplexity is the exponential of the average cross-entropy loss. Roughly, it is how many tokens the model is choosing between at each step: lower is better. It is useful for comparing models with the same tokenizer on the same text, but it does not measure whether answers are helpful.

import math
perplexity = math.exp(mean_cross_entropy_loss)

What do scaling laws say about model size and data?

Loss falls predictably as you add parameters, data and compute. The Chinchilla study found that, for a fixed compute budget, about 20 training tokens per parameter is compute-optimal. Many current models train on far more tokens than that, because a smaller model trained longer is cheaper to run.

How do you estimate the compute needed to train a model?

A common rule of thumb is about 6 floating-point operations per parameter per training token: 2 for the forward pass and 4 for the backward pass. So compute is roughly 6 times parameters times tokens, which you divide by the GPUs' realistic throughput to estimate time.

params = 7e9
tokens = 2e12
flops = 6 * params * tokens   # about 8.4e22 FLOPs

What happens to data before pretraining?

Raw web and code data is extracted to text, filtered for language and quality, cleaned of boilerplate, deduplicated (exact and near-duplicate), scrubbed of personal data where possible, checked against benchmark test sets, then mixed in chosen proportions such as web, code, books and maths, and tokenized.

What is a base model versus an instruct model?

A base model has only been pretrained: it continues text and does not reliably follow instructions. An instruct or chat model has had SFT and preference tuning so it answers in a conversation format. You usually fine-tune an instruct model for chat tasks and a base model when you want full control of the format.

What problems does fine-tuning solve well, and what does it not solve?

Fine-tuning is good at teaching a consistent format, tone or style, a narrow task such as classification or extraction, and making a small model match a large one on that task. It is poor at adding fresh or frequently changing facts; for those, retrieval is cheaper and easier to update.

What should you try before fine-tuning?

Try better prompts with clear instructions and a few examples, then structured output, then retrieval for missing knowledge. Build an evaluation set first. Fine-tune only when those plateau below the quality you need, or when you need lower cost or latency from a smaller model.

How much data do you need for supervised fine-tuning?

For style or format, a few hundred to a few thousand high-quality examples often work; research such as LIMA showed strong results from about 1,000 carefully chosen examples. Harder tasks need more. Quality and diversity matter more than size, so clean and deduplicate before adding more.

How do you format SFT data for a chat model?

Store each example as a list of messages with roles, such as system, user and assistant, usually in JSON Lines. During training the tokenizer's chat template turns the messages into the exact token format the model expects, including special role tokens.

{"messages": [
  {"role": "system", "content": "You are a support assistant."},
  {"role": "user", "content": "How do I reset my password?"},
  {"role": "assistant", "content": "Open Settings, then Security, then Reset password."}
]}

Why does the chat template matter so much?

Each model family expects specific special tokens around roles and turns. Training or serving with a different template makes the model see unfamiliar input, which quietly lowers quality. Use the tokenizer's own apply_chat_template for training and inference so both match.

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

How do you use synthetic data for fine-tuning?

A strong model generates instructions and answers, or rewrites and expands real examples, and you filter them with rules, deduplication and an LLM judge before training. It is cheap and scales, but it copies the teacher's mistakes and style, so mix in real data and check the licence terms of the teacher model.

Why mask the prompt tokens out of the loss?

You want the model to learn to produce answers, not to predict the user's question or the system prompt. Setting the labels of prompt tokens to an ignore value (-100 in Hugging Face) means only the assistant tokens contribute to the loss, which usually gives better results on small datasets.

labels = input_ids.clone()
labels[:prompt_length] = -100  # ignored by the loss

What is sequence packing?

Packing joins several short examples into one sequence up to the maximum length, so batches are not mostly padding. It can speed training a lot. Use attention masking that stops examples from seeing each other, or accept a small amount of cross-example attention.

Which hyperparameters matter most in fine-tuning?

Learning rate first: roughly 1e-5 to 2e-5 for full fine-tuning and about 1e-4 to 2e-4 for LoRA are common starting points. Then the number of epochs (often 1 to 3), effective batch size, warmup steps and the schedule, usually cosine decay. Change one at a time against a validation set.

How do you notice overfitting during fine-tuning?

Training loss keeps falling while validation loss rises, and the model starts repeating training phrases or loses general ability. Reduce epochs, lower the learning rate, add more varied data or stop at the checkpoint with the best validation score.

What is gradient accumulation?

When the batch you want does not fit in GPU memory, you run several smaller micro-batches, add up their gradients and update the weights once. Effective batch size equals micro-batch size times accumulation steps times the number of GPUs.

How would you fine-tune a model with the Hugging Face stack?

Load the model and tokenizer with transformers, add LoRA adapters with peft, and run training with TRL's SFTTrainer, which applies chat templates, masking and packing. Log to a tracker, evaluate checkpoints on your validation set, then merge or save the adapter.

from trl import SFTConfig, SFTTrainer

trainer = SFTTrainer(
    model=model,
    train_dataset=train_ds,
    eval_dataset=eval_ds,
    peft_config=lora_config,
    args=SFTConfig(output_dir="out", num_train_epochs=2, learning_rate=2e-4),
)
trainer.train()

How does LoRA change a weight matrix?

LoRA freezes the original weight W and learns two small matrices, A and B, whose product has the same shape as W but rank r. The layer uses W plus (alpha divided by r) times B times A. B starts at zero, so training begins from exactly the original model.

What do the LoRA rank and alpha settings control?

Rank r sets how many directions the adapter can learn: 8 to 64 is typical, higher for harder tasks. Alpha scales the update; the effective scale is alpha divided by r, and alpha equal to r or twice r is a common default. Too high a scale with a high learning rate makes training unstable.

from peft import LoraConfig

lora_config = LoraConfig(
    r=16, lora_alpha=32, lora_dropout=0.05,
    target_modules="all-linear", task_type="CAUSAL_LM",
)

Which layers should LoRA target?

Early work adapted only the attention query and value projections. Later results, including the QLoRA paper, found that adapting all linear layers, attention and MLP projections, gives quality closer to full fine-tuning for a small increase in trainable parameters.

What are the three ideas inside QLoRA?

The frozen base model is stored in 4-bit NormalFloat (NF4), a data type suited to normally distributed weights. Double quantization also compresses the quantization constants. Paged optimizers move optimizer state to CPU memory during spikes. LoRA adapters train in 16-bit on top, so a large model fits on one GPU.

import torch
from transformers import BitsAndBytesConfig

bnb = BitsAndBytesConfig(
    load_in_4bit=True, bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16,
)

When would you choose full fine-tuning over LoRA?

When you have a large dataset, a big change in behaviour or domain, and enough GPUs, full fine-tuning can reach slightly higher quality. LoRA is cheaper, faster, forgets less, and produces small adapter files, so it is the default for most product fine-tuning.

What does merging a LoRA adapter mean?

Merging adds the learned update into the base weights, producing a normal model file with no adapter and no extra inference cost. Keeping adapters separate lets one server load a single base model and switch between many small task adapters per request.

merged = peft_model.merge_and_unload()
merged.save_pretrained("support-model-merged")

What other parameter-efficient methods exist besides LoRA?

Prompt tuning and prefix tuning learn soft prompt vectors while freezing the model; adapters insert small trainable layers; DoRA splits weights into magnitude and direction and applies LoRA to the direction. LoRA and its variants are the most used because they merge cleanly and perform well.

How much GPU memory does full fine-tuning need?

With mixed precision and the Adam optimizer, a common estimate is about 16 bytes per parameter before activations: 16-bit weights and gradients, a 32-bit master copy of the weights, and two 32-bit Adam moment buffers. A 7-billion-parameter model needs roughly 112 GB plus activations, so it does not fit on one consumer GPU.

params = 7e9
bytes_per_param = 2 + 2 + 4 + 4 + 4   # weights, grads, master copy, Adam m, Adam v
print(params * bytes_per_param / 1e9, "GB")  # about 112 GB

What is the difference between fp16 and bf16 training?

Both use 16 bits. fp16 has more precision but a small range, so gradients can underflow and you need loss scaling. bf16 keeps the same exponent range as fp32 with less precision, so it rarely overflows and needs no loss scaling. Use bf16 on GPUs that support it.

What is gradient checkpointing?

Instead of keeping every layer's activations for the backward pass, the model keeps only some and recomputes the rest when needed. It cuts activation memory a lot, allowing longer sequences or bigger batches, at the cost of roughly 20 to 30 percent more compute.

What are data, tensor and pipeline parallelism?

Data parallelism copies the model to every GPU and splits the batch. Tensor parallelism splits individual weight matrices across GPUs, which needs very fast links. Pipeline parallelism places different layers on different GPUs and streams micro-batches through. Large training runs combine all three.

What do DeepSpeed ZeRO stages 1, 2 and 3 do?

Plain data parallelism stores a full copy of optimizer states, gradients and weights on every GPU. ZeRO stage 1 shards the optimizer states across GPUs, stage 2 also shards gradients, and stage 3 also shards the weights themselves, gathering them layer by layer when needed. Each stage saves more memory but adds communication.

What is FSDP?

Fully Sharded Data Parallel is PyTorch's built-in way to shard weights, gradients and optimizer state across GPUs, similar to ZeRO stage 3. Each GPU holds only a slice and gathers full layers just in time for the forward and backward pass, so models far bigger than one GPU can train.

What does FlashAttention improve?

Standard attention writes a full sequence-by-sequence score matrix to GPU memory. FlashAttention computes exact attention in tiles that stay in fast on-chip memory, so memory grows linearly with sequence length and training and inference run faster, especially for long contexts.

How is a reward model trained for RLHF?

People compare two or more answers to the same prompt and pick the better one. A model, usually initialised from the SFT model with a scalar output head, is trained so the preferred answer gets a higher score than the rejected one. That score later guides reinforcement learning.

What does the beta parameter in DPO control?

DPO trains directly on chosen and rejected answer pairs, comparing the model with a frozen reference model. Beta sets how far the model may move from that reference: a low beta allows bigger changes and risks drift, a high beta keeps it close to the starting model.

from trl import DPOConfig, DPOTrainer

trainer = DPOTrainer(
    model=model, ref_model=None,  # None: TRL builds the reference copy
    args=DPOConfig(output_dir="dpo", beta=0.1),
    train_dataset=pairs,  # prompt, chosen, rejected
    processing_class=tokenizer,
)

What is GRPO and why is it used for reasoning models?

Group Relative Policy Optimization samples several answers per prompt, scores them, and uses each answer's score relative to the group average as its advantage. It needs no separate value model, which saves memory, and works well with verifiable rewards such as correct maths answers or passing unit tests.

What is reinforcement learning with verifiable rewards?

Instead of a learned reward model, the reward comes from a program that checks the answer: a maths answer matched against the key, code run against tests, or output validated against a schema. Rewards are harder to game, which is why this approach drives much recent progress on reasoning.

What is reward hacking?

The model finds ways to raise the reward without actually getting better, such as writing longer answers because the reward model likes length, or special-casing tests. Counter it with a penalty for drifting from the reference model, better reward design, varied evaluation and human review of samples.

What is catastrophic forgetting and how do you limit it?

Fine-tuning on a narrow dataset can erase general skills the model had, such as following other instructions or reasoning. Limit it with LoRA instead of full tuning, a lower learning rate, fewer epochs, and by mixing some general instruction data into the training set. Always re-run general benchmarks.

How do you evaluate a fine-tuned model?

Compare it with the base model and with a well-prompted larger model on a held-out test set from your task, using exact metrics where possible and an LLM judge or human review for open answers. Also run a few general benchmarks to catch forgetting, and check latency and cost.

What is benchmark contamination?

Contamination happens when test questions, or close copies, appear in the training data, so scores rise without real skill. Check by searching training data for test items, keep private test sets, and prefer recent or freshly written evaluation questions.

Compare GPTQ, AWQ and GGUF.

GPTQ is a post-training method that quantizes weights layer by layer while compensating for the error it adds. AWQ protects the small set of weights that matter most for activations and quantizes the rest. GGUF is a file format used by llama.cpp with many quantization levels, popular for running models on CPUs and laptops.

What is knowledge distillation for LLMs?

A smaller student model is trained to imitate a larger teacher, either on the teacher's generated answers or on its full output probabilities. It produces a model that is much cheaper to serve while keeping much of the teacher's quality on the target tasks.

How are fine-tuned models served efficiently?

Inference servers such as vLLM use continuous batching, which adds new requests to the batch as others finish, and PagedAttention, which stores the attention cache in small blocks to avoid wasted memory. They can serve many LoRA adapters on one base model and run quantized weights.

What is model merging?

Model merging combines the weights of several fine-tuned models built from the same base, for example by averaging them or adding their weight differences, to get one model with several skills without more training. Results vary, so evaluate the merged model like any new one.