Deep Learning · 40 topics
What is a neural network, and why do we need non-linear activation functions?
A neural network stacks layers of neurons; each neuron computes a weighted sum of its inputs plus a bias, then applies an activation function.
Without non-linear activations, any number of layers collapses into a single linear transformation, so the network could only learn linear relationships.
Non-linearities (ReLU, GELU, tanh) let the network approximate complex functions; depth lets it build features hierarchically (edges, then shapes, then objects).
Explain backpropagation step by step.
Forward pass: compute activations layer by layer and the final loss, caching intermediate values.
Backward pass: apply the chain rule from the loss back through each layer to get the gradient of the loss with respect to every weight and bias.
For a linear layer Y = XW + b with upstream gradient dY: dW = X transposed times dY, db = sum of dY over the batch, dX = dY times W transposed. Matching tensor shapes is a good sanity check.
The optimizer then updates the parameters using those gradients. Autograd frameworks (PyTorch, JAX) build and traverse this computation graph automatically.
What is cross-entropy loss, and why is it used with softmax?
Softmax turns raw scores (logits) into probabilities that sum to 1; cross-entropy measures how much probability the model gave to the correct class: loss = negative log of the predicted probability of the true class.
It penalizes confident wrong predictions heavily and gives strong gradients when the model is wrong.
Combined, the gradient with respect to the logits simplifies to predicted probability minus the one-hot label, which is simple and numerically stable.
Use binary cross-entropy with sigmoid for two classes or multi-label tasks. In practice use the fused "with logits" loss functions rather than applying softmax yourself.
Why is softmax implemented in a numerically stable way?
Exponentiating large logits overflows (inf) and tiny ones underflow to zero, producing NaN losses.
Subtracting the maximum logit before exponentiating leaves the result unchanged mathematically but keeps values in a safe range.
Similarly, compute log-softmax directly (log-sum-exp trick) instead of taking log of softmax.
Why can't we initialize all weights to zero, and what are Xavier and He initialization?
With identical weights every neuron in a layer computes the same output and receives the same gradient, so they never become different (the symmetry is never broken).
Xavier (Glorot) initialization scales weights by the number of inputs and outputs to keep activation variance stable; it suits tanh and sigmoid.
He initialization uses a larger variance (scaled by inputs) to account for ReLU zeroing half the activations.
Good initialization keeps signals and gradients from shrinking or exploding at the start of training.
What are an epoch, a batch, a batch size and an iteration?
Epoch: one full pass over the training dataset.
Batch size: the number of examples used for one gradient update.
Iteration: one parameter update, so iterations per epoch = dataset size divided by batch size.
Example: 10,000 samples with batch size 100 gives 100 iterations per epoch.
Explain the vanishing and exploding gradient problems, and how do you identify them?
Vanishing: gradients shrink as they are multiplied backward through many layers (common with sigmoid or tanh), so early layers barely learn. Symptoms: loss plateaus, early-layer weights barely change.
Exploding: gradients grow exponentially, causing huge updates. Symptoms: loss spikes or becomes NaN, gradient norms jump.
Identify by logging gradient norms per layer and monitoring the loss.
Fixes: ReLU-family activations, He or Xavier initialization, normalization layers, residual connections, gated units (LSTM, GRU), and gradient clipping for exploding gradients.
What does the universal approximation theorem say, and what does it not say?
A network with one hidden layer and enough neurons can approximate any continuous function on a bounded domain to any accuracy.
It does not say that training will find those weights, how many neurons are needed (it can be enormous), or that the network will generalize.
That is why depth, architecture, data and optimization matter in practice: deep networks often represent complex functions far more efficiently than a single wide layer.
How does the Adam optimizer work, and what are its caveats?
Adam keeps two running averages per parameter: the first moment (mean of gradients, like momentum) and the second moment (mean of squared gradients). Both are bias-corrected at the start.
The update divides the momentum term by the square root of the second moment, so each parameter gets its own adaptive step size.
Typical defaults: learning rate 1e-3, beta1 0.9, beta2 0.999, epsilon 1e-8.
Caveats: it can generalize slightly worse than well-tuned SGD with momentum on some vision tasks, uses extra memory (two states per parameter), and its L2 penalty interacts badly with the adaptive scaling.
AdamW decouples weight decay from the gradient update and is the standard choice for transformers.
What is the difference between SGD with momentum, RMSprop and Adam?
SGD with momentum accumulates a velocity of past gradients, smoothing noisy updates and speeding up along consistent directions.
RMSprop divides the step by a running average of squared gradients, adapting the step size per parameter.
Adam combines both ideas.
Adaptive methods converge fast with little tuning; SGD with momentum needs more learning-rate tuning but is still common for CNNs.
How does the learning rate affect training, and what are schedules and warmup?
Too high: loss oscillates or diverges. Too low: slow progress or getting stuck.
Schedules reduce the learning rate over time: step decay, cosine decay, or one-cycle.
Warmup starts with a very small learning rate and ramps up over the first steps; it stabilizes training when Adam's moment estimates are still noisy and for large models like transformers.
Finding a good value: run a learning-rate range test, then tune alongside batch size.
How does batch size affect training?
Small batches give noisy gradients (a mild regularizing effect) and need less memory but use hardware less efficiently.
Large batches give smoother gradients and better GPU utilization, but may generalize worse and usually need a proportionally larger learning rate (often with warmup).
If memory limits the batch size, use gradient accumulation: sum gradients over several small batches before one optimizer step.
What is the difference between batch normalization and layer normalization?
Batch norm normalizes each feature across the batch, then applies learnable scale and shift; it speeds up convergence, allows higher learning rates, and acts as a mild regularizer.
It depends on batch statistics, so it struggles with small batches and sequences, and uses running averages at inference (remember model.eval() in PyTorch).
Layer norm normalizes across the features of each example, independent of batch size, so transformers and RNNs use it (often in the pre-norm position or as RMSNorm).
How do dropout, weight decay, early stopping and data augmentation prevent overfitting?
Dropout randomly zeroes activations during training so the network cannot rely on single neurons; it is disabled at inference.
Weight decay (L2) keeps weights small, which limits model complexity.
Early stopping halts training when validation loss stops improving and restores the best checkpoint.
Data augmentation creates varied training examples (flips, crops, noise, text perturbations), effectively enlarging the dataset.
Also helpful: more data, smaller models, label smoothing, and transfer learning.
What is gradient clipping, and when do you use it?
It caps the gradient magnitude, either by value or (more commonly) by rescaling the global gradient norm to a maximum such as 1.0.
It prevents exploding gradients and sudden loss spikes, especially in RNNs, transformers, and training with large learning rates.
Log the gradient norm so you can see how often clipping is triggered.
What is mixed precision training, and what are its risks?
It runs most operations in 16-bit floats (FP16 or BF16) while keeping a master copy of weights in FP32, cutting memory use and speeding up training on modern GPUs.
FP16 has a narrow range, so it needs loss scaling to avoid gradient underflow; BF16 has the range of FP32 and usually needs no loss scaling.
Risks: numerical instability in some operations (softmax, normalization, loss), so keep those in higher precision.
Your model's loss is not decreasing. How do you debug it?
Start simple: overfit a tiny subset (a few batches). If the model cannot memorize it, there is a bug in the model, loss, or data pipeline.
Check data and labels: correct shuffling, label alignment, normalization, augmentation applied only to training data.
Check the code: loss and activation choice, output shapes, gradients flowing to all layers (not all zeros), train versus eval mode, frozen layers by mistake.
Tune the learning rate (the most common culprit), then watch the gradient norm histograms and weight update ratios.
Change one thing at a time, fix random seeds, and compare against a known-good baseline.
Training loss goes down but validation loss goes up. What do you do?
This is overfitting: add regularization (dropout, weight decay), augmentation, more data, early stopping, or reduce model capacity.
Also check for a train-validation mismatch (different preprocessing, distribution shift, leakage in the opposite direction) and a validation set that is too small.
Confirm with learning curves, then apply one fix at a time and re-measure.
How does a convolutional neural network work? Explain padding, stride and pooling.
A convolution slides small learned filters over the input, computing dot products to detect local patterns; early layers find edges and textures, deeper layers find parts and objects.
Weight sharing (the same filter everywhere) and local connectivity give far fewer parameters than fully connected layers and translation-tolerant features.
Stride is the filter's step size (larger stride shrinks the output); padding adds border values to preserve size and keep edge information.
Pooling (max or average) downsamples feature maps, reducing computation and adding small-shift invariance. Output size = (input - kernel + 2 x padding) / stride + 1.
Dilated convolutions widen the receptive field without extra parameters.
What are residual networks (ResNets), and why do skip connections help?
A residual block outputs F(x) + x: the layers learn the residual change, and the input is added back through a skip connection.
Gradients flow directly through the identity path, which eases vanishing gradients and lets networks reach hundreds of layers.
Very deep plain networks actually trained worse than shallower ones (the degradation problem); residual learning fixed that, since the block can easily learn an identity mapping.
The same idea appears in every transformer block.
Why do we prefer LSTMs or GRUs over vanilla RNNs, and what are the LSTM gates?
A vanilla RNN reuses the same weights at each step, so gradients vanish or explode over long sequences and it forgets distant context.
An LSTM adds a cell state with three gates: the forget gate (what to discard), the input gate (what new information to store), and the output gate (what to expose as the hidden state). The additive cell-state path keeps gradients alive.
A GRU merges these into two gates (update and reset), with fewer parameters and similar performance on many tasks.
Both are sequential, so they cannot parallelize across time steps, which is a main reason transformers replaced them in NLP.
What is a sequence-to-sequence model, and what problem did attention solve?
An encoder reads the input sequence into a representation and a decoder generates the output sequence step by step (translation, summarization).
Early versions squeezed the whole input into one fixed-size vector, which became a bottleneck for long sentences.
Attention lets the decoder look back at all encoder states at each step and weight them by relevance, removing the bottleneck; transformers later made attention the core of the whole model.
What is an autoencoder, and what are its uses?
An encoder compresses the input into a lower-dimensional latent code and a decoder reconstructs the input; training minimizes reconstruction error.
Uses: dimensionality reduction, denoising (denoising autoencoders), anomaly detection (high reconstruction error flags unusual inputs), and pretraining.
Variational autoencoders (VAEs) learn a probabilistic latent space, so new samples can be generated by sampling from it.
How does a GAN work, and what is mode collapse?
A generator creates fake samples from noise while a discriminator tries to tell real from fake; they train against each other in a minimax game until the fakes look real.
Training is unstable: the two networks can fall out of balance, and gradients may vanish.
Mode collapse is when the generator produces only a few kinds of outputs that fool the discriminator, ignoring the diversity of the data.
Mitigations: Wasserstein loss with gradient penalty, minibatch discrimination, spectral normalization, and careful learning-rate balance.
How do diffusion models work, and how do they compare with GANs?
Forward process: gradually add Gaussian noise to a training image over many steps until it is pure noise. The model learns to reverse this by predicting the noise at each step; generation starts from random noise and denoises step by step (often guided by a text prompt).
Compared with GANs: training is stable and covers the data distribution well (more diverse), but sampling is slower because of many denoising steps (reduced with faster samplers and distillation).
GANs are fast at inference but harder to train and prone to mode collapse.
What is transfer learning, and what is the difference between feature extraction and fine-tuning?
Transfer learning reuses a model pretrained on a large dataset for a new task, saving data, time and compute.
Feature extraction freezes the pretrained layers and trains only a new output head; it is safe with very small datasets.
Fine-tuning also updates some or all pretrained layers with a small learning rate (often unfreezing gradually); it adapts better but can overfit small data or forget earlier knowledge.
Choose based on dataset size and how similar the new domain is to the pretraining data.
What is data augmentation, and how do you choose augmentations?
It applies label-preserving random transformations to training data to increase variety and reduce overfitting.
Images: flips, crops, rotations, color jitter, cutout or mixup. Text: synonym replacement, back-translation. Audio: noise, time and pitch shifts.
Pick transformations that match real-world variation (do not flip digits or medical images where orientation matters), apply them to training data only, and check that validation performance improves.
Explain self-attention step by step. What is its computational complexity?
Each token's embedding is projected by learned matrices into a query (Q), key (K) and value (V).
Scores = Q times K transposed, divided by the square root of the key dimension; softmax turns each row into attention weights; the output is the weighted sum of the value vectors.
Every token can directly attend to every other token, so long-range dependencies are one step away and all positions compute in parallel.
Complexity is quadratic in sequence length for both compute and memory (an n by n score matrix), which is why long contexts are expensive.
Why do we divide the attention scores by the square root of the key dimension?
Dot products of high-dimensional vectors grow in magnitude with the dimension, pushing softmax into saturated regions where one weight is near 1 and gradients almost vanish.
Dividing by the square root of d_k keeps the variance of the scores roughly constant, so softmax stays well-behaved and training is stable.
Why does a transformer use multi-head attention instead of a single head?
Several heads run attention in parallel on different learned projections of the same input, so each can specialize (for example syntax, coreference, nearby versus distant tokens).
Their outputs are concatenated and projected back to the model dimension.
The total cost stays about the same because each head works in a smaller subspace (model dimension divided by number of heads).
Why do transformers need positional encoding, and what is RoPE?
Attention treats the input as a set, so without position information "dog bites man" and "man bites dog" look the same.
The original transformer added fixed sinusoidal vectors; many models learn positional embeddings instead.
Rotary position embedding (RoPE) rotates the query and key vectors by an angle that depends on position, so attention scores depend on relative distance; it extends better to longer contexts. ALiBi instead adds a distance-based bias to the scores.
What is masked (causal) self-attention, and where is it used?
A mask sets scores for future positions to negative infinity before softmax, so each token can attend only to itself and earlier tokens.
It is required in decoders and language-model training, where the model predicts the next token and must not see the answer.
It also allows training on all positions of a sequence in parallel in a single forward pass.
What is the difference between encoder-only, decoder-only and encoder-decoder transformers?
Encoder-only (BERT): bidirectional attention, produces contextual representations; used for classification, embeddings, and search.
Decoder-only (GPT, Llama, Claude): causal attention and next-token prediction; the dominant design for text generation and chat.
Encoder-decoder (T5, original transformer): the encoder reads the input, the decoder generates the output while attending to it through cross-attention; used for translation and summarization.
What advantages do transformers have over RNNs, LSTMs and CNNs?
Parallel training over the whole sequence (RNNs are sequential), which scales well on GPUs.
Direct long-range connections between any two tokens, instead of passing information through many steps.
Scale well with data and parameters, and transfer well via pretraining.
Costs: quadratic attention memory, large data and compute needs, and weaker built-in locality bias than CNNs (so vision transformers need more data or augmentation).
What is the KV cache, and how do you estimate its memory?
During generation, each new token needs the keys and values of all previous tokens. The cache stores them per layer so they are computed once instead of recomputed at every step, turning each step's cost from recomputing everything into one new token's work.
Memory estimate: 2 (K and V) x layers x sequence length x model dimension x batch size x bytes per element. Example: 32 layers, dimension 4096, 4,096 tokens, batch 1, FP16 comes to about 2 GB.
It grows linearly with context length and batch size, so for long contexts it, not the weights, often exhausts GPU memory.
Reduce it with multi-query or grouped-query attention (share K and V across heads), KV-cache quantization, paged attention (vLLM), and cache eviction.
What is FlashAttention, and why is it faster?
Standard attention writes the full n by n score matrix to GPU high-bandwidth memory and reads it back, and the bottleneck is memory traffic, not arithmetic.
FlashAttention tiles the computation so blocks of Q, K and V stay in fast on-chip SRAM and uses an online softmax, never materializing the full matrix.
The result is the same exact output, with much lower memory use and faster speed, enabling longer sequences.
How does quantization affect inference speed, memory and quality?
Quantization stores weights (and sometimes activations or the KV cache) in lower precision such as INT8 or INT4 instead of FP16: INT8 halves and INT4 quarters the weight memory.
Smaller models fit on fewer GPUs, and memory-bound decoding speeds up because less data moves per token.
Quality can drop, especially at 4-bit or below or with outlier-heavy activations; post-training methods (GPTQ, AWQ) and quantization-aware training reduce the loss.
Always evaluate on your own task before and after quantizing.
What is LoRA, and why is it used for fine-tuning?
Low-Rank Adaptation freezes the pretrained weights and adds small trainable low-rank matrices (A times B) to selected layers, such as attention projections.
Only a tiny fraction of parameters train, so memory and compute drop sharply, checkpoints are small, and adapters can be swapped per task.
QLoRA goes further by keeping the frozen base model in 4-bit precision while training the adapters.
Quality is often close to full fine-tuning on style, format and domain adaptation, though not always for large knowledge changes.
You get a GPU out-of-memory error while training. What do you do?
Reduce the batch size and use gradient accumulation to keep the effective batch size.
Use mixed precision (BF16 or FP16), activation (gradient) checkpointing, and a memory-efficient attention such as FlashAttention.
Shorten sequence length or crop images, use a smaller model or LoRA, and use an 8-bit or paged optimizer.
For larger models, shard across GPUs (data, tensor, pipeline parallelism, or ZeRO and FSDP) and offload to CPU as a last resort.
Profile first: parameters, gradients, optimizer states and activations each take memory, and Adam alone adds about two extra copies of the weights.
How is a large language model trained end to end?
Pretraining: next-token prediction on a huge text corpus teaches language, knowledge and reasoning patterns.
Supervised fine-tuning (SFT): train on curated instruction and response pairs so the model follows instructions.
Preference alignment: RLHF (a reward model plus reinforcement learning) or direct methods such as DPO adjust the model toward human-preferred answers.
Then evaluation, safety testing, and optimization for serving (quantization, batching, caching).