Machine Learning Fundamentals · 41 topics
What is the bias-variance tradeoff?
Bias is error from overly simple assumptions: the model misses real patterns (underfitting).
Variance is error from sensitivity to the training data: the model fits noise (overfitting).
Expected error = bias squared + variance + irreducible noise. Making a model more complex lowers bias but raises variance, and the reverse.
Diagnose with learning curves: train and validation error both high means high bias; low train error with high validation error means high variance.
Fix high bias: richer features, a more flexible model, less regularization. Fix high variance: more data, regularization, simpler model, bagging, early stopping.
What is overfitting, and how do you detect and prevent it?
Overfitting: the model memorizes noise in the training data and performs poorly on unseen data. A typical sign is 95% training accuracy but about 70% validation accuracy.
Detect: compare training and validation metrics, plot learning curves, and use cross-validation.
Prevent: more (or augmented) data, regularization (L1/L2, dropout), simpler models, pruning trees, feature selection, early stopping, and ensembling.
Underfitting is the opposite: both training and validation performance are poor, so add capacity or better features.
What is the difference between L1 and L2 regularization?
Both add a penalty on coefficient size to the loss to discourage overfitting.
L1 (Lasso) penalizes the sum of absolute weights; it drives some weights to exactly zero, so it performs feature selection and gives sparse models.
L2 (Ridge) penalizes the sum of squared weights; it shrinks all weights smoothly toward zero without zeroing them, and handles correlated features well.
Elastic Net mixes both. The penalty strength (lambda or alpha) is a hyperparameter tuned with cross-validation.
What is cross-validation and why is it used?
It splits training data into k folds, trains on k-1 folds, validates on the remaining one, and averages the k scores.
It gives a more reliable estimate of generalization than a single split and makes better use of limited data; it is also used for hyperparameter tuning.
Use stratified k-fold for classification with imbalanced classes.
For time series use forward-chaining splits (for example scikit-learn's TimeSeriesSplit); random shuffling would leak future information.
What is data leakage and how would you detect it?
Leakage is when information unavailable at prediction time sneaks into training, producing unrealistically good validation scores that collapse in production.
Common causes: scaling or imputing on the full dataset before splitting, features derived from the target, duplicates across train and test, and random splits of time-ordered data.
Detect: suspiciously high validation scores, one feature with outsized importance, and a large train-to-production gap.
Prevent: split first, fit all preprocessing on the training set only (use a pipeline), split by group or time when rows are not independent, and review every feature for availability at prediction time.
What is the difference between supervised, unsupervised, semi-supervised and reinforcement learning?
Supervised: learns from labeled examples to predict labels (classification, regression).
Unsupervised: finds structure without labels (clustering, dimensionality reduction, anomaly detection).
Semi-supervised: uses a small labeled set plus a large unlabeled set, useful when labeling is expensive.
Reinforcement: an agent learns a policy by interacting with an environment and maximizing cumulative reward.
Do we need to scale features? Which models care?
Scale when the algorithm uses distances or gradient-based optimization: KNN, SVM, k-means, PCA, regularized linear models, and neural networks.
Tree-based models (decision tree, random forest, gradient boosting) are not affected, because splits depend on ordering, not magnitude.
Options: standardization (zero mean, unit variance) or min-max scaling; use robust scaling when there are outliers.
Fit the scaler on training data only and reuse it on validation and test data.
How do you handle missing data?
First understand why it is missing (random, depends on other features, or depends on the value itself) and how much is missing.
Options: drop rows or columns if little is lost; impute with mean, median or mode; model-based imputation (KNN or iterative); or use models that handle missing values natively (XGBoost, LightGBM).
Add a missing-indicator feature when missingness itself carries signal.
Fit imputers on training data only to avoid leakage.
How would you handle a dataset where only 1% of examples are positive?
Do not use accuracy (predicting all negatives gives 99%). Use precision, recall, F1, and PR-AUC.
Data level: resample the training set only (oversample the minority class, undersample the majority, or SMOTE); never resample validation or test data.
Algorithm level: class weights or a cost-sensitive loss, and gradient boosting with a scale-positive-weight setting.
Use stratified splits, then tune the decision threshold for the business tradeoff between false positives and false negatives.
Consider framing it as anomaly detection if positives are extremely rare.
How does linear regression work, and what are its assumptions?
It fits a line (or hyperplane) that minimizes the sum of squared errors between predictions and actual values, solved with the normal equation or gradient descent.
Assumptions: a linear relationship, independent errors, constant error variance (homoscedasticity), roughly normal residuals, and little multicollinearity among features.
Check residual plots and the variance inflation factor (VIF); outliers can distort the fit heavily.
Fix problems with transformations, polynomial features, regularization (Ridge or Lasso), or removing correlated features.
How does logistic regression differ from linear regression?
Linear regression predicts a continuous value; logistic regression predicts the probability of a class.
It passes a linear combination of features through the sigmoid function, giving an output between 0 and 1, then applies a threshold (0.5 by default) to pick a class.
It is trained by minimizing log loss (cross-entropy) with gradient descent, not squared error.
Coefficients are interpretable as changes in log-odds. Multi-class problems use softmax.
How does a decision tree work, and how do you stop it from overfitting?
It repeatedly splits the data on the feature and threshold that best separates the target, using Gini impurity or entropy (information gain) for classification and variance reduction for regression.
Easy to interpret and needs no scaling, but a deep tree memorizes the training data (high variance).
Control with max depth, minimum samples per leaf or split, pruning, or by moving to an ensemble.
How does a random forest improve on a single decision tree?
It trains many trees independently, each on a bootstrap sample of rows and a random subset of features at every split, then averages predictions (or takes a majority vote).
Averaging decorrelated trees sharply reduces variance while keeping low bias.
Out-of-bag samples give a free validation estimate; feature importance comes as a by-product.
Trade-offs: less interpretable than one tree and slower at prediction time with many trees.
What is the difference between bagging and boosting?
Bagging trains models in parallel on random resamples and combines them; it mainly reduces variance (random forest).
Boosting trains models sequentially, each one focusing on the errors of the previous ones; it mainly reduces bias (AdaBoost, gradient boosting, XGBoost).
Boosting often reaches higher accuracy but is more sensitive to noisy labels and overfitting, so it needs careful tuning.
How does gradient boosting work, and which XGBoost hyperparameters matter most?
It builds shallow trees one after another; each new tree is fit to the residual errors (the negative gradient of the loss) of the current ensemble, and its output is added with a small learning rate.
Key hyperparameters: learning rate (eta), number of trees, max depth, min child weight, subsample and colsample ratios, and L1/L2 regularization terms.
Lower learning rate with more trees generalizes better; use early stopping on a validation set to choose the number of trees.
Modern libraries (XGBoost, LightGBM, CatBoost) add regularization, handle missing values, and train fast on tabular data.
When would you choose logistic regression over a random forest?
When you need interpretability, calibrated probabilities, or a fast and simple baseline.
When data is limited or the relationship is roughly linear; a forest can overfit small datasets.
When features are high-dimensional and sparse (for example text with TF-IDF).
Choose a forest or boosting when there are nonlinear interactions and enough data, and always compare both on cross-validated metrics.
How does the k-nearest neighbors (KNN) algorithm work?
To classify a point, find the k closest training points by a distance metric (usually Euclidean) and take a majority vote (or average for regression).
There is no training step (a lazy learner), so prediction is slow on large datasets.
It needs feature scaling and suffers in high dimensions (curse of dimensionality).
Small k gives high variance; large k gives high bias. Choose k with cross-validation.
How does k-means clustering work, and how do you choose k?
Pick k centroids, assign each point to the nearest one, recompute each centroid as the mean of its points, and repeat until assignments stop changing.
It minimizes within-cluster variance, assumes roughly spherical clusters of similar size, is sensitive to scaling, outliers and initialization (use k-means++ and several restarts).
Choose k with the elbow method (inertia vs k), silhouette score, or domain knowledge.
What is PCA and when would you use it?
PCA finds orthogonal directions (principal components) that capture the most variance, and projects the data onto the top few.
Use it for dimensionality reduction, noise removal, visualization, and removing correlated features before a model.
Standardize features first; components are linear and less interpretable than original features.
Choose the number of components by cumulative explained variance (for example 95%).
How does an SVM work, and what is the kernel trick?
A support vector machine finds the hyperplane that separates classes with the maximum margin; only the points closest to the boundary (support vectors) define it.
The soft-margin parameter C trades off a wide margin against misclassified training points.
The kernel trick (for example RBF) computes similarities in a higher-dimensional space without explicitly mapping the data, allowing nonlinear boundaries.
Needs scaling and becomes slow on very large datasets.
How does Naive Bayes work, and why is it called naive?
It applies Bayes' theorem: the class probability is proportional to the prior times the product of per-feature likelihoods.
It is called naive because it assumes features are conditionally independent given the class, which is rarely true.
Despite that, it is fast, works well with small data and text classification (spam filtering), and needs little tuning; use Laplace smoothing for unseen words.
What are precision, recall and F1-score, and when do you prefer one over the other?
Precision = TP / (TP + FP): of the items predicted positive, how many are truly positive.
Recall = TP / (TP + FN): of the truly positive items, how many were found.
F1 is the harmonic mean of the two; it punishes a big gap between them, which a plain average would hide.
Prefer recall when a miss is costly (cancer screening, fraud detection). Prefer precision when a false alarm is costly (spam filter moving real email, content takedowns).
Choose the decision threshold to match the business cost of each error type.
What is a confusion matrix, and why is accuracy misleading on imbalanced data?
A confusion matrix counts true positives, false positives, true negatives and false negatives for each class.
All other classification metrics are computed from it.
With 1% positives, a model that always predicts negative scores 99% accuracy yet finds nothing, so look at precision, recall, F1, and PR-AUC instead.
What is ROC-AUC, and how does it differ from PR-AUC?
The ROC curve plots true positive rate against false positive rate across thresholds; AUC is the probability that a random positive is ranked above a random negative (0.5 is random, 1.0 is perfect).
It is threshold-independent, but on heavily imbalanced data it can look optimistic because the false positive rate is diluted by the huge number of negatives.
PR-AUC (precision-recall curve) focuses on the positive class and is more informative when positives are rare.
Your model has high AUC-ROC but low precision. What is going on, and what do you do?
Likely cause: class imbalance, so a small false positive rate still means many false positives relative to true positives.
Check PR-AUC and the precision-recall curve rather than ROC.
Raise the decision threshold to trade recall for precision, and check calibration of predicted probabilities.
Improve the model with better features, class weights or resampling, and confirm the production class balance matches validation.
Why might validation performance be misleading?
Data leakage between train and validation, or duplicates across splits.
Validation set not representative of production (distribution shift, different time period or users).
Repeated tuning on the same validation set overfits to it; keep an untouched final test set.
A metric that does not match the real objective, or a validation set too small for a stable estimate.
Fix with proper splits (time- or group-based), a holdout test set, confidence intervals, and online monitoring after launch.
What is a loss function, and how do you pick one?
A loss measures how wrong predictions are on one example; training minimizes its average (the cost).
Regression: mean squared error (penalizes large errors, sensitive to outliers), mean absolute error (robust), Huber (a mix).
Classification: log loss or cross-entropy for probabilistic outputs, hinge loss for SVMs.
The loss must be differentiable for gradient descent; the evaluation metric (such as F1) may differ from it, so check both.
How does gradient descent work, and what is the difference between batch, mini-batch and stochastic variants?
It repeatedly moves parameters in the opposite direction of the gradient of the loss, scaled by the learning rate.
Batch uses the whole dataset per update: accurate but slow and memory-heavy.
Stochastic (SGD) uses one example per update: fast and noisy, which can help escape poor minima.
Mini-batch uses a small subset per update: the standard compromise between speed and stability.
How does the learning rate affect training, and how do you tune it?
Too high: the loss oscillates or diverges. Too low: training is slow and can stall in a poor region.
Use a learning-rate range test or grid search, then a schedule (step decay, cosine decay, warmup) to start higher and reduce over time.
Optimizers such as momentum, RMSprop and Adam adapt the effective step size; AdamW decouples weight decay and is common for transformers.
Larger batch sizes typically allow a larger learning rate.
How do you tune hyperparameters?
Grid search tries every combination (expensive); random search often finds good settings faster; Bayesian optimization (for example Optuna) uses past trials to pick the next.
Always evaluate with cross-validation or a separate validation set, never the test set.
Tune the most influential parameters first (learning rate, depth, regularization strength) and use early stopping to cut bad runs.
Log every experiment (parameters, data version, metrics) so results are reproducible.
How do neural networks learn? Explain backpropagation.
Forward pass: inputs flow through layers (weighted sums plus activation functions) to produce a prediction, and a loss compares it with the target.
Backward pass: the chain rule computes the gradient of the loss with respect to every weight, layer by layer from the output back to the input.
An optimizer (SGD, Adam) then updates the weights using those gradients; repeating over many mini-batches and epochs reduces the loss.
Explain the vanishing and exploding gradient problems. How can they be mitigated?
Vanishing: in deep networks, repeated multiplication by small derivatives (for example sigmoid or tanh saturating) makes gradients near the early layers almost zero, so they barely learn.
Exploding: repeated multiplication by large values makes gradients blow up, causing unstable training and NaN losses.
Mitigate: ReLU-family activations, careful initialization (Xavier or He), batch or layer normalization, residual (skip) connections, LSTM or GRU gates for sequences, and gradient clipping for exploding gradients.
What are common activation functions, and what is the dying ReLU problem?
Sigmoid squashes to (0, 1) and saturates; tanh is zero-centered but also saturates; both suffer from vanishing gradients in deep nets.
ReLU (max of 0 and x) is cheap and speeds up training, but a neuron whose output is always zero stops learning (dying ReLU).
Leaky ReLU, GELU and Swish reduce that issue; softmax turns outputs into class probabilities.
What is batch normalization, and how does it differ from layer normalization?
Batch norm normalizes each feature across the mini-batch to zero mean and unit variance, then applies learnable scale and shift; it speeds up and stabilizes training and reduces sensitivity to initialization.
It depends on batch statistics, so it works poorly with tiny batches and sequence models, and behaves differently at training and inference time (running averages).
Layer norm normalizes across the features of each example, independent of batch size, so it is the standard choice in transformers.
How does dropout work, and why does it help?
During training, each neuron is randomly zeroed with probability p, so the network cannot rely on specific neurons; it acts like training an ensemble of thinned networks.
It reduces overfitting; at inference dropout is turned off and activations are scaled appropriately (inverted dropout handles this during training).
Other regularizers: weight decay, early stopping, data augmentation.
How do CNNs work, and why are they good for images?
Convolution filters slide over the input and detect local patterns (edges, textures, then parts and objects in deeper layers).
Weight sharing and local connectivity mean far fewer parameters than fully connected layers and translation-tolerant features.
Pooling reduces spatial size; the final layers classify. Transfer learning with pretrained backbones (ResNet, EfficientNet) is the usual starting point.
What is the difference between RNNs, LSTMs and Transformers?
RNNs process tokens one at a time, carrying a hidden state; they are slow to train (no parallelism over time) and struggle with long dependencies because of vanishing gradients.
LSTMs and GRUs add gates that control what to remember and forget, handling longer dependencies better.
Transformers use self-attention to relate every token to every other token in parallel, scaling better on modern hardware and capturing long-range context; the cost is attention memory and compute that grows quadratically with sequence length.
How does the attention mechanism work in a Transformer?
Each token is projected into a query, key and value vector. The attention score between two tokens is the dot product of the query and key, scaled by the square root of the key dimension, passed through softmax to give weights.
The output for a token is the weighted sum of all value vectors, so it blends information from the tokens most relevant to it.
Multi-head attention runs several of these in parallel to capture different relationships; positional encodings supply word order, and a causal mask prevents decoder models from seeing future tokens.
What are embeddings, and where are they used?
Embeddings are dense vectors that represent items (words, sentences, images, users, products) so that similar items sit close together in vector space.
They are learned by models (word2vec, transformer encoders) and used for search, recommendations, clustering, classification, and retrieval in RAG systems.
Compare them with cosine similarity or dot product, and choose the embedding model to match the domain and language.
What is transfer learning, and what is the difference between fine-tuning and feature extraction?
Transfer learning reuses a model pretrained on a large dataset for a new task, which needs far less data and compute than training from scratch.
Feature extraction freezes the pretrained layers and trains only a new head on top.
Fine-tuning also updates some or all pretrained weights, usually with a small learning rate; it adapts better but risks overfitting on small data and costs more.
For large language models, parameter-efficient methods such as LoRA update small added matrices instead of all weights.
What is the difference between a generative and a discriminative model?
Discriminative models learn the boundary or the conditional probability of the label given the input (logistic regression, SVM, most classifiers).
Generative models learn how the data itself is produced, so they can create new samples (Naive Bayes, GANs, diffusion models, large language models).
Discriminative models usually win on pure classification accuracy; generative models enable synthesis, simulation and handling of missing data.