Statistics and Mathematics · 43 topics
What is conditional probability? How is independence different from mutual exclusivity?
Conditional probability P(A|B) = P(A and B) / P(B): the probability of A given that B has happened.
Independent events: knowing B does not change A, so P(A and B) = P(A) x P(B) and P(A|B) = P(A).
Mutually exclusive events cannot happen together, so P(A and B) = 0. If both have non-zero probability, they are dependent, because one occurring rules out the other.
Addition rule: P(A or B) = P(A) + P(B) - P(A and B).
State Bayes' theorem and work through a medical test example.
Bayes' theorem: P(A|B) = P(B|A) x P(A) / P(B), which flips a conditional probability and updates a prior belief with evidence.
Example: a disease affects 1% of people; the test detects it 99% of the time and gives a false positive 5% of the time. For a positive test, P(disease|positive) = (0.99 x 0.01) / (0.99 x 0.01 + 0.05 x 0.99) = 0.0099 / 0.0594, about 16.7%.
The surprise comes from the low base rate: most positives are false positives. This is the same reason precision drops on imbalanced datasets.
Naive Bayes classifiers and Bayesian updating build on this theorem.
What are expectation and variance? What is linearity of expectation?
Expectation E[X] is the probability-weighted average of the values X can take.
Variance Var(X) = E[(X - mean)^2] = E[X^2] - (E[X])^2 measures spread; standard deviation is its square root.
Linearity of expectation: E[X + Y] = E[X] + E[Y] always, even when X and Y are dependent; also E[aX + b] = aE[X] + b.
Var(aX + b) = a^2 Var(X). Var(X + Y) = Var(X) + Var(Y) only when X and Y are uncorrelated (independent).
Which probability distributions should you know, and when do you use each?
Bernoulli: one yes/no trial. Binomial: number of successes in n independent trials (mean np, variance np(1-p)).
Poisson: counts of rare events in a fixed interval (mean equals variance), such as requests per minute. Exponential: waiting time between Poisson events; it is memoryless.
Uniform: all values in a range equally likely. Normal (Gaussian): bell curve, arises from sums of many small effects; defined by mean and variance; about 68%, 95% and 99.7% of values lie within 1, 2 and 3 standard deviations.
Beta distribution models probabilities (priors for rates); the multinomial and categorical generalize binomial and Bernoulli to many classes.
What is the law of large numbers, and how does it differ from the central limit theorem?
Law of large numbers: as the sample size grows, the sample mean converges to the true population mean.
Central limit theorem (CLT): as the sample size grows, the distribution of the sample mean approaches a normal distribution with mean equal to the population mean and standard deviation sigma divided by the square root of n.
One says where the sample mean ends up; the other describes the shape of its fluctuations around that value.
Explain the central limit theorem. When does it hold and when does it fail?
The CLT holds for sums or means of independent, identically distributed observations with finite variance, regardless of the original distribution; a sample size of about 30 or more is a common rule of thumb for moderately skewed data.
It underpins confidence intervals and many hypothesis tests, which is why t-tests work even on non-normal data with enough samples.
It can fail or converge slowly for heavy-tailed distributions (infinite variance, such as Cauchy), strongly dependent observations (time series, clustered data), and extremely skewed data or rare events with small samples.
It is about the sampling distribution of the mean, not about the raw data becoming normal.
What is the difference between covariance and correlation?
Covariance measures how two variables move together, but its size depends on their units and has no fixed range.
Correlation (Pearson) is covariance divided by the product of standard deviations, so it lies between -1 and 1 and is unit-free.
Correlation captures only linear relationships: a perfect nonlinear relationship (like y = x squared on symmetric data) can have zero correlation. Spearman's rank correlation handles monotonic relationships and outliers better.
Correlation does not imply causation.
Explain the Monty Hall problem.
You pick one of three doors; the host, who knows where the car is, opens another door with a goat and offers a switch.
Switching wins with probability 2/3 and staying wins with 1/3.
Reason: your first pick is right 1/3 of the time, and the host's reveal moves all the remaining 2/3 probability onto the other closed door. Listing the three equally likely starting cases confirms it.
How do you solve classic probability puzzle questions?
Define the sample space, count favorable outcomes, or use the complement (probability of at least one = 1 minus probability of none).
At least one head in 3 fair coin flips: 1 - (1/2)^3 = 7/8.
Two dice sum to 7: 6 favorable outcomes out of 36 = 1/6.
Expected number of flips to get the first head with a fair coin is 2; expected rolls to get a six on a fair die is 6 (geometric distribution: 1/p).
Birthday paradox: with just 23 people the chance of a shared birthday is above 50%, because the number of pairs grows quadratically.
State assumptions out loud, and sanity-check with a quick simulation if allowed.
What is a p-value, and what does it not mean?
The p-value is the probability of seeing results at least as extreme as the observed ones, assuming the null hypothesis is true.
A small p-value (below the chosen significance level, often 0.05) means the data would be surprising under the null, so we reject it.
It is not the probability that the null hypothesis is true, not the probability that the result happened by chance, and not a measure of effect size or importance.
A tiny p-value with a huge sample can reflect a trivial effect, so always report effect size and a confidence interval.
Explain hypothesis testing step by step.
State the null hypothesis (no effect or difference) and the alternative hypothesis (there is an effect).
Choose the significance level alpha (for example 0.05) and the test appropriate for the data (t-test, z-test, chi-square, ANOVA).
Compute the test statistic and p-value from the sample.
Decide: if p is below alpha, reject the null; otherwise fail to reject it (this is not proof that the null is true).
Check assumptions (independence, normality or large sample, equal variances) and report effect size.
What is a confidence interval, and how do you interpret a 95% interval?
A confidence interval is a range of plausible values for a population parameter, built from sample data (estimate plus or minus critical value times standard error).
A 95% interval means that if we repeated the sampling process many times, about 95% of the intervals built this way would contain the true value. It does not mean there is a 95% probability that this particular interval contains it.
Larger samples give narrower intervals; a higher confidence level gives wider intervals.
What are Type I and Type II errors, significance level and statistical power?
Type I error (false positive): rejecting a true null hypothesis; its probability is alpha.
Type II error (false negative): failing to reject a false null; its probability is beta.
Power = 1 - beta: the probability of detecting a real effect.
Power rises with larger sample size, larger true effect, higher alpha, and lower noise. Decide the sample size before running an experiment with a power analysis.
Tradeoff: lowering alpha reduces false positives but raises false negatives for the same sample size.
When do you use a t-test, z-test, chi-square test and ANOVA?
z-test: compare a mean when the population variance is known or the sample is very large.
t-test: compare means when the variance is estimated from the sample (one-sample, two-sample, or paired).
Chi-square test: test association between two categorical variables or goodness of fit to an expected distribution.
ANOVA: compare means across three or more groups (an F-test); follow up with post-hoc tests to see which groups differ.
Non-parametric alternatives (Mann-Whitney U, Wilcoxon) when normality or outlier assumptions fail.
How would you design and analyze an A/B test?
Define one primary metric (such as conversion rate), guardrail metrics, and a clear hypothesis; randomize users into control and treatment.
Pick the minimum detectable effect, significance level and power, then compute the required sample size and run duration before starting.
Run for full business cycles (for example whole weeks), do not stop early when the p-value dips below 0.05 (peeking inflates false positives), and check for sample ratio mismatch.
Analyze with a two-proportion z-test or t-test, report the effect size with a confidence interval, and consider practical significance, not just statistical significance.
Watch for novelty effects, interference between users, and multiple variants.
What is the multiple comparisons problem, and how do you handle it?
Testing many hypotheses at alpha 0.05 makes false positives likely: with 20 independent tests, the chance of at least one false positive is about 64%.
Bonferroni correction divides alpha by the number of tests (simple but conservative); Holm is a less conservative step-down version.
Benjamini-Hochberg controls the false discovery rate, which is more suitable when testing many features or metrics.
Pre-register the primary hypothesis and limit the number of comparisons.
What is the difference between correlation and causation, and how can you establish causality?
Correlation means two variables move together; it can arise from causation in either direction, a confounder, or chance.
Randomized controlled experiments (A/B tests) are the gold standard because randomization balances confounders.
When experiments are impossible: difference-in-differences, instrumental variables, regression discontinuity, propensity-score matching, and controlling for confounders, each with assumptions to defend.
What is the difference between maximum likelihood estimation (MLE) and maximum a posteriori (MAP)?
MLE picks the parameters that make the observed data most likely: maximize P(data | theta).
MAP also includes a prior: maximize P(data | theta) x P(theta), which is the posterior up to a constant.
MAP with a Gaussian prior equals L2 regularization, and with a Laplace prior equals L1 regularization.
With lots of data the likelihood dominates and the two estimates converge.
What is the difference between frequentist and Bayesian statistics?
Frequentist: parameters are fixed unknowns; probability describes long-run frequencies of data; results are expressed as p-values and confidence intervals.
Bayesian: parameters have probability distributions; combine a prior belief with the likelihood to get a posterior; results are credible intervals and direct probability statements about parameters.
Bayesian methods handle small data and incorporate prior knowledge naturally; the cost is choosing priors and heavier computation (MCMC, variational inference).
How do outliers, skewness and the choice of mean versus median affect analysis?
The mean is pulled by outliers; the median is robust. For incomes such as 25, 28, 30, 32, 35 and 5,000 (thousands), the mean is dominated by the outlier while the median stays representative.
Right-skewed data has mean above median; consider log transforms, winsorizing, or robust statistics.
Detect outliers with the interquartile range rule (beyond 1.5 times the IQR), z-scores, or visualization; decide whether they are errors or real signal before removing them.
Kurtosis describes tail heaviness: heavy tails mean more extreme values than a normal distribution.
What is sampling bias, and how do you avoid it?
Sampling bias happens when the sample does not represent the population: selection bias (who gets in), survivorship bias (only successes observed), non-response bias, and measurement bias.
Avoid it with random or stratified sampling, clear inclusion criteria, follow-up on non-responders, and checking sample demographics against the population.
In ML the equivalent is training data that does not match production data, causing poor generalization.
What is the dot product, and how does it relate to cosine similarity?
The dot product of two vectors is the sum of products of their components; geometrically it equals the product of their lengths times the cosine of the angle between them.
A zero dot product means the vectors are orthogonal; a large positive value means they point in similar directions.
Cosine similarity = dot product divided by the product of the norms; it ignores vector length and measures only direction (range -1 to 1), which is why it is the default for comparing embeddings.
For unit-normalized vectors, dot product and cosine similarity are identical, and Euclidean distance is a monotonic function of them.
How does matrix multiplication work, and what are the shape rules?
Multiplying an (m by n) matrix by an (n by p) matrix gives an (m by p) matrix; the inner dimensions must match, and each output entry is a dot product of a row and a column.
It is associative and distributive but not commutative: AB is generally not BA, and one order may not even be defined.
Cost is about m x n x p operations, which is why neural networks batch inputs into matrices and run on GPUs.
Transpose rule: (AB) transposed = B transposed times A transposed. Checking shapes is the quickest way to catch bugs in code.
What are eigenvalues and eigenvectors, and what is their intuition?
An eigenvector v of a matrix A satisfies A v = lambda v: applying A only stretches or flips it by the factor lambda (the eigenvalue), without changing its direction.
Find eigenvalues by solving det(A - lambda I) = 0, then solve (A - lambda I) v = 0 for each eigenvector.
Example: for the triangular matrix [[3, 1], [0, 2]], the eigenvalues are the diagonal entries, 3 and 2.
Intuition: eigenvectors are the principal directions of a transformation, and eigenvalues are the scaling along each direction. Eigenvalues of a Hessian describe curvature of a loss surface; those of a covariance matrix describe variance along principal directions.
What is singular value decomposition (SVD), and how does it differ from eigendecomposition?
SVD factors any matrix (any shape) as U times Sigma times V transposed, where U and V are orthogonal and Sigma holds non-negative singular values in decreasing order.
Eigendecomposition applies only to square matrices (and diagonalizable ones); SVD always exists.
Singular values are the square roots of the eigenvalues of A transposed times A.
Uses: low-rank approximation (keep the top k singular values for the best rank-k approximation), PCA, recommender systems, noise reduction, and computing pseudo-inverses.
How are PCA, eigenvectors and SVD connected?
Center the data, compute the covariance matrix; its eigenvectors are the principal components and its eigenvalues are the variance captured along each.
Equivalently, the right singular vectors of the centered data matrix are the principal components, and the squared singular values (divided by n-1) are the variances. SVD is the numerically preferred way to compute PCA.
Explained variance ratio = eigenvalue divided by the sum of all eigenvalues, which guides how many components to keep.
What are rank, linear independence and invertibility?
Vectors are linearly independent if none can be written as a combination of the others; the rank of a matrix is the number of independent columns (or rows).
A square matrix is invertible only if it has full rank, its determinant is non-zero, and none of its eigenvalues is zero; otherwise it is singular.
Multicollinear features make X transposed times X nearly singular, causing unstable regression coefficients; fix by removing features or using regularization.
Low-rank structure is exploited in compression and in LoRA, which adds a low-rank update to a weight matrix.
When does a matrix inverse exist, and why do we avoid computing it explicitly?
The inverse exists only for square, full-rank matrices; non-square or singular matrices use the pseudo-inverse (via SVD).
Linear regression's normal equation is w = (X transposed X) inverse times X transposed y, but in practice it is better to solve the linear system with a decomposition (QR, Cholesky) or use gradient descent, which is faster and more numerically stable.
Explicit inversion costs about n cubed operations and amplifies rounding errors when the matrix is ill-conditioned (a large condition number).
What are the L1, L2 and Frobenius norms, and how do they relate to regularization?
L1 norm: sum of absolute values of components. L2 norm: square root of the sum of squares (the usual length). Frobenius norm: the L2 norm of a matrix's entries.
Penalizing the L1 norm of weights (Lasso) promotes sparsity; penalizing the squared L2 norm (Ridge, weight decay) shrinks weights evenly.
Geometrically, the L1 ball has corners on the axes (so solutions land on sparse points), while the L2 ball is round.
Gradient clipping and normalization layers also use norms.
What is a positive definite matrix, and why does it matter in ML?
A symmetric matrix is positive definite if x transposed A x > 0 for every non-zero x, equivalently all eigenvalues are positive (positive semi-definite allows zero).
Covariance matrices are positive semi-definite; kernel (Gram) matrices must be too.
If the Hessian of a loss is positive definite at a point, that point is a local minimum; a convex quadratic has a positive semi-definite Hessian everywhere.
Cholesky decomposition is a fast, stable way to solve systems with positive definite matrices.
What are orthogonal matrices and orthonormal bases, and why are they useful?
An orthogonal matrix Q has orthonormal columns, so Q transposed times Q = I and its inverse is simply its transpose.
It preserves lengths and angles (a rotation or reflection), so it does not change norms, which is numerically very stable.
Orthonormal bases make coordinates easy to compute (just dot products); PCA and SVD produce them, and orthogonal initialization helps keep gradients stable in deep networks.
How do you choose between cosine similarity, dot product and Euclidean distance for embeddings?
Cosine similarity: direction only, best when vector magnitude is not meaningful (text embeddings).
Dot product: considers magnitude too; used in attention and when training objectives encode confidence in magnitude.
Euclidean distance: absolute closeness in space, suitable for clustering with k-means; sensitive to scale and degrades in high dimensions.
Match the metric used to train the embedding model, and normalize vectors if the index uses inner product to emulate cosine.
What is a gradient, and how does gradient descent use it?
The gradient is the vector of partial derivatives of a function with respect to each input; it points in the direction of steepest increase and is perpendicular to the level curves.
Gradient descent minimizes a loss by stepping against the gradient: theta_new = theta_old - alpha x gradient of the loss, where alpha is the learning rate.
Too large an alpha overshoots or diverges; too small converges slowly. For a smooth function, convergence needs alpha below about 2 divided by the Lipschitz constant of the gradient.
Stochastic and mini-batch variants estimate the gradient from a subset of the data.
How does the chain rule power backpropagation, and what is a Jacobian?
For a composition f(g(x)), the derivative is f'(g(x)) times g'(x); a network is a long composition of layers, so the loss gradient is a product of layer-wise derivatives computed from the output backward.
The Jacobian is the matrix of all first-order partial derivatives of a vector-valued function (rows are outputs, columns are inputs); backpropagation multiplies Jacobians (in practice, vector-Jacobian products) layer by layer.
Long products of Jacobians whose values are below or above 1 are the mathematical cause of vanishing and exploding gradients.
What is the Hessian, and what does it tell you about the loss surface?
The Hessian is the matrix of second derivatives, H[i][j] = second partial derivative of f with respect to x_i and x_j; it describes curvature.
At a critical point (zero gradient): all positive eigenvalues mean a local minimum, all negative mean a local maximum, mixed signs mean a saddle point.
Newton's method uses the Hessian to take better steps, but computing and inverting it costs order n squared memory and n cubed time, which is impractical for networks with millions of parameters; Adam and quasi-Newton methods approximate curvature cheaply.
In high-dimensional deep learning losses, saddle points are far more common than bad local minima.
What is a convex function, and why does convexity matter for optimization?
A function is convex if the line segment between any two points on its graph lies above the graph; equivalently its Hessian is positive semi-definite.
Every local minimum of a convex function is a global minimum, so gradient descent is guaranteed to reach the optimum (with a suitable learning rate).
Linear regression, logistic regression (with cross-entropy) and SVMs have convex losses; neural networks do not, so training finds good but not provably optimal solutions.
Why use cross-entropy instead of mean squared error for classification?
With a sigmoid or softmax output, MSE makes the loss non-convex in the weights and its gradient shrinks when the output saturates, so a confidently wrong prediction learns slowly.
Cross-entropy comes from maximum likelihood for a Bernoulli or categorical model; its gradient with respect to the logits is simply the predicted probability minus the label, so it stays large when the model is wrong.
Logistic regression with cross-entropy is convex, so training is well-behaved.
What are entropy and cross-entropy, and what is the entropy of a biased coin?
Entropy H(p) = minus the sum of p(x) log p(x): the average uncertainty or information (in bits with log base 2) of a distribution; it is maximal for a uniform distribution and zero for a certain outcome.
A fair coin has 1 bit of entropy; a coin that lands heads 90% of the time has about 0.47 bits, since outcomes are more predictable.
Cross-entropy H(p, q) = minus the sum of p(x) log q(x): the average number of bits needed to encode events from the true distribution p using a code built for q.
It is used as the classification loss, with p the true labels and q the model's predicted probabilities.
What is KL divergence, and how does it relate to cross-entropy?
KL(p || q) = sum of p(x) log(p(x) / q(x)): the extra information lost when q is used to approximate p.
It is always zero or positive, and zero only when p equals q.
Relationship: H(p, q) = H(p) + KL(p || q). Since the entropy of the true labels H(p) is constant, minimizing cross-entropy is the same as minimizing the KL divergence from the true distribution to the model.
Used in VAEs, knowledge distillation, t-SNE, and the KL penalty that keeps a model close to a reference model in RLHF.
Is KL divergence a distance metric?
No: it is asymmetric (KL(p || q) differs from KL(q || p)) and does not satisfy the triangle inequality.
Forward KL (p to q) is mode-covering: q is pushed to cover all regions where p has mass. Reverse KL (q to p) is mode-seeking: q concentrates on a single mode.
If a symmetric measure is needed, use Jensen-Shannon divergence (a symmetrized, bounded form) or the Wasserstein distance.
What is mutual information, and how is information gain used in decision trees?
Mutual information measures how much knowing one variable reduces uncertainty about another: I(X; Y) = H(Y) - H(Y | X); it is zero if they are independent and captures nonlinear dependence, unlike correlation.
Information gain for a decision tree split is the entropy of the parent node minus the weighted entropy of its children; the tree picks the split with the highest gain (or lowest Gini impurity).
It is also used for feature selection.
What are Lagrange multipliers, and where does constrained optimization appear in ML?
To optimize f(x) subject to a constraint g(x) = 0, form the Lagrangian f(x) + lambda times g(x) and set its gradient to zero; at the solution the gradients of f and g are parallel.
Inequality constraints use KKT conditions.
ML examples: the SVM margin problem and its dual (which leads to kernels), and regularization seen as a constrained problem (minimize the loss subject to a bound on the weight norm).
How do you compute the derivative of the sigmoid and softmax, and why does it matter?
Sigmoid: s(x) = 1 / (1 + e^(-x)) has derivative s(x) times (1 - s(x)), with a maximum of 0.25 at x = 0; it approaches zero for large positive or negative inputs, which explains vanishing gradients.
Softmax p_i = e^(z_i) / sum of e^(z_j) has derivative p_i x (1 - p_i) with respect to its own logit and minus p_i x p_j with respect to another logit.
Combined with cross-entropy, the gradient with respect to the logits collapses to p minus y, which is why the fused loss is both simple and stable.