Search problems

Loading…

Practice for AI, ML and GenAI engineering interviews. Answer out loud or in writing, then learn from the reference solution.

Start practicing

Practice

All problems
Easy warm-ups
Hard challenges
System design
Blog

Your space

Dashboard
Saved problems
Privacy
Terms

Account

Create free account
Log in

Loading…

AI Infrastructure and Scalability Interview Questions | Buncho

Library Dashboard Saved for later

Topics

Python for AI/ML75 Machine Learning Fundamentals10 ML Algorithms0 ML System Design1 Deep Learning5 Statistics and Mathematics1 Generative AI and LLMs122 Embeddings13 RAG and Vector Databases47 AI Agents and Tool Calling50 MCP0 Multimodal AI29 LLM Evaluation and Testing27 AI Safety, Ethics, and Responsible AI36 AI System Design41 Fine-Tuning25 AI Infrastructure and Scalability29 LLM Gateways0 Production AI Engineering39 AI Engineering Case Studies15

of 565

Solved

Easy0/73
Med.0/297
Hard0/195

Topic

AI Infrastructure and Scalability interview questions

GPUs and accelerators, inference optimisation, parallelism, serving, scaling, and cost of AI systems.

All topics565 Python for AI/ML75 Machine Learning Fundamentals10 ML Algorithms0 ML System Design1 Deep Learning5 Statistics and Mathematics1 Generative AI and LLMs122 Embeddings13 RAG and Vector Databases47 AI Agents and Tool Calling50 MCP0 Multimodal AI29 LLM Evaluation and Testing27 AI Safety, Ethics, and Responsible AI36 AI System Design41 Fine-Tuning25 AI Infrastructure and Scalability29 LLM Gateways0 Production AI Engineering39 AI Engineering Case Studies15

Search problems

DifficultyProblem type

29 problems

See how often each question is asked, and which companies asked it.Part of Buncho Pro.Unlock

TitleTopicTypeTimeAsked Companies Level

Not started1.How does a Google TPU work?AI Infrastructure and Scalability · Conceptual · 12 min

Asked %

Companies

AI Infrastructure and Scalability

Conceptual

12 min

Pro feature

Medium

Not started2.What is the role of load balancing in AI serving infrastructure?AI Infrastructure and Scalability · Practical exercise · 12 minAsked %CompaniesAI Infrastructure and ScalabilityPractical exercise12 minPro featurePro featureHard

Not started3.Do the roofline math: how many tokens/sec can one H100 produce for a 70B model at batch size 1?AI Infrastructure and Scalability · Practical exercise · 12 minAsked %CompaniesAI Infrastructure and ScalabilityPractical exercise12 minPro featurePro featureHard

Not started4.What is pipeline parallelism?AI Infrastructure and Scalability · Conceptual · 10 minAsked %CompaniesAI Infrastructure and ScalabilityConceptual10 minPro featurePro featureHard

Not started5.How do you monitor and profile LLM inference in production (TTFT, inter-token latency, GPU utilization)?AI Infrastructure and Scalability · Practical exercise · 14 minAsked %CompaniesAI Infrastructure and ScalabilityPractical exercise14 minPro featurePro featureHard

Not started6.What is model parallelism vs data parallelism in distributed training?AI Infrastructure and Scalability · Conceptual · 12 minAsked %CompaniesAI Infrastructure and ScalabilityConceptual12 minPro featurePro featureMedium

Not started7.How do you manage GPU memory for serving multiple models?AI Infrastructure and Scalability · Practical exercise · 14 minAsked %CompaniesAI Infrastructure and ScalabilityPractical exercise14 minPro featurePro featureHard

Not started8.What is speculative decoding, and how does it speed up inference?AI Infrastructure and Scalability · Conceptual · 14 minAsked %CompaniesAI Infrastructure and ScalabilityConceptual14 minPro featurePro featureHard

Not started9.How do you select GPUs for LLM inference?AI Infrastructure and Scalability · Practical exercise · 12 minAsked %CompaniesAI Infrastructure and ScalabilityPractical exercise12 minPro featurePro featureMedium

Not started10.What is tensor parallelism, and how does it help serve large models?AI Infrastructure and Scalability · Conceptual · 12 minAsked %CompaniesAI Infrastructure and ScalabilityConceptual12 minPro featurePro featureHard

Not started11.How do you handle cold start latency for serverless AI deployments?AI Infrastructure and Scalability · Practical exercise · 12 minAsked %CompaniesAI Infrastructure and ScalabilityPractical exercise12 minPro featurePro featureHard

Not started12.What are TTFT, TPOT, and throughput, and how do they trade against each other?AI Infrastructure and Scalability · Conceptual · 10 minAsked %CompaniesAI Infrastructure and ScalabilityConceptual10 minPro featurePro featureMedium

Not started13.How does EAGLE (feature-level speculative decoding) work?AI Infrastructure and Scalability · Conceptual · 10 minAsked %CompaniesAI Infrastructure and ScalabilityConceptual10 minPro featurePro featureHard

Not started14.How do CUDA kernels work?AI Infrastructure and Scalability · Conceptual · 14 minAsked %CompaniesAI Infrastructure and ScalabilityConceptual14 minPro featurePro featureHard

Not started15.What is model sharding, and when would you use it?AI Infrastructure and Scalability · Conceptual · 10 minAsked %CompaniesAI Infrastructure and ScalabilityConceptual10 minPro featurePro featureMedium

Not started16.Estimate the GPU memory needed to serve a 70B model (weights, KV cache, activations). Does it fit on a single 80 GB GPU?AI Infrastructure and Scalability · Practical exercise · 14 minAsked %CompaniesAI Infrastructure and ScalabilityPractical exercise14 minPro featurePro featureHard

Not started17.How does Medusa (multi-head speculative decoding) work?AI Infrastructure and Scalability · Conceptual · 10 minAsked %CompaniesAI Infrastructure and ScalabilityConceptual10 minPro featurePro featureHard

Not started18.How do you implement auto-scaling for AI workloads?AI Infrastructure and Scalability · Practical exercise · 14 minAsked %CompaniesAI Infrastructure and ScalabilityPractical exercise14 minPro featurePro featureHard

Not started19.How do you optimize inference for edge and mobile deployment?AI Infrastructure and Scalability · Practical exercise · 14 minAsked %CompaniesAI Infrastructure and ScalabilityPractical exercise14 minPro featurePro featureHard

Not started20.How does continuous batching improve LLM inference throughput?AI Infrastructure and Scalability · Conceptual · 12 minAsked %CompaniesAI Infrastructure and ScalabilityConceptual12 minPro featurePro featureMedium

Page 1 of 2