Estimate the GPU memory needed to serve a 70B model (weights, KV cache, activations). Does it fit on a single 80 GB GPU?
Interview insights are lockedSee how often this is asked and which companies asked it with Buncho Pro.Compute weights, KV cache per token, activations and overhead for a Llama-70B-class model across precisions, and decide on GPUs.