Do the roofline math: how many tokens/sec can one H100 produce for a 70B model at batch size 1?
Interview insights are lockedSee how often this is asked and which companies asked it with Buncho Pro.Use bandwidth and FLOPs limits to bound batch-1 decode speed, and show how batching, quantisation and speculation change it.