Implement multi-head attention, then convert it to grouped-query attention
Interview insights are lockedSee how often this is asked and which companies asked it with Buncho Pro.Write a PyTorch multi-head attention module with a causal mask, then modify it to support n_kv_heads < n_heads (GQA), with MQA as a special case.