Implement a token-bucket rate limiter for an LLM API where the cost of a request scales with its tokens, then make it distributed with Redis
Interview insights are lockedSee how often this is asked and which companies asked it with Buncho Pro.The provider limits you to a budget of tokens per minute, and a request costs roughly prompt_tokens + max_output_tokens.
- Write a token bucket where each request consumes its estimated token cost (not 1 unit).
- Run the application on many servers sharing one quota. Make the limiter distributed with Redis so that two servers can never both spend the same tokens.
- Explain how you reconcile the estimate with the actual usage.