Route requests to the smallest capable model, and use a larger one only when needed.
Cache: prompt caching for long static prefixes, exact-match and semantic caching for repeated queries.
Reduce tokens: shorter prompts, fewer retrieved chunks, capped outputs; batch non-urgent work and run independent calls in parallel.
Stream tokens for perceived speed, set timeouts with fallbacks, and monitor cost per request and per feature with budgets and alerts.
Loading…