Diagnose high latency in an LLM chat application
Interview insights are lockedSee how often this is asked and which companies asked it with Buncho Pro.Users of an AI chat feature complain that it feels slow. Monitoring shows a p50 of 5.2 s and a p95 of 9.8 s. The team wants a plan to bring the experience under control.
Use the Scenario data (a trace of one slow request and the service configuration) to work out where the time goes, which fixes to prioritise, and how you would verify and protect the improvement.
Scenario data
Logs and configuration from the incident. Use them as evidence in your answer.
Trace of one slow request (p50 = 5.2 s, p95 = 9.8 s)
span start duration
auth_and_session 0 ms 40 ms
load_chat_history 40 ms 120 ms
embed_query 160 ms 180 ms
vector_search 340 ms 650 ms
rerank_top_50 990 ms 1,900 ms cross-encoder on CPU, 50 candidates
build_prompt 2,890 ms 15 ms
llm_first_token 2,905 ms 2,300 ms prompt is about 6,500 tokens
llm_generate 5,205 ms 4,100 ms about 520 output tokens, stream=false
output_moderation 9,305 ms 700 ms runs after the full answer exists
save_to_db 10,005 ms 90 ms
total 10,095 ms