LLM Observability: Metrics, Python & Best Practices | Buncho
AI Engineering / Generative AI
LLM Logging & Observability: What AI Engineers Should Know for Interviews
Prepare for AI Engineering interviews with practical examples of LLM logging, monitoring, latency (P50, P95, P99), error handling, token usage, cost tracking, RAG evaluation, and hallucination detection.
Key takeaways
LLM logging records events, monitoring tracks system health, and observability helps identify root causes.
Track P50, P95, P99 latency, error rates, and time to first token to understand performance.
Monitor input and output tokens, cost per request, and cost per user to control LLM expenses.
Building an LLM application is easy compared to debugging it in production.
Imagine your platform suddenly takes 20 seconds to evaluate an answer. Some users get incorrect feedback, and your API costs keep increasing.
Is the model slow? Is retrieval returning irrelevant documents? Are retries causing extra API calls?
That's where LLM logging and observability come in. As an AI Engineer, you need to know not just how to build an LLM application, but also how to debug and monitor it.
These three terms are commonly asked about in AI Engineering interviews.
Logging - What happened? Records events, errors, and request details.
Monitoring - Is the system healthy? Tracks metrics such as error rate, latency, and cost.
Observability - Why did it happen? Helps identify the root cause using logs, metrics, traces, and evaluation results.
Example: Your RAG application returns an incorrect answer. Logs show the request succeeded, but tracing reveals that retrieval returned irrelevant documents. The problem is in the retrieval pipeline, not necessarily the LLM.
P50 = 1 second: Half of requests finish within one second.
P95 = 5 seconds: 95% finish within five seconds.
P99 = 15 seconds: 99% finish within 15 seconds.
Why does P95 matter more than average latency?
An average can hide slow requests. P95 is the latency threshold at or below which 95% of requests complete. The remaining 5% take at least as long as that threshold, subject to the percentile calculation method. P95 is useful for detecting slow user experiences that averages can hide.
For streaming applications, also track time to first token (TTFT) - how long the user waits before the first generated token arrives.
3. Implement LLM logging and observability using LangChain#
We'll build a pipeline that lets you monitor:
Latency: How long retrieval and LLM generation take.
Logging: Request IDs, errors, model names, and retry attempts.
Token usage and cost: Track model usage per request.
RAG quality: Inspect retrieved documents and evaluate faithfulness.
Reliability: Handle timeouts, retries, and failures.
Tracing: Use LangSmith to inspect the full chain execution.
This is a useful starting point, but a few additional components are needed for a complete observability system.
P50/P95/P99 dashboards and alerting: Export latency and error metrics to Prometheus or another metrics backend. Calculate percentiles from request-level measurements rather than averaging percentiles.
Retry visibility:max_retries=2 configures bounded retries in the model integration; it does not automatically log every individual attempt in the application logs. Add provider callbacks or SDK instrumentation if you need per-attempt details.
Faithfulness evaluation: Log and evaluate retrieved evidence against generated claims using a validated evaluator and human-reviewed test cases. Logging retrieved documents alone does not measure faithfulness.
Privacy controls: Avoid exposing sensitive prompts, retrieved content, and responses in logs or tracing platforms without appropriate redaction, access controls, and retention policies.
Streaming latency: If your application streams answers, measure time to first token (TTFT) separately from total generation time.
LLM applications can fail even when your code is correct. Common issues include rate limits (HTTP 429), provider errors (HTTP 5xx), timeouts and tool failures.
Know these three techniques:
Retries: Retry temporary failures with exponential backoff and jitter. Set a retry limit and avoid retrying permanent errors.
Timeouts: Prevent requests from waiting indefinitely for a slow provider.
Fallbacks: Switch to another model or return a graceful error when the primary provider is unavailable. Check that the fallback meets your quality and cost requirements.
Also track both individual failed attempts and the final outcome. A request that succeeds after a retry is different from one that fails completely.
LLM costs depend on factors such as the model, input tokens, output tokens, and applicable pricing.
To control costs:
Remove irrelevant conversation history.
Retrieve fewer, more relevant document chunks.
Limit unnecessary output length.
Track token usage and estimated cost per request.
Cache reusable results when doing so is safe.
Exact-match caching reuses results for matching cache keys.
Semantic caching can reuse results for queries with similar meanings, but similarity alone doesn't guarantee that an answer is appropriate. Account for context, user permissions, and freshness.
Track cache hit rate alongside answer quality. A high hit rate isn't useful if users receive incorrect or outdated answers.
Q1. Your LLM application has high P95 latency. How would you debug it?
I'd compare P50, P95, and P99 latency and inspect distributed traces to break each request into retrieval, model generation, tool execution, and post-processing. I'd check provider latency, token counts, queueing, retries, and timeouts to identify the bottleneck. Then I'd optimize the slowest stage and validate the improvement against latency and quality metrics.
Q2. Your API error rate suddenly increases. What would you check?
I'd inspect error types, HTTP status codes, provider status, rate limits, timeouts, and retry counts. I'd then identify whether the issue is in the provider or our application.
Q3. Your RAG application produces hallucinations. How would you investigate?
I'd inspect retrieved chunks and measure retrieval relevance, then evaluate whether the answer's claims are supported by the retrieved evidence. I'd distinguish retrieval failures from prompt-construction and generation failures, use a human-reviewed evaluation set, and compare quality before and after any changes.
Q4. How would you reduce LLM costs without hurting quality?
I'd analyze input and output token usage, remove irrelevant context, optimize retrieval, and cache safe reusable responses. I'd compare answer quality before and after each change.
Q5. What would you log for every LLM request?
At minimum: request ID, timestamp, model, latency, status, error details, and token usage. For RAG and agent applications, I'd also trace retrieved documents and tool executions. Sensitive payloads should be protected or redacted.
For AI Engineering interviews, remember five things: latency, reliability, cost, quality, and root-cause analysis.
When an LLM application fails, you should be able to explain what you'd measure, where you'd look for the problem, and how you'd fix it without blindly changing the model.
That's the difference between building an LLM demo and engineering an AI system for production.
Practice what you learned
Try these interview questions on the same topic and get AI feedback on your answers.
Comments (0)
Sign in to join the discussion.