Agentic AI
How to Evaluate Agents Locally with Arize Phoenix
A hands-on guide to running Phoenix locally: trace an agent over OpenTelemetry, then score it with LLM-as-judge and code evaluators.
7 min read
17 articles
A hands-on guide to running Phoenix locally: trace an agent over OpenTelemetry, then score it with LLM-as-judge and code evaluators.
AgentOps is the practice of running agents in production: observability, evaluation, guardrails, cost governance, and human oversight.
A tour of the reasoning research: how prompting elicits it, how researchers categorize the techniques, and why one prominent position paper says it is not reasoning at all.
A production case study measured fourteen prompt modifications one by one. The results are humbling: naming the model helped, few-shot examples hurt, and templates cost accuracy.
Cranking temperature does not make synthetic data diverse. Injecting randomness into the prompt does. Here is the technique, with code.
You can prompt a model to produce labeled examples on demand: test fixtures, eval sets, cold-start training data. Here is how to do it well and where the sharp edges are.
Your few-shot examples carry hidden biases. The distribution and order of them can nudge the model toward a label before it reads the input. Here is how to spot it and fix it.
Models hallucinate because they are built to sound coherent, not to be right. Here are three prompt-level moves that reduce made-up answers, plus where prompting stops working.
Hand-picking chain-of-thought examples is a guess. Active-Prompt uses model disagreement to find the questions where a human annotation actually moves the needle.
One chain of thought can take a wrong turn. Self-consistency samples several reasoning paths for the same question and votes on the answer, beating a single greedy pass on reasoning tasks.
Telling a model to reason before it answers genuinely improves accuracy on multi-step problems. How standard and zero-shot chain-of-thought work, when to use them, and the cost.
Examples are the most effective context you can add to a prompt, until they aren't. When few-shot beats zero-shot, how many to use, how to pick them, and where examples won't help.
A practical guide to LLM observability: the traces and metrics to log, the handful of alerts worth waking up for, and why traditional monitoring misses what matters.
If you can already ship software, the way into AI engineering is not another tutorial. It is building five systems that force you to handle failure, measure quality, and run things in production.
A practical playbook for making LLM apps cheaper and faster without hurting answers: where the cost and latency actually go, and the levers that work.
Rerankers re-score your top candidates with a model that actually reads the query and the document together. Here is how they work and when they pay off.
You cannot improve an agent you cannot measure. A deep, practical guide to evaluating agents with LangSmith: tracing, datasets, evaluators, experiments, and what makes agent evaluation different.