Tag

Evaluation

21 articles

From Demo to Product · Part 10Who Owns Quality: The Benevolent Dictator and AI Incident ResponseOn an AI product, quality is a judgment call that needs one owner, and an incident whose postmortem output is a new eval exampleAugust 2, 20269 minAugust 2, 20269 min
From Demo to Product · Part 7LLMOps and the Data FlywheelLaunch is the start of the work. How you operate an AI product, and how production usage compounds into a moat.August 2, 20268 minAugust 2, 20268 min
From Demo to Product · Part 3Evals Are the Product Discipline (and Writing Them First Backfires)Why eval-driven development done like TDD fails, and what error-analysis-first evaluation looks like insteadJuly 25, 20266 minJuly 25, 20266 min
From Demo to Product · Part 1The Afternoon and the QuarterWhy an AI demo takes an afternoon and an AI product takes a quarter, and what lives in the gapJuly 25, 20265 minJuly 25, 20265 min
Agentic AIHow to Evaluate Agents Locally with Arize PhoenixTrace and judge your agent on your own machine, no hosted service requiredJuly 12, 20267 minJuly 12, 20267 min
Agentic AIWhat Is AgentOps? Operating AI Agents in ProductionHow MLOps and LLMOps extend to autonomous, multi-step agentsJuly 12, 20266 minJuly 12, 20266 min
LLM IntegrationWhat the Research Actually Says About Prompting ReasoningThe taxonomy of reasoning elicitation, and the uncomfortable retrieval-vs-reasoning debate every agent builder should sit withJuly 3, 20264 minJuly 3, 20264 min
LLM IntegrationA Real Prompt Engineering Case Study: 65.6 to 91.7 F1 on Job ClassificationWhat a production classification system teaches about which prompt tweaks actually move the needleJuly 3, 20264 minJuly 3, 20264 min
LLM IntegrationDataset Diversity: Fixing Repetitive Synthetic GenerationsWhy your generated dataset all sounds the same, and the seeding trick that fixes itJuly 3, 20267 minJuly 3, 20267 min
LLM IntegrationGenerating Synthetic Data with PromptsTurn a model into a data factory for tests, evals, and cold-start training, without spending a month labelingJuly 2, 20266 minJuly 2, 20266 min
LLM IntegrationBias in Prompting: How Your Prompt Design Skews the ModelThe distribution and order of your few-shot examples can quietly bias the model. Here is how to test for it.July 2, 20266 minJuly 2, 20266 min
LLM IntegrationFactuality: Prompting to Reduce HallucinationThree prompt-level moves that cut made-up answers, and where prompting stops being enoughJuly 2, 20266 minJuly 2, 20266 min
LLM IntegrationActive-Prompt: Stop Guessing Which Few-Shot Examples to AnnotateUse the model's own uncertainty to pick which questions are worth a human-written reasoning chainJuly 1, 20266 minJuly 1, 20266 min
Prompt Engineering, Properly · Part 6Self-Consistency: Sampling Your Way to Better AnswersRun the same reasoning prompt several times, then take the majority answerJune 29, 20265 minJune 29, 20265 min
Prompt Engineering, Properly · Part 5Chain-of-Thought PromptingStandard and zero-shot CoT, when step-by-step reasoning actually helps, and what it costsJune 29, 20267 minJune 29, 20267 min
Prompt Engineering, Properly · Part 4Zero-Shot vs Few-Shot PromptingWhen examples help, how many to use, how to pick them, and where few-shot quietly breaks downJune 29, 20267 minJune 29, 20267 min
AI InfrastructureObservability for LLM Apps: What to Log, What to Alert OnYour 500s and latency graphs will look fine while the product quietly gives wrong answers. LLM observability is about catching the failures that do not throw.June 24, 20267 minJune 24, 20267 min
AI InfrastructureFive Projects to Actually Master AI Engineering (for Experienced Engineers)Skip the toy chatbots. Build the five systems that teach the hard parts: retrieval, agents, evals, cost, and reliability.June 23, 20267 minJune 23, 20267 min
LLM IntegrationCutting LLM Cost and Latency Without Wrecking QualityMeasure first, then reach for caching, routing, smaller models, and the structural fixes that actually move the numbersJune 22, 20267 minJune 22, 20267 min
Production RAG · Part 2Reranking in RAG: Cross-Encoders and When They Are Worth the LatencyA reranker can rescue a mediocre retriever or waste 200ms on an already-correct answer. Knowing which is the whole skill.June 20, 20265 minJune 20, 20265 min
Agentic AI in Practice · Part 5Evaluating Agents with LangSmith: A Complete GuideWhy "it worked when I tried it" is not evaluation, and how to measure agents properly with tracing, datasets, evaluators, and experimentsJune 19, 202611 minJune 19, 202611 min