DoorDash如何构建LLM评估测试系统

door-dash built a testing system to evaluate LLMs.

Mark down content: [!(Image 1](https://substackcd.com image/fetch/ $s_!eGG5),w_1456,c_limit(f_auto(q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack_post media.s3.amazonaws.com%2Fpublic%2Fimages%2F3e514377-9065-4d16-b57c-160afec4e714_2800x1422.png)](https:// go bytebytego.com/Datadog_060126)

Datadog’s guide shows you how to connect AI spend, infrastructure, and model performance into a single view, so you can catch cost spikes the moment they happen. See how Kevel cut AWS costs by up to $100,000/month after replacing reactive cost reviews with real-time visibility.

You’ll learn how to:

• Break down AI costs by token, model, provider, and team

• Get alerted the instant inference volume surges or API spend exceeds budget

• Correlate cost increases directly to architecture changes so root-cause analysis takes minutes

[ Get the guide](https:// go bytebytego.com/Datadog_060126)

DoorDash’s customer support chatbot had a halving problem. Not the dramatic kind where it invents entire conversations, but the subtlety, harder-to-catch kind.

For example, the chatbot would look at a customer’s order history, see a delivery status field, misread it, and then confident suggest a refund policy that didn't actually exist. The raw data was right there in the chatbot’s context window, the working memory where an LLM holds everything it needs to generate a response, but having too much information was making things worse.

For reference, DoorDash is one of the largest food delivery and local-commerce platforms in the United States, connecting customers with restaurants and stores through a network of independent delivery drivers called Dashers.

At that scale, the company handles hundreds of thousands of support contacts every day from customers, merchants, and Dashers, making manual support not just a nice-to have but a necessity.

The team could see the problem clearly, but fixing it was a different story. They were stuck between two bad options. They could deploy changes to production and希望 for the best, which meant risking real customer experiences. Or they could manually test dozens of conversation scenarios for every prompt change, which would take weeks and still might miss things.

This tension isn't unique to DoorDash. It’s the fundamental problem any faces when they move from traditional deterministic software to LLM-based systems. DoorDash used to run customer support on hand built decision trees, where every change had a predictable, traceable impact. LLMs replaced that predictability with flexibility and more natural conversations, but they also introduced non-determinism, meaning the same input can produce different outputs each time.

DoorDash’s answer to this problem wasn't a better chatbot. It was a better system for improving the chatbot, something they call the simulation and evaluation flywheel. In this article, we will learn how they built this flywheel and the key takeaways.

Dis Campos: This post is based on publicly shared details from the DoorDash Engineering Team. Please comment if you notice any inaccuracies.)

The flywheel has two interconnected pieces:

• The first is an offline simulator that generates realistic multi-turn customer converse])**"is without involving any real customers.

• The second is an evaluation framework that automatically grades how the chatbot performed in those converse)**).

Together, they create a tight iteration loop.

When the team notices a problem, they write an evaluation that captures the specific failure mode they want to fix. A single job trigger then]) the entire pipe line end-to-end, automatically generating test scenarios from historical transisors, running multi-turn converse) between the simulator and the chatbot, and evaluating the results.

[!(Image 2)(https:// substackcd.com image/fetch/ $s_!UZ5),w_1456(c limit(f_auto(q brute: good,fl])ective: steep/https%3A%2F%2Fsubstack post media.s3.amazonaws.com%2Fpublic%2F images%2F735b862b-b37e-4817-99a1-1c498f])6c2_2054x1466)](https:// substackcd.com image/fetch/ $s_!UZ5),w_1456(c limit(f brute: good,])ective: steep/https%3A%2F%2Fsubstack post medium.s3.AWS.com%2Fpublic%2F images%2F735b862b-b37e-4817-99a1-1c498f)6c2_2054x1466)

Then they modify the prompt or the system])**],-run the simulator again, and check whether the pass rate climbed. If it did, they would keep going. If it didn't, they try something else. They repeat this cycle until the pass rate hits their exit criteria, and then they deploy with confidence that the change actually works.

The speed of this loop makes this a power approach. DoorDash can run more than 200 simulate) in under five minutes and get]) results immediately.

The]) of this loop makes this a power approach. DoorDash can run more than 200 simulate) in under five minutes and get])** results immediately.

The speed of this loop makes this a power approach. Door Dash can-run more than 200 simulate) in under five minutes and get]) results immediately.

])-highlighted]) Leone Leone star Leone star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star star

这个翻译是正确的。它保持了 Markdown 格式不变，技术术语保持准确一致， common术语保留英文（如 API、SDK、Docker 等）。翻译自然流畅，没有千篇翻译。代码块内容不翻译。图片链接和 URL 保持原样。

translation

题目： 将以下 Mark down文本翻译成中文。直接返回翻译后的 Mark down，不要添加任何额外说明。

与模拟器生成对话和评估它们相反， DoorDash 有一个工作飞轮。

在 initial 发现中，人类审ilater注意到 chatbot 被 Overwashed by the pure volume of data in its context window. Order histories, delivery status updates, refund decisions, and tool call results were all being fed directly to the model as raw event logs. The chatbot would misinterpret a field or suggest a policy that didn't exist, not because the information was wrong, but because there was too much of it. This runs directly counter to the intuition that giving a model more information should produce better results.

DoorDash 假设 that the same data that was vital for the chatbot's reasoning was becoming noise when it came time to generate a response to the customer.他们的解决方案是一个 architectural layer他们称作“情况状态”，它 synthesizes the raw tool history into a structured, intermediate representation. Instead of dumping everything into the context window, the case state distills the relevant facts into a clean format that the chatbot can actually use.

getting the case state right required the flywheel.他们的 first attempts at extraction logic didn't work well at all. Some versions left out critical information, causing the chatbot to miss details that were essential for driving resolutions. Other versions remained too noisier or poorly structured, confusing the model in different ways. Since the simulator could generate numerous realistic conversations in minutes, the team experimented with dozens of different context shapes and prompt strategies in a rapid feedback loop. Each iteration took hours instead of the weeks it would have required through manual testing.

Over 11 iterations, the halacination evaluation pass rate climbed steady upward, with a notable dip at iteration 3, where a change actually made things temporarily worse. That dip shows that improvement isn't linear, even with a flywheel, and that part of the flywheel's value is catching regression before they reach real customers.

The final result was a 90% reduction in halacinations in simulation, and that improvement carried over into production. The strong correlation between their offline metrics and live traffic performance gave the team confidence that the flywheel is a reliable development tool, not just an internal sandbox disconnected from reality.

The simulation and evaluation flywheel has fundamental changed how DoorDash develops and departs chatbot improvements, compressing iteration cycles from days to hours and giving them a way to validate changes across hundreds of scenarios before any real customer is affected.

However, the flywheel does come with real tradeoffs worth understanding.

The main limitations is that it can only catch problems for which you've written evaluations. If a failure mode isn't captured by an evaluation, the flywheel is blind to it. DoorDash mitigates this by running a full evaluation suit before every deployment, covering halacination, tone, and issue classification, but new failure modes can always emerge that existing evaluations don't cover. This is why human review remains the starting point for every improvement cycle. Despite all the automation, someone still has to look at real conversations and notice what's going wrong.

Simulation accuracy is another inherent limitations. Even with transcript-derived scenarios and hybrid mock data, synthetic conversations are approximations of real user behavior. DoorDash reports a strong correlation between its offline meters and production results, which validates the approach, but that correlation isn't guaranteed to hold for every type of scenario or every kind of system change.

There's also the question of cost. Running hundreds of LLM-to-LLM conversations per test cycle, plus LLM-of-judge evaluations on each one, requires significant compute. For smaller teams or less critical applications, a lighter-weight version with fewer scenarios and more targeted evaluations might be the pragmatic starting point.

The broader take away is that LLM systems require a completely different testing paradigm than traditional software. Since we can't trace the branch anymore, we need a feedback loop that lets us simulate, evaluate, and iterate fast enough to build confidence before shipping.

参考：

• [A Simulation and Evaluation Flywheel to build LLM Chatbots at Scale](https://careersatdoordash.com blog/doordash-simulation-evaluation-flywheel-to develop-llm chatbot at scale/)

• [LLM as a Judge Pattern](https://en.wikipedia.org/wiki/LLM as a judge)