Your Agent Works in the Demo. Now Make It Survive Monday.
TL;DR · AI 摘要
生产环境中代理系统需通过‘harness engineering’解决权限控制与安全问题,而非仅依赖模型能力。
核心要点
- Agent = Model + Harness 的工程范式强调环境约束而非模型限制
- OpenAI案例显示模型可生成百万行代码但需人工审核拉取请求
- 权限控制应通过代码定义可读/写/发送的边界而非依赖人工审批
结构提纲
按章节快速跳转。
- §引言
演示环境与生产环境的差距催生了harness engineering新范式
- ·核心概念
Agent由模型与环境约束共同构成,Harness定义操作边界
模型生成百万行代码时需通过环境设计而非人工审核保障安全
- ·工程原则
应通过代码约束模型权限而非限制其思考能力
- ›部署挑战
Forward-deployed engineers需解决生产环境中的权限与反馈循环问题
思维导图
用一张图看清主题之间的关系。
查看大纲文本(无障碍 / 无 JS 友好)
- Agentic AI工程化
- 核心概念
- Harness Engineering
- Agent = Model + Harness
- 案例研究
- OpenAI实践
- Palantir FDE部署
- 工程原则
- 环境约束优先
- 自动化反馈循环
金句 / Highlights
值得收藏与分享的关键句。
Agent = Model + Harness 的工程范式强调环境约束而非模型限制
OpenAI案例显示模型可生成百万行代码但需人工审核拉取请求
权限控制应通过代码定义可读/写/发送的边界而非依赖人工审批
Your Agent Works in the Demo. Now Make It Survive Monday. | Towards Data Science
Agentic AI
Your Agent Works in the Demo. Now Make It Survive Monday.
The demo is the easy part
Gursimar Singh
November 3, 2026
11 min read
Prefer us on Google
Share
Image by Getty Images via Unsplash
Anyone can build an AI agent in an afternoon. Give a model some tools, write a loop, watch it call an API and summarise a web page. It feels like magic.
Then you put it in front of a real business. It touches a customer record it should not touch. It runs for forty minutes and forgets what it was doing. It passes your test suite because it wrote the tests. Finance wants to know who approved the change, and the honest answer is "the model, probably."
That gap between "it works on my laptop" and "it works on Monday morning with real data" has picked up a name: harness engineering . (You'll also hear "agentic engineering" and "context engineering" for overlapping ideas. The label matters less than the discipline.)
The idea is simple. An agent is not just a model. It is a model plus everything wrapped around it: the loop that drives it, the tools it can reach, the permissions that stop it, the files that remember for it, the tests that check it, and the people who sign off on it. That wrapper is the harness. Thoughtworks' shorthand : Agent = Model + Harness.
The term spread after OpenAI published a write-up in February 2026 on shipping an internal product with zero hand-written code. Three engineers driving Codex produced on the order of a million lines and roughly 1,500 merged pull requests in five months. Their conclusion was not "the model is amazing." It was that the engineering had moved into designing environments, specifying intent, and building feedback loops. Tellingly, humans could review pull requests but were not required to ( OpenAI ). The safeguards were in the harness, not in the reviewer.
One principle runs through everything below, so I'll state it once: constrain what the model can touch, not what it can think about. Let it plan and reason freely. Decide in code what it is allowed to read, write, spend and send.
This post is for anyone building agents who keeps hitting the same question at work: should the model decide this, should code enforce it, or does a human need to approve it? You'll want basic familiarity with LLMs, tool calling and reading Python. No specific framework required.
1. Start where the agent will live
The most useful lens on production agents comes from the people who sit inside the customer's building and make the thing work: forward-deployed engineers. Palantir invented the role; the AI labs have adopted it.
Hiring data gives a rough sense of scale. A September 2026 count of public job boards found 77 of Palantir's 310 open roles were forward deployed, and OpenAI listed 21 of 780, with Anthropic running the smallest FDE footprint of the group ( Valletta Software ). Treat the raw numbers with care. The same analysis found most of Palantir's FDE postings had been open more than six months. But the direction is clear: labs are investing in deployment, not just models.
An FDE's job is translation. The business says "clean up our supplier data" or "answer refund questions." The FDE turns that into a system with clear answers to three questions:
- How much can the agent do alone? Read only? Draft but not send? Act under a spending or record limit?
- Where are the walls? Which systems, tables and actions are off limits regardless of what the model concludes?
- Where does a human sign off? Which decisions are too costly, regulated or irreversible to leave to a model?
None of these are prompt questions. They are product, risk and operations questions, and you answer them with the business owner, the compliance person and the engineer who will be paged at 2am.
A good exercise before writing code: get engineering and the business in one room and ask each person what the agent should never do, and what it should always ask about first. The disagreements you surface in that hour save weeks.
2. Open the hood: what real frameworks actually do
Deep-research agents are a good place to study this because they went through three phases in about two years:
- Model-driven. One loop, one model, a bag of tools. The model decides everything. Flexible, but it wanders, loops and forgets.
- Code-based pipelines. Engineers took control back. A fixed graph: plan, then research, then write. Predictable, but anything outside the graph is impossible.
- Harnesses. The model plans and sequences freely, inside walls that code enforces: sandboxes, permissions, memory files, sub-agents with their own context, checks before and after each action.
Note what changes between phase two and three. The model gets more say over the steps, not less. What gets locked down is the blast radius.
DeerFlow: a pipeline that grew into a harness
ByteDance's DeerFlow v1 (2025) was a LangGraph pipeline: a Coordinator took the request, a Planner broke it into steps, a Researcher and a Coder did the work, a Reporter wrote it up, and a human could review the plan before execution.
DeerFlow 2.0 is a ground-up rewrite that calls itself a "super agent harness." It's still built on LangGraph and LangChain. The framework didn't change; the shape did. A lead agent spawns sub-agents on the fly, each with scoped context, tools and termination conditions. Execution runs in a sandbox that can be local, Docker or Kubernetes. Skills are Markdown files describing a workflow. Memory persists across sessions ( DeerFlow repo , deployment guide ).
Same team, same problem. They moved from "code decides the steps" to "model decides the steps, code decides the boundaries."
Claude Code: a small loop inside serious infrastructure
The core loop is almost boring: assemble context, call the model, run the tools it asks for, append results, repeat until it stops asking. You could write it in fifty lines. The value is around it:
- Permissions. Before each tool call, the permission mode and allow/deny rules decide whether to auto-approve, ask, or block. This is the actual enforcement layer.
- Hooks. Lifecycle events (PreToolUse, PostToolUse, Stop and others) run your own shell commands or checks, in your process rather than the model's context. This is where logging and policy checks live ( Claude Code harness layers ).
- Sub-agents. Separate context window, hand back a summary, so a big research pass doesn't flood the main thread.
- Context management. Project memory files (CLAUDE.md) load into context; compaction kicks in as the window fills.
Anthropic's guidance on long-running agents shows the same thinking. Each session starts with no memory, so the work is split: an initialiser sets up the environment, a feature list, a progress file and a git repo once; a coding agent then makes one increment of progress per session and leaves notes for the next ( Anthropic ). That's one approach to memory: files and git. DeerFlow's persistent memory store is another. Neither is a magic model feature.
MiroFlow: built for the leaderboard, and a lesson in reading one
MiroMind's MiroFlow is an open-source research framework: a main agent augments the query, plans, delegates to specialist sub-agents, calls tools over MCP and synthesises. Its February 2026 paper describes an agent graph for orchestration, an optional deep-reasoning mode and a workflow built for reproducible runs ( arXiv ).
Its headline numbers are strong, and it publishes reproducible code and full traces, which is more than most. But read the table. With Claude 3.7 Sonnet, MiroFlow scores 82.4 on GAIA validation, and that figure combines answers from several agents. The average over three single runs, on the same model, is 73.1. On the held-out test set it's 67.3 averaged over single runs and 73.1 with the ensemble ( arXiv ). Some comparison rows in the same table are text-only subsets.
None of that is dishonest. It's a nine-point gap between the headline and the typical run, in a framework that is unusually transparent. Ask the same question of any framework that doesn't publish its traces.
The same paper tried to reproduce two rival frameworks. OWL came in 15.8 points below its reported GAIA score. AgentOrchestra, which also claimed 82.4, came in 27.8 points below. The authors say this doesn't mean those claims were false, only that they were hard to reproduce. That's the point: a headline number you can't rerun is a press release.
How to read benchmark claims
A NeurIPS 2025 paper audited popular agent benchmarks and found flaws in task setup and grading that can over- or under-estimate performance by up to 100% in relative terms. τ-bench, for example, counted empty responses as successes. The authors published an Agentic Benchmark Checklist ( NeurIPS 2025 ); what follows is the short version.
When you see a framework claim, ask:
- Validation set or held-out test set?
- Full benchmark or a subset?
- Single run, best of N, average of N, or majority vote over N?
- What does the grader actually check? Could a wrong answer pass?
- Same model, tools and budget as the comparison?
Read the trace, not the README
The best habit you can build is reading agent traces. Run a framework on a real task and walk every step: what was in context, which tool it chose, what came back, where it got confused, where the harness caught it and where it didn't. One ugly trace teaches more than an architecture page.
3. Case study: the data-cleansing agent meets real data
Data cleansing is a good test: it sounds dull, it's everywhere, and it punishes sloppy agents fast.
Picture years of messy master data. Duplicate suppliers, inconsistent addresses, product codes typed five ways, fields that mean different things by region. The ask: "use AI to clean it up."
Version one: let the model do it
Hand the model a batch, a description of "clean," and read/write tools. It demos beautifully. Then:
- Not repeatable. Same batch twice, different answers.
- Confidently wrong. Two similarly named suppliers get merged; they're separate legal entities.
- Burns money on easy work. Tokens spent normalising dates and trimming whitespace that a regex does for free.
- Unauditable. A record changes with no rule, reason or approver attached.
- Touches what it shouldn't. Write access to the whole table means one bad step hits live data.
Version two: redesign around who owns what
The fix is not a better prompt. It's splitting the work by who should own each decision.
- Deterministic code takes the certain work: format normalisation, schema validation, exact-match dedupe, reference lookups. If a rule can be written and tested, it goes in code.
- The model takes the fuzzy middle. Is "Acme Ltd, Leeds" the same as "ACME Limited (Leeds Branch)"? Which of three conflicting addresses is current? The model proposes, with a reason and a confidence.
- Domain experts own the risky calls: legal-entity merges, anything financial or regulated, low-confidence matches, anything above an agreed volume. Their decisions feed back as rules or examples.
- The harness ties it together. Writes go to staging. Every change carries its source, rule or reasoning, and approver. Batches replay and roll back.
The model did less in version two and the system got much better. Reliability came from shrinking the model's job to the part only a model can do, and wrapping it in checks. And the business got something it could sign off on: boundaries, a review queue, an audit trail.
One honest cost: the review queue is now your bottleneck. If experts can't keep up, the agent stalls or reviewers rubber-stamp. Budget for the humans before you budget for the tokens.
4. Design the harness before you write the agent
Try this on something real: a refund-approval agent, invoice matching, IT-ticket triage.
Step 1: list every decision. Not the steps. The decisions. "Which records match?" "Is this refund valid?" "Should I email the customer?"
Step 2: give each decision an owner. Two axes: can it be written as a testable rule, and how expensive is a wrong answer to catch or undo?
- Rule-expressible → code. Don't pay a model for arithmetic, validation or lookups.
- Needs judgement, cheap to catch or undo → model, inside guardrails.
- Needs judgement, expensive or irreversible → human approves. The model prepares the decision; a person makes it.
Step 3: set the autonomy boundary. A ladder: Suggest (recommends only) → Draft (prepares, human clicks go) → Act within limits (hard caps on amount, volume, scope) → Act and report (logs, humans review after). Start lower than the team wants. Climb when the traces earn it.
Step 4: enforce the boundary in code, not the prompt. "Please don't delete records" in a system prompt is a wish. A database role with no delete permission is a guarantee. Permission checks before every tool call; sandboxes and staging instead of live writes; hard limits on spend, runtime, retries and blast radius; validation after every action. Thoughtworks frames this as guides (controls before the action) and sensors (checks after it); some are cheap and computational, some are LLM reviewers and cost more. You want both.
Step 5: design the human checkpoint like a product. A reviewer shown a wall of JSON will rubber-stamp. Show the proposed action, the evidence, the model's reason and confidence, and one-click approve, reject or edit. Feed every correction back into rules, examples or tests. Measure queue depth and time-to-decision; those are your real throughput limits.
Step 6: make it observable and reversible. Log every step as a replayable trace. Attach reason and approver to every change. Make rollback boring. If you can't answer "what did the agent do and why" in five minutes, you're not in production yet.
What to take with you
- The model is the engine. The harness is the product.
- Constrain what the model can touch, not what it can think about.
- If a rule can be written and tested, it doesn't belong in a prompt.
- A boundary in a prompt is a suggestion. A boundary in code is a boundary.
- Humans approve what is costly or irreversible, and the review queue is a system you have to staff and measure.
- Every agent mistake is a missing piece of harness. Fix the harness, not just the run.
- Trust benchmarks you can reproduce, on splits you understand, reported as averages.
- Autonomy is earned, one level at a time.
Building an agent is easy. Making it behave when nobody is watching is the real engineering.
I hope you’ve enjoyed this and learned something new. I’m always open to suggestions and discussions on LinkedIn . Hit me up with direct messages.
Till the next one, happy exploring!