The model is not the point: why enterprise AI is won or lost in the harness
AI is still judged on benchmarks, and lately on what its tokens cost. Inside a company, three numbers matter more: what AI earns when things go well, what it costs when they go wrong, and what it costs to run. Those three numbers depend less on the model than on what surrounds it: the harness. This article sets out the measure that counts, defines the harness, and gives eight questions for judging any tool before putting it into production.
The real measure: expected value, minus cost
Enterprise AI is still in its early days, and it shows in the way it gets judged. Companies know they have to use it; few know how, and fewer still what it brings them. We have seen some measure AI’s contribution by the number of tokens consumed, which amounts to counting an expense as a result.
Most of the news, meanwhile, is about models and their benchmarks: one model gains three points in mathematics, another gains one in reasoning. More recently, people have started to talk about the cost of a task, since two models do not use the same number of tokens to do the same thing. That is progress, but it still measures the case where everything goes right.
The question that matters is expected value: what a task earns when it succeeds, minus what it costs when it fails, each weighted by its probability. Subtract what the task itself costs, in tokens, compute and time, and you get AI’s real contribution. One rare error can wipe out hundreds of successes if it sends the customer list to the wrong person. This is how we judge AI at PipeDuck, and it is what puts security before cost, and cost before capabilities.
p, the probability of success
The model provides part of it, the part benchmarks measure. Rules written in code and human approval provide the rest.
The cost of a failure
The term that can mean ruin. The harness bounds it: limited rights, isolation, approval before any irreversible action.
The cost of the task
Paid on every run, successful or not: tokens, compute, time. Code where code is enough, the right model at each step, a spending cap.
Something matters even more than expected value. A business can absorb violent swings: a fall of 80% followed by a rise of 500% leaves it 20% above where it started. It cannot absorb zero. An error that leaks customer data, commits a payment that cannot be recovered, or breaks the trust of a major account ends the game: there is no climbing back. The physicist Ole Peters formalized this gap: what is true on average across many companies is not true for each one over time, because a total loss cuts off everything that follows. He calls it the ergodicity problem1. Ruin therefore has to be impossible, not merely improbable, and that is the first job of a harness.
Three questions no benchmark asks
Take a common case: an AI reads customer emails, finds the order and replies. The demo takes an afternoon to build, and it is convincing. Then the project goes before the steering committee.
The head of security asks whose rights this AI uses to read the orders, and what it would do with an email telling it to send out the customer list. Finance asks what a month of running it will cost, and what stops it from costing ten times as much. The head of operations asks what happens the day an answer is wrong, and who will have approved it.
None of these questions is about the model. A benchmark measures what a model can do; it says nothing about what the model is allowed to do, what that costs, or who is accountable for it. Even reliability parts ways with it: a March 2026 study of ten models shows that on long tasks, the ranking by capability and the ranking by reliability diverge sharply2. Each of these questions targets one term of the contribution, and the first two hardly depend on the model at all.
The harness is everything around the model
The image comes from working horses: a harness connects the horse to what it pulls, and it is through the harness that the horse is steered. Applied to AI, the word means the infrastructure that frames a model: the loop that calls it, the tools it is allowed to use, the context it is given, the isolated environment where code runs, human approvals, traceability, budget and error recovery. The model reasons; the harness decides what that reasoning is allowed to touch.
It carries more weight than people expect, even on benchmarks. In February 2026, the LangChain team took its agent from 52.8% to 66.5% success on Terminal-Bench 2.0 by changing only the harness, with the same model3. Yet most writing treats the harness as an engineering project. PipeDuck starts from a different idea: the harness can be a product, in which a company’s rules are configured instead of programmed.
Your company already has a harness, for its employees
No company hands bank signing authority to a new hire on day one, however brilliant. It gives them the access their job requires, asks for a second signature on commitments above a threshold, keeps a record of what was done and by whom, and separates the environments where people test from those where the company is committed. That is internal control, and it is exactly what a harness applies to a model.
| For an employee | For an AI | In PipeDuck |
|---|---|---|
| Access limited to what the job requires | Data and tools limited to the task | Each AI step sees only its instructions and has only the tools checked for it |
| A second signature above a threshold | Human approval before acting | An approval step with named approvers; a workflow that uses a non-safe action is approved before it runs, and again after every change |
| An audit trail | Traceability | Each step keeps its inputs, its result, its logs and who started it; every change to the workflow is versioned |
| A commitment ceiling | A budget | An AI spending cap checked before every AI step |
| Separated environments | Isolation | Code runs in an isolated container with filtered network access; database passwords stay out of the code |
| Written, reviewed procedures | Deterministic steps | Rules are code, reviewed and approved by a person before they can run |
Eight questions to ask before putting an agent into production
They separate a production-ready harness from a demo, whatever tool you choose. None of them is settled by choosing a better model.
- Whose rights does the AI act with?
- What data does it see, which tools can it call, and who decided?
- What happens if an email or a document gives it instructions?
- Who approves an irreversible action, and does that approval lapse when the process changes?
- Where does the code run, and what can it reach?
- Who can say, three months later, what happened and who approved it?
- What is the spending cap, and is it checked before the call?
- Can you change the model, for one step or for all of them, without rewriting the process?
Choosing a model takes an afternoon, and the choice gets revisited every quarter. The rules around it commit you for years: that is where enterprise AI is won or lost, and that is where we put our work.
Sources
- O. Peters, “The ergodicity problem in economics”, Nature Physics 15, 1216-1221, 2019.
- A. Khanal, Y. Tao, J. Zhou, “Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents”, arXiv 2603.29231, March 2026.
- LangChain, “Improving Deep Agents with harness engineering”, blog, February 17, 2026.