The Forward Deployed Playbook · EP01
Measure the outcome, not the sprint
A forward deployed engineer proves a deployment worked by measuring the business number the customer bought it to move, such as time to resolve a case, error rate or cost per transaction, against a baseline taken before go-live. Story points, velocity and closed tickets measure delivery. They cannot show whether the business improved, and an AI deployment can hit every sprint goal while changing nothing.
Why don’t story points tell you whether a deployment worked?
Because they measure the team, not the customer. A story point is an estimate of effort, and velocity is the rate at which estimated effort gets marked done. Neither one contains any information about what happened to the process the system was built for. The moment a team is judged on velocity, it learns to optimise velocity, which is Goodhart’s Law doing exactly what it says: the metric stops measuring the thing.
AI deployments make the gap wider. A model can be integrated, demoed and signed off while the people who were meant to use it route around it. That is the pattern behind the AI project failure numbers: MIT’s NANDA initiative reported in August 2025 that 95% of enterprise generative AI pilots it studied showed no measurable impact on profit and loss. Most of those pilots were delivered. Few were measured against the business.
What should a forward deployed engineer measure instead?
One outcome the customer would recognise without a definition, one or two guardrails that must not get worse, and nothing that only engineers understand. Typical pairs:
| Deployment | Outcome metric | Guardrail |
|---|---|---|
| Support agent or copilot | Median time to resolution | Reopen rate, escalation rate |
| Document extraction | Cost per document processed end to end | Field-level error rate on a weekly audit sample |
| Claims or case triage | Cases cleared per analyst per day | Wrongly cleared cases found on review |
| Internal knowledge assistant | Time to answer a question that used to need an expert | Answers flagged wrong by experts |
| Code or content generation | Lead time from request to accepted result | Rework rate after acceptance |
Notice what is missing: model accuracy, token counts, latency. Those matter for running the system, and later episodes cover them. They are inputs to the outcome, not the outcome.
How do you set a baseline before go-live?
Measure the current process the way the new system will be measured, on real cases, before anyone changes how they work. In practice:
- Agree the metric in discovery, in the customer’s words, with the person who owns the budget.
- Pull or sample the current numbers for at least two weeks. If the data does not exist, time and count a sample by hand.
- Write the baseline down with its date, sample size and method, and get the customer to sign it off.
- Set the target as a change against that baseline, not as an absolute number lifted from a vendor deck.
The signed baseline is the part teams skip, and it is the part that saves the engagement. Six months later, nobody remembers how slow the old process was.
How do you know the change came from the system?
Pick the strongest design the customer’s operations allow, in this order:
- Holdout: route a random share of cases around the system and compare. The cleanest answer, and often politically hard.
- Staggered rollout: switch on team by team or region by region, and compare teams that have it with teams that do not yet.
- Shadow mode: run the system alongside the old process without acting on its output, and score what it would have done.
- Before and after: the weakest design, because seasonality and staffing move the number too. Use it only with a long baseline.
What changed when we measured our own agent deployments this way?
I run agents across our own portfolio of content sites and services, so I am both the vendor and the customer, which removes every excuse. In early November 2026 we shipped the work to give each site a written voice, audience and vocabulary that agents load before writing. The ticket view said done: pull requests merged, a daily job installed. The outcome metric we set instead was the share of in-scope sites that are fully grounded. The first daily run counted 1 of 85.
That number is uncomfortable, and it is the point. A velocity report would have shown a productive week. The outcome report shows a baseline and a gap, which is what tells us where the next week goes. The same rule now applies to our editorial pipeline: 236 ideas moved into the new queue is an output; articles that rank and get read are the outcome we report.
What should you report to the customer?
One page, every two weeks: the outcome metric against the signed baseline, the guardrails, what changed in the system since the last report, and what you will change next. Put the delivery numbers in an appendix if anyone asks for them. If you want to turn the outcome into money for a sponsor, the AI ROI hub and its calculator take it from there.
Sources: Fortune on MIT NANDA, “The GenAI Divide” (18 August 2025) · grounding coverage figures from our own operations log, first daily run, November 2026.
Measuring AI outcomes: common questions
What is the difference between an output metric and an outcome metric?
An output metric counts what the team produced: tickets closed, story points, features shipped, documents processed by the system. An outcome metric counts what changed for the business because of it: time to resolve a case, error rate on a process, cost per transaction. Outputs tell you the team is busy. Only outcomes tell you the deployment worked.
When should you set the baseline for an AI deployment?
Before the first user touches the system, and ideally in the first week of discovery. Once the system is live you can no longer measure the old process cleanly, because people change how they work as soon as a new tool appears. A baseline taken from memory or from a vendor benchmark is not a baseline.
What if the customer has no data on the current process?
Then measuring the current process is the first deliverable. Sample it by hand for two weeks: time a set of cases, count the errors in a batch, log the exceptions. A rough baseline built from fifty real cases beats a precise number nobody can defend.
How many metrics should a deployment track?
One outcome the customer bought the system to move, one or two guardrails that must not get worse (quality, escalation rate, cost), and the operational numbers you need to run it. More than that and nobody reads the report.
