AutoBuddy Finds Agents Spot Owed Work but Fail During Execution

Microsoft researchers tested 17 models as always-on agents in a simulated company. They found most of the owed work but lost the most credit in helpdesk, CRM, finance and HR systems, so a pilot has to score discovery and execution separately.

By Rajesh Beri·November 21, 2026·10 min read
Share:
An office desk at dusk with a laptop showing an open helpdesk ticket queue, a sticky note half-peeled off the screen bezel, and a printed CRM account record on the desk with one line circled in red pen.

Illustration generated using AI

Score an always-on agent pilot on what it finds and what it finishes as two separate numbers: Microsoft researchers found those are different skills, and the misses cluster in helpdesk, CRM, finance and HR systems.

That is the practical reading of AutoBuddy, a paper submitted to arXiv on November 9 by ten authors listed under Microsoft. It evaluates agents that have to find their own work across a simulated company's mail, chat, calendar, files and business systems, and then do it. Its abstract says the strongest models "fail mainly during execution, most often on business-system operations." Across 17 models the best end-to-end score was 0.698 and the worst 0.234. Where the credit went missing tells you more than the ranking does, and it tells you what to measure before one of these agents writes to a system of record.

Vendors are selling this now. Microsoft's Ignite announcement describes Autopilot as an agent that is "always on, and working for you" and acts "without waiting for a prompt." Databricks says Genie's coming autonomous mode will run "in the background using its own compute" and email users when they are away (our coverage of the Genie launch on November 12). AutoBuddy gives you published evidence to test those pitches against.


What Did the AutoBuddy Study Actually Test?

AutoBuddy tests whether an agent can work out what a person owes their colleagues and then finish it, graded on the state of the workspace afterwards. A proactive (or always-on) agent is one that reads your mail, chat and calendar on its own schedule and decides what to do without being handed a task. Salesforce's architecture guide defines the ambient version as agents that "operate continuously in the background of a user's workflow without requiring explicit commands." That is a harder job than answering a prompt.

The paper builds one simulated organisation covering six months, March 15 to September 15, 2026. It contains 120 ground-truth tasks spread across 94 actor-days and 44 employees, in four families:

Task family Tasks Systems involved
Business systems 49 (41%) Zendesk, HubSpot, QuickBooks, BambooHR
Collaboration 54 Mail, chat, calendar, files
Research 9 Mock Weights & Biases, Kubernetes, issue tracker
Headcount review 8 Files and mail

The business-system tasks split into 26 customer support, 15 sales, 5 finance and 3 HR. Those tools come from AutomationBench, an earlier benchmark by two Zapier researchers of multi-app business workflows whose authors reported that "Even the best frontier models currently score below 10%." Each of AutoBuddy's 17 models ran three times on the same 120 tasks inside one harness, the GitHub Copilot SDK, which went generally available on June 2 and is being renamed the Microsoft Copilot SDK.

The grading is the part worth copying. Discovery only counts as a match when the agent cites the record that holds the original request. Execution is graded by diffing the workspace before and after the agent's session. The authors write that "No language model is used in matching or in grading." An agent that says it updated the ticket gets no credit unless the ticket changed.

Why Do Finding Work and Finishing It Score So Differently?

They score differently because they are different capabilities, and the study measured them separately. Across the 17 models, discovery precision correlated with execution at r = 0.27, and recall at r = 0.65. The authors call those coefficients "noisy with so few models", which is fair for 17 data points. The rank swaps are harder to argue with.

Model Discovery F1 Execution End-to-end
claude-opus-5.5 0.925 0.739 0.698
gpt-5.6-sol 0.958 0.676 0.654
grok-4.7 0.736 0.676 0.635
DeepSeek-V4-Pro 0.758 0.697 0.592
gpt-6.1-sol 0.944 0.548 0.541
gpt-6-luna 0.860 0.392 0.325
claude-haiku-4.5 0.391 0.471 0.234

Source: AutoBuddy, Table 1, mean of three runs.

Grok-4.7 and DeepSeek-V4-Pro ranked 14th and 13th on discovery F1 but 6th and 3rd on execution. Gpt-6.1-sol had the highest recall in the study, 0.989, and still landed mid-table because its execution score was 0.548. The authors put it plainly: "Recognising what work is owed and completing it are distinct capabilities."

So when a vendor demo shows the agent producing a tidy list of things you owe people, you have seen the discovery half. The list says little about whether the agent will move the HubSpot deal stage or send the QuickBooks invoice correctly.


Where Do Always-On Agents Lose the Most Credit?

They lose it in business systems. For the five strongest models, business-system tasks were about 40% of the work but 48% to 56% of the credit those models lost. Per-task scores on business systems ran 0.45 to 0.56, against 0.67 to 0.75 on collaboration tasks for the same five models.

The paper's breakdown of the 4,943 discovered tasks that went to execution, pooled across every run, shows how the failures happen:

  • 49.8% earned full credit.
  • 17.9% ended with the agent stopping to ask or reporting itself blocked.
  • 15.0% hit the right target with wrong or incomplete content.
  • 10.2% acted on the wrong record, audience or channel.
  • 5.3% were claimed as done with no gradable change in the workspace.
  • 1.9% got the answerability or format wrong.

An enterprise-apps owner should worry most about two of those. A wrong-record write in a CRM or helpdesk is an error someone has to go and find, and nothing about it looks broken. A claimed-but-not-done task is worse, because the person who trusted the agent stops chasing it. Inside business systems, stopping to ask accounted for 28.9% of executions, and the authors note that two same-day requests often concern the same account, which the agents read as conflicting rules.

The spread between models is wide. Stopping to ask made up 55.1% of gpt-6-luna's executions and 41.0% of gpt-6.1-sol's, against 7.3% for claude-opus-5.5. Gpt-5.4-mini claimed completion without acting on 27.5% of its executions. Our Visa agentic commerce benchmark coverage found a similar gap between an agent reporting success and the transaction being right.

Is Stopping to Ask Actually a Failure?

Sometimes it is the right call, and the benchmark's scoring does not distinguish the two cases. In AutoBuddy every write was pre-authorised, so a stop counted against the agent. In your environment, an agent that pauses before issuing a customer credit on an ambiguous instruction may be doing exactly what your controls team wants.

Read that way, the benchmark is generous to the agents. It removed the approval step a real deployment adds, and business-system work still lost the most credit. Put an approval gate in front of those writes and you should expect more stops. That gate is still the right default: OWASP's guidance on excessive agency says to "require a human to approve high-impact actions before they are taken" and to limit an agent's permissions in other systems "to the minimum necessary." AT&T runs nearly 500 agentic workflows with human review kept in place for the same reason.

A pilot therefore has to sort each stop into one of two outcomes: a justified escalation (the instruction really was ambiguous, or the action needed sign-off) or a stall (the agent had what it needed and gave up). A single success rate lumps them together and hides whether you have a cautious agent or a broken one.


Why Do Agents Reopen Work That Is Already Done?

Agents reopen answered requests because they read the original ask and miss the later reply that closed it. Of 1,663 false positives across all 51 runs, 60% "re-opened a request that the workspace shows was already answered on an earlier day." Another 28% picked up a request addressed to a colleague. Of 2,304 already-closed items the agents raised, 43% were treated as open.

Some models were far worse than others: grok-4.7 reopened 88% of the closed items it raised, and DeepSeek-V4-Pro 99%. At company scale that is a noise problem before it is an accuracy problem. An always-on agent that re-sends an answered email or re-files a resolved ticket teaches people to ignore it.

Part of the fix is product design and part is measurement. Ask the vendor how the agent decides a thread is closed, and count reopened items as their own metric in the pilot.

What Should an Always-On Agent Pilot Scorecard Measure?

It should measure discovery and execution separately, per system, against each system's own record of what changed. AutoBuddy's grading method translates into six lines:

  1. Discovery recall: of the requests a human reviewer confirms the person owed, how many did the agent surface?
  2. Discovery precision, with reopened closed items and colleagues' requests counted separately.
  3. Full-completion rate per system, with helpdesk, CRM, finance and HR scored apart from mail and documents.
  4. Wrong-target rate: writes to the wrong record, recipient or channel, taken from the system's audit log.
  5. Claimed-without-change rate: tasks the agent reported done where the audit log shows nothing happened.
  6. Stops, split into justified escalations and stalls, judged by the person who owned the request.

Grade every line on world state. If the pilot measures success by asking the agent, or by asking another model to read the transcript, you are measuring the agent's self-report, and the 5.3% claimed-but-not-done figure says that self-report cannot be taken on trust.

You do not need production data to run it. HubSpot lets developers create "up to 10 test accounts" to test integrations "without affecting any real HubSpot data," and Zendesk says "Support Enterprise includes one sandbox." Run it one system at a time. On the study's numbers, an agent can be ready to draft replies in mail well before it is ready to change a deal stage in HubSpot or send an invoice from QuickBooks. Our piece on setting ChatGPT for Sales write access walks through the CRM permission side.


What to Do Before You Turn One On

This Week:

  1. List every business system your always-on agent pilot (Copilot, Agentforce, Claude, ServiceNow or a custom build) can write to today, and the scopes behind each connection.
  2. Ask the vendor for discovery and execution numbers as separate figures. If it has only one success rate, that tells you how it measures.
  3. Pull 20 to 30 real requests from last month that a person owed and closed, and keep them as your ground-truth set.

This Month:

  1. Run discovery read-only first. AutoBuddy runs discovery in read-only sessions and hands write tools only to the execution step; copy that split.
  2. Turn on writes for collaboration tools (drafts, documents, calendar) before any helpdesk, CRM, finance or HR system, and point the first business-system writes at a HubSpot test account or Zendesk sandbox.
  3. Score the six scorecard lines from audit logs, not from the agent's summary.

Before You Scale:

  1. Keep approval on business-system writes until the per-system wrong-target and claimed-without-change rates are near zero on your own ground-truth set.
  2. Re-run the scorecard every time the vendor swaps the underlying model. Execution ranged from 0.739 to 0.392 across the models in the table above, and you usually do not choose which one you get.

The Bigger Picture

AutoBuddy has real limits. It is one simulated company, 120 tasks, one harness and three runs per model, and it reports no cost, token or latency figures. A vendor will point that out, fairly. The method is still stricter than what most pilots use, which is a demo plus a satisfaction survey.

Robotic process automation programmes hit a version of this: the happy path demoed well and the exceptions were where the real work turned out to be. Always-on agents are heading into the same systems of record, with more freedom to pick their own work. Start with last month's closed requests and score the agent against them before it writes to anything a customer can see.

Continue Reading

Share:

Frequently Asked Questions

What is the AutoBuddy benchmark?

AutoBuddy is a Microsoft research benchmark, submitted to arXiv on November 9, 2026, that tests proactive agents in a simulated six-month workplace. Agents must find 120 owed tasks across mail, chat, calendar, files and business systems such as Zendesk, HubSpot, QuickBooks and BambooHR, then complete them. Grading diffs the workspace state and uses no LLM judge.

Where do always-on AI agents fail most often?

In business systems. For the five strongest models, business-system tasks were about 40% of the work but 48% to 56% of lost credit. Common failures were stopping to ask, writing to the wrong record, audience or channel (10.2% of executions) and claiming a task was done with no change made (5.3%).

Does a good task list mean an agent will complete the work?

No. AutoBuddy found discovery precision correlated with execution at only r = 0.27 across 17 models. Gpt-6.1-sol had the highest recall in the study, 0.989, but an execution score of 0.548. Measure finding the work and finishing it as separate metrics.

How should I score a proactive AI agent pilot?

Score discovery recall and precision, full-completion rate per system, wrong-target rate, claimed-without-change rate, and stops split into justified escalations and stalls. Grade each from system audit logs rather than the agent's own summary, and test business-system writes in a sandbox such as a HubSpot test account or Zendesk sandbox first.

Newsletter

More on Agentic AI, every Sunday

The week's enterprise AI news, Agentic AI included, in one Sunday email.

Weekly, free. Unsubscribe any time. Privacy

Latest Articles

View All →