Before You Build Another AI Agent, Ask This First
Not every AI problem needs an agent. Leaders need to know whether the work is deterministic or probabilistic before chasing ROI.
📌THE POINT IS: A lot of companies are trying to turn every AI opportunity into an agent opportunity, and that is where the strategy starts to fall apart.
Some business problems are deterministic: they need the same answer, the same process, and the same result every time. Other problems are probabilistic: they require research, judgment, ambiguity, and recommendations. The ROI comes from knowing the difference and using the right kind of AI for the job.
A couple of weeks ago I had an epiphany: many AI strategies I’ve heard lately are basically the same as pounding a square peg into a round hole.
It’s not anybody’s fault. For the past few years, we’ve heard experts say we should use AI to automate repeatable tasks. We’ve heard that agents will come online quickly and cheaply to do what humans do, only faster. We’ve heard that businesses should be looking for every routine process they can hand over to AI.
The problem is that people have not always thought deeply enough about the amount of work it takes to force an agent to do truly repeatable tasks correctly, accurately, and consistently every time.
I use an agent at work to build a morning report of email and meeting activity, and I’ll be darned if that report is not a little different every day despite how detailed the skills and instructions are. That is fine for my email management activity. It will not hold water for regulated companies that need transactions handled the same way every time.
What is adding fuel to this fire is our new friend: tokenomics.
Tokenomics is the analysis of how much money your agent will save compared with the token costs required to run it. Companies are learning that they can build agents to do a lot of repeatable work, but at scale, the cost to run those agents can outweigh the human or traditional software alternatives they are used to.
That is not just a theory. A July 2026 KPMG report, covered by TechRadar, found that only 35% of organizations have full visibility into AI operating costs and that nearly half have delayed, paused, or shrunk AI strategies over cost concerns. Rob Fisher, KPMG’s Global Head of Advisory, put it plainly:
“AI is now as much a financial management priority as it is a technology one.”
The epiphany I had is that we, as leaders, need to start looking at our business problems through the lens of deterministic or probabilistic.
If a problem is truly repeatable and needs to be done the same way every time according to structured rules, then it is a deterministic problem. You should use AI to help build the tool, workflow, or system that can be customized to your business and scaled across your enterprise. The ROI comes from building that system faster with AI coding tools, implementing features more quickly as you scale, and still getting the human efficiency and redistribution benefits you were hoping for.
This is where AI helps you create the machine. It does not need to be the machine forever.
Meanwhile, when you find a probabilistic problem that requires research, judgment, decision-making, synthesis, and response generation, go with an agent. You can create the bones of that agent quickly with skills files and instructions. You can work with it side by side with humans. And as you roll it out to pilot teammates who reinforce what is right or wrong, you strengthen its context, memory, and usefulness over time.
Anthropic makes a similar distinction in its December 2024 engineering article, “Building effective agents”. They describe workflows as systems where tools follow predefined paths, while agents dynamically direct their own process. Their advice is a useful gut check for leaders:
“Workflows offer predictability and consistency for well-defined tasks, whereas agents are better when flexibility and model-driven decision-making are needed at scale.”
That is the square peg and round hole problem in one sentence.
Let’s look at customer service for a great example.
Everyone has called into a customer service center. We call these Level 1 centers. The person on the other end usually has limited scripts, limited permissions, and access to the most common actions. Most of the work these people do is deterministic, with a light touch of traditional classification.
They ask mostly canned questions. They look up what to do in that circumstance. They try the one or two options they have to complete the transaction. If they cannot help you, they gather your information, open a ticket, and move it up to Level 2.
You should not automatically build an AI agent to do this work.
You can save a lot of time by using AI development tools and old-fashioned data science to build a scalable system that does these things and accelerates response times. You may not eliminate all of the people on the Level 1 team, but you can triage many of the items they would normally work on and free human operators to focus on items that do not follow the happy path.
At Level 2, the job is inherently more probabilistic.
The human agent has to research what happened, look for similar issues, determine how those issues were handled in the past, reference materials about related problems, come up with a recommendation, and then try solutions until one hopefully works.
This is what a probabilistic problem looks like. It involves research, analysis, judgment, and decision-making.
An AI agent that does most of those steps overnight and is ready with recommended actions for each ticket in the queue can make the Level 2 agent capable of handling many more cases than before. As the Level 2 agent confirms the AI agent’s recommendations, they train the system and help build confidence scores. In time, business leaders may decide that recommendations above a certain confidence score can be handled automatically, giving human agents more time for the novel and ambiguous problems that need real attention.
This is also why the ROI conversation has to get more precise.
A 2026 arXiv paper by Longju Bai, Zhemin Huang, Xingyao Wang, Jiao Sun, Rada Mihalcea, Erik Brynjolfsson, Alex Pentland, and Jiaxin Pei, “How Do AI Agents Spend Your Money?”, found that agentic coding tasks can consume dramatically more tokens than simpler code reasoning or chat tasks. The authors call agentic tasks “uniquely expensive,” and their findings reinforce the bigger point: leaders need to understand cost behavior before scaling agentic work.
You have heard me mention “repeatable activities” a few times.
Repeatable tasks are like the penny machine in a souvenir shop. You put the penny in, turn the crank, and out pops a smashed penny with an image on it. You insert another penny, crank again, and get another souvenir.
That is repeatable.
Tasks related to coding, bug fixes, incident response, and anything else that sounds like the Level 2 example above are not repeatable in that same way. They are also not routine just because they happen often. They require thought, analysis, action, testing, additional resolution, and validation before they go into production.
Every bug is a new mini software project. It may not take long, but it still involves ambiguity.
Make sure you are using AI coding tools to accelerate this work. But I would be careful about fully outsourcing these kinds of tasks to AI agents yet.
There is real promise here, but only if leaders stop treating AI like one big hammer. The MIT NANDA report on enterprise AI adoption, covered by Tom’s Hardware in August 2025, found that only about 5% of enterprise generative AI pilots achieved rapid revenue acceleration. The report’s lead author, Aditya Challapally, explained the successful group this way:
“They pick one pain point, execute well, and partner smartly.”
The question shouldn’t be, “Where can we use agents?”
The better question is: “What kind of problem are we solving?”
Five things AI leaders should do today
Classify your work before choosing the tool. Separate deterministic problems from probabilistic ones before you talk about agents, copilots, automation, or vendors.
Use AI to build deterministic systems faster. If the process needs the same result every time, use AI coding tools, data science, and workflow automation to build the system instead of running every transaction through an agent.
Reserve agents for ambiguity. Agents are best when the work requires research, judgment, synthesis, recommendations, and adaptation.
Measure tokenomics before scaling. Do not wait until the bill arrives. Model expected usage, compare it with human and software alternatives, and build cost visibility into the strategy from the beginning.
Keep humans in the learning loop. The best agent deployments will not be “set it and forget it.” They will be side-by-side systems where humans validate, correct, reinforce, and gradually expand the agent’s authority.
The leaders who win with AI will not be the ones who put an agent on everything.
They will be the ones who understand the work deeply enough to know when they need a deterministic system, when they need a probabilistic partner, and when they are just trying to pound a square peg into a round hole.
That is the kind of AI strategy I am most interested in: practical, measurable, and tied to the way work actually happens. If you are thinking through these questions inside your own organization, stick around. I’ll keep writing about the gap between AI hype and AI that actually earns its keep.
Like and Share this newsletter to spread the word about practical data and AI strategies for executive leaders!