Self-Improving Agents Still Need Human Feedback Loops
2026-06-01 · 4 min read · Janaina Maia
The phrase “self-improving agent” sounds like the human is disappearing from the system. I think the opposite is true. The more an AI product claims it can improve itself, the more carefully we need to design the human feedback loop around it.
OpenAI published a case study with Thrive Holdings about Tax AI, an agent used by accounting firms to prepare complex tax returns. The system processed 7,000 returns during a pilot, saved practitioners about a third of their preparation time, and improved over the season as real production feedback was turned into structured evaluations and product changes.
That is the part worth paying attention to. The story is not simply “the model got smarter.” The product got better because domain experts were close to the workflow, errors were captured, outcomes were measured, and the team built a loop that could turn messy real-world use into better behaviour.
Self-improvement is not magic.
In enterprise AI, “self-improving” can be a dangerous phrase if it makes teams imagine an agent quietly getting better by itself. Real improvement needs instrumentation. It needs examples of success and failure. It needs a way to compare versions. It needs practitioners who can say, “this is technically correct but practically wrong,” which is often the most valuable feedback in high-stakes work.
That matters because many AI failures are not obvious crashes. They are plausible outputs with a missing assumption, a bad source, an edge case, or a workflow detail the model did not understand. If the product cannot capture those moments, the organisation is relying on vibes instead of learning.
Evals are a product design concern.
An evaluation, or eval, is a repeatable way to test whether an AI system is doing the job well enough. A simple analogy: if the agent is a new team member, evals are the structured practice scenarios and quality checks you use before trusting them with more responsibility.
Design leaders should care about evals because they shape what the product treats as quality. If the eval only measures completion, the agent may optimise for getting something done. If it also measures correctness, uncertainty, review burden, policy fit, and user correction patterns, the product starts learning the things that actually matter in the workflow.
The review surface is where learning happens.
The strongest design implication is that review cannot be an afterthought. When a human checks an agent’s work, the product should make it easy to mark what was wrong, what was missing, what was accepted, and what required judgement. That feedback should not disappear into a comment box. It should become structured evidence for improving the system.
This is especially important in domains like finance, law, engineering, healthcare, security, and enterprise operations. In those areas, the expert reviewer is not just approving output. They are teaching the system what good looks like in context.
Design implications.
- Capture corrections as data: make human edits, rejections, and approvals structured enough to improve the product.
- Measure review burden: do not only ask whether the agent completed the task. Ask how much expert effort it took to make the work usable.
- Keep domain experts close: practitioners should shape the agent’s quality criteria, not only react to finished demos.
- Separate confidence from authority: a better score does not automatically mean the agent deserves more autonomy.
- Design for version comparison: teams need to see whether the agent is genuinely improving, not just changing behaviour.
My take.
The next maturity step for agentic products is not agents that say more confident things. It is agents inside systems that learn from real expert review without hiding the judgement, risk, or accountability.
Self-improving agents need humans more than the name suggests. They need humans in the loop, but not as rubber stamps. They need humans as teachers, reviewers, boundary setters, and owners of quality.
If the feedback loop is weak, the agent will not improve. It will only repeat its mistakes with better phrasing.