Your AI Makes Great Predictions. Nobody Trusts Them.
2026-01-30 · 11 min read · Janaina Maia
I once sat in a meeting where a brilliant AI model was presented to a group of senior engineers. Accurate. Well-validated. Technically sound. The data scientist presenting it was thorough.
The engineers rejected it.
Not because it was wrong. Because they couldn't understand why it was right. They couldn't explain its recommendations to their clients. Couldn't defend them in a regulatory review. Couldn't trust them enough to bet their professional reputation on them.
That moment changed how I think about AI products.
Explainability Is a Design Problem, Not a Technical One
Most explainability efforts focus on SHAP values, attention weights, feature importance scores. Useful for data scientists debugging models. Useless for the senior engineer trying to explain to a client why the AI recommended a specific foundation type.
The question isn't "how does the model work?" It's "how do we help different people understand and trust the output?"
Different People Need Different Explanations
Domain Experts (Engineers, Scientists)
They don't care about the model architecture. They want to know: what data drove this? How does it relate to my experience? Where does it agree or disagree with established practice?
Give them data-driven narratives: "Based on borehole data from GH-101 through GH-120, showing consistent clay layers at 3-7m depth with SPT values averaging 12, the model recommends shallow foundation."
Decision Makers (Managers, Clients)
Executive summary. What's the recommendation, how confident are you, what are the risks, what's the impact on cost and timeline?
Regulators and Auditors
Traceability. Methodology. Data lineage. Compliance mapping. The full paper trail.
End Users (Field Workers, Technicians)
Simple and actionable. "Use Method A because the soil is softer than typical for this area." That's it.
Explanation Patterns That Work
The Narrative Explanation
Convert model reasoning into natural language that follows the expert's own thinking process.
Instead of: "Feature importance: Clay content (0.42), SPT N-value (0.31)..."
Show: "The recommendation is primarily driven by the high clay content (62%) found in the upper layers, supported by relatively low bearing capacity indicated by SPT values averaging 12."
Same information. One a data scientist can read. The other a senior engineer can present to a client.
The Counterfactual
Show what would need to change for a different recommendation. This is incredibly powerful.
"For deep foundation to be recommended, clay content would need to be below 30% OR SPT values would need to exceed 25."
Counterfactuals answer the most natural human question: "What would change your mind?"
The Analogous Case
"This recommendation is similar to the approach used for [Project X] which had comparable conditions and was completed in 2023."
Analogies to known projects leverage existing knowledge, provide tangible reference points, and make AI reasoning feel grounded — not abstract.
Progressive Detail
Layer explanations from simple to complex. Let each audience self-select their depth:
- One-liner: "Recommended: shallow foundation (high confidence)"
- Summary: "Based on consistent clay layers across 20 boreholes"
- Full explanation: Narrative with data, methodology, alternatives
- Technical detail: Model parameters, weights, validation
Most people stop at level 2. Auditors go to 4. Everyone gets what they need.
Common Mistakes
- Over-explaining: Drowning users in technical detail because you can. More information ≠ better understanding.
- Wrong audience: Showing SHAP values to project managers.
- Explanation theater: Providing explanations that sound good but don't actually reflect how the model works. Users eventually notice.
The most accurate AI in the world is useless if the people who need to act on it don't trust it. And they won't trust what they can't understand. That's a design problem. Own it.