AI features break the assumption every test pyramid was built on: that the same input produces the same output. Once a large language model sits inside your product, "green build" stops meaning "correct behavior," and teams that don't adjust their pyramid end up in one of two bad places—a wall of flaky end-to-end tests nobody trusts, or no automated coverage at all and a rotation of humans manually checking AI output before every release. Neither scales past a handful of engineers.
The fix isn't abandoning the pyramid. It's re-drawing where the layers sit and adding one new layer that traditional QA doesn't have a name for yet.
Keep the base deterministic
Everything below your AI layer—auth, permissions, billing, data writes, the plumbing that moves information in and out of the model—needs to stay exactly as strict as it was before you added AI to the product. This is the part teams get wrong first: they see "AI feature shipped" and start treating the whole feature as inherently unpredictable, so they loosen assertions everywhere, including in code paths that were never nondeterministic to begin with.
A billing calculation that happens to sit next to an AI summarization call is still a billing calculation. Unit and integration tests here should be as boring, strict, and fast-failing as any traditional backend test suite. If this layer isn't rock solid, you can't tell whether a production incident came from the model or from your own code—and you'll waste a week debugging the wrong system.
Add an evaluation layer, not "more E2E"
This is the layer most teams are missing, and it's not a bigger version of your existing E2E suite. An evaluation layer runs a fixed set of "golden" prompts or scenarios against the model on every build, and checks properties rather than exact strings:
- Does the output contain the required fields?
- Does it validate against a JSON schema?
- Does it correctly refuse an out-of-scope request?
- Is it inside your latency budget?
You store the pass rate over time, the same way you'd track a performance metric, so that when a model version changes, a prompt gets tweaked, or a vendor silently updates their API, you see the drift on a dashboard instead of hearing about it from an angry customer. Treat this evaluation suite as a first-class CI citizen with its own quality gate—not an optional nightly job that quietly goes red and stays red.
Separate product tests from model tests
The single fastest way to end up with a flaky suite nobody trusts is testing your wrapper code and the model's actual output in the same test. They fail for completely different reasons and need completely different remediation.
Your wrapper—the code that handles timeouts, retries, caching, PII redaction, and fallback responses when the model is unavailable—should be tested the traditional way, with deterministic unit tests and mocked model responses. The model's actual reasoning and output quality belongs in the evaluation harness described above, run separately, with its own thresholds and its own tolerance for variance.
When teams collapse these into one suite, a single flaky model response takes down tests that had nothing to do with the model, and engineers start ignoring red builds altogether—which is worse than having no tests, because now you're paying for the suite and getting no signal from it.
Human review is a sampling layer, not a safety net
No evaluation harness catches everything, especially in the first few months after launch when you're still discovering failure modes. The answer isn't manually reviewing every AI output before it ships—that doesn't scale past a demo. It's reviewing a fixed, small sample of production traffic every week, on a schedule, with someone accountable for reading it and feeding anything interesting back into your golden prompt set.
One to five percent of weekly volume is usually enough to catch drift and edge cases before they become a pattern. Founders who skip this layer entirely tend to find out about failure modes the expensive way: from a customer complaint or a bad screenshot on social media, well after the failure has been happening quietly for weeks.
What this looks like assembled
A pyramid that survives AI features has four layers instead of three:
- Deterministic base. Fast, strict unit and integration tests for everything that isn't the model.
- Wrapper suite. Mocked model responses, validating your own timeout, retry, redaction, and fallback logic.
- Evaluation harness. Golden scenarios against the real model on every build, scored and tracked over time.
- Sampled human review. A thin, scheduled read of real production traffic, feeding back into layer three.
None of these layers substitute for the others. The moment teams try to make end-to-end UI tests do the job of the evaluation layer, the suite becomes too flaky to trust and gets switched off—which is how "we have tests for the AI feature" quietly turns into "we have no tests for the AI feature" within two release cycles.
If your team is adding AI to an existing product and isn't sure which of these layers you're missing, that's usually a half-day conversation, not a research project. See how I approach QA architecture engagements.
On the tooling side: the evaluation-layer thinking above is what PixellPeep—the AI-assisted UI testing product I build at Aarohii—exists to automate. Useful context if you'd rather buy that layer than build it.