QA automation has always had a trust problem. Teams build test suites, run them in CI, and then discover — in production — that the tests weren’t actually testing what mattered. The suite was green. The product was broken.
AI tools are making this problem easier to create at scale and harder to detect.
The test confidence illusion
Coverage percentage is not a proxy for confidence. I’ve seen codebases with 80% coverage and serious undetected defects, and codebases with 40% coverage and a strong safety record. The difference is almost never the number.
The difference is whether the tests are testing the right things, from the right perspective, in conditions that resemble production.
AI-assisted test generation amplifies this gap. It’s very good at producing tests that pass. It’s not reliably good at producing tests that would fail if something important broke.
The reason is that AI-generated tests are largely learned from the implementation. They test that the code does what it does — not that it does what it should.
What earns actual confidence
Tests written from the outside in. The most valuable tests are written against the contract — the expected behavior from a user’s or caller’s perspective — not against the implementation. These tests survive refactoring and catch regressions that implementation-derived tests miss.
Tests that exercise realistic failure modes. Happy-path coverage is easy to generate and provides less value than it appears. The tests that find real bugs are the ones that test what happens when dependencies fail, inputs are malformed, or state is unexpected.
Tests that are maintained. A test that’s been disabled, skipped, or left broken because “it’s flaky” isn’t a test. It’s technical debt with a testing label. The maintenance discipline around a test suite matters as much as writing tests in the first place.
Tests connected to real incident history. After every incident, the question should be: “What test, if it existed, would have caught this?” The tests that come out of that process are more valuable than almost anything generated by coverage tooling.
Where AI helps in QA
Generating test scaffolding. AI is good at producing the boilerplate structure for a new test module — the arrange/act/assert skeleton, the mock setup, the file organization. This removes friction from writing the first test.
Identifying coverage gaps. Given a function or module, AI tools can surface paths that don’t have test coverage and suggest inputs that would exercise them. This is genuinely useful for finding the cases a developer didn’t think to test.
Parameterizing tests. Expanding a single test into a table-driven test with a broad set of inputs is tedious work that AI handles well.
Mutation testing assistance. AI can suggest small code mutations that should cause tests to fail — a fast way to verify that tests are actually detecting the behaviors they’re supposed to.
Where AI doesn’t help — and why
Deciding what matters. Which user journeys are critical? Which failure modes are most costly? Which edge cases have actually appeared in production? This is domain knowledge and organizational context that no AI tool has access to.
Writing integration and end-to-end tests. These tests require deep knowledge of the system’s architecture, its dependencies, and its production behavior. AI-generated E2E tests are typically superficial and brittle.
Maintaining test quality over time. Generating tests is easy. Keeping them meaningful as the system evolves requires human judgment about what the tests are supposed to be protecting.
A practical framework
When I’m evaluating QA automation in a codebase, I ask five questions:
- What has failed in production in the last 12 months? Are those scenarios covered?
- What would need to be true for a developer to feel safe merging a significant change? Are those conditions in the test suite?
- What percentage of tests fail on a typical CI run? High flakiness rates mean the suite is providing false confidence.
- When did someone last intentionally break the code to verify a test would catch it? If the answer is “never,” the suite’s reliability is unknown.
- What would have to break for the test suite to miss it entirely? Understanding the blind spots is as important as understanding the coverage.
The goal
QA automation should make engineers confident that they can change things — not anxious that they might break something, and not falsely reassured that everything is fine.
That confidence isn’t a metric. It’s a feeling that comes from tests that have earned their credibility by catching real problems, being maintained carefully, and testing the right things for the right reasons.
AI tools can help build that suite faster. They can’t substitute for the discipline that makes it trustworthy.