There is a question I’ve heard in different forms throughout my career:
“How many bugs did testing find?”
It sounds like a reasonable question. After all, we hire testers to find defects. We build automation frameworks to catch regressions. We execute test cases, record failures, and report results.
But the longer I’ve worked in software quality engineering, the less satisfied I’ve become with this way of measuring testing.
Imagine two teams preparing to release a new application.
Team A finds 150 defects during testing.
Team B finds only 12.
Which team did a better job?
The answer depends on what those numbers hide.
Perhaps Team A found 150 minor visual inconsistencies, while Team B uncovered a single defect that prevented customers from completing their purchases. Perhaps Team A tested thousands of low-risk scenarios, while Team B investigated the one integration that could bring the entire application down.
Or perhaps Team B simply didn’t test enough.
The number of bugs found tells us what testing discovered. It doesn’t tell us what testing failed to discover, what risks remain, or whether the product is ready for its users.
That distinction changed how I think about testing.
I no longer see its primary purpose as finding defects. I see it as helping engineering teams understand what they are about to release, what could go wrong, and how much confidence they should place in the outcome.
And that is a much bigger responsibility.
1. The bug is a symptom. The risk is the real story.
Let’s take a familiar example: an e-commerce application.
A customer browses a product, adds it to the cart, enters a delivery address, chooses a shipping option, and proceeds to checkout.
A traditional test case might look like this:
Verify that the product can be added to the cart.
Verify that the cart displays the correct price.
Verify that the delivery address can be entered.
Verify that a shipping option can be selected.
Verify that the customer can proceed to checkout.
Every step passes. The test is green.
We could execute this scenario a hundred times across different browsers and proudly report that checkout testing is complete.
But what if the application calculates the delivery fee incorrectly when a customer changes the address after selecting a shipping option?
What if the cart displays the right total, but the checkout page uses a stale value?
What if a customer clicks the checkout button twice because the first click appears unresponsive, creating two orders?
What if the product becomes unavailable between the time it is added to the cart and the time the customer places the order?
These are not merely additional test cases. They represent different ways the application can fail in the real world.
The distinction matters because software does not fail according to our test-case structure. It fails according to the interactions between its components, its data, its users, and its environment.
A test case may pass while the customer experience remains broken.
That is why I try to look beyond the question, “Did this feature work?”
I ask:
What assumptions does this feature make?
What happens when those assumptions are no longer true?
Which failure would cause the greatest harm to the user or the business?
What evidence do we have that the system handles that failure correctly?
This changes the nature of testing. Instead of treating every scenario as equally important, we investigate the conditions under which the product could fail in meaningful ways.
Finding a bug is an outcome of testing. Understanding the risk behind it is the engineering work.
2. A real engineering lesson: the green pipeline that wasn’t enough
I’ve seen a familiar pattern in automation projects.
A team invests considerable effort in building a regression framework. The framework automates login, navigation, form submissions, API validations, and the main business workflows. Tests run in the CI/CD pipeline whenever new code is merged.
Over time, the team reaches a point where most regression runs are green.
That is progress. Repetitive checks become faster, feedback arrives earlier, and engineers can detect many unintended changes before they reach production.
But green pipelines can create a dangerous sense of completeness.
Consider a hypothetical example based on common e-commerce engineering problems.
A team has automated the product catalogue and checkout workflow. Every night, the suite runs against a test environment and reports a 98% pass rate.
One morning, a developer changes an API response to improve performance. The response still contains all the expected fields, and the automated API tests pass.
The frontend also loads correctly.
However, the change alters how product availability is represented when inventory is temporarily unavailable. The frontend interprets the new response incorrectly and continues displaying an unavailable product as purchasable.
The existing tests don’t catch it because they validate the normal response, not the transition between available and unavailable states.
The pipeline is green. The application is not behaving correctly.
What went wrong?
Not necessarily the automation framework. The framework did exactly what it had been designed to do.
The gap was in the assumptions behind the tests.
The team had automated the expected behaviour but had not adequately tested the system’s behaviour when a dependency returned a different, still-valid response.
The useful response isn’t simply to add another test and move on. It is to ask why the original coverage missed the risk.
I would investigate several things:
API contract: Did the tests define the meaning of the availability field, or only check that it existed?
State transitions: Did we test a product moving from available to unavailable?
Integration behaviour: Did we verify how the frontend interpreted the API response?
Failure visibility: Would the test suite detect this problem before release?
Prevention: Could contract tests or stronger response validation prevent similar defects from recurring?
The additional test matters. But the improved understanding matters more.
Once the team identifies the underlying assumption, it can strengthen the contract, update the test strategy, and potentially prevent an entire category of failures.
That is a better outcome than simply adding one more line to the defect tracker.
Good testing doesn’t just catch yesterday’s bug. It improves our ability to catch tomorrow’s.
3. Automation can execute checks. It cannot automatically define quality.
I’m a strong believer in test automation. It is one of the most effective ways to increase repeatability, shorten feedback loops, and protect existing functionality as software evolves.
But I’ve also learned that automating a test and knowing what deserves to be tested are two different engineering problems.
Suppose we have an automated test that verifies whether a user can reset a password.
The test checks that:
The user can request a reset link.
The reset link opens the correct page.
The user can enter a new password.
A success message appears.
The user can log in using the new password.
The test passes.
That is useful evidence.
But it doesn’t necessarily tell us whether the reset link expires at the correct time, whether it can be reused, whether error messages reveal sensitive information, or whether the flow behaves properly when the email service is delayed.
It doesn’t tell us whether the experience is accessible to someone navigating with a keyboard or screen reader.
And it certainly doesn’t tell us whether the requirements themselves adequately describe the expected security behaviour.
These questions require different forms of investigation.
Some are well suited to automation. Others need exploratory testing, security expertise, accessibility evaluation, or a deeper understanding of the system’s design.
The goal isn’t to force every testing activity into an automated script.
The goal is to choose the right technique for the risk.
I think of automation as a way to scale the checks we already understand. It also frees us to spend more time investigating questions that are difficult to reduce to predictable steps.
When we automate without questioning our assumptions, we can produce a very efficient system for confirming that the wrong things still work.
That is not a framework problem. It is a quality strategy problem.
4. Testing is about reducing uncertainty
Every software release contains some uncertainty.
We may not know exactly how users will behave. We may not have tested every possible combination of data. Third-party services may respond differently under load. A seemingly harmless code change may affect a component that nobody expected it to touch.
We cannot eliminate all uncertainty through testing.
What we can do is reduce the uncertainty that matters most.
Imagine we are preparing to release a new search feature for an online store.
We could test whether a search for a product name returns the expected results. That establishes a useful baseline.
Then we could investigate questions that reveal more about the system:
What happens when the search query is empty?
What happens when the query contains special characters?
How does the system behave when there are thousands of matching products?
What happens if the search service times out?
Does pagination remain correct when products are added while the user is browsing?
Does the feature remain responsive under a realistic traffic spike?
Each question explores a different source of uncertainty.
We might discover a defect. We might discover that the system handles the condition correctly. Or we might discover that the team has no clear requirement for how the application should behave.
All three outcomes provide useful information.
The third is particularly important. Testing sometimes exposes gaps in product decisions long before it exposes defects in code.
For example, if a search service becomes unavailable, should the application display an error, show cached results, or offer a retry? A tester cannot reliably validate the behaviour until the team agrees on what the behaviour should be.
That conversation is part of quality engineering.
Testing produces evidence about the system. Engineering teams use that evidence to make better decisions.
The value is not limited to the number of failures uncovered. It also lies in the assumptions challenged, the risks clarified, and the decisions made before release.
5. The hardest bugs are often hidden in the gaps between components
As systems become more distributed, I find it increasingly useful to think about the boundaries between components.
A frontend can work correctly in isolation. An API can return valid responses. A database can store the expected values. A background job can execute successfully.
Yet the complete user journey can still fail.
Consider an online store that sends order confirmations through a background processing service.
The checkout API creates an order, stores the order details, and publishes a message for the notification service. The notification service then sends an email.
Each component may pass its individual tests.
But what happens if the message is delivered twice?
What happens if the database transaction succeeds but publishing the message fails?
What happens if the email service is temporarily unavailable?
What happens if a retry sends the same confirmation twice?
These are system-level questions. They cannot always be answered by checking each component independently.
A test strategy focused only on individual features may miss the interactions that determine whether the entire workflow is reliable.
I would want to test the boundaries deliberately:
Validate that the API and its consumers agree on the message contract.
Verify that retries don’t create unintended side effects.
Check how the system recovers from partial failures.
Confirm that duplicate messages are handled safely where required.
Observe whether failed processing is visible through logs, metrics, and alerts.
This is where testing starts to overlap with architecture, observability, resilience, and operational readiness.
The tester is no longer simply asking whether a button works. They are investigating whether the system can maintain its intended behaviour when individual parts behave imperfectly.
And in modern software, imperfect behaviour is not an exceptional event. It is an engineering condition we need to design for.
6. A bug-free release is not the same as a quality release
Imagine a team completes a release candidate and reports zero open defects.
That sounds encouraging.
But I would still want to know a few things before interpreting the result:
How much of the critical user journey has been exercised?
Which high-risk scenarios remain untested?
Were the tests run against an environment representative of production?
Did the team validate performance, accessibility, security, and compatibility where relevant?
Were any tests skipped, unstable, or blocked by missing dependencies?
What changed in the release, and what parts of the system could those changes affect?
A release with zero known defects can be a good release. It can also be a release with insufficient evidence.
The difference is not visible in the defect count.
This is why I don’t think quality can be represented by a single testing metric.
Pass rate, defect count, automation coverage, escaped defects, and execution time can all tell us something useful. None of them tells the entire story.
For example, 95% automation coverage sounds impressive until we discover that the untested 5% contains the most critical user journeys.
Likewise, a test suite with a 99% pass rate may be less useful than one with a lower pass rate if the latter exposes meaningful failures while the former is dominated by repetitive, low-value checks.
Metrics need context. Otherwise, teams can become very good at improving the numbers without improving the product.
I would rather have a clear explanation of the remaining risks than a dashboard full of green indicators that nobody can interpret.
A useful release assessment should communicate not just whether the tests passed, but what the team learned, what remains uncertain, and why the available evidence supports the release decision.
That is the difference between reporting test execution and communicating quality.
7. The tester’s role is evolving from defect detection to decision support
For a long time, testing was often treated as a phase that followed development.
Developers wrote the code. Testers executed the test cases. Defects were reported, fixed, and retested. The product moved toward release.
That workflow still exists, but it is not sufficient for every modern engineering environment.
Quality concerns begin much earlier.
When requirements are being discussed, someone needs to challenge ambiguous acceptance criteria.
When architecture is being designed, someone needs to consider failure modes and integration risks.
When code is being reviewed, someone needs to think about edge cases and unintended consequences.
When a pipeline is running, someone needs to distinguish a genuine product failure from an unreliable test.
And when a release is approaching, someone needs to communicate what the team knows and what it does not know.
This is why I see quality engineering as a discipline that extends beyond the execution of tests.
It combines technical investigation, risk analysis, automation, system understanding, and communication.
It also changes the questions I believe a strong QE engineer should ask.
Instead of only asking, “Which test cases should I execute?” I want to ask:
What is the most important thing that could go wrong?
How would we know if it happened?
What evidence would convince us that the system behaves correctly?
Which checks should run automatically, and which require human investigation?
How can we prevent this category of failure from returning?
These questions make testing more connected to engineering decisions.
They also make it easier for quality engineers to contribute earlier, collaborate more effectively with developers, and help teams build reliable software rather than merely inspect it before release.
8. What changes when AI enters the testing workflow?
AI is making it easier to generate test ideas, draft automation code, analyse logs, summarise failures, and explore unfamiliar codebases.
These are valuable capabilities.
Imagine giving an AI assistant an API specification and asking it to suggest negative test cases. It might identify missing fields, invalid values, boundary conditions, and unexpected response codes in seconds.
Or imagine feeding it a failed CI pipeline, the relevant logs, and a code change. It might help narrow down the likely cause and suggest a debugging path.
That can save engineering time.
But there is a distinction I don’t want us to lose: generating a test is not the same as knowing that the test is valuable.
An AI assistant may generate dozens of plausible test cases that all exercise essentially the same behaviour. It may misunderstand a requirement. It may write an assertion that passes while the application violates an important business rule.
It may also produce automation code that runs successfully but checks the wrong outcome.
The same principle applies to AI-generated bug reports. A convincing explanation is not proof that the suspected defect exists.
The engineer still needs to validate the expected behaviour, examine the evidence, and understand the consequences of a failure.
In fact, as generating code and tests becomes cheaper, deciding what to test and evaluating whether the results are meaningful may become even more important.
My preferred approach is to use AI to accelerate investigation, not outsource judgment.
Let AI propose the scenarios. Let it help analyse the failures. Let it draft the repetitive parts of the framework.
Then validate the important assumptions, execute the checks, inspect the evidence, and decide what the results actually mean.
The objective is not to produce more tests simply because we can.
It is to produce better evidence with less wasted effort.
9. How I would measure testing differently
If I were evaluating a team’s testing effectiveness, I wouldn’t discard traditional metrics. I would put them into a broader picture.
What I would examineWhat it helps us understandCritical user-journey coverageWhether important workflows have meaningful protectionRisk coverageWhether testing addresses the failures with the greatest potential impactEscaped defectsWhich important failures reached users and what we can learn from themRegression feedback timeHow quickly engineers discover unintended changesTest reliabilityWhether teams can trust the results produced by the suiteRecovery and resilienceWhether the system behaves acceptably when dependencies failTime to diagnose failuresHow quickly the team can move from a failure signal to a useful explanationUnresolved release risksWhat the team still needs to understand before making a release decision
No metric is perfect, and none should become a target in isolation.
The point is to connect measurement to outcomes.
If automation coverage increases but release confidence does not, we need to understand why.
If escaped defects decrease, we should investigate which engineering practices contributed to the improvement.
If test execution becomes faster, we should ask whether the team is getting earlier feedback without sacrificing meaningful coverage.
The most useful metrics encourage better questions. They do not replace the need to ask them.
10. The question I want every tester to ask
After years in software testing and automation, I still care about finding bugs. Of course I do.
A defect discovered before release can save users frustration, engineers rework, and businesses unnecessary cost.
But I no longer believe that defect discovery, by itself, captures the purpose of our work.
A tester who finds ten defects without understanding their impact may provide less value than a tester who discovers a single critical weakness, explains how it could affect users, and helps the team prevent it.
A robust automation framework is valuable. A framework that checks the wrong assumptions is not.
A green pipeline is encouraging. A green pipeline backed by meaningful evidence is far more useful.
And a release with no known defects is only as reassuring as the testing and investigation behind it.
The real objective is not to prove that software has no bugs. For any sufficiently complex system, that is an unrealistic promise.
The objective is to understand the system well enough to make informed decisions about its behaviour, limitations, and risks.
That means knowing what we tested, why we tested it, what the results tell us, and what remains unknown.
Software testing is not about finding every bug. It is about finding the failures that matter, exposing the risks that are easy to overlook, and helping teams release software with justified confidence.
That is how I want to approach quality engineering.
Not as the final checkpoint where we count defects, but as an engineering discipline that helps us build better systems from the beginning.
And perhaps the most useful question after a test run isn’t:
“How many bugs did we find?”
It is:
“What do we understand about the product now that we didn’t understand before?”
That is where testing starts to create its greatest value.
A question for the community
When your team finishes a regression run, what carries more weight in the release decision: the number of tests that passed, the risks that were investigated, or the evidence that the most important user journeys work as expected?
I’d be interested to hear how other QE engineers, SDETs, developers, and engineering leaders approach this.
If you work in quality engineering, I’d love to hear your perspective in the comments.




