Council Post: When AI Learns How To Pass The Test, What Are We Actually Testing?

Margarita Simonova is the founder of ILoveMyQA.com.

getty

In QA, we are used to a fairly simple idea: Define what should happen, test it and see whether the product behaves the way we expect. AI makes that relationship less straightforward. We can evaluate an AI system against the same benchmarks, test sets and evaluate scoring criteria again and again. Then we change the model, the prompts or the surrounding logic until the results improve.

At first glance, that sounds exactly like what we want. But there is another possibility: The system may simply be getting better at satisfying the way we measure it. A better score does not always mean a better product.

Passing The Test Is Not The Same As Solving The Problem

We already see versions of this in traditional QA. If a team is measured heavily on automation coverage, automation coverage usually goes up. If success means running more tests, teams get very good at running more tests. None of that guarantees that customers are seeing fewer problems. AI adds another layer because the behavior we measure can become the behavior we optimize.

Take an AI customer support system. Suppose the company measures it mainly by resolution rate and how quickly conversations are closed. The numbers start improving, and from a dashboard perspective, everything looks great. But then you look closer and find that some customers are getting quick answers that do not actually solve their problems. The conversation was technically “resolved,” but the customer still had to contact support again.

The AI performed well against the metric. It did not perform well for the customer. That is the kind of gap QA needs to look for.

What Did We Actually Ask The System To Optimize?

Metrics are useful. The problem starts when we forget they are only a representation of what we care about. An AI coding agent might be evaluated on whether the code it produces passes an existing test suite. Naturally, passing those tests becomes an important target.​ Did it interpret the requirement correctly? Did it introduce a problem outside the test coverage? Does it behave properly when the data is messy or the user does something unexpected?

The same issue appears in other products. A sales agent rewarded for booked meetings may become too aggressive. A recommendation engine rewarded for engagement may keep people clicking without necessarily improving the experience. A support agent rewarded for speed may learn to end conversations too quickly.

In each case, the metric can improve while the product moves in the wrong direction. That is why QA cannot only ask whether the system met the target. We also need to ask whether we picked the right target in the first place.

​The Same Test Gets Less Useful Over Time

A fixed test set is useful because it gives us something consistent to compare against. But if we keep testing the system against the same scenarios while actively optimizing it around those scenarios, eventually the result becomes less informative.

It is similar to giving someone the same exam over and over. Their score will probably improve. At some point, though, you need a different exam if you want to know whether they actually understand the subject.​ Teams should keep introducing new scenarios, different wording, unusual inputs and cases taken from real production behavior. Otherwise, it becomes too easy to build confidence around a test set the system already knows how to handle.

QA Should Be Looking For What The Score Missed

This is where I think QA becomes especially important. If an AI system scores 95% on an evaluation, I am less interested in celebrating the 95% than understanding the remaining 5%.

What failed? Was it random, or was there a pattern? Did the failures affect a particular type of customer? What happens when the wording changes? What happens when information is incomplete? What happens when two business rules conflict? And perhaps most importantly: Are there situations we never included in the evaluation at all?

These are not new questions. Good QA has always looked beyond the happy path. AI just gives us more reasons to do it.

Do Not Let One Score Become The Release Decision

An AI evaluation score can be useful, but it should not become the entire definition of quality. Teams should look at several types of evidence: evaluation results, production behavior, customer feedback, previous incidents, human review and new scenarios that were not part of the original test set.

For higher-risk products, it can also help to separate the people optimizing the system from the people deciding whether it is ready. That independence matters. When the same team defines the score, improves the score and decides that the score is good enough, it becomes easy to miss the assumptions underneath it.

What Leaders Should Ask

Leaders do not need to understand every detail of an AI evaluation. But they should ask the following:

• What does this score actually represent?

• Could the system perform well against this metric and still create a bad customer experience?

• How often do we add new scenarios?

• Are we testing situations based on real customer behavior and production failures?

• Who is responsible for challenging the evaluation itself?

• What other evidence do we use before deciding the system is ready?

Those questions tell me much more about the maturity of an AI testing process than a single percentage on a dashboard.

The Test Is Not The Goal

AI systems are going to keep getting better at whatever we ask them to optimize. That includes our tests. So the question cannot stop at, “Did it pass?” We also need to understand why it passed and whether passing actually tells us what we think it does.

If the system learns how to satisfy the measurement without satisfying the real intent behind it, the score becomes much less useful. The purpose of testing is not to help AI get better at taking tests. It is to give us enough evidence to trust what happens when there is no test case in front of it.

Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?