How to Test AI Systems: Cases, Checks, and Regression Tests

Testing an AI system is different from testing deterministic code. The same input can produce different outputs on repeated runs, and "correct" is often a matter of degree rather than a binary pass or fail. The goal is not to prove the system is perfect but to catch regressions, surface failure modes, and build enough confidence to ship.

Every test run teaches you something about the system. Tools like fenbs keep failure lessons and review decisions in a record every future session reads, so the team does not repeat the same investigations. For testing at the automation layer, Selenium AI test automation covers a related approach.

What makes AI testing different

Ordinary software tests compare actual output against expected output precisely. AI system tests check whether the output has the right properties: the right format, the right fields, a sensible value in the right range, or the absence of a known bad pattern. Exact string matching rarely works.

Non-determinism is the main difficulty. A test that passes today may fail tomorrow on identical input. Temperature settings, model updates, and subtle changes to the prompt can all shift behavior. Treat repeated failures on the same input as a signal worth investigating. Treat a single failure as something to log and monitor.

Speed and cost also matter. A test suite that calls a commercial model API for every run is expensive. Keep a small, fast test set for frequent runs and a larger evaluation set for less frequent checks. Know which tests run in which context.

Defining test cases from behavior

Start with the behaviors the system is supposed to exhibit, not with the inputs you happen to have. A useful behavior list answers the question: what would convince you the system is working well enough to use?

• Core cases: the most common inputs the system will see, with the properties the output must have.
• Edge cases: inputs at the boundaries of what the system handles, such as very short inputs, very long ones, inputs with unusual formatting, or inputs that include instructions.
• Adversarial cases: inputs designed to trigger known failure modes, such as ambiguous phrasing, conflicting instructions, or topics outside the system's scope.
• Regression cases: inputs that caused failures in the past, kept permanently in the test set.

Write each case with a clear expected behavior, not just an input. "The output should be a valid JSON object with the summary and category fields" is a testable expectation. "The output should be good" is not.

Writing evaluation checks

Evaluation checks are functions that take a model output and return a pass or fail. Common checks include: JSON schema validation, presence or absence of specific substrings, maximum length, specific field values, and absence of known bad patterns like hallucinated names or incorrect dates.

For qualitative properties like tone, factual accuracy, and relevance, a second model call is a useful check. Ask a separate model to evaluate whether the output meets the requirement, using a clear rubric. This is called an LLM-as-judge approach. It is not perfectly reliable, but it scales to properties that a regular function cannot check.

Record which checks each test case uses. When a check fails, you need to know whether the input, the expected behavior, or the check itself was wrong. See system prompt examples for patterns that reduce the need for complex output parsing.

Running evaluations regularly

Run the core test set after every significant prompt or model change. Run the full evaluation set before any deployment. Run regression cases whenever a failure is reported in production.

Track results over time. A system that passes 90% of checks is not the same as one that passes the same 90% every time. Fluctuating results on the same cases signal an instability worth investigating. A steady result with a clear failure profile is easier to work with than an erratic one.

Compare results across model versions and prompt versions. When a model update changes behavior, the evaluation tells you whether the change is an improvement, a regression, or a mix. That comparison guides the decision to adopt the new version.

Tracking failures and recording lessons

When a test fails, investigate before fixing. Understand whether the input is valid, whether the expected behavior was correctly specified, and whether the failure is consistent or intermittent. Then fix the right thing: the system, the prompt, the expected behavior, or the test.

Record the failure, the root cause, and the resolution. A team that does not record its AI testing failures will repeat the same investigations. The second investigation of the same failure is wasted time. The third is a sign that something needs to be in a more visible place.

Regression testing after changes

Every time you change a prompt, a model setting, or the retrieval logic, you create a regression risk. A change intended to improve one behavior may silently break another. Regression tests catch this before users do.

Keep every input that caused a failure in the past as a permanent regression case. Do not remove a case because the current version handles it correctly; remove it only if the case no longer represents a real input the system will see. A regression set that grows over time builds cumulative protection against the failures the system has actually experienced.

Automate regression checks where you can. A check that runs automatically on every code change is more reliable than one that runs when someone remembers. For broader automation testing patterns, see test automation.

How to do this with fenbs

1. Create a free account. Add the AI system as a project with its purpose, the model it uses, and any existing known constraints.
2. Add tasks for each test scenario to write: core cases, edge cases, and adversarial cases.
3. When a test fails, record it as a lesson in fenbs: the input, the failure mode, the root cause, and the fix. Link it to the affected task.
4. Record decisions about the evaluation approach in Decisions and rules: which model version is in use, which prompt version passed evaluation, which checks are required.
5. Connect your assistant over MCP from Integrations. At the start of each testing session, ask it to read the project context, including lessons from previous failures.