How do you test an AI application?

Build an evaluation set of representative (input, expected output) pairs with explicit acceptance criteria. Run the set before and after every prompt or configuration change. Test each failure category explicitly: hallucinations, wrong refusals, scope violations, and retrieval failures.

What should an AI evaluation set contain?

Real or realistic examples drawn from the application's intended use. Each example needs an input, a description of what a good answer looks like, and any specific failure modes to check for. Ten to twenty examples cover most important behaviors to start.

How do I know when an AI prompt change causes a regression?

Run the full evaluation set before and after the change. Compare the results for every example. An example where the behavior changed for the worse is a regression; one where it changed for the better is an improvement. Run everything, not just the examples related to the change.

Where should I store AI acceptance criteria and test decisions?

In a place the whole team can find and update. Some teams use the project's version control alongside the eval set. Others use a project management tool. What matters is that everyone is working from the same criteria, and that changes are recorded with the reason for the change.

How to Test AI Applications with Repeatable Examples

Testing an AI application means building evaluation sets, checking failure categories, and running regression checks when prompts or context change. Here is a practical approach that produces results you can act on.

July 1, 2026Updated October 21, 20266 min read

Testing an AI application is different from testing deterministic code. The same input can produce slightly different outputs on different runs, and "correct" often depends on judgment rather than an exact match. The solution is a structured evaluation process: a set of representative examples with defined acceptance criteria, a way to run them consistently, and a record of what changed and what the results were.

The decisions behind your acceptance criteria — what counts as a good answer, what counts as a failure — are as important as the tests themselves. A tool like fenbs keeps those decisions alongside the open tasks they create, so the next session starts with the same standards the team agreed on.

Build a repeatable evaluation set

An evaluation set is a collection of (input, expected output, acceptance criteria) triples. Start small: ten to twenty real examples cover most of the important behavior. Each example should be a case that has happened or is likely to happen — not a contrived edge case invented to increase coverage numbers.

Collect examples from real usage or realistic scenarios. For a customer support bot, use real support tickets. For a code review tool, use real pull requests.
Write the expected output or a description of what a good answer looks like. For open-ended outputs, a rubric ("mentions the return policy, does not make up a phone number, is two sentences or fewer") is more useful than an exact string.
Note the failure modes you are specifically checking for: hallucinations, refusals that should not happen, out-of-scope responses that should be refused but are not.
Store the evaluation set in version control alongside the application. It should change when the application changes, just like tests.

Categories of failures to test

AI application failures tend to fall into a small number of categories. Test each one explicitly:

**Hallucination**: the model states something false confidently. Test with questions that have a clear correct answer and check that the answer is right.
**Refusal failure**: the model refuses a request it should handle. Test with valid, in-scope requests and confirm it responds.
**Scope failure**: the model handles a request it should refuse. Test with out-of-scope or policy-violating requests and confirm it declines.
**Retrieval failure**: in a RAG system, the model answers from the wrong source or ignores the retrieved context. Test by checking whether the answer can be found in the retrieved chunks.
**Instruction failure**: the model ignores or misapplies a system prompt instruction. Test each constraint in the system prompt with an input designed to trigger it.

Regression checks for prompts and context

Any change to the system prompt, the retrieval configuration, or the model version can change the application's behavior across inputs you did not test directly. Run the full evaluation set after every change, not just the examples related to the thing you changed.

Run the evaluation set before making a change to establish a baseline.
Make the change (prompt update, model version, retrieval tweak).
Run the evaluation set again and compare the results.
For any example where the result changed, decide whether the change is an improvement or a regression.
If it is a regression, either revert the change or update the evaluation set to reflect the new accepted behavior.

This is how a small evaluation set becomes a safeguard. The set is only useful if you run it consistently and act on what it shows.

Testing tools for AI applications

Tools commonly used for testing AI applications
Tool	What it does	Good for
Langfuse	Traces, evaluations, and prompt versioning.	Teams using LangChain or custom pipelines who want observability and eval tracking.
Braintrust	Evaluation experiments, scoring, and dataset management.	Running structured eval experiments and tracking scores over time.
PromptFoo	Open-source prompt testing and comparison.	Comparing prompts, models, and configurations with custom tests.
Pytest + custom assertions	Python test framework with application-specific checks.	Teams that prefer code-level tests and want full control over the evaluation logic.

These tools run tests; they do not store the decisions behind the acceptance criteria or the follow-up tasks that failures create. Those belong in your project record.

How fenbs helps

Keep what failures taught you

Every eval failure is a lesson: what the input was, what the model produced, and what the correct behavior should be. fenbs Lessons keep those findings attached to the project so the next iteration starts from the failures the last one exposed. Decisions and rules hold the acceptance criteria so they do not have to be rediscovered. Create a free account and connect your AI from Integrations.

Create a free account

Free for one person · No credit card · Works with Claude, ChatGPT, Cursor and more

How to do this with fenbs

Create a free account and add the AI application as a project. Write the acceptance criteria for each major feature as Decisions — what counts as a good answer and what counts as a failure.
When evaluation runs produce failures, create tasks in fenbs for each one. Note the input, the actual output, and what the correct behavior should be.
Record lessons from failures: what the model did wrong, what change fixed it, and what to check for in future prompt changes.
Before a prompt change, run the evaluation set and save the baseline results as a project note. After the change, run again and record the delta.
Connect a supported AI app to fenbs over MCP from Integrations. Ask it to read the project context — including the acceptance criteria and recent lessons — before working on prompt improvements.

The what is an AI brain article covers the broader question of how AI applications can access shared context. The how to use AI at work guide addresses practical integration for teams. The agent harness topic hub has more on building reliable AI-powered workflows.