Make an AI Chatbot: Define, Build and Test Its First Job

Making an AI chatbot that works comes down to three things: a single job the chatbot is responsible for, a small set of approved sources it draws on, and a clear rule for what to say when it cannot answer. Most chatbot problems trace back to skipping one of these three. fenbs keeps the design decisions, test results, and lessons learned from a chatbot build in one place, so the next person working on it starts with the full picture.

This guide covers the design and build process for a first chatbot job. For choosing a platform, see the building an AI chatbot guide.

Start with one bounded job

Write the chatbot's job in one sentence, and write what is explicitly out of scope in another. "This chatbot helps customers track their order status and understand the return policy. It does not handle account changes, complaints, or billing questions." That pair of sentences is the design spec. Every decision during the build should be checked against it.

A chatbot with a narrow scope is easier to test and easier to improve. When a user asks something outside the scope, the chatbot can say clearly where to go next rather than attempting a guess. An honest "I can't help with that, but you can contact the billing team at [address]" is a better experience than a confident wrong answer.

Design the input sources

List the documents or data sources the chatbot will draw on. For each one, ask: is it current? Is it authoritative? Does it contain the correct answer to the questions the chatbot will face? A return policy document from two years ago is worse than no source at all if the policy has changed.

Format matters too. Sources that are clearly structured, with headings and short sections, are easier for a language model to cite accurately than long unstructured prose. If you can improve the source document before connecting it, do so.

Write a fallback rule first

Before configuring anything else, write the fallback rule: what the chatbot says when it cannot answer confidently. Put it in the system prompt. "If you are not confident in the answer based on the provided documents, say 'I'm not sure about that' and direct the user to [specific contact or resource]."

Test the fallback by asking questions the sources do not cover. If the chatbot generates an answer anyway, tighten the system prompt or lower the confidence threshold if your platform supports it. A chatbot that refuses to guess is more trustworthy than one that fills gaps with plausible-sounding fabrications.

Build and test in steps

Connect the first source and test it before connecting more. Ask ten questions the source should answer. Record the answers and whether they are correct, acceptable, or wrong. Fix problems before adding complexity. A wrong answer that exists with one source is not solved by adding more sources.

When the first source works, add the second and test again. Focus on whether the chatbot correctly chooses between sources and whether adding the new material introduced any conflicts. Two sources that contradict each other on the same point will produce inconsistent answers.

Handling questions outside the scope

Even with a clear scope, users will ask the chatbot things outside it. Some will be obvious, like asking a returns chatbot for a shipping address. Others will be borderline: questions the chatbot almost covers but not quite.

For clearly out-of-scope questions, the fallback rule handles them. For borderline questions, decide whether they reveal a gap in the scope or a gap in the sources. A question the chatbot should answer but gets wrong needs a better source. A question the chatbot should not answer at all needs the scope definition tightened in the system prompt. Both are improvements, but they require different fixes.

Record design decisions before you forget them

A chatbot build accumulates many small decisions: why a source was included or excluded, why the fallback was worded a particular way, which test questions revealed a gap. Those decisions are easy to forget and hard to reconstruct later.

Record each significant decision when you make it. When a test answer was wrong, note what the question was, what went wrong, and what was changed. That record becomes a test log and a design document simultaneously. The AI chatbot for business guide covers broader deployment considerations once the build is stable.

How to do this with fenbs

1. Create a free account. Add a project for the chatbot build.
2. Add tasks for each step: writing the job scope, selecting sources, configuring the fallback, and running the test set.
3. Open Decisions and rules and record the job scope, approved sources, and fallback rule. These are the constraints every session must follow.
4. After each test run, add the results to the relevant task: which questions passed, which failed, and what was changed.
5. When a lesson from the build should change how future chatbots are designed, record it as a lesson in the project for future reference.