AI Chatbot With a Custom Knowledge Base: Build and Check It

An AI chatbot with a custom knowledge base answers questions from documents you provide — a help center, a policy manual, a product catalog — rather than from general training data. The practical challenge is not connecting the chatbot; most platforms make that straightforward. The challenge is making sure the answers are grounded in your documents and knowing when they are not.

Tools like fenbs keep the approved instructions that govern the chatbot, the improvements you decide on after testing, and the lessons from each iteration in one record, so the team that maintains the chatbot starts from current knowledge rather than last month's memory.

Choose your sources carefully

The chatbot answers from the documents you give it. Outdated content, internal jargon, or documents written for an expert audience will produce answers that mislead users. Choose sources by asking: if a user reads this answer, is it accurate, current and understandable to them?

• Prefer shorter, focused documents over long general ones. A fifty-page policy manual produces worse retrieval than ten focused policy pages.
• Exclude documents with contradictory information. If two documents say different things about the same policy, the chatbot will too.
• Remove outdated versions before uploading. A document about a discontinued product will confuse users and the chatbot alike.
• Write clear headings. Most knowledge base platforms chunk documents by section; a document without headings is one large chunk that retrieves poorly.

Start with a small set of the most frequently asked questions and the documents that answer them. Test the chatbot on those questions before expanding the knowledge base. Adding more documents fixes a missing-answer problem; it does not fix a wrong-answer problem.

Connect the chatbot to your knowledge base

Most chatbot platforms — including offerings from major AI providers and dedicated customer support tools — offer a knowledge base connection. You upload documents or point the platform at a URL, and it processes the content into a form the chatbot can search. This retrieval step is often called RAG (retrieval-augmented generation), meaning the chatbot retrieves relevant passages and uses them to form its answer.

The setup process differs by platform, but the key decisions are the same: which documents to include, how to handle content that changes, and what the chatbot should say when it cannot find an answer in the knowledge base. Set an explicit fallback — "I don't have information about that; contact support at this address" — rather than letting the chatbot guess from training data.

For adjacent questions about building custom knowledge bases for other tools, see n8n AI agent knowledge base, Zendesk AI agent knowledge base, and the general AI knowledge base guide.

Write the system instructions

System instructions tell the chatbot how to behave: what it is, what it helps with, how to handle questions outside its knowledge base, and what tone to use. Write them as if writing a job description for a support agent. Be specific about what the chatbot should and should not do.

Write down the instructions as a decision with a reason. When the instructions change after testing, record the change and the reason. Six months from now, someone on the team will ask why the chatbot behaves a certain way. A written decision is a much better answer than "we think it was changed at some point."

Test for unsupported answers

The most important test for a knowledge-base chatbot is what it does when asked something not in the knowledge base. A well-configured chatbot says it cannot find an answer and points to another resource. A misconfigured one answers from general training data — with confidence, and often incorrectly.

Run at least three types of test questions: questions clearly answered in the knowledge base, questions adjacent to the knowledge base (related topic but not covered), and questions completely outside the knowledge base. Check that answers to the first type cite or quote the relevant document, and that answers to the second and third types trigger the fallback.

When the chatbot produces an answer you cannot trace to a document, that is a miscalibration in either the retrieval settings or the system instructions. Record the test case and the result. That record is the input to the next improvement.

Handle knowledge base updates

A knowledge base that is not maintained becomes a liability. When a document it contains is updated, the old version needs to be replaced. When a policy changes, the chatbot needs to reflect it before users encounter the wrong answer.

Establish a review cycle for the documents in the knowledge base: who reviews them, how often, and what triggers an immediate update. That process is a standing rule for the team, not a one-time task. Record it in Decisions and rules so the next person who joins the team knows to follow it.

How to do this with fenbs

1. Create a free account. Add a project for the chatbot with a description of its purpose and intended users.
2. Add tasks for the knowledge base setup steps: source selection, document upload, instruction writing, and testing.
3. Record the system instructions in Decisions and rules, with the reason for each significant choice.
4. Connect your AI app over MCP from Integrations. Ask it to read the project context and lessons before each session.
5. After each testing round, record what you found and the resulting change as a lesson and a decision update.

When fenbs is not the right fit here

fenbs keeps the project record behind the chatbot — instructions, decisions and lessons — but it does not host the knowledge base or build the chatbot itself. Use a knowledge base or chatbot platform for that. fenbs is useful when more than one person maintains the chatbot, or when the instructions and improvements need to survive across AI sessions and team changes.