Azure AI Search Knowledge Base: Build and Test One

Azure AI Search is Microsoft's cloud search service that lets you index your own documents, configure semantic ranking and then query the index to get answers grounded in your content rather than in the model's training data. A knowledge base built on it powers AI assistants that answer questions about your policies, products or internal documentation without hallucinating facts. Tools like fenbs sit alongside it differently: not as a search index, but as a shared record of the source decisions, configuration choices and project context that your team and its AI assistants need to work from.

What Azure AI Search does for a knowledge base

Azure AI Search (previously Azure Cognitive Search) takes documents you provide, breaks them into searchable chunks and lets you query them using keyword search, vector search or a combination. When you pair it with a language model, the model reads the retrieved chunks and generates an answer based only on what it found — this is retrieval-augmented generation, or RAG.

The advantage over asking the model a question without retrieval is grounding: the model cites its answer in the documents you indexed, so you can check whether the answer matches the source. A well-configured knowledge base reduces hallucinations on domain-specific questions. See Microsoft's Azure AI Search documentation↗ (opens in a new tab) for current capabilities and pricing.

Prepare your source documents

1. Decide which documents belong in the knowledge base. Not every document deserves to be indexed. Include documents that contain answers people actually need, are accurate and current, and have a clear owner who will update them. Exclude drafts, duplicates and documents whose accuracy you cannot verify.
2. Choose a document format Azure AI Search can process. Supported formats include PDF, Word, PowerPoint, Excel, HTML and plain text. For documents in unsupported formats, convert them first.
3. Clean the documents before indexing. Remove headers, footers and boilerplate that repeats across every page. Those repeated phrases appear in many chunks and reduce retrieval quality by diluting the signal.
4. Decide on chunk size. Azure AI Search splits documents into sections for retrieval. Smaller chunks (a paragraph or two) return more precise matches; larger chunks return more context. A typical starting point is 512 tokens per chunk with some overlap.
5. Record which documents you included and why. Write that as a decision, not just a config file comment: who chose to include this source, when it was last verified and when it should be reviewed again.

Configure the index and semantic retrieval

Create an index in the Azure portal or using the Azure AI Search REST API. Define the fields the index will store: content, source URL, document title, creation date and chunk metadata are common starting fields. Azure AI Search can generate embeddings for vector search if you enable it with a supported embedding model.

Semantic ranking uses a secondary model to re-rank results based on meaning rather than keyword frequency. Enable it for knowledge base use cases: queries are typically full questions, and semantic ranking significantly improves result quality over keyword-only ranking. It adds cost per query, so check Azure's pricing page↗ (opens in a new tab) before enabling it at scale.

If you are building a knowledge base through Azure AI Foundry (the portal that combines model deployment, search and grounding), the setup wizard handles index creation and the RAG connection in one flow. Check the Azure AI Foundry documentation↗ (opens in a new tab) for current setup steps, since the interface updates frequently.

Test grounded answers before going live

Testing a knowledge base means checking retrieval quality and answer quality separately. Retrieval quality is whether the right chunks come back for a given question. Answer quality is whether the model produces a correct, grounded answer from those chunks.

Build a test set of ten to twenty questions your users are likely to ask, along with the expected answers and the documents those answers should come from. Run each query against the index and check whether the expected document appears in the top results. Then check whether the model's answer matches the source. Keep the test questions and their expected answers in a document you can rerun after any configuration change.

Common failure modes: the relevant chunk is not being returned (check the chunk size and the search mode), the model answers from its training data rather than the retrieved chunks (check whether grounding instructions are in the system prompt), or the answer is correct but uncited (configure the model to always cite the source document).

Decisions worth recording before you go to production

A knowledge base accumulates configuration choices that are easy to forget six months later. Which document sources are authoritative? Who is responsible for keeping them current? What topics is the knowledge base not designed to answer? What happens when the model cannot find a relevant chunk — does it say so, or does it fall back to training data?

Each of these is a decision, not just a setting. Record them with the person who made them and the reason, and link to the relevant index configuration. When the index is updated or a new document type is added, those decisions become the checklist for the change review.

How to do this with fenbs

1. Create a free account and add a project for your Azure AI Search knowledge base build.
2. Record the source document decisions as Decisions and rules: which sources are included, who approved them, when they were last verified and the review schedule.
3. Add the configuration choices as rules: chunk size, search mode, semantic ranking on or off, and the fallback behavior when retrieval returns no relevant results.
4. Connect a supported AI assistant over MCP from Integrations to help plan and document the knowledge base structure. Ask it to read the project rules before making any index configuration suggestions.
5. After each test cycle, add a task note with what was tested, what passed and what still needs fixing. Update Where we left off so the next session starts from the current state of the build.

fenbs does not host a search index or store your documents. It keeps the human-approved decisions and project context that make the knowledge base maintainable over time. For a guide to the topic of connecting AI tools to outside knowledge sources, see the connect AI topic hub.