Elastic AI Assistant Knowledge Base: Organize and Check Sources

The Elastic AI Assistant uses a knowledge base stored in Elasticsearch to help answer questions about your environment: security alerts, infrastructure data, observability signals and your own policies or runbooks. What the assistant can usefully say depends directly on the quality and completeness of what is in that knowledge base.

The investigation tasks, reviewed decisions, and lessons that come out of using the assistant are a different kind of knowledge — they belong with the project, not inside the index. Tools like fenbs keep that record outside the source data, so the next session starts from the current state of the work rather than the last alert.

What the Elastic AI Assistant knowledge base holds

The Elastic AI Assistant knowledge base can hold entries you add manually and, on supported Elastic plans, entries generated automatically from your environment. Typical sources include: runbooks and response playbooks, alert documentation, system architecture notes, and policies your team has written for specific detection rules.

The assistant retrieves relevant entries when you ask a question and includes them as context. An empty or thin knowledge base produces generic answers. A well-maintained one produces answers specific to your environment: "This alert matches the pattern in runbook R-12, which says to isolate the host and open a Jira ticket."

Check Elastic's own documentation↗ (opens in a new tab) for the current list of supported entry types and any plan requirements, as these change across Elastic releases.

Prepare knowledge before adding it

A knowledge base entry is most useful when it answers one specific question or covers one specific procedure. Long documents that cover multiple topics retrieve poorly: the retrieval step finds the document but pulls in a lot of irrelevant text along with the relevant passage.

• Break large runbooks into separate entries by alert type or procedure step.
• Write clear titles. The assistant uses the title to decide whether an entry is relevant to a question.
• Remove superseded versions. If a runbook was updated, delete the old entry before adding the new one.
• Use the language your team actually uses in queries. If your analysts ask about "lateral movement," that phrase belongs in the entry, not only the formal MITRE technique name.

For a broader look at building AI knowledge bases, see AI chatbot with a custom knowledge base and the AI knowledge base overview.

Validate what the assistant retrieves

Adding an entry does not guarantee the assistant will retrieve it. After adding a batch of entries, test the assistant with the questions those entries are meant to answer. Check whether the assistant cites the entry, uses the right procedure, or ignores it and gives a generic response.

If an expected entry is not retrieved, the usual causes are: the entry title does not match the terms you used in the query, the entry content is too long and the relevant passage is not near the top, or the entry was added to the wrong index or connector. Elastic's documentation on knowledge source configuration covers the specific setup for your version.

When an entry is retrieved but the assistant's answer is still wrong, the entry content itself may need revision. Record the test case, the expected answer, and the actual answer. That record drives the improvement without having to remember the detail three weeks later.

Separate what goes in the index from what goes in the project record

The knowledge base holds reusable reference content: runbooks, playbooks, policies. A project record holds the work: which alerts were investigated, what was decided about each one, what is still open, and what was learned. Mixing the two makes both harder to use.

Say an alert fires and the investigation determines it is a false positive caused by a new deployment pattern. The runbook for that detection rule should be updated to note the false positive condition. The decision about the specific alert — including the evidence, the conclusion, and who signed off — lives in the project record, linked to the task for that investigation.

This separation also means the assistant's knowledge base stays clean. If you write investigation notes into the knowledge base, the assistant will start citing those notes as if they are policy guidance. Keep the two categories distinct.

Keep knowledge entries current

A knowledge base that is not maintained drifts from reality. Runbooks become outdated after system changes. Alert thresholds change. New tools are added. Establish a review cycle: which entries are reviewed, how often, and who is responsible. Record that as a standing rule for the team.

One practical approach is to review entries whenever a related alert fires and produces a wrong answer. That ties maintenance to real cases rather than a calendar, and it produces a record of why each entry was updated.

How to do this with fenbs

1. Create a free account. Add a project for the security operations or observability work the assistant supports.
2. Add tasks for each runbook or policy you are adding to the knowledge base. Record the status: draft, reviewed, or live.
3. After each test session, add a lesson if you found a gap or a retrieval failure: what the test question was, what the assistant said, and what changed.
4. Record the maintenance cadence as a decision in Decisions and rules, with the person responsible.
5. Connect your AI app over MCP from Integrations. Ask it to read the project context before the next investigation session.