Semantic Kernel Agent Memory: State, Storage, and Recall

Semantic Kernel is Microsoft's open-source SDK for integrating AI models into applications. It supports C#, Python, and Java. For agents built with Semantic Kernel, memory includes chat history for conversation continuity, text memory plugins for storing and retrieving information, and process state for longer-running workflows.

Each memory mechanism serves a different job and has different persistence behavior. Chat history scopes to a session by default. Text memory requires a backing store to persist. Project decisions made by people belong outside the SDK entirely. Tools like fenbs keep that record in a place any tool can read, not locked to a single SDK or runtime.

Chat history in Semantic Kernel

ChatHistory is the primary way a Semantic Kernel application maintains conversation continuity. It accumulates messages from the user and the assistant, which the kernel includes in subsequent calls so the model can reference earlier turns.

ChatHistory is in-memory by default: the object lives for the lifetime of the application instance. When the process exits or the object goes out of scope, the history is gone. To persist it across restarts, you serialize the history to storage — a database, a file, or a cloud store — and reload it at startup.

As the history grows, it approaches the model's context window limit. Semantic Kernel provides utilities for truncating history to a maximum token count. Older messages are removed first, which means detail from early in a long conversation may be lost.

Memory plugins and vector stores

Semantic Kernel supports memory plugins that let an agent save and retrieve information from a vector store (a database that stores text as numerical embeddings for semantic similarity search). The agent can call a memory plugin to store a fact and retrieve related facts by meaning rather than exact match.

Supported vector stores include Azure AI Search, Qdrant, Pinecone, Chroma, and others. Each requires setup and ongoing maintenance. The agent reads and writes to these stores through plugin calls, which means the retrieval is only as accurate as the index and the search query.

Process state in Semantic Kernel Process Framework

Semantic Kernel also includes a Process Framework for building multi-step workflows. Processes maintain state across steps, and that state can be persisted to a store, which lets a workflow pause and resume across sessions. This is a more structured form of persistence than chat history but requires designing your logic around the process model.

Setting up a vector store for memory plugins

To use a Semantic Kernel memory plugin with a vector store, you configure a text embedding model and a storage connector. The embedding model converts text to vectors when you save a memory; it runs the same conversion on queries when you search. The embedding model you use at save time must match the one you use at search time, or the vectors will not compare correctly.

Index your memories in consistent units. A memory that stores a single task or decision retrieves cleanly for relevant queries. A memory that stores a long session transcript retrieves for many unrelated queries and adds noise to the results. Start with a simple collection and refine the chunk size based on retrieval quality, not on what is easiest to write.

Testing what survives a restart

For each memory mechanism you use, test the restart behavior explicitly. Stop and restart the application, then ask the agent to recall something from the previous session. If the answer is empty or generic, the state did not survive. If the answer is accurate, the persistence layer is working.

Also test the context window boundary. Run a conversation long enough that early messages should be truncated, then ask about something from the beginning. Accurate recall means the message survived truncation; inaccurate recall means it was removed. Knowing this boundary helps you decide what must be stored in a memory plugin rather than left in chat history.

For comparing in-session memory to retrieval-based memory, see agent memory vs RAG. For security questions about what goes into a vector store, see agent memory security.

What does not belong in the SDK's memory

Project decisions, standing rules, and task state are not agent runtime artifacts. They are facts that people chose and that every session must respect, regardless of which SDK version is running or whether the vector index was rebuilt. Storing them only inside Semantic Kernel means they depend on that SDK staying available and configured.

These facts also need attribution: who set the rule, when, and why. Chat history and vector stores do not preserve that structure naturally. An external record does.

How to do this with fenbs

1. Create a free account and add the project your Semantic Kernel application supports.
2. Add the decisions and rules to fenbs as Decisions, noting who set each one.
3. Add the active tasks so agents start from an accurate picture of the current work state.
4. Configure your Semantic Kernel application to load fenbs project context at session start using the MCP connection at Integrations.
5. After significant runs, record completed work and remaining steps in fenbs rather than relying on chat history to carry that state forward.

For building memory into a Semantic Kernel agent from scratch, see how to build agent memory. For how MCP connects agents to external memory, see agent memory MCP.