Agent Memory Architecture: Storage, Retrieval, and Control

An AI agent's memory is not one thing. It spans the context window that holds the current conversation, external databases the agent can query, logs of past episodes, and shared records that multiple agents read. Each layer serves a different purpose, has different limits, and is managed differently. Understanding which layer to use for which kind of information is the foundation of a reliable agent system.

Most discussions of agent memory focus on the technical layers — context windows, vector stores, key-value caches. What those layers do not reliably hold is the explicit, structured record of a project in motion: agreed decisions, active tasks, standing rules, and where things stand. A tool like fenbs fills that role: it is the project record layer that sits alongside the runtime memory architecture.

The four memory types

Agent memory research identifies four types, borrowed loosely from cognitive science. Each maps to a different use in AI systems:

• **Working memory**: what the agent is actively reasoning over right now. In practice, the context window.
• **Episodic memory**: records of specific past experiences — what happened in a previous session, what the agent tried and what the outcome was.
• **Semantic memory**: general facts, knowledge, and relationships that do not belong to a specific episode — a domain vocabulary, a product catalog, a policy document.
• **Procedural memory**: how to do things — the agent's skills, tool instructions, and behavioral patterns.

Working memory — the context window

The context window is an agent's working memory. Everything the agent needs for the current reasoning step must fit here: the conversation so far, the system prompt, any retrieved documents, tool results, and the user's message. The model can only "see" what is in the context window.

Context windows have grown significantly — modern models support hundreds of thousands of tokens — but size alone does not solve the problem. A large context still has limits, and a full context slows down inference. The useful principle is load what is needed, not everything that exists. That means having good retrieval from other layers.

What should always be in the context window at session start: the agent's role and constraints (system prompt), the most relevant current task, standing rules, and where things were left. These prime the agent before it takes any action.

External memory — retrieval and storage

External memory stores information outside the model and retrieves it on demand. Three patterns are common:

The explicit project record is often the missing layer. Vector stores retrieve relevant passages; they do not reliably answer "what did the team decide?" or "what is the current task status?" A structured record with explicit decisions and tasks fills that gap. The agent memory vs RAG article covers the distinction in more depth.

Episodic memory — recording what happened

Episodic memory records specific past events: a session log, the steps an agent took on a task, the outcome of a tool call. This is the layer that lets an agent or a team ask "what did we try?" and "did it work?"

Raw logs are one form of episodic memory, but they are often too detailed for a session's context window. The useful version is a distilled record: the task, what was attempted, what succeeded, what failed, and the next step. This is precisely what "Where we left off" captures in a project record.

For multi-session projects, episodic memory prevents repeated work. If an agent tried approach A and it failed, the next session should know that before it tries again. Recording failure modes as Lessons turns episodic memory into something the next agent can actually use.

Shared memory for multi-agent systems

When multiple agents work on the same project — different specialists, parallel tasks, or successive sessions — shared memory becomes essential. Without it, agents repeat each other's work, contradict each other's decisions, or lose track of what has already been tried.

Shared memory should have clear ownership rules: which agents can write, which can only read, and which writes require a human to sign off. In most project contexts, people decide and agents record — assistants can propose, but a person confirms decisions before they become standing rules.

In fenbs, Decisions and rules appear first in the AI context every connected agent reads. A rule recorded once applies to every agent on the project. Agents can record a person's decision but cannot decide themselves — that separation keeps the project's governance from drifting as agents change. The agent memory MCP article explains how agents access shared memory over MCP.

How to do this with fenbs

1. Create a free account and add your project. Write the decisions and rules that every agent session should follow as Decisions and rules in fenbs.
2. Use your existing vector store, database, or key-value store for the runtime memory the agent needs during a session.
3. Connect your AI apps to fenbs over MCP from Integrations. At the start of each session, ask the agent to read the project context: rules, active tasks, and Where we left off.
4. After each session, ask the agent to save a distilled progress note: what was done, what was checked, what comes next. This is the episodic record for the next session.
5. When the project teaches something worth keeping, record it as a Lesson. When a decision is made or changed, update the Decisions record. Over time, the project record becomes the most reliable part of the memory architecture.

The what is agent memory article covers the foundational concepts. The how to build agent memory article goes into implementation details for specific storage patterns. The agent harness topic hub collects the full library of related guides.