Building LLM Memory From Scratch

Building LLM memory from scratch means designing the storage, retrieval and write logic that lets your AI application carry facts from one session to the next. The model itself forgets everything when the conversation ends; the memory layer you build is what keeps the useful parts. For shared project work, tools like fenbs offer an existing MCP-accessible layer: tasks, decisions and handoff state that any connected assistant reads without custom code.

The goal is not a large archive. A useful memory layer is small, current and easy to query. Start with the minimum that makes a fresh session as useful as a session that already knows the context. See the agent harness topic for more on the software that surrounds a working AI model.

Decide what to remember before writing any code

The most common mistake in building LLM memory is storing too much. Every stored fact is a retrieval problem later. Before writing storage logic, answer three questions: what does a future session need to know that the current one discovered? What would it be harmful to forget? What will go stale quickly enough to cause problems if it is wrong?

• Decisions and rules: the choices people made that still apply, with the reason. These are the highest-value facts to carry forward.
• Task state: what is in progress, what is waiting and what was completed in the last session. A sentence per task is usually enough.
• Corrections: facts that were wrong and are now fixed. Storing the corrected version alongside a note about the previous error helps an assistant avoid repeating the mistake.
• Skip raw transcripts: they are large, expensive to search and rarely the right granularity. Store conclusions, not conversations.

Design the storage layer

For most from-scratch implementations, a relational database or a structured document store is the right starting point. You want named fields you can filter and update precisely: project identifier, fact type, text, timestamp and an expiry or version field. Avoid a single large text blob per project; it is hard to update one fact without rewriting the whole thing.

Add a namespace for every project or user from day one. Without it, facts from one context will mix with another as your application grows. A simple string prefix on every row is enough to start. Vector stores (which search by meaning rather than by key) are worth adding later if you need to retrieve from large unstructured text, but they are overkill for most minimal builds.

Write the retrieval function

The retrieval function runs at the start of every agent session. It should return only what the session actually needs, not the full store. A good retrieval function answers: what are the current rules? What is in progress? What should the next session do first?

1. Filter by namespace or project identifier. Never return facts from a different project.
2. Return rules and decisions first. An assistant that reads rules before task state is less likely to act against a known constraint.
3. Limit task state to the active and recent items. Completed tasks from three sessions ago are rarely useful context.
4. Cap the output size. A large memory dump fills the context window and leaves less room for the actual work. Aim for the key facts a person could read in two minutes.
5. Return a timestamp or version number with each fact so the assistant can note when something was last confirmed.

Selective writes: when to record and what to say

Write to the memory layer at natural stopping points, not after every model turn. Useful write moments are: when a task changes status, when a person confirms or overturns a decision, and when a session ends with unfinished work. A write that fires too often produces noise; one that fires too rarely loses useful state.

Format matters as much as timing. Write facts as complete sentences with enough context to stand alone: "The header image was removed from the landing page on October 21, 2026 because it slowed the mobile load" is far more useful than "header removed." The future session reading this will have no access to the conversation where the decision was made.

For patterns in larger systems, LLM memory management covers pruning, ranking and expiry strategies. If you are working with LangChain, LLM memory with LangChain explains the built-in memory modules.

Maintenance: keep the record accurate

A memory layer that no one maintains becomes a liability. Facts go stale; decisions get reversed; tasks marked as in progress stay that way for weeks. Build a correction path from the start: a person or a session with the right access can update or delete a stored fact directly, without going through the agent's normal write path.

Also add an expiry or review date to facts that are likely to change. A rule that applies this week may not apply next month. An assistant that reads a rule with an expired date should flag it for review rather than acting on it silently.

The what is agent memory guide explains the broader memory landscape across different AI tools. For the kind of memory that travels between sessions and across model changes, portable AI memory covers the design principles.

How to do this with fenbs

1. Create a free account. You get a kanban board with projects, tasks and Decisions and rules ready to use.
2. Add your project. Write a goal sentence and add the active tasks with their current status. This is the initial state a connected assistant will read.
3. Connect your AI app over MCP from Integrations. Approve the connection in the browser sign-in.
4. At the start of each session, ask the assistant to call the context tool. It reads Rules, Where we left off, tasks in progress and any notes you have saved.
5. At the end of each session, ask the assistant to save progress: what changed, what remains and the next step. You can also correct or delete any stored fact directly from the board.

fenbs does not import chat history or train on your sessions. It holds the structured work record: what you decided, what is in progress and what the next session needs to start from. If you need memory inside a product you are building for users, you need a custom layer; fenbs is for the work the team and their AI apps do together.