Hindsight LLM Memory: What It Stores and How to Evaluate It
Hindsight stores information outside a language model and retrieves it for later requests. Here is how to assess its recall, corrections and operational control before relying on it.
Updated 5 min read
Hindsight LLM memory is an external memory system that stores information supplied by an application and retrieves relevant material for later language-model requests. It gives an app a way to carry useful information across conversations without putting its entire history into every prompt. The practical questions are what gets saved, how a correction reaches future answers and who operates the service.
A team may also need an explicit record of its work. fenbs keeps projects, decisions and attributed changes available to connected assistants; a memory engine and a shared work record solve different parts of remembering.
What does Hindsight keep outside the model?
The Hindsight retain documentation (opens in a new tab) describes turning supplied conversations and documents into structured, searchable memories. It distinguishes information about the outside world from the memory-owning assistant’s experiences. The application supplies the material, so useful memory starts with deciding which sources belong there.
Say you are building an assistant for an equipment rental business. It might need the customer’s stated requirements and the outcome of a previous recommendation. A draft suggestion, a confirmed booking and a returned item have different meanings. Your application should preserve those distinctions when preparing the information it sends.
This does not mean the language model itself permanently learned your rental history. The model receives selected information when the application requests it. If that distinction is unfamiliar, what an LLM is explains the model; our memory tools hub covers the surrounding tools.
What is the difference between recall and reflect?
Hindsight separates looking up material from generating a response using it. Its recall documentation (opens in a new tab) describes combining searches for meaning, exact terms, relationships and time. That matters when a question names a particular item, describes an earlier event in different words or asks what happened before a change.
Its reflect operation (opens in a new tab) gathers evidence from stored memory and produces a contextual response with source references. Treat that response as a generated interpretation. Inspect the supporting material when the answer affects a commitment or when two records disagree.
For the rental example, retrieving the customer’s earlier requirements is one job. Explaining why a different machine may now be appropriate is another. Evaluate them separately: a sensible explanation can still rest on the wrong customer’s information, while a correct retrieved record can still be interpreted poorly.
How should you handle a changed fact?
Corrections need an identifiable source. Hindsight’s document API (opens in a new tab) associates memories with documents, lets an application retrieve original text and supports replacing content by retaining it again with the same document ID. It also documents removing a document and its associated memories.
Give a changing source a stable identity. If a rental requirement changes from indoor use to outdoor use, update the intended source rather than adding an unlabeled statement beside it. If you need the earlier requirement as history, keep a separate dated event explaining when it applied and why it changed.
Then ask the old question again, ask a differently worded version and ask a question about the earlier period. These checks reveal whether your application can distinguish current truth from historical truth. A successful write request alone does not show that every later answer will use the correction correctly.
What operational control should you check?
The Hindsight source repository (opens in a new tab) publishes the project under the MIT license and points to self-hosting and managed deployment options. Choose an operating model before connecting important information. Running software yourself gives you responsibilities for its configuration, access, backups and updates.
- Identify who can submit, retrieve, replace and delete information in your deployment.
- Trace where text goes during extraction, search and answer generation, including any external model provider.
- Test recovery from a backup and confirm which information that backup retains.
- Keep source identifiers and enough request evidence to investigate a wrong answer without logging unnecessary sensitive content.
- Assign a person to review configuration changes and failed processing jobs.
Separate operating control from content authority. Being able to delete a database entry does not tell your colleagues who approved a policy or whether that policy still applies. If the question is how the assistant reads and writes during work, the Hindsight agent-memory guide covers that separate workflow.
How can you tell whether recall is useful for your app?
Use a small collection of fictional records with answers you can check. Include similar names, an outdated instruction, a correction, a missing answer and a question about the order of events. Write the expected result before testing so a fluent answer does not quietly become your definition of success.
| Test | What to inspect |
|---|---|
| Two customers request similar equipment | Whether the answer uses the intended customer’s source. |
| A requirement changes | Whether the current answer uses the replacement and explains any relevant history. |
| The requested fact was never supplied | Whether the app admits the gap instead of filling it with a plausible detail. |
| A source contains a proposed rule | Whether the answer labels it as a proposal rather than an approval. |
Record response time and the material returned as well as the final answer. A memory setup has to fit the task’s waiting time and review process. Repeat these examples after changing the model or configuration, using the same expected answers. Portable AI memory also requires preserving meaning when the assistant changes.
Where should decisions and change history live?
Memory extraction can help an application find something that happened. An approved decision has an additional requirement: a person chose it, for a stated scope, and later work needs to respect it. Keep a proposal separate from a decision and a one-time exception separate from an ongoing rule.
For example, an assistant may suggest allowing overnight returns. A manager’s approval is a different event. Record who decided, the reason and when it applies, then make the current answer easy to find. Our guide to AI memory for teams explains why shared ownership and correction matter here.
In fenbs, people decide, assistants can record their decisions and History attributes each change. That gives the team a visible work record to consult alongside any memory engine it builds; see how it works.
Keep the explanation behind a changed answer
Record the work in fenbs with its tasks, decisions and progress. History shows who changed what, while Decisions and rules preserve what a person agreed. Start with one decision that would otherwise be buried in a conversation.
Create a free accountFree for one person · No credit card · Works with Claude, ChatGPT, Cursor and more
How to do this with fenbs
- Create a free account, free for one person with no credit card, and add the project whose work needs a shared record.
- Add a task for evaluating your memory setup. Put its intended behavior in the note and its comparison steps in the plan.
- Record the person’s choices in Decisions and rules, including which sources may be saved and who reviews corrections.
- Connect a supported assistant using Integrations. MCP is a standard way for an AI app to use an outside tool; the connection uses a browser sign-in.
- Ask the assistant to read project context before working and save progress before stopping. Record what was checked, what failed and the next step.
fenbs is not the right fit when you need a memory engine embedded in your own application, document indexing or vector search. It keeps the shared work surrounding that application. You can evaluate Hindsight for product memory while keeping the decisions and unfinished work understandable to the next person.