Databricks Agent Memory: Sessions, Storage, and Shared Context

Agent memory in a Databricks workflow persists when it is written to a Delta table, a managed store, or another durable layer before the notebook session or job run ends. By default, Python variables disappear when a cluster shuts down or a job finishes.

The choice of persistence layer affects query speed, schema flexibility, and who else can read the memory. Tools like fenbs keep the project brief, task list, and human-approved decisions outside the data platform, so the agent picks up shared context at the start of each run without needing a dedicated Databricks table for it.

Why session state alone is not enough

A notebook session keeps variables alive as long as the cluster is running. Once you detach the notebook, restart the cluster, or hand the job to another engineer, those variables are gone. An agent that stores conversation history or extracted facts only in session variables will start fresh every time the cluster restarts.

The solution is to flush memory to a durable store at the end of each run and reload it at the start of the next. This is a deliberate step, not automatic. Design the flush and reload into your workflow from the beginning, because retrofitting it later means auditing every place where the agent writes something it expects to find later.

Delta tables as a persistent memory store

Delta tables are the most natural persistence layer in Databricks. Write a row for each memory item: a session ID, a timestamp, a key, and a value. Use MERGE to update existing entries without duplicating them. Read back the latest value for each key at the start of a run.

Delta's transaction log gives you a full history of what the agent wrote and when. You can query earlier versions with VERSION AS OF if you need to understand what the agent knew at a specific point in a run. This audit trail is useful when a retrieved fact turns out to be stale and you want to know when it was last updated.

Unity Catalog and metadata context

Unity Catalog stores table schemas, column descriptions, and access policies in one place. An agent with read access to Unity Catalog can discover what tables and columns exist, which helps when the agent needs to generate queries or route data to the right destination. Write short descriptions to Unity Catalog columns and tables so the agent can read them before writing a query, rather than guessing schema from column names alone.

Unity Catalog is metadata, not operational memory. It tells the agent what exists, not what the agent has done. Keep the two separate: Unity Catalog for schema discovery, a Delta table for session history and extracted facts. For a wider look at agent memory design, see what is agent memory and agent memory vs context.

Testing memory recall between runs

Run a four-step check after setting up your persistence layer.

1. Write: run the agent and confirm it writes at least one memory item to your Delta table.
2. Terminate: shut down the cluster or detach the notebook to clear all session state.
3. Restart: start a fresh cluster and run the agent again from the beginning.
4. Read: confirm the agent loads the memory items written in the previous run before doing any new work.

If step four returns an empty result, the agent is not reading the Delta table at startup. Check the read query and the cluster's Unity Catalog permissions. If it returns stale data, check whether the write step is using MERGE correctly or whether it is writing duplicate rows. For a deeper look at memory persistence and retrieval patterns, see how to build agent memory.

Choosing the right memory layer for your workflow

Short-lived memory that only one run needs can stay in a Python dictionary for that run. Memory that must survive a cluster restart goes into a Delta table. Memory that other people or agents need to read goes into a Delta table registered in Unity Catalog with appropriate access grants.

Consider the size of each memory item. A short string key and a JSON value fit easily in a Delta row. A large embedding or a binary file does not belong in a Delta table row; write it to object storage (DBFS or an external bucket) and store the path in the table. See agent memory vs RAG for guidance on when retrieval augmentation is a better fit than direct memory lookup.

How to do this with fenbs

Set up your Databricks memory layer for operational data and use fenbs for the shared project record that travels with the work across sessions and team members.

1. Create a free account and add your project to fenbs with a short goal description and the current tasks.
2. Connect your assistant from /integrations and ask it to call fenbs at the start of each Databricks job to read Rules and Where we left off.
3. Create a Delta table for operational memory with columns for session ID, timestamp, key, and value.
4. Add a flush step at the end of your notebook or job that writes the current memory state to the Delta table.
5. Add a load step at the start that reads the latest memory state into a Python dictionary before the agent does any work.