Originally presented as a live talk on January 7, 2026
Background
So this is a survey paper about agent memory, but to lay the groundwork I’d like to start with human memory.
In common parlance I think most people have long-term memory in mind when we talk about human memory. Folks with a bit of psych knowledge will likely distinguish between short-term and long-term, and will probably only distinguish those two by time, maybe like remembering what you had for breakfast as a short-term memory - which is wrong by the way.
The reality is more complex. For one, short-term memory is better described as working memory, which is the number of things you can keep in your head at once. A very simple test of someone’s working memory is giving them some random numbers or words and then asking them to immediately repeat them back. Most people can do 5-7. That working memory is a fundamental constraint on how much information you can take in and process and potentially convert into long-term memory. It also gets used in basically all cognitive activity, like reading or playing video games or driving a car. Stuff outside the working memory gets lost.
For our purposes, the context window is the working memory of the agent. That includes the kv cache, which we’ll talk about in a minute.
Now for long-term memory, we have a granular breakdown here that doesn’t map directly onto agents, like there’s no one component that has to store facts vs another that has to store tasks. But keeping all the different versions of long-term memory in mind is going to help us pick which form of memory to augment our agent with.
One insight this chart doesn’t capture is how long-term memory complements working memory. The more concepts and ideas you have in long-term memory, which has indefinitely high capacity, the more powerful your working memory can be, since it can manipulate those preexisting concepts and ideas rather than constructing them on the fly from scratch.
A good example is math. If you already know how to add automatically, like without any working memory effort, then you can learn to multiply by using the working memory to process the concept and do the problem. But if you’re not good at adding, or you only know how to count, then you’re going to fill up your working memory before you can start to grasp multiplication.
In agent world, if your agent has a memory of a certain skill or process let’s say, then you don’t have to spend precious context window explaining it and giving examples. Of course if your memory mechanism is just retrieving text, then you don’t get any gains, but as we’ll see there are other forms of memory.
Now we kind of glossed over this, but LLMs and agents don’t intrinsically have memory. That’s part of what makes benchmarks and leaderboards work, you can ask the same question repeatedly to test the model - no risk of memorization, unless the benchmark is in the training data of course.
But a lot of chatbot websites do have memory. For example, ChatGPT rolled out memory to all users by default in April 2024. The way it works is there’s a system that reviews your responses for facts and preferences, then records those as text in a database. The LLM itself does not remember anything, it just gets access to this database that the memory system is building and refining in the background. The memory system is surely also ML-based, but it isn’t the GPT you’re talking to.
Then when you’re chatting in the future, it probably does a combination of system prompting and RAG. In other words, it’s going to put some memories at the top of every conversation, but for others it will fetch based on context. So let’s say you mentioned once you like concise responses and the memory system captures that. Now every chat might say in the system prompt that the user prefers concise responses, which will guide the model’s behavior and also could come up in conversation. But let’s say you also noted you’re a vegetarian, and in one chat you’re asking for restaurant recommendations in a foreign city. The memory system will find that note about you being vegetarian and provide it to GPT so you get more relevant results.
Of course that’s in addition to the memory of past conversations, where the model can look through earlier chats and get caught up. But that’s more like a Google search than proactive memories.
Anyway, this type of memory is pretty rudimentary. It’s just a certain type of tool use at the end of the day, reading and writing to this little database. And it’s only capturing facts, which as we saw from our chart of memory types is a small slice of what memory entails.
So memory in consumer chatbots is kinda new, but the idea of supplying external information to a model at inference time is not. Other than just typing in whatever information you want to provide as a prompt, the most established method for providing relevant information is RAG: retrieval augmented generation.
The basic idea is simple: take a bunch of data, put it into a form you can search, and then run a search with your prompt to get any relevant information. Kind of like doing a Google search and looking at the top search results before writing your own response to a question.
Of course if you have a bunch of relevant information in front of you, that’s going to help your response. The tricky parts are how to put the data into a form you can search, and doing a good job searching.
The first part is at the top here: chunking, which means splitting up long documents into much smaller pieces; and embedding model, which turns the chunks of text into the same type of vector a model uses when it takes in text. The original chunk, plus the vector that represents its meaning, go into the vector store.
The second part is the connection between the top and the bottom, where you turn the prompt into a vector with the embeddings and then compare the prompt vector with the chunk vectors. The more similar the prompt and chunk vectors, the more relevant the chunk is to the prompt. The most relevant ones come back as text, then the LLM responds based on the prompt and the retrieved text.
The original RAG paper is from 2020, over two years before ChatGPT came out, so the memory solutions we’ll see often descend from or still use RAG.
I also want to quickly recap agents. The term “agent” has picked up a lot of cruft and buzz, but it’s really pretty simple: it’s any system that takes actions in an environment and gets observations back from the environment, in a loop until it has completed its task. Usually when people talk about agents nowadays they mean an LLM with access to tools that has a system prompt telling it to be agentic, like to complete tasks for the user.
So in this framework, an “action” for our purposes is going to be using a tool, like web search or file access etc. Requesting or writing memories could be an action, and receiving a memory back could be an observation. However, not all agentic memory is tool use.
Now for the most technical parts, we need to get familiar with embeddings and the kv cache.
First we have to recall what a Transformer-style LLM is, which is a series of matrix multiplications. Training a LLM is just adjusting the numbers in these matrices. Using a LLM is just turning words into numbers, into vectors, that you multiply through all these matrices.
One of those matrices is the embeddings, which turns input tokens into vectors. Each token in the model’s vocabulary will have a unique place in the embeddings, a unique vector that pops out when you plug the token into the embeddings. You can also reverse the process, feeding in a vector and getting out the corresponding token, which is what happens at the end of the LLM when you’re ready to switch back from numbers to words.
While tokens are discrete though, embeddings space is continuous. So if you’re doing that reverse process and you just pick a series of numbers to go in the vector, it’s pretty unlikely it will match up exactly to one token. But that vector will still have a meaning - it will just be a mixture of meanings that doesn’t map cleanly to one single token. Through a certain lens, you could view that mixture of meanings as a kind of memory. And if you wanted to save that memory you could make a special new token that maps to that mixed vector.
I think of this sort of like how a smell can instantly evoke something deep and subtle in a way that’s hard to describe, or that words couldn’t exactly access. The meaning is there but sometimes it takes a special input to activate.
Now for the kv cache.
So in the attention part of the LLM, you take your input and multiply it into three different quantities: the query, the key, and the value. That’s what the middle section is showing, that each input token gets turned into Q K and V.
The actual formula for attention is also in this section: Q times the transpose of K, with a transformation called a softmax. That tells you how important each combination of input tokens is. Then that whole thing multiplied with V to give the meaning of that combination.
The key fact we need to focus on is that to generate the next token, you only need the query for the last token, but you need the key and value for all the prior tokens. So that means if you keep the keys and values in memory, you don’t need to compute them again. That is the kv cache.
The kv cache is like the model’s conscious state of mind, carrying over and changing with each computation. But when you turn off the model the kv cache goes away, like when you go to sleep and lose consciousness.
Since the kv cache is where information about past tokens lives and persists, if you change the kv cache you’re changing the model’s understanding of prior inputs. The normal way the kv cache changes is by continuing to process inputs, like your normal stream of consciousness changes as you continue experiencing. But if we go in there and fiddle with the kv cache in a different way, we can alter the model’s “state of mind” and “thoughts” about the past - its memory.
The Paper
So first we should clarify what agent memory entails, because it does overlap significantly with many other concepts.
First is our old friend RAG, which we covered before. The general idea of reading and writing to a database works for agent memory too, so the main difference is what goes in that database. Traditional RAG would be a static, external corpus like a bunch of company documents, whereas agent memory would be stuff the agent chose to remember or was given to remember, usually with some way of updating and consolidating the stored memories.
Next is context engineering, which is the science of optimizing the context window. Using our human analogies from before, that’s going to be the working memory. So deciding what goes in the context window, how it’s structured, when to remove it maybe, when it’s about memories in the context window that’s going to be agent memory. Of course there can be other things in the context window that aren’t memory, like tool use, so that’s in the non-overlapping part of the Venn diagram.
Finally there is LLM memory, which again will have tons of overlap because the LLM is the heart of the agent. Prompting strategies and KV cache stuff is shared, but LLM architecture stuff like linear attention isn’t related to agent memory. Basically anything intrinsic to the LLM falls under LLM memory, while any ancillary system or technique will be part of agent memory.
So as you can see from this graphic, there is a shitload of agent memory techniques. We are absolutely not going to go over every one of them, but we’ll talk about their categories and maybe mention a few of the most clever techniques.
Let’s start with the categories in red, the “functions” as the graphic calls them. We already saw these in the human context so there’s not much to add here, just reviewing. Factual memory is gonna be facts and figures. Experiential memory is gonna be principles, strategies, tactics, tips, and tricks. And then working memory will be anything in the current state like the chain of thought or the kv cache.
The other axis, forms, uses LLM-specific terms but has human analogs.
Token-level is like taking notes, assuming for now that our model isn’t multimodal. That is by far the most common and most researched memory technique. It’s also nice because it’s legible to humans, you can go in and see what the agent is remembering.
Parametric is like actually forming a memory in humans. It’s in the model’s weights, or the weights of an adapter or ancillary model. That’s great for memories you want globally incorporated, like a tone or a strategy etc, and it’s going to be faster than token-level. The major downside is it’s illegible, although in some cases that could be useful.
Latent is like your current state of mind. I will spend a bit more time on this later because it’s the least intuitive and most technical of the three.
They don’t show it in this graphic but they also have categories for structure, like unstructured vs a graph or tree vs a pyramid of increasing abstraction. I’m going to ignore that one for the rest of this presentation but if you’re interested it’s in there.
So to expand on the Latent form, there are really two different mechanisms you can use.
One is the embedding space. As I mentioned in the background slides, embeddings encode meanings into vectors. That means any vector with the same size as the embeddings matrix will have some sort of meaning. In the case of tokens, we have a pretty good idea of what the vector version means, because we know what the token means, like if the token is “dog” then we know what the vector is going to represent.
So the trick with the latent embeddings is to take something more complicated, embed it, and save that vector for later. In this drawing they show an auxiliary model producing that vector or those vectors for later, the bit labeled “latent embeddings”. Then when you use your main model, you pass in your prompt AND the latent embeddings, and suddenly your model gains these memories, these extra meanings as context with the prompt.
The other mechanism is the kv cache. Again, it’s like a state of mind, except in models you can swap them in and out instantly. If all you do is pop in a saved kv cache from before, that’s what they call Reuse. If you’re somehow fiddling with a saved kv cache, they call that Transform. Either way, if you swap in a previous state of mind with all these memories, again the prompt is going to gain new context.
In both cases, you’re using the power of models to distill context into a more compact representation. That’s really helpful if your source material is highly compressible but doesn’t contain obvious stuff to trim out like boilerplate text; you let the model do the compressing once and then reuse it many times.
One other thing to note is that compressibility depends on the model. Like think about two college students taking the same upper-level math class. If one of them is a freshman who doesn’t have the prerequisites and another is a math grad student just filling in a hole in his knowledge, the freshman is going to take way more notes because he can’t rely on the shorthand of earlier or related concepts. Whereas the grad student probably will take a few really compact notes that only he can decipher because they draw on his richer background knowledge.
Here’s a bit more about the relative advantages and best uses of each memory form.
For token-level, there are a few advantages and use cases. First, a lot of data natively is token-level, like anything you’d keep in a database, or facts and figures, reference material etc. Just like how a human doesn’t need to memorize or internalize things like dates or addresses, agents can use this external memory effectively. Second, it’s going to be legible as we mentioned. And third, it’s by far the simplest to implement.
For parametric, the best fit is broad, conceptual, implicit knowledge. They mention role-playing on here, that’s an easy example since character traits are going to be broad and kind of filter everything, not something to look up at clearly defined times.
For latent, as a middle ground between token and parametric you get some of the good and some of the bad of each, but the main way to think of it is a more efficient representation of some prior input. So it’s going to shine in resource-constrained environments. And as they mention it’s a better fit for multimodal because it can fuse the modalities, whereas for tokens you need separate tokens for each modality.
Now regardless of form, you have to deal with memory formulation, evolution, and retrieval.
Formulation means turning raw data into memories, as the graphic notes. That’s probably the most intuitive of the three here and frankly isn’t that specific to agents, like there’s already a lot of best practice about data structures.
Evolution is more challenging, as anyone who has maintained a knowledge base or wiki or any form of documentation knows. They have another graphic about that so we’ll save it for the next slide.
Retrieval is the most agent-specific in my opinion. Like human memory is an unconscious process so it’s a little funny to have to design it for agents.
When to retrieve is the biggest one. Like if the user says “Hello” that probably requires no memory, but if they want help planning a vacation then clearly you want to know how old they are, whether they have kids etc. But you can’t enumerate that whole list of course. Heuristics work okay, retrieving every time and then just letting the agent ignore everything if it’s not helpful is fine but wasteful, so letting the agent decide is often the way to go. And then you can use RL to improve the decision making.
What and how to retrieve is also interesting. The naive approach is to use the prompt directly as a search query, but people have developed way fancier methods: rewriting the prompt to be a better query, breaking down the prompt into its elements to be sub-queries, writing a hypothetical document based on the prompt that then serves as a query. You can also use a sub-agent just to do the retrieving, letting it iterate and experiment until it’s satisfied. All depends on your quality, speed, and cost requirements.
You also need to process the results before sending them back to the agent. Like if there are duplicates or near-duplicates for example, or conflicts. You also may want to rerank or filter down to save on context or tokens.
Jumping back to memory evolution, there are tons of techniques for consolidating, updating, and forgetting memories. Again, all unconscious work we take for granted in our brains that needs designing for agents.
Consolidating means taking new information and cleaning it up, then putting it into the right place in the existing body of memory. After that you have to check the rest of the body for conflicts or updates, and decide how you want to resolve them: deletion, archive, timestamping etc.
Now eventually all minds fill up, and some forgetting occurs. How to forget gracefully usually depends on the case, but time limits, frequency minimums, and value filters all work, often in combination.
The paper wraps up with a peek into the future, including this somewhat confusing graphic.
One thing they look forward to is improved memory generation, like taking the base information from the memory store and enriching it on the fly, then storing it enriched later on. For example, if it’s a data analysis agent and it tends to have GTM folks as users, it might add new sales strategies to its memories about key tables.
More autonomous management is also on the table. Instead of relying on rules like a time cutoff for forgetting or a priority in conflicts, the agent can make the decision using its own intelligence and the rest of its memories.
Of course once you introduce an ability, you can start improving on it with RL. In the long-term I would expect, and the authors note on this graphic, no human prior on the memory system - something optimized for machine intelligence, not human intelligence. They call RL-driven memory “the next major stage”.
Multi-agent memory is a relatively new challenge, now that agents are actually practical. Here I do expect human priors to provide guidance for a while, since there is a lot of best practice and tooling for sharing knowledge in organizations. Again though, the optimal system for agents to coordinate will probably look strange to humans.
Relatedly, one concern now that will only grow over time is auditability, traceability, legibility - anything that allows humans to understand and debug agents related to memory.
So What?
