What Is AI Agent Memory? The Design Decisions That Keep It Safe

A model has no recall between conversations. Everything it knows about you arrives in its context window and is gone when the session ends. Agent memory is the layer that persists something across sessions and puts the right part back in context next time. It is not the model remembering. It is a storage and retrieval system wrapped around a stateless model, and the choices made in building it decide whether the feature is personalization or a privacy incident.

What is AI agent memory?

Agent memory is a persistent store that sits outside the model and feeds selected content back into the context window on a later session. It holds facts about the person, the project, and the history between them. The model itself contributes nothing to it, because the model cannot. Each conversation starts cold, with only the context it receives.

The result is that an agent's continuity comes from the system around it, not from the model. Two agents with identical models behave differently if one has a good memory layer and the other has none. The quality of the persistent store, what goes in, who can read it, and how it is retrieved, is what separates a useful memory from a liability.

Why does a stateless model need an external memory system?

A language model predicts the next token from the context it is given. That context is fixed per request and does not survive the request. There is no hidden state the model carries forward, so any fact about the user must re-enter as text.

That makes memory a retrieval problem, not a storage problem. The hard part is deciding which stored fact belongs in a given conversation and pulling it in without bloating the window.

How does memory work in Askpert?

On Askpert, buyers rent AI Experts built by sellers, and memory is scoped to the pair of the two: what an Expert remembers about you is separate from what every other Expert remembers, and separate from what it remembers about anyone else.

A session with an Expert closes automatically after 24 hours of inactivity, or 7 days at most. Closing triggers an extraction pass that reads the conversation and writes durable memories, capped at 20 per pair and sorted into five categories: relationship, profile, preference, history, and project. The next session starts with an index of what is stored rather than the memories themselves, so a returning buyer gets continuity without paying for it on every turn. The rest of this piece is the four decisions that shaped that design, and each one generalizes past our implementation.

Where does naive agent memory become a privacy incident?

The naive design stores everything the agent saw and lets the agent's author read it. Both choices look reasonable and both are dangerous. If the store captures internal tool results, it can leak the seller's protected assets. If the author can query the store, the personalization feature becomes a surveillance product. The interesting engineering in a memory system lives entirely in what you refuse to store and who you refuse to show it to.

Why must memory not become a laundering channel?

The product promise at askpert.dev is that a seller's private assets, their prompts, documents, and tools, execute server-side and never reach the buyer. An agent's internal tool results pass through the conversation record. If the extraction pass read those raw rows, it would copy fragments of protected seller material into a buyer-owned memory store, and the protection would leak out through the back door.

The fix is to read only the buyer-visible projection of the transcript, exactly what the buyer could already see. Nothing the agent saw internally is eligible to become a memory. This constraint was caught by an adversarial design review before any code was written. The rule holds for anyone building on top of a code agent: never persist what the user could not already observe.

Why can the agent's author not read the memories?

Access is enforced in the database itself with row-level security keyed to the buyer, and it is seller-blind. The seller who authored the Expert cannot query what any buyer's memories contain. The intuitive architecture, where the agent's author owns the data their agent collects, quietly turns a personalization feature into a surveillance product.

Enforcing this at the database layer matters because it survives application bugs. A permission check in application code can be mis-wired on one request path. Row-level security is a property of the data, so a query from the wrong principal returns nothing regardless of how the application calls it.

Why is recall progressive rather than a dump?

The naive design loads every stored memory into the context window at the start of every conversation. That burns context, costs money on every single turn, and buries the relevant memory among nineteen irrelevant ones. Loading twenty memories is largely noise, because a single conversation calls for a fraction of them.

Ours mirrors how skills load progressively. The agent sees a compact index of what is available and pulls the full content of a specific memory only when the conversation actually calls for it. Progressive recall keeps the window lean and keeps the right piece at hand. The index is cheap, and the pull is deferred until it earns its tokens.

Why did Askpert not mine existing conversations?

When the feature shipped, there were existing conversations already in the database. Extracting memories from them retroactively would have meant harvesting personal information from people who had no idea the feature was coming. Consent cannot apply to data collected before consent was possible.

Every pre-existing session was marked skipped. Memory only accumulates from sessions that happened after the buyer could know about it. That is the honest way to roll out a retention feature, and it is the version that does not erode trust on day one.

What control does the buyer get?

A buyer sees a memory page listing everything stored, a global off switch, and a per-Expert mute. That third control matters: a buyer can allow memory with one Expert and refuse it with another, because trust is per-partner, not per-product. The controls are not decorative. They are the point, because memory is the feature where the user's right to see, stop, and narrow is not a courtesy but the thing that keeps it from becoming surveillance.

How should you evaluate an agent platform's memory?

Ask four questions of any platform you are evaluating or building. Does the store capture only what the user could already observe? Can the author of the agent read the memories? Is recall progressive or is it a full dump on every turn? And was anything retroactively mined from before the feature existed? A platform that answers all four narrowly has treated memory as a retrieval system with a privacy model. A platform that hedges on any one of them has a feature that is a privacy incident waiting to happen.