AI Agent Memory: What to Remember, What to Forget, and When to Ask Again

An AI assistant remembers that you prefer window seats.

That feels useful. The next time you book a flight, it does not make you repeat yourself.

Six months later, you are travelling with a broken ankle. The assistant silently applies the old preference, books a window seat, and makes the trip harder.

The memory was accurate when it was created. It was relevant to the task. It was also wrong at the moment it mattered.

This is the part of AI agent memory that gets less attention. Storing information is easy. Deciding when that information deserves to influence a future action is the real product problem.

For me, good agent memory is not a giant archive of everything the user has ever said. It is a small, inspectable system that can answer five questions:

1. Why was this fact saved?
2. Where did it come from?
3. How long should it remain useful?
4. What happens when newer information conflicts with it?
5. Should the agent trust it now, or ask again?

If a memory system cannot answer those questions, it is not really memory. It is old context with authority.

Memory is not the same as conversation history

The word “memory” is often used for several different things.

A chat application may keep the last twenty messages and call that memory. An agent framework may persist every tool call in a session. A product may extract user preferences into a profile. A coding agent may write a progress note before starting a new context window.

All of those mechanisms carry information forward, but they solve different problems.

I separate them like this:

• Working context is what the model can see during the current run.
• Conversation history is the ordered record of previous turns in a thread.
• Task state is the structured progress of a workflow: completed steps, pending approvals, files changed, or the current checkpoint.
• Long-term memory is selected information that may be useful in a future conversation or task.
• Live state is information that belongs to another system and should be checked at the source when needed.

This distinction matters because the storage rule for each one is different.

Conversation history may be retained for continuity. Task state may need strong consistency and explicit versioning. Long-term memory needs selection and expiry. Live state may not belong in memory at all.

The OpenAI Agents SDK session documentation is a useful concrete example of conversation history: a session loads earlier items before a run and stores new items afterward. It also allows applications to limit or filter the history passed into the next run. That is session continuity, not automatically a trustworthy profile of the user.

Keeping those concepts separate prevents a common mistake: treating every old message as a fact worth remembering forever.

A transcript is evidence, not truth

People speak casually.

They explore ideas, change their minds, make jokes, paste someone else's text, describe hypothetical situations, and correct themselves three messages later.

If an agent turns every sentence into permanent memory, it will eventually build a confident but fictional version of the user.

Consider these messages:

I might move to Berlin next year.

My brother is allergic to peanuts.

Let's assume our budget is $20,000 for this exercise.

I hate weekly reports when they are just status theatre.

None of them safely becomes a universal fact such as “lives in Berlin,” “has a peanut allergy,” “company budget is $20,000,” or “never wants weekly reports.”

The message is real. The interpretation may be wrong.

That is why I prefer memory extraction to be conservative. The agent should save less than it can extract. When the meaning is ambiguous, keeping the original message in conversation history is often safer than promoting an interpretation into long-term memory.

The five kinds of information I treat differently

One memory policy for everything is too blunt. I use different rules for different kinds of information.

Working memory for the current run

Working memory contains the details needed right now: the user's request, the current plan, recent tool results, relevant documents, and intermediate outputs.

Most of it should disappear when the run ends.

A raw search result, a temporary file path, or the output of a one-time database query may be essential for the current step and useless tomorrow. Persisting all of it creates noise, cost, and risk without improving the next task.

Session memory for conversational continuity

Session memory lets the user say “make the second option shorter” without explaining what the second option was.

It should preserve the local thread: recent messages, decisions made in the conversation, unresolved questions, and tool outcomes that still matter.

Long conversations eventually need pruning or compaction. The goal is not to keep every token visible. It is to preserve enough continuity that the next turn makes sense.

Anthropic describes this broader problem as context engineering: the work is not only writing a prompt, but choosing which information deserves a place in the limited context available to the model. Its practical techniques include compaction and structured notes for long-running work.

Long-term memory for durable preferences and facts

Long-term memory should earn its place.

Examples may include:

• the user's preferred language;
• their normal measurement system;
• an accessibility need they explicitly asked the product to remember;
• a stable project convention;
• a recurring output format;
• a confirmed relationship between a person and an account or workspace.

Even these facts need scope. “Use concise answers” may apply to work updates but not to legal explanations. “Use TypeScript” may be a rule for one repository, not every future project.

Task memory for work that spans multiple runs

Long-running agents need a reliable handoff to their future selves.

For a coding task, that may include the goal, completed changes, failing tests, open questions, and the next safe step. For research, it may include sources already checked, claims still unsupported, and the current outline.

This is closer to a project checkpoint than a personality profile.

The most useful task memory is usually structured and verifiable. “Implemented authentication” is weak. “Added OAuth callback handling in auth/callback.ts; tests X and Y pass; token refresh remains unfinished” is actionable.

Live state that should be fetched again

Some facts become dangerous when remembered.

Account balance, inventory, subscription status, permission level, delivery address, medication list, exchange rate, appointment availability, and the final price of an order should normally come from the current source of truth.

The agent may remember how to find the information. It should not assume the old value is still valid.

This is the boundary I care about most:

Memory can guide the next check. It should not replace the check when the world may have changed.

Every remembered fact needs a lifecycle

I do not think of memory as a row that is either present or absent. I think of it as a small lifecycle.

A candidate appears. The system decides whether it is useful enough to keep. If it is stored, it receives a source, scope, and time boundary. Later it may be retrieved, confirmed, updated, contradicted, expired, or deleted.

That lifecycle makes memory less magical and much easier to debug.

The test I use before storing a memory

Before promoting a detail into long-term memory, I ask a short series of questions.

Will this likely help with a future task?

Interesting is not the same as useful.

The user mentioning a favourite film once does not justify a permanent memory in an invoicing product. Their preferred invoice currency probably does.

Memory should serve the product's job, not curiosity about the user.

Did the user state it clearly?

Explicit preferences are safer than inferred ones.

“Please always show distances in kilometres” is a good candidate. Inferring that the user dislikes miles because they corrected one conversion is weaker.

When an inference could materially affect future behaviour, ask for confirmation before saving it.

Is it stable enough to reuse?

Some facts are naturally durable. Others decay quickly.

A preferred language may remain useful for years. A travel itinerary may be relevant for one week. A temporary project role may end next month. A restaurant preference may depend on who is joining dinner.

The expected lifetime should influence both storage and retrieval.

Does it have a clear scope?

“Use a formal tone” is incomplete.

Does that apply to client emails, all writing, one project, or only the document being edited now?

A scoped memory is far less likely to surprise the user:

{
  "fact": "Use a formal tone",
  "scope": "client_acme/outbound_email"
}

Would saving it surprise the user?

An assistant remembering the user's language is usually unsurprising. Remembering a passing comment about a medical issue, family conflict, or financial situation may feel invasive even if the information could improve a future answer.

The more sensitive or personal the detail, the stronger the case for explicit consent, narrow scope, short retention, or no storage at all.

Store the source, not only the conclusion

A memory without provenance is difficult to trust and difficult to correct.

At minimum, I want a long-term memory record to carry:

• the normalized fact;
• the source event or user statement;
• who or what created it;
• when it was created;
• when it was last confirmed;
• its scope;
• an expiry or review time when relevant;
• whether it was explicit, inferred, or imported;
• sensitivity and access rules;
• active, disputed, expired, or deleted status.

For example:

{
  "key": "flight.seat_preference",
  "value": "window",
  "scope": "personal_travel",
  "source": "user_explicit",
  "createdAt": "2026-04-12T09:20:00Z",
  "lastConfirmedAt": "2026-04-12T09:20:00Z",
  "reviewAfter": "2026-10-12T00:00:00Z",
  "sensitivity": "low",
  "status": "active"
}

The dates do not make the fact true. They make its age visible.

That gives the product a better choice than blindly using or deleting it. The agent can say, “I have window seat saved as your preference. Is that still right for this trip?”

Confidence should not become fake precision

It is tempting to attach a score such as 0.87 to every memory.

Sometimes a ranking score is useful internally. But a decimal can make a vague inference look scientific.

I prefer understandable states:

• explicit: the user directly asked the product to remember it;
• confirmed: the system repeated it and the user agreed;
• observed: it came from a reliable product event;
• inferred: the system derived it from behaviour or language;
• disputed: newer information conflicts with it;
• expired: it should not influence action without a refresh.

Those labels tell the product what to do. “Inferred and six months old” suggests asking. “Explicit and confirmed yesterday” may be safe to apply in the same scope.

New information should not silently overwrite old information

People change their minds. Systems also receive contradictory data.

Suppose the user once said, “Send invoices to my personal email,” and later says, “Use the finance address for invoices.”

The system could replace the old value. But for an important action, I want a short audit trail:

• what was active before;
• what new evidence arrived;
• which value became active;
• whether the user confirmed the change;
• which future workflows will use it.

This is especially important when multiple channels can update the same fact. The latest timestamp is not always the most authoritative source. A casual chat message should not necessarily override an administrator-controlled billing setting.

Conflict resolution needs source priority, scope, and sometimes a human decision.

Retrieval is part of memory quality

Even a clean memory store can produce bad behaviour if retrieval is careless.

The agent should not receive every remembered fact on every turn. That wastes context and introduces irrelevant influence.

When selecting memories, I consider:

• relevance to the current task;
• scope match;
• freshness;
• source quality;
• sensitivity and the current agent's permissions;
• whether the fact conflicts with newer data;
• the cost of being wrong.

This is closely related to context optimization. The goal is not maximum context. It is the smallest set of trustworthy information that helps the model make the next decision.

A preference for concise status updates may help when drafting a project summary. It should not appear in the context for checking a payment status.

Summaries can preserve mistakes

Compaction and summaries are necessary for long conversations, but they have a subtle failure mode.

A summary turns many events into a smaller interpretation. If that interpretation is wrong, future runs may no longer see the original evidence needed to correct it.

For example, a conversation contains:

1. the user considers cancelling a trip;
2. the agent discusses cancellation fees;
3. the user decides to keep the booking.

A weak summary says: “User wants to cancel the trip.”

The summary is shorter. It is also wrong.

For important state, I keep structured decisions separate from narrative summaries. The summary can help with orientation, but the actual booking status should come from the travel system, and the final user decision should remain a distinct event.

Forgetting is a product feature

Memory discussions often focus on recall. I think forgetting deserves equal design attention.

Information should leave active memory when:

• its time to live has ended;
• the user deletes it;
• its source is no longer valid;
• a newer confirmed fact replaces it;
• the product no longer needs it;
• the account, project, or relationship that defined its scope has ended;
• policy or consent no longer permits its use.

Expiry does not always mean immediate physical deletion. A system may need a short audit or recovery window. But expired information should stop influencing the agent by default.

Current agent tooling increasingly exposes this idea directly. For example, encrypted sessions in the OpenAI Agents SDK support encryption and time-to-live behaviour for stored session items. The specific implementation will vary, but the product principle is broader: stored history needs security and an exit path.

Users need a way to see and correct memory

Invisible memory creates invisible frustration.

When an agent repeatedly uses an outdated preference, the user should not have to discover the exact phrase needed to make it forget.

For meaningful long-term memory, I want the product to support:

• “What do you remember about me?”
• editing a saved fact;
• deleting one memory without erasing everything;
• disabling memory for a conversation or category;
• seeing why a fact was used;
• knowing whether a correction applies to one task, one project, or the whole account.

The interface does not need to expose an internal vector database. It needs to expose understandable facts and controls.

The best correction flow is immediate. If the user says, “That is no longer my address,” the product should fix the underlying memory, not merely apologize in the current answer.

Sensitive memory needs stricter boundaries

Long-term memory concentrates personal information.

That makes authorization, encryption, retention, deletion, tenant isolation, and auditability part of the feature—not infrastructure details to add later.

I treat these as separate questions:

1. Is the agent allowed to read this memory?
2. Is it allowed to use the memory for this task?
3. Is it allowed to reveal the memory in its response?
4. Is it allowed to update or delete the memory?

Those permissions are not equivalent.

A scheduling agent may use a private accessibility preference to avoid unsuitable venues without repeating the sensitive detail in a group message. A support agent may see an account identifier but should not expose it to another user in the same shared workspace.

Memory retrieval should pass through the same authorization model as the source data. A session ID or user ID is a lookup key, not proof that the requester is allowed to read the stored history.

Memory can be poisoned

Agents do not receive information only from trusted users.

They read documents, web pages, tickets, messages, tool outputs, and content written by other people. If any of that content can create long-term memory, an attacker may be able to plant instructions or false facts that affect future runs.

I do not allow retrieved third-party content to become authoritative memory by default.

The memory write path should know:

• whether the source is the account owner, another user, a tool, or external content;
• whether the source is allowed to update this field;
• whether the content is a fact, an instruction, or merely text being processed;
• whether a confirmation step is required;
• which agents may retrieve the result later.

This is the same basic lesson as tool access: capability is not authority. In MCP Is Not the Product, I argued that connecting an agent to a tool does not decide when it should act. Giving an agent a memory store does not decide what it should believe.

Shared memory needs ownership rules

Multi-agent systems make memory boundaries harder.

Should a research agent see preferences learned by a support agent? Can a coding agent update a project decision written by a product agent? Does a delegated agent return notes to the parent, write directly to shared memory, or both?

I prefer explicit ownership:

• personal preferences belong to the user profile;
• project conventions belong to the project;
• conversation continuity belongs to the thread;
• workflow checkpoints belong to the run or task;
• operational facts remain in their source systems;
• delegated agents return proposed memory updates to the parent unless they have narrow permission to write directly.

This prevents one specialised agent from casually changing the worldview of the whole system.

It also makes incidents traceable. The memory record should identify which agent, workflow version, and source event created or changed it. That information belongs in the same observability story described in AI Agent Observability.

One stale memory can change an entire workflow

Imagine a purchasing agent that remembers a project manager's approval limit as $5,000.

The limit was correct last quarter. The company later reduced it to $2,000, but the agent keeps the old value in long-term memory.

A new purchase request arrives for $3,500.

The agent retrieves the old limit, skips escalation, and prepares the order. The model behaves consistently. The tool call is valid. The workflow is still wrong.

The fix is not a better prompt.

Approval authority is live organisational state. The agent may remember where to check it, but it should fetch the current limit before deciding. If the lookup fails, it should stop or ask for help rather than fall back to an old value.

This is where memory design, human approval, and safe action meet. The higher the cost of stale information, the less authority memory should have. The approval patterns in Human-in-the-Loop AI Agents are especially important when remembered context changes who, what, or how much an agent is allowed to affect.

What I measure in a memory system

Storage size is not the main quality metric.

I care more about:

• how often a retrieved memory helps complete the task;
• how often the user corrects a remembered fact;
• stale-memory incidents;
• conflicts between memories and live sources;
• memories retrieved but ignored by the agent;
• sensitive-memory access denials;
• expiry and deletion completion;
• duplicate or contradictory facts per scope;
• how often the agent asks for confirmation at the right time;
• task quality with and without retrieved memory.

That final comparison matters. A memory feature can feel sophisticated while quietly making answers worse.

I also trace memory reads and writes. For each run, I want to know which memories were considered, which ones entered model context, which fact influenced an action, and whether any memory was created, changed, or deleted.

A practical architecture for a small team

You do not need a universal memory platform on day one.

A useful first version can have four pieces:

1. Session history for recent conversational continuity.
2. Structured task state for resumable workflows.
3. A small long-term memory table for explicit, scoped facts with source and expiry metadata.
4. Live tool lookups for anything operational or safety-critical.

Add a retrieval layer that filters by user, tenant, scope, permission, freshness, and relevance before any memory reaches the model.

Add a write policy that is stricter than the read policy. The agent may propose a memory, but sensitive or ambiguous facts can require user confirmation before they become active.

Then test real failure cases:

• the user changes a preference;
• two sources disagree;
• an old memory expires during a long task;
• a delegated agent proposes a bad update;
• a user deletes a memory;
• live data contradicts memory;
• the memory store is temporarily unavailable;
• a shared workspace user tries to access someone else's memory.

The unavailable-store case is important. The agent should be able to say “I cannot access your saved preferences right now” rather than inventing continuity.

What I check before shipping agent memory

Before I trust a memory feature, I want clear answers to these questions:

• What exact problem does each memory type solve?
• Which information is temporary, session-level, task-level, or long-term?
• Which facts must always come from a live source?
• Can every long-term memory point back to its source?
• Are inferred memories visibly different from explicit ones?
• Does every memory have a scope?
• Which memories expire, and what happens after expiry?
• How are conflicts resolved?
• Can users inspect, correct, and delete meaningful memories?
• Do authorization checks happen before retrieval?
• Can external content create memory without confirmation?
• Are memory reads and writes included in traces?
• Can a failed memory service degrade safely?
• Do our evals include stale, conflicting, poisoned, and deleted memory cases?

If those answers are vague, the product probably has storage but not a memory design.

Conclusion

An agent that remembers everything will not feel intelligent for long.

It will repeat old assumptions, carry mistakes across conversations, surface personal details in the wrong context, and use yesterday's state to make today's decision.

Useful memory is selective.

Keep conversation history for continuity. Keep task state for reliable progress. Save only durable, scoped facts that are likely to help again. Record where they came from. Give them an expiry path. Let the user correct them. Fetch changing facts from the systems that own them.

And when a remembered fact is old, uncertain, sensitive, or expensive to get wrong, ask again.

That small moment of uncertainty is not a failure of intelligence.

It is what makes the system trustworthy.

FAQ

What is AI agent memory?

AI agent memory is the information an agent can carry across steps, runs, or conversations. It may include recent conversation history, structured task state, or selected long-term facts and preferences. These should be managed separately because they have different lifetimes and trust requirements.

Should an AI agent remember every conversation?

No. Full history can help with continuity, but not every message is a durable fact. People speculate, change their minds, and discuss temporary situations. Long-term memory should be selected conservatively and kept within a clear scope.

What is the difference between context and memory?

Context is the information available to the model during the current call. Memory is information stored so it can be retrieved later. A memory becomes part of context only when the system selects it for a particular run.

When should an agent ask again instead of using memory?

It should ask or verify when a fact is old, inferred, disputed, sensitive, outside its original scope, or costly to get wrong. Operational facts such as permissions, prices, balances, inventory, and approval limits should usually be checked at their live source.

How long should agent memory last?

There is no universal duration. The lifetime should match the information: a temporary task detail may last minutes, a trip preference may need confirmation for each journey, and an explicitly saved language preference may remain useful for years. Every category should have a review, expiry, or deletion rule.

Can vector databases be used for agent memory?

Yes, especially for similarity-based retrieval, but storage technology does not solve memory policy. The system still needs source tracking, scope, freshness, authorization, conflict handling, deletion, and rules for deciding when retrieved information is safe to use.