Why agents need memory

An LLM is stateless — by default it remembers nothing between calls. For a multi-step task, that's a problem: the agent has to know what it already tried, what it learned, and what the goal was. Memory is how we give a stateless model continuity. There are two distinct kinds, and conflating them causes a lot of confusion.

Short-term memory

This is the running message history of the current task — every reasoning step, tool call, and result so far. It's what lets the agent build on its own progress within a single run. The catch: it lives in the context window, which is finite. A long-running agent generates a lot of history, and eventually it won't all fit.

The standard fix is summarization: when the history grows large, compress older turns into a short summary and keep the recent detail. The agent remembers the gist of step 2 without carrying its full transcript into step 40.

Long-term memory

This is knowledge that persists across sessions — user preferences, facts learned last week, past results. You can't keep it all in the prompt, so it's stored externally (often in a vector database) and retrieved with RAG when relevant to the current step. The agent "remembers" your name not because it's in the context, but because it looked it up.

Keep it lean

The instinct to give an agent more memory usually backfires. Irrelevant context distracts the model, dilutes its attention, and costs tokens on every call. The discipline is selectivity:

• Store only what's genuinely useful.
• Retrieve only what's relevant to the current step.
• Summarize aggressively.

More memory isn't smarter — it's noisier. A focused agent that recalls the right three facts beats one drowning in fifty irrelevant ones.

Try this: Sketch the memory an agent would need to plan a week of meals: short-term (what it's chosen so far this run) vs. long-term (your allergies and dislikes, remembered across sessions). Separating the two is exactly the design decision real agents make.