An agent that remembers by pasting its entire history into the next prompt does not have memory. It has a growing bill, a hard token ceiling, and a latency curve that gets worse with every turn. Real memory is a storage decision: what you persist, where you put it, and how you fetch only the relevant slice back at inference time. That is a database problem, and treating it as a prompt problem is how agents that demo well fall over in week two.

The seductive thing about context stuffing is that it works at first. Early in a session the history is short, everything fits, and the agent looks like it remembers. Then the conversation grows, the prompt grows with it, and three things happen at once: cost climbs because you resend everything every turn, latency climbs because the model reads more each time, and eventually you hit the context window and older facts silently fall off the front. The failure is not a bug you can patch. It is the architecture.

Two kinds of memory, two kinds of store

"Memory" is really two different needs, and conflating them is where designs go wrong.

Session and working memory: a key-value store

The recent turns, the current task state, the user's short-term preferences, these are looked up by a known key: a session ID, a user ID, a thread ID. You want fast reads and writes on that key and a time-to-live so stale sessions expire on their own. That is a DynamoDB table, not a vector search. Partition on the session or user, keep the running state as an item, set a TTL, and read it back at the start of each turn. No embeddings, no similarity, just a fast keyed lookup of "where were we."

Long-term semantic memory: a vector store

The facts an agent should recall across sessions, prior decisions, learned preferences, relevant past exchanges, are not looked up by key. They are looked up by meaning: "what do I know that is relevant to this new message." That is a semantic search over embeddings, which is what a vector store is for. On AWS the pragmatic choice is Aurora Serverless v2 with pgvector: your embeddings live next to relational data, you query them with SQL, and you are not standing up a separate specialized database to do it.

The pattern: retrieve, do not accumulate

Once memory is a database, each turn stops appending and starts querying. The loop becomes:

on each turn:
  1. read session state by key      (DynamoDB: where were we)
  2. embed the new user message
  3. semantic search long-term store (pgvector: what is relevant)
  4. assemble a bounded prompt:
       system + tools
       + top-k retrieved memories
       + recent turns from session state
       + new message
  5. call the model
  6. write new facts back to the stores

The prompt is now bounded no matter how long the relationship with the user runs. A conversation on its thousandth turn sends the same size prompt as its tenth, because you fetch the relevant handful of memories instead of carrying all of them. Cost and latency go flat instead of climbing. This is the same retrieval discipline that makes RAG work, applied to the agent's own history instead of a document corpus.

What you have to decide, that a prompt hides

Moving memory into a database forces choices context stuffing let you ignore, and those choices are the actual engineering:

  • What is worth remembering. Not every turn is a memory. Write durable facts and decisions, not chit-chat, or your store fills with noise that retrieval then surfaces.
  • When to forget. TTLs on session state, and a policy for aging out or superseding long-term facts, so an old preference does not outrank a corrected one.
  • How much to retrieve. Top-k is a knob. Too little and the agent forgets; too much and you are back to a bloated prompt with the retrieval step adding no value.
  • Consistency across the two stores. Session and semantic memory can disagree. Decide which wins when they do.

The takeaway

Context stuffing is not memory, it is deferring a storage decision until the token bill and the context ceiling make it for you. Split the problem: fast keyed session state in DynamoDB, semantic long-term recall in pgvector on Aurora, and a per-turn retrieval step that assembles a bounded prompt from both. The moment memory becomes a database with a retrieval step instead of an ever-growing prompt, cost and latency stop scaling with conversation length, and you get to decide what the agent remembers instead of letting the window decide what it forgets.

Read this next

For the database and infrastructure side of running Aurora and DynamoDB at scale, the cloud field notes live at ercan.cloud, and the hub is at ercanermis.com.