February 11, 2026
By David Szarzynski · Founding Engineer
Your agent works in the demo. It answers questions, runs tools, completes tasks. Then a user comes back the next day, and the agent has no idea who they are.
The conversation buffer is gone. The user repeats themselves. The agent makes the same mistakes it made yesterday. This is the memory problem, and every team building production agents hits it eventually.
Most agent frameworks give you conversation memory within a session. Nothing survives after the session ends. Building persistent memory, the kind that makes an agent useful across days and weeks, requires a different set of engineering decisions than most teams expect.
This post covers the architecture of a persistent memory system, the decisions at each layer, and where the real complexity hides.
The memory problem is the gap between what an agent knows within a single session and what it needs to remember across sessions to be useful.
Frameworks like LangChain, CrewAI, and AutoGen provide conversation buffers that work within a session. Messages accumulate, the LLM sees recent history, and the agent can reference what was said minutes ago. When the session ends, everything disappears. (For a deeper look at memory types, see What Is AI Agent Memory?.)
Persistent memory requires solving four engineering problems simultaneously: storing information durably, retrieving it efficiently, keeping it current as source data changes, and assembling it into context at the right time.
Each sounds straightforward in isolation. The complexity comes from the interactions between them. Your storage strategy constrains your retrieval options. Your retrieval quality depends on your ingestion pipeline. Your context assembly determines whether any of it actually helps the LLM reason better.
A persistent memory system is a five-component pipeline that transforms raw data from conversations, documents, and APIs into context an LLM can use.
Component | Function | Key Decisions |
|---|---|---|
Data sources | Where information originates | Conversations, documents, APIs, user actions |
Processing pipeline | Transforms raw data into storable units | Chunking strategy, embedding model, entity extraction |
Storage layer | Where memories persist | Vector DB, graph DB, relational DB, or hybrid |
Retrieval engine | Finds relevant memories for a query | Semantic search, keyword search, graph traversal |
Context assembly | Formats memories for the LLM prompt | Token budgeting, metadata inclusion, structured formatting |
These components form a chain, and weaknesses cascade downstream. Poor chunking produces poor embeddings. Poor embeddings produce irrelevant retrieval results. Irrelevant results waste your context window and degrade agent performance.
The following sections walk through the decisions in the order you will hit them.
Three primary approaches exist, each suited to different query patterns.
Vector database only (Pinecone, Weaviate, Qdrant). Store document chunks as embeddings and retrieve by semantic similarity. Simple to set up, well documented, fast for similarity search. The limitation: vectors capture meaning but not relationships. You cannot easily answer "who does Alice work with?" from embeddings alone.
Graph database (Neo4j, Amazon Neptune). Model entities and their relationships explicitly. Strong for queries about connections, hierarchies, and temporal sequences, but more complex to set up because you need an entity extraction pipeline feeding the graph. Graphiti, the open-source temporal knowledge graph engine behind Zep, reports accuracy improvements of up to 18.5% on the LongMemEval benchmark by combining semantic search with temporally aware graph retrieval (arXiv 2501.13956).
Hybrid (vector + graph + key-value). Production systems usually combine approaches. Vector search handles semantic similarity, graph traversal handles relationships, and key-value stores handle fast lookups for user preferences and recent context. More infrastructure to manage, but this combination covers the full range of queries a production agent needs.
If your agent needs... | Choose | Why |
|---|---|---|
Semantic search over documents | Vector DB | Fast similarity matching, simplest setup |
Relationship queries ("who works with whom?") | Graph DB | Explicit entity and relationship modeling |
User preferences, recent context | Key-value store | Sub-millisecond lookups |
All of the above | Hybrid | Production agents rarely need just one approach |
Start with a vector database. Add graph and key-value layers when your use case demands them.
What you ingest depends on your agent's purpose. At minimum: conversation history from your own agent's sessions. Beyond that: user documents, plus external sources like email, Slack, CRM, and calendar.
The hard parts people underestimate:
Chunking strategy. How you split documents directly affects retrieval quality. Start with recursive character splitting at 400-512 tokens with 10-20% overlap, which keeps paragraphs and sentences together as coherent units. Semantic chunking (grouping sentences by embedding similarity) can lift retrieval accuracy substantially, up to 70% over naive fixed-size splitting in one published benchmark, but it requires running an embedding model on every sentence. Start simple, upgrade when metrics justify the cost.
Embedding model selection. No single model dominates across all content types. OpenAI's text-embedding-3-large is a strong general-purpose option. Cohere's embed models lead on multilingual tasks. For self-hosted deployments, BGE-M3 offers competitive quality at zero API cost. The MTEB leaderboard tracks current benchmarks, though rankings shift frequently.
Incremental updates. You need to re-index when source data changes, not just on first load. That means tracking document versions, detecting changes, and running partial re-indexing without downtime.
OAuth and permissions. If your agent connects to user accounts (email, calendar, CRM), you are now building an OAuth integration layer. Each provider has different auth flows, token refresh patterns, and rate limits. At Hyperspell, we have seen teams spend more engineering time on OAuth plumbing than on the memory system itself. This is typically where the build-vs-buy decision becomes real.
Retrieval determines whether the right memories reach the LLM. Three levels of sophistication, each building on the last.
Simple retrieval. Embed the query, find the top-K nearest vectors. Works for basic use cases where documents are uniform and recency does not matter.
# Simple semantic search (pseudocode)
query_embedding = embed(user_query)
results = vector_db.search(query_embedding, top_k=10)
context = format_results(results)
Better retrieval. Combine semantic search with keyword search. BM25 handles exact term matching while vector search captures semantic relationships. Merge results with reciprocal rank fusion, then rerank with a cross-encoder. Reranking vendor ZeroEntropy reports that its cross-encoder reranker improves NDCG@10 by 28% over baseline retrievers, and cross-encoder reranking delivers similar-magnitude gains across the industry.
# Hybrid search with reranking (pseudocode)
semantic_results = vector_db.search(embed(query), top_k=50)
keyword_results = bm25_index.search(query, top_k=50)
merged = reciprocal_rank_fusion(semantic_results, keyword_results)
reranked = cross_encoder.rerank(query, merged, top_k=10)
Best retrieval. Add temporal weighting so recent memories rank higher. Add source weighting so CRM data scores higher for sales agents and code repos score higher for engineering agents. Layer in graph traversal for relational queries. A customer support agent needs different retrieval than a research assistant.
Hyperspell's retrieval engine combines indexed search, live search (real-time queries to source APIs), and knowledge graph traversal, so teams get this full range without building each layer from scratch.
You have retrieved relevant memories. Now format them so the LLM can use them.
Budget your context window. Reserve space for the system prompt, the user's current query, tool definitions, and the agent's working memory. What remains is your memory budget. Stuffing the window is expensive and slow: input cost scales roughly linearly with tokens, and very large prompts add real latency. Filling a million-token window on every request is not a viable production pattern.
Structure your context. Labeled sections outperform unstructured text dumps. Tell the LLM what each memory is, when it was recorded, and where it came from.
## Retrieved Memories
### From CRM (updated 2026-02-03, high relevance)
- Customer: Acme Corp, Enterprise plan, renewed Q1 2026
- Primary contact: Jane Smith, VP Engineering
### From recent conversations (2026-02-04)
- User asked about API rate limits for production deployment
- User mentioned migrating from AWS to GCP next quarter
Include metadata. Source, timestamp, and retrieval confidence help the LLM weigh conflicting information. A CRM record from yesterday should override a conversation memory from three months ago.
For long-running agents, compress older memories into summaries while keeping recent interactions at full fidelity. This manages your token budget without losing the context that matters most right now.
It depends on where memory sits in your product strategy and how quickly you need to ship.
Build when memory is your core product differentiator, you have unusual requirements no existing platform supports, or you have a dedicated infrastructure team with time to invest. Building gives you full control and zero vendor dependency. Expect 3-6 months of engineering for a production-quality system, plus permanent maintenance for embedding model updates, schema changes, and scaling (Datagrid's build-vs-buy guide walks through the trade-offs well).
Buy when speed to market matters more than custom optimization, you need many data source integrations, or you would rather spend engineering time on your agent's core capabilities. Platforms like Hyperspell, Mem0, and Zep each take a different architectural approach.
If memory is not the product you are selling, building it yourself usually means paying months of engineering time for infrastructure your users never see.
OAuth integrations alone can consume weeks per provider. If your agent needs email, Slack, CRM, and calendar access, that is four separate implementations, four sets of pagination logic, and four ongoing maintenance burdens. This is often the tipping point. For a Hyperspell-specific walkthrough, see the Quickstart guide.
Persistent memory transforms an AI agent from a stateless tool into a system that learns with every interaction. The architecture is not conceptually complex: five components, each with well-understood options. The complexity lives in the details. Chunking, embedding selection, retrieval tuning, OAuth flows, and context window management all demand careful decisions.
Whether you build or buy, invest in memory early. It compounds. At Hyperspell, we have watched teams go from concept to production memory in days rather than months by using a platform. But even building from scratch, the architecture in this post gives you a clear map.
Start with a vector database and simple retrieval. Add hybrid search and reranking when quality plateaus. Layer in graph storage when your agent needs relationship awareness. The best memory system is the one that ships and starts learning from real usage.
A production-quality memory system typically takes 3-6 months of senior engineering time, an investment comparable to dedicating one or two full-time engineers for a couple of quarters, plus ongoing infrastructure and maintenance costs that grow with your data sources. The largest cost drivers are OAuth integrations for external data and retrieval tuning for quality.
Start with pgvector or Chroma for prototyping. For production, Pinecone, Qdrant, and Weaviate are the most established options. Pinecone offers fully managed infrastructure with minimal operational overhead. Qdrant excels at complex metadata filtering. Weaviate provides strong built-in hybrid search. Choose based on your scale, query patterns, and team preferences.
Not always. Vector databases handle semantic search well for most use cases. Add a graph database when your agent needs to answer relationship queries ("who does Alice work with?") or when temporal ordering of events matters. Hybrid systems that combine vector search with knowledge graph traversal deliver the best retrieval quality for complex, multi-source agent workloads.
Build incremental re-indexing into your ingestion pipeline from the start. Track document versions and detect changes rather than re-indexing everything on a schedule. For sources like email and Slack, use webhooks or polling with change detection. Include timestamps in stored memories so the retrieval engine can weight recent information higher.
Persistent memory requires five components: data sources, processing pipeline, storage, retrieval engine, and context assembly. Weakness in any layer cascades downstream.
Start with a vector database and recursive character chunking at 400-512 tokens. Upgrade to semantic chunking and hybrid storage only when metrics justify the complexity.
Hybrid retrieval (semantic search + keyword search + cross-encoder reranking) meaningfully outperforms vector-only search.
Structure retrieved memories with labels, timestamps, and source metadata so the LLM can weigh relevance and resolve conflicts.
OAuth integrations for external data sources are often the largest time sink when building from scratch. Budget weeks per provider.
The build-vs-buy decision hinges on whether memory infrastructure is your core differentiator. If not, a platform can save 3-6 months of engineering.