June 24, 2026
By David Szarzynski · Founding Engineer
Your agent said "I don't have information about that" to a question whose answer is sitting in a Google Doc the user saved ten minutes ago. Or worse: it answered confidently with last quarter's number while the user was looking at the new dashboard.
These are the moments that break trust in an AI product. They are also the moments where most teams reach for the wrong diagnosis. "The retrieval is bad." "The model is being weird." "We probably need a better embedding model." Sometimes those are right. More often the failure was at a step nobody instrumented.
At Hyperspell we run a memory layer across 50+ source types in production. The pattern we see is that most "memory" bugs are actually one of seven distinct failures, and each one has a different fix. This is the field guide we wish we'd had when we started. Each failure mode has the same shape: what it looks like, the most common root causes, a diagnostic you can run in five minutes, and the structural fix.
Most agent observability work treats agent failures broadly, spanning memory, reflection, planning, action, and system-level operations (Zhu et al., "Where LLM Agents Fail"). Microsoft's industry taxonomy treats memory poisoning as its own security class (Microsoft AI Red Team, "Taxonomy of Failure Modes in Agentic AI Systems"). This piece zooms into the memory column and unpacks the operational failures teams actually triage.
When an agent fails, the symptom (wrong answer, missing answer, weirdly stale answer) doesn't tell you where in the chain the break happened. A taxonomy maps symptoms to likely causes, gives you a confirmation step, and points to a fix. The seven modes below run roughly in pipeline order. Earlier failures cascade into later ones, which is part of why naive triage misses the root cause.
In plain English: "The agent forgot" is a symptom, not a diagnosis. There are at least seven different reasons it might forget, and each one needs a different fix.
The agent says it doesn't know something that is definitely in the system. Or it answers from general knowledge when it should have used your data. The memory exists. The agent never sees it.
The query embedding lands in a different neighborhood than the document embedding. This is the vocabulary gap: a user asks how to "unstage a file," the document calls it "removing from staging area," and the two embeddings don't meet. The document is chunked badly, with the relevant passage split across two fragments where neither is self-contained. Or the similarity threshold is too aggressive and filters the right chunk out of the recall set.
This is also the most common bucket statistically. Practitioner analyses estimate that when RAG systems give wrong answers, retrieval is the failure point about 70 percent of the time, not generation (ChatRAG).
Look up the memory by ID directly, not semantically, and confirm it exists. Re-run the failing query with the similarity threshold dropped to zero. Does the right document show up at all? If yes, you have a threshold or ranking problem (Failure mode 2). If no, you have a retrieval miss.
Hybrid search (BM25 plus vectors) recovers the exact tokens and rare strings that vector-only retrieval misses. Lower the similarity threshold and add a downstream reranker. Re-chunk with semantic boundaries instead of fixed sizes, and add overlap so a single concept isn't split. For multi-source memory layers, source weighting lets you tune which integrations contribute candidate chunks before ranking (Hyperspell docs).
The right memory was retrieved. It was sitting in the recall set. It just ranked too low to make it into the context. Less relevant memories ranked above it, and the agent answered from those instead, often confidently.
The embedding model isn't tuned to your domain. Generic embeddings can't reliably tell that "AAPL" and "Apple Inc." should rank above "Apple pie recipe" for a finance query. There's no reranker in the pipeline, so the recall set is reasonable but the order is wrong. Or the query is genuinely ambiguous and a different interpretation has stronger matches.
A subtler version: the ranking inside the recall set is fine, but the assembled context places the most relevant memory in the middle of a long prompt. Liu et al. showed that performance is highest when relevant information sits at the start or end of the input and degrades significantly in the middle, even for models marketed as long-context (Liu et al., "Lost in the Middle"). That's still a ranking failure in effect, even if every retrieval score looked fine.
Examine the full recall list, not just the top K. Check whether the right document is anywhere in the top 50. If it is, you have a ranking problem. If it isn't, you have a retrieval miss. For each top-ranked document, write down why it ranked high. If "I don't know" comes up more than once, the embedding model is the suspect.
Two-stage retrieval is the standard answer: pull the top 20 to 50 cheaply with embeddings, then rerank with a cross-encoder that scores each candidate against the original query with full attention. Rerankers feed the raw query-document pair into the transformer, so they lose less information than a single-vector cosine similarity (Pinecone, "Rerankers and Two-Stage Retrieval"). Add custom metadata to memories so the ranker can also weigh source authority, recency, or domain (Hyperspell docs).
In plain English: If the right answer is in the top 50 but not the top 5, you don't have a retrieval problem. You have a ranking problem, and reranking is the highest-ROI fix in this taxonomy.
The agent gives a confidently outdated answer. The user knows the current state. The memory hasn't caught up. Or the answer is technically correct in 2024 and dangerously wrong in 2026.
Sync delay between the source system and the memory store. A connector that silently stopped syncing two weeks ago. A cache layer holding old vectors after the source updated. No invalidation when supersedence happens, so old facts coexist with new ones in the recall set and ranking picks one of them.
Compare the memory's last-updated timestamp against the source's last-updated timestamp. Check connector health: when did each integration last successfully ingest? Push a known recently-updated document through the system end-to-end and confirm the agent sees the update. If your integrations don't expose a "last successful sync" signal, that itself is a staleness diagnostic gap.
The principled answer is bi-temporal modeling: track event time (when a fact was true) separately from ingestion time (when the system observed it). Zep's Graphiti architecture handles supersedence this way: when a new event invalidates an old fact, the system sets invalid_at on the old edge instead of deleting it, so live answers update while history is preserved (Rasmussen et al., "Zep: A Temporal Knowledge Graph Architecture for Agent Memory"). Reduce sync intervals where freshness is critical. Where it isn't, surface "as of" timestamps in the response so the user can judge. AWS AgentCore exposes a similar split at the API level: short-term events expire on a configurable schedule, while long-term insights persist across sessions (AWS Bedrock AgentCore Memory documentation).
The agent says it doesn't know something. A different user with broader access asks the same question and gets the answer. Or the agent gives a partial, awkward answer that omits the relevant document without flagging the omission.
The current user genuinely doesn't have permission. This is correct behavior; the failure is in how it's surfaced. Or the permission filter is misconfigured and blocking content the user should see. Or the permission cache is stale: the user got access yesterday, the cache hasn't refreshed.
There's also an adversarial version. Memory poisoning is when malicious content gets stored in the agent's memory and recalled later, steering future behavior. Microsoft's taxonomy treats this as a distinct security class with documented mitigations: require external authentication for memory updates, restrict which components can write to memory, and validate the structure and format of stored items (Microsoft AI Red Team taxonomy).
Re-run the query as an admin user. Does the answer appear? Compare the user's effective permissions on the source system against what the memory layer enforces. Check the permission cache TTL.
Enforce permissions at query time, not just at ingest, because permissions change after a document is indexed. Surface "you may not have access to that information" instead of a vague "I don't know" so users understand the failure shape. Validate stored item structure on the way in to defend against poisoning, not only at retrieval.
The agent gives different answers to the same question on different runs. Or it confidently states a fact that contradicts something the user knows is current.
Multiple memories on the same topic with no resolution strategy, so the system effectively picks one at random based on retrieval order. Conflicting facts across sources, where the CRM says one customer status and the support tool says another. Soft conflicts: similar but not identical statements that retrieval can't tell apart.
Galileo calls this "context and memory corruption" and notes that poisoned or stale entries quietly steer future actions because nothing reconciles them (Galileo, "7 AI Agent Failure Modes").
Retrieve all memories on the topic, not just the top K, and read them. Look for explicit contradictions, version mismatches, or cross-source disagreement. If you find them, the question stops being "which one is right" and starts being "how does the system decide."
Detect conflicts at ingest, not at query time. Resolution strategies, in rough order of robustness: source authority, timestamp recency, explicit user flag, bi-temporal invalidation (Zep paper). When resolution isn't confident, surface the conflict in the response: "the CRM and the email thread disagree on this. Using the CRM as the more recent source." Users tolerate uncertainty when it's named.
The right memories were retrieved. They ranked correctly. They were fresh. They were permission-allowed. The answer is still wrong. This is the most frustrating failure mode because every upstream metric looks healthy.
Context truncated, with the relevant memory landing past the token budget. Position-in-context bias, with the relevant memory landing in the middle of a long prompt where models systematically underweight information (Liu et al.). Poor formatting the model can't parse, like deeply nested JSON with no schema hint. The relevant fact buried alongside ten irrelevant chunks the model gives roughly equal attention to.
The Mem0 evaluation work makes the cost angle explicit: a system that retrieves the right thing but assembles 50,000 tokens of context still costs and degrades like a system that retrieved the wrong thing. Their framework measures accuracy, cost in tokens per query, and latency together, because optimizing one without the others lies about real performance (Chhikara et al., "Mem0").
Log the final context sent to the LLM for the failing request. Token-count it: did your relevant memory survive truncation? Position-check it: did it land in the middle of the prompt? Strip everything except the relevant memory and rerun. If the model now answers correctly, you have an assembly problem. If it still doesn't, the failure is downstream in utilization or upstream in ranking.
Lift the context budget where you can. Compress aggressively where you can't. Place high-relevance memories at the start or end of the assembled context, not the middle. Filter the recall set down to what's actually needed for this query: precision over volume. Use structured formatting with consistent fields and clear separators so the model doesn't burn attention on layout.
The right memory was retrieved, ranked, fresh, permission-allowed, fit in the context, well-formatted. The model still didn't use it. The fact is sitting in the prompt and the model answered around it.
The prompt didn't tell the model to prefer retrieved context over its parametric knowledge. The retrieved context contradicted the model's prior, and the model "trusted itself" instead. Or the retrieved context was true but irrelevant to the actual question, so the model correctly ignored it (which means upstream selection was wrong, even though everything looked clean).
Check the response text for entities, names, numbers, or quotes from the retrieved memory. None present? Utilization failure. Run the prompt with the memory present and absent. If the answer doesn't change, the model isn't using the memory either way.
Prompt the model explicitly: "Answer based only on the provided context. If the context doesn't contain the answer, say so." Reduce noise in the context, because utilization improves when signal-to-noise is high. The Mem0 framework calls this out directly: accuracy depends on what the model actually uses, not what was retrieved.
The taxonomy isn't just for postmortems. It's a debugging chain you walk down for any failed response. Each step is a yes/no question. The first "no" is your failure mode.
Was the right memory in the system? → Coverage / ingest gap (out of scope here)
Was it retrieved? → Retrieval miss (Mode 1)
Was it ranked highly enough? → Ranking failure (Mode 2)
Was it fresh? → Staleness (Mode 3)
Was the user allowed to see it? → Permission block (Mode 4)
Did anything contradict it? → Conflict (Mode 5)
Did it survive context assembly? → Assembly failure (Mode 6)
Did the model actually use it? → Utilization failure (Mode 7)
Most teams instrument the first two steps and call it observability. The full chain is what separates "we have logs" from "we can debug." Cascading failures, where a single root cause propagates through subsequent decisions, are common in agent systems and very hard to untangle without per-step signals (Zhu et al.).
For multi-source memory layers, the system can record per-step signals (which integration produced the candidate, what rank it landed at, what timestamp it carried, what permission filter applied, what made it into the final assembly) without each team rebuilding the trace pipeline themselves (Hyperspell docs). That's the operational case for managed memory infrastructure: not "we retrieve better than you would" but "we make the chain debuggable end-to-end."
Retrieval miss is the most common in raw frequency. Practitioner analyses estimate retrieval is the failure point in roughly 70 percent of wrong RAG answers (ChatRAG). Ranking failure is the most often misdiagnosed, because teams blame retrieval when the right document was in the recall set all along.
Look at the recall set with the similarity threshold dropped to zero. If the right document appears anywhere in the top 50, you have a ranking problem and reranking is the fix. If it doesn't appear at all, you have a retrieval miss and the fix is upstream in chunking, embeddings, or hybrid search.
Memory poisoning is when malicious content is stored in an agent's memory and recalled later, steering future behavior. Microsoft's failure-mode taxonomy treats it as a distinct security class. Mitigations: require external authentication for memory updates, restrict which components can write to memory, and validate the structure and format of stored items.
Either an assembly failure (the context didn't survive truncation, or it landed mid-prompt where models underweight information, per Liu et al.) or a utilization failure (the model received the context but ignored it). Log the final prompt and check whether your context made it in, then check whether any entities from your context appear in the response.
Memory failures have at least seven distinct shapes: retrieval miss, ranking failure, staleness, permission block, conflict, assembly failure, and utilization failure.
Practitioner analyses estimate retrieval, not generation, is the failure point in roughly 70 percent of wrong RAG answers (ChatRAG).
If the right answer is in the top 50 but not the top 5, you have a ranking problem, not a retrieval problem. Reranking is the highest-ROI fix.
Position-in-context bias means information in the middle of a long prompt is systematically underweighted by language models (Liu et al.).
Bi-temporal modeling (event time and ingestion time tracked separately) is how production memory systems handle staleness without losing history (Zep paper).
Memory poisoning is a named security class with documented mitigations: authenticated writes, restricted components, structural validation (Microsoft taxonomy).
"The agent forgot" is a symptom. Walking the chain (retrieval, ranking, freshness, permissions, conflicts, assembly, utilization) tells you which one to fix.
The next time an agent "forgets" something it should know, don't reach for "the LLM is being weird" until you've ruled out the seven failures above. Each one has a different signal, a different diagnostic, and a different fix. Teams that walk the chain ship more reliable agents than teams that swap embedding models in a panic.
If you're building on a managed memory layer, the question to ask isn't "is your retrieval good?" It's "can I see, per request, where in the chain something failed?" That's the difference between an agent product and an agent demo.
ChatRAG. "5 Proven Methods to Debug RAG Systems When They Give Wrong Answers." March 2026. https://www.chatrag.ai/blog/2026-03-11-5-proven-methods-to-debug-rag-systems-when-they-give-wrong-answers
Chhikara, P. et al. "Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory." 2025. https://arxiv.org/abs/2504.19413
Galileo. "7 AI Agent Failure Modes and How to Prevent Them." 2026. https://galileo.ai/blog/agent-failure-modes-guide
Liu, N. et al. "Lost in the Middle: How Language Models Use Long Contexts." TACL 2024. https://arxiv.org/abs/2307.03172
Microsoft AI Red Team. "Taxonomy of Failure Modes in Agentic AI Systems." April 2025. https://www.microsoft.com/en-us/security/blog/2025/04/24/new-whitepaper-outlines-the-taxonomy-of-failure-modes-in-ai-agents/
Pinecone. "Rerankers and Two-Stage Retrieval." https://www.pinecone.io/learn/series/rag/rerankers/
Rasmussen, P. et al. "Zep: A Temporal Knowledge Graph Architecture for Agent Memory." 2025. https://arxiv.org/abs/2501.13956
AWS. "Add memory to your Amazon Bedrock AgentCore agent." https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/memory.html
Zhu, K. et al. "Where LLM Agents Fail and How They Can Learn From Failures." 2025. https://arxiv.org/abs/2509.25370
Hyperspell. Product documentation. https://docs.hyperspell.com