May 20, 2026
By David Szarzynski · Founding Engineer
Traditional unit tests follow a clean pattern. Input goes in, function runs, expected output comes out. If the test passes, the code works.
Memory systems break this pattern. The output of any query depends on everything the system has ingested over days or weeks. The question is no longer "does this function return the right value?" It becomes "given everything my agent has seen so far, does it retrieve the right context for this query?"
This is not a theoretical problem. A comprehensive survey of agent memory research found that existing works "differ substantially in their motivations, implementations, and evaluation protocols." There is no standard testing methodology for memory systems. The field is fragmented, and most teams are building their QA practices from scratch.
At Hyperspell, we process memory queries across 50+ workspace integrations. That operational experience has taught us patterns for testing stateful systems: what to test, how to test it, and how to make tests reproducible.
Traditional software testing assumes determinism. Same input, same output. Memory systems violate this assumption in four ways.
State accumulation. The output of a memory query depends on what has been ingested previously. Two identical queries can return different results if one runs against a fresh memory store and the other against one with three weeks of indexed data.
Time dependency. Results change as new data syncs. A query that returns correct results today may return different results tomorrow after a new batch of Slack messages is indexed.
Non-determinism. Retrieval ranking and LLM processing add variance. Research on testing agentic systems notes that the same test can pass one run and fail the next simply because the agent chose a different reasoning path. This applies directly to memory systems where retrieval ranking is similarity-based.
Combinatorial scale. You cannot exhaustively test all possible memory states. As data volume grows, the space of possible retrieval results grows with it. Brute-force testing is not an option.
Dimension | Traditional tests | Memory tests |
|---|---|---|
State | Stateless or reset between runs | Accumulated over time, affects every result |
Determinism | Same input, same output | Same input, different output depending on context |
Time sensitivity | Results are stable | Results change as data syncs |
Failure diagnosis | Clear stack trace | Requires inspecting retrieval context and ranking |
Traditional tests ask "does this function work?" Memory tests ask "given everything the system knows right now, does it find the right information?" That shift from stateless to stateful is what makes memory QA fundamentally different.
Memory testing breaks down into five distinct problems, each with its own failure modes.
Does data get stored as expected? When you ingest a document that mentions a specific person, project, or date, does that information appear in query results with the correct metadata?
A basic ingestion test: add a known document using the memories API, query for a specific entity mentioned in that document, and verify the entity appears in results. If it does not, your ingestion pipeline has a gap.
from hyperspell import Hyperspell
client = Hyperspell(api_key="API_KEY", user_id="test-user")
# Ingest a known document
status = client.memories.add(
text="Project Alpha launched on March 15. Lead engineer: Sarah Chen.",
metadata={"collection": "test-fixtures"}
)
# Query for the entity
response = client.memories.search(
query="Who is the lead engineer on Project Alpha?",
sources=["vault"],
options={"filter": {"collection": "test-fixtures"}}
)
# Verify the expected entity appears in results
assert any("Sarah Chen" in doc.text for doc in response.documents)
Given a query, are the right memories retrieved? Are they ranked appropriately?
The standard metrics are Precision@k, Recall@k, and NDCG@k. Precision measures how many of the top k retrieved items are relevant. Recall measures how many of all relevant items appear in the top k. NDCG rewards relevant items appearing higher in the ranking.
The MemoryAgentBench benchmark identified four core competencies to evaluate: accurate retrieval, test-time learning, long-range understanding, and conflict resolution. Its finding was sobering: on multi-hop conflict resolution, every tested method scored in the single digits, 7% at best. Retrieval accuracy alone does not tell you whether your memory system works.
In multi-tenant memory systems, one user's data must never appear in another user's retrieval results. Permission tests are binary: either the filter works or it leaks data.
Test by creating two isolated user contexts, ingesting different data for each, and verifying that cross-user queries return nothing. This is one area where you want 100% pass rates, not statistical confidence intervals.
When a source document changes, does the memory reflect the change? When a document is deleted, does it disappear from retrieval results?
Test pattern: update a source document, wait for sync, then verify the memory reflects the new version and the old version is no longer retrievable. You can monitor sync progress using the indexing progress endpoint.
If a user connects a new data source, how long until that data is queryable?
Freshness tests measure the time between ingestion and availability. Set a threshold, ingest test data, poll for availability, and fail the test if the threshold is exceeded.
The five dimensions above tell you what to test. The categories below tell you how.
Test individual components in isolation: the chunker, the embedder, the metadata parser. This is the one category where traditional testing patterns apply directly. Verify that a specific document format parses into the expected number of chunks with the correct metadata attached. Unit tests should be fast, deterministic, and run on every commit.
Test the full pipeline end-to-end. Ingest a document, query for it, and verify retrieval. These are slower but catch issues that unit tests miss, like mismatches between how the chunker splits text and how the embedder represents it.
Because memory systems are non-deterministic, run multiple trials per scenario. One assessment framework for agentic AI recommends running each scenario several times independently and averaging the results. A test that fails one out of three runs is a signal worth investigating, even if it passes the majority.
This is where memory testing diverges most from traditional QA. You need a ground truth dataset: a set of queries paired with the documents that should be retrieved for each.
Evidently AI's RAG evaluation guide describes three approaches to building ground truth:
Pre-labeled documents. Manually map test queries to correct source documents. Labor-intensive but precise.
Post-retrieval assessment. Run queries against your system, then have reviewers label the actual results as relevant or not. Captures real-world retrieval behavior.
LLM-based scoring. Use an LLM to judge whether retrieved chunks are relevant to queries. Strong LLM judges can approach human-level agreement on chunk relevance.
The most scalable approach is synthetic generation: start from your knowledge base, use an LLM to generate realistic questions answerable from specific chunks, and use those question-document pairs as your test suite.
Hyperspell's Evaluation API provides a built-in feedback loop for this. Every query returns a query_id, and each retrieved highlight includes a highlight_id. You can score both programmatically:
# Score a query result (scale: -1 to 1)
client.evaluate.score_query(
query_id=response.query_id,
score=1.0
)
# Score an individual highlight with a comment
client.evaluate.score_highlight(
highlight_id=highlight.highlight_id,
score=-1.0,
comment="Retrieved outdated project status"
)
These scores feed back into retrieval ranking over time, creating a continuous improvement loop without manual re-tuning.
Golden queries with expected results, run on every deploy. These catch the specific case where a system update degrades retrieval quality for previously working queries.
Version your test datasets alongside your code. What counted as a correct answer last month may change after new data ingestion. Cognee's memory benchmarking work found that LLM-based scorers like DeepEval are prone to variance, which is why they rely on repeated runs and bootstrapped confidence intervals across evaluation cycles.
The hardest part of memory testing is not writing the tests. It is creating the conditions for tests to run reliably.
Create a canonical test dataset: a fixed set of documents, emails, and messages that represent known entities and relationships. Ingest this dataset before each test run to create a known starting state.
The dataset should be small enough to ingest quickly but rich enough to cover your scenarios. Include edge cases: documents with conflicting information, overlapping entities, and time-sensitive data.
Each test run must operate against an isolated memory state. Tests that share state interfere with each other, producing flaky results that erode trust in the suite.
Pattern: create a dedicated test user per test suite, ingest fixtures for that user, run tests, then tear down the user's data after completion.
In plain English: Think of it like setting up a clean database for each test run. Your memory system needs the same isolation guarantees you already expect from your application database tests. If two tests can see each other's data, your results are meaningless.
Freshness and sync correctness tests require time to pass. Waiting for real time is impractical in a CI pipeline. Mock timestamps and sync schedules so your tests can simulate "24 hours since last sync" without the wait.
Retrieval accuracy is necessary but not the whole picture. The agent's final output depends on how the LLM uses the retrieved context, and that introduces its own failure modes.
A memory system that retrieves all the right documents can still produce wrong answers if the LLM misinterprets the context or ignores relevant information.
LLM-as-judge evaluation is the emerging standard. Use a separate model to evaluate whether the agent's response is correct, complete, and faithful to the retrieved context. Hyperspell's Traces API records full agent interactions for later evaluation against these criteria.
Sample-based manual review remains essential. Automated metrics correlate with quality but do not replace human judgment on subjective dimensions like helpfulness, tone, and contextual appropriateness.
A practical approach: sample 5-10% of production queries weekly, have reviewers label relevance and correctness, and use the results to calibrate your automated metrics.
Compare memory system versions in production by routing a percentage of traffic to the new version and measuring downstream metrics: task completion rate, user satisfaction, follow-up query frequency. This produces the most reliable evaluation signal, but requires sufficient traffic and careful experimental design.
Pre-deploy testing catches regressions. Production testing catches the problems that only appear at scale with real data.
Canary queries. Run synthetic queries with known answers continuously against production. If a canary query starts returning wrong results, something in the memory system has changed.
Anomaly detection. Track precision and recall over time. Set alerts for when retrieval metrics degrade beyond a threshold. Gradual degradation often signals data quality issues or sync failures rather than code bugs.
User feedback loops. Thumbs up/down on agent responses is a noisy but valuable signal. Aggregate it over time to identify queries and topics where retrieval quality is consistently poor.
For production monitoring patterns in depth, see our guide on memory observability.
Testing memory requires new patterns, not new tools. The shift from stateless to stateful changes what you test, how you build test data, and how you interpret results.
A practical starting path:
Start with retrieval evaluation. Build a ground truth dataset of 50-100 query-document pairs. Measure precision and recall. This gives you a baseline.
Add regression tests. Pick your 20 most important queries and lock in expected results. Run them on every deploy.
Build fixtures and isolation. Create a canonical test dataset and ensure each test run starts clean.
Expand to integration tests and end-to-end evaluation. These require more infrastructure but catch problems the earlier stages miss.
The field is early. Research confirms that evaluation protocols are fragmented and no standard has emerged. Teams that invest in test infrastructure now will compound their advantage as memory systems mature.
Hyperspell's Evaluation API and trace recording are designed to make this testing loop accessible without building custom evaluation infrastructure from scratch.
Run multiple trials per test scenario and average results. Use bootstrapped confidence intervals to account for LLM variance. Focus on statistical properties like average precision and recall distribution, not exact-match expectations.
Start with Precision@k, Recall@k, and NDCG@k for ranked retrieval quality. Add F1-score for entity extraction accuracy. For end-to-end quality, use LLM-as-judge scoring. Together these cover both the retrieval and generation stages of a memory-backed agent.
Three approaches: manually label which documents are relevant to each test query, review actual system outputs post-retrieval, or use LLM-based relevance scoring at scale. Synthetic generation (creating questions from known documents) is the most scalable method and requires no manual annotation.
Benchmarking measures system-level performance across standardized tasks (like MemoryAgentBench). Testing is what you build into your development workflow: unit tests, integration tests, regression suites, and evaluation pipelines that run on every deploy. Benchmarking tells you how your system compares. Testing tells you whether your latest change broke something.
Traditional testing patterns break for memory systems because state is accumulated, not static. Same input, different output depending on context.
Test five things: ingestion correctness, retrieval accuracy, permission enforcement, sync correctness, and freshness guarantees.
Four test categories cover different failure modes: unit tests for deterministic components, integration tests for the full pipeline, retrieval evaluation against ground truth, and regression tests on every deploy.
Reproducible testing requires fixtures, isolation, and time simulation. Without these, flaky tests erode trust in the suite.
Handle non-determinism by running multiple trials and using confidence intervals, not exact-match expectations.
Start with retrieval evaluation (50-100 ground truth pairs), then expand to regression tests, integration tests, and end-to-end evaluation.