> ## Documentation Index
> Fetch the complete documentation index at: https://docs.maximem.ai/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Maximem Synap is a managed memory layer for AI agents: it ingests conversations and documents, extracts structured memory, and returns scoped context to the agent at retrieval time.
> The Python and JavaScript/TypeScript SDKs are the supported way to use Synap. Write integration code with SDK calls, not raw HTTP requests.
> Start with the Quickstart: https://docs.maximem.ai/getting-started/quickstart.md. Every SDK method is documented in the SDK reference: https://docs.maximem.ai/sdk-reference/overview.md.

# Benchmarks

> The public benchmarks Synap is evaluated on, how each answer is scored, what to check before you compare two numbers, and how to measure recall and latency on your own data.

Synap is evaluated on two public benchmarks for long-term conversational memory: LongMemEval and LoCoMo. This page explains what each one tests, how an answer is scored, and what to check before you compare two benchmark numbers. It then shows how to run a small recall and latency test against your own Synap instance, which tells you more about your workload than a public benchmark can.

> [!NOTE]
>
> Results are not listed on this page yet. Each figure will be added with its dataset, the questions scored, the answer and judge models, the number of runs behind it, and the run date, so you can compare it fairly with other published numbers.

## What Synap is evaluated on

| Benchmark | What it tests | Questions scored |
| - | - | - |
| **LongMemEval** ([Wu et al., ICLR 2025](https://arxiv.org/abs/2410.10813)) | Questions about long chat histories that span many sessions. The `LongMemEval_S` release gives about 115,000 tokens of history per question. | All 500 |
| **LoCoMo** ([Maharana et al., ACL 2024](https://arxiv.org/abs/2402.17753)) | Questions about 10 very long conversations, roughly 300 turns each (the `locomo10` release). | 1,540: every category except adversarial |

Both benchmarks group their questions into categories of very different sizes. The overall score counts every question equally, so a category with 30 questions moves it far less than one with 133:

| Benchmark | Category | Questions |
| - | - | - |
| LongMemEval | Multi-session | 133 |
| LongMemEval | Temporal reasoning | 133 |
| LongMemEval | Knowledge update | 78 |
| LongMemEval | Single-session user | 70 |
| LongMemEval | Single-session assistant | 56 |
| LongMemEval | Single-session preference | 30 |
| LoCoMo | Open-domain | 841 |
| LoCoMo | Multi-hop | 321 |
| LoCoMo | Single-hop | 282 |
| LoCoMo | Temporal | 96 |
| LoCoMo | Adversarial (not scored) | 446 |

### Why LoCoMo's adversarial category is left out

Adversarial questions are written so that the conversation holds no answer. They test whether a system declines to answer, not whether it remembers. Including them changes an overall LoCoMo score substantially, so a LoCoMo number is only comparable with another number scored over the same four categories.

## How an answer is scored

For every question:

1. The benchmark's conversation history is ingested into Synap.
2. An answer model asks Synap for context about the question and writes an answer from what comes back.
3. A judge model compares that answer with the benchmark's reference answer and marks it correct or wrong. There is no partial credit.
4. Accuracy is the number of correct answers divided by the number of questions scored.

The answer model and the judge model sit outside Synap, and changing either one changes the score for the same memory system. That is why each result will name both.

## Before you compare two numbers

Memory benchmark numbers are easy to compare unfairly. Check these for every number you see, ours included:

* **Dataset and variant.** LongMemEval ships in more than one variant. The `oracle` variant gives the model only the sessions that contain the answer, so it skips the search through unrelated history that `LongMemEval_S` requires.
* **Questions scored.** The full set or a sample, and for LoCoMo, with or without the adversarial category.
* **Answer model.** A larger answer model can raise the score with no change to the memory layer.
* **Judge.** Correct-or-wrong or graded, and which model.
* **Runs.** One run, a mean across runs, the most frequent result, or the best result. A single run is a sample, so ask for the spread.
* **Date.** Memory systems change between releases, and their numbers change with them.

## Test Synap on your own data

A public benchmark measures conversations that someone else wrote. Whether memory helps your agent depends on your facts, your phrasing and the questions your agent asks. A small recall test measures that directly: plant facts you know, ask for them back, and count how many return.

**1. Write the test cases**

Write each fact your agent must remember as a short conversation. Add the question your agent would ask and a word or number the answer must contain. Pick distinctive answers such as names, places and times: the check looks for an exact string, and a stored memory can word a fact differently from your conversation.

**2. Ingest under a new user**

Use a new `user_id` for every run, and wait for each ingestion to finish before you fetch.

**3. Fetch, check and time**

Fetch with each question, look for the expected answer in the returned items, and time each fetch.

```python Python theme={null}
# export SYNAP_API_KEY=synap_...   SYNAP_INSTANCE_ID=inst_...
import statistics
import time
import uuid
from maximem_synap import MaximemSynapSDK

# (conversation that states the fact, question, string the answer must contain)
CASES = [
    ("User: I'm moving to Lisbon in March.\nAssistant: Congratulations!", "Where is the user moving?", "lisbon"),
    ("User: My daughter is allergic to peanuts.\nAssistant: Noted.", "What is the user's daughter allergic to?", "peanut"),
    ("User: Our standup is at 9:30 every weekday.\nAssistant: Got it.", "When is the user's standup?", "9:30"),
]

sdk = MaximemSynapSDK()
await sdk.initialize()
user_id = f"recall-test-{uuid.uuid4()}"
# B2B instance: also pass customer_id to create() and fetch().

for conversation, _, _ in CASES:
    result = await sdk.memories.create(
        document=conversation, document_type="ai-chat-conversation", user_id=user_id
    )
    await sdk.memories.wait_for_completion(result.ingestion_id)

hits, latencies_ms = 0, []
for _, question, expected in CASES:
    start = time.perf_counter()
    ctx = await sdk.user.context.fetch(
        user_id=user_id, search_query=[question], response_shape="ranked"
    )
    latencies_ms.append((time.perf_counter() - start) * 1000)
    found = any(expected in item.content.lower() for item in ctx.items)
    hits += found
    print("HIT " if found else "MISS", question)

print(f"recall {hits}/{len(CASES)}, median fetch {statistics.median(latencies_ms):.0f} ms")
await sdk.shutdown()
```

```typescript TypeScript theme={null}
// export SYNAP_API_KEY=synap_...   SYNAP_INSTANCE_ID=inst_...
import { randomUUID } from 'node:crypto';
import { SynapClient } from '@maximem/synap-js-sdk';

// [conversation that states the fact, question, string the answer must contain]
const CASES: [string, string, string][] = [
  ["User: I'm moving to Lisbon in March.\nAssistant: Congratulations!", 'Where is the user moving?', 'lisbon'],
  ['User: My daughter is allergic to peanuts.\nAssistant: Noted.', "What is the user's daughter allergic to?", 'peanut'],
  ['User: Our standup is at 9:30 every weekday.\nAssistant: Got it.', "When is the user's standup?", '9:30'],
];

const sdk = new SynapClient();
await sdk.initialize();
const userId = `recall-test-${randomUUID()}`;
// B2B instance: also pass customer_id to create() and fetch().

for (const [conversation] of CASES) {
  const result = await sdk.memories.create({
    document: conversation,
    document_type: 'ai-chat-conversation',
    user_id: userId,
  });
  await sdk.memories.wait_for_completion(result.ingestion_id);
}

let hits = 0;
const latenciesMs: number[] = [];
for (const [, question, expected] of CASES) {
  const start = performance.now();
  const ctx = await sdk.user.context.fetch({
    user_id: userId,
    search_query: [question],
    response_shape: 'ranked',
  });
  latenciesMs.push(performance.now() - start);
  const found = (ctx.items ?? []).some((item) => (item.content ?? '').toLowerCase().includes(expected));
  if (found) hits += 1;
  console.log(found ? 'HIT ' : 'MISS', question);
}

const sorted = [...latenciesMs].sort((a, b) => a - b);
const mid = Math.floor(sorted.length / 2);
const median = sorted.length % 2 ? sorted[mid] : (sorted[mid - 1] + sorted[mid]) / 2;
console.log(`recall ${hits}/${CASES.length}, median fetch ${median.toFixed(0)} ms`);
await sdk.shutdown();
```

A new `user_id` per run keeps memories from earlier runs out of the results. In Python it also keeps a repeated read from being answered by the SDK's local cache, which would make latency look lower than it is. `response_shape="ranked"` returns every memory type in one list, `items`, so the check reads one list instead of five.

### Turn the test into a measurement

* **Use more cases, and run more than once.** Ingestion uses language models to extract memories, so results can differ between runs. Three cases show that the test works; use enough cases to cover your agent's real questions, run the test several times, and report the range rather than the best run.
* **Test with your production retrieval settings.** Retrieval `mode` (`fast` by default, or `accurate`) and `precision_level` (`high` by default, or `medium`) change both recall and latency. See [Ingestion and retrieval modes](/concepts/retrieval-modes).
* **Measure latency from where your agent runs.** Each timing includes the network round trip from your machine, so a laptop far from your Synap region measures distance as much as retrieval. Read the median and the higher percentiles over many fetches, not a single call.
* **Check a miss before you count it.** `wait_for_completion` returns when ingestion ends, including when it failed, so check the status it returns if a case misses on every run. In Python, `ctx.retrieval` is set when the server reports a degraded read, for example after a timeout; rerun those questions instead of scoring them as misses.
* **Score the final answer too.** Recall tells you the fact came back. To know whether your agent uses it, pass the context to your model and grade its answer against the expected one, the same way the benchmarks above do.
* **Keep test questions out of your prompts.** If your agent's prompt contains the same facts or questions as examples, the test measures the prompt, not memory.

## Related pages

> **[Ingestion and retrieval modes](/concepts/retrieval-modes)**
>
> The speed and thoroughness tradeoff your test should match.

> **[Performance and limits](/resources/performance-limits)**
>
> SDK timeouts, and where to read live latency for your instance.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.