How Ingestion Works

Your application produces content all the time: live conversations, support tickets, product docs, CRM records. Ingestion is the process that turns that raw content into structured, retrievable memory. There are two paths into Synap, and they share the same underlying pipeline:

Runtime ingestion feeds content in as it is generated during live agent interactions, one turn at a time.
Bootstrap ingestion loads pre-existing data in bulk: historical conversations, documentation, migrations from another system.

Both paths converge on the same processing pipeline. The difference is how you call them and what defaults make sense for each.

There is a third way content reaches this pipeline, and you do not call it directly. Conversation turns sent over the real-time Listen stream with instance.send_message() are persisted to conversation history, and when the conversation compacts, those raw turns are promoted into the same pipeline described below: compaction promotion.It is deferred but not conditional: compaction fires at 3,000 tokens, 10 messages, or 5 minutes of inactivity, so every conversation reaches this pipeline. That is why the Agent Integration needs no ingestion call: agents report turns and Synap forms the memory. Use the two paths below for content that is not a conversation turn, and never for text you already streamed, which would extract it twice.

The ingestion pipeline

Every document you send, via either path, flows through the same stages before it becomes a memory you can retrieve:

Categorization

The pipeline reads the document’s document_type and selects the appropriate extraction logic. A chat conversation, an email, and a PDF are each processed differently.

Extraction

Synap analyzes the content to pull out the things worth remembering: facts, decisions, preferences, action items, and the entities involved. The ingestion mode does not change this step today (see below).

Chunking

Larger documents are segmented into coherent units so that retrieval can return precise, relevant passages rather than whole documents.

Entity resolution

Extracted entities (people, projects, organizations) are matched against entities already in the store, so references to “Sarah” or “Project Atlas” link to the same entity across many documents. See Entity Resolution.

Storage

The resulting structured memories are written to the correct scope, indexed for both vector and graph retrieval, and made available to future queries.

Ingestion is asynchronous on both paths. The SDK call returns quickly with an identifier, and processing continues in the background. Memories become available for retrieval once the pipeline finishes.

Ingestion mode

mode accepts long-range (the default) or fast on both ingestion paths. The server accepts either value, but in the current release nothing downstream reads it: both run the same pipeline, so the choice does not change what is extracted or how long a write takes. You can leave it out. See Ingestion mode. The ingestion mode is also distinct from the retrieval mode you pick when querying, which does change what a fetch does (fast = vector search; accurate = vector search + graph traversal + LLM subquery decomposition + reranking). See Retrieval Modes.

Document types

document_type tells the pipeline what kind of content it is looking at and which extraction logic to apply. Both paths accept the same set:

Document Type	Description	Extraction Focus
`ai-chat-conversation`	Chat between a user and an AI agent. The most common type for runtime ingestion.	Speaker identification, preference detection, decision tracking, entity extraction
`document`	General text documents, articles, or notes.	Topic extraction, entity extraction, key facts
`email`	Email content including headers and body.	Sender/recipient extraction, action items, references
`pdf`	PDF document content (text extracted).	Structured content extraction, section awareness
`image`	Image descriptions or OCR-extracted text.	Visual entity extraction, scene understanding
`audio`	Transcribed audio content.	Speaker diarization, topic segmentation
`meeting-transcript`	Meeting transcriptions with multiple speakers.	Action items, decisions, attendee tracking, topic flow

During runtime you will use ai-chat-conversation almost exclusively. The other types show up most often in bootstrap loads and specialized pipelines.

Scoping every document

user_id and customer_id determine where a memory is stored in the scope hierarchy, which in turn controls who can retrieve it later. On B2C agents, customer_id is resolved automatically and you only pass user_id; on B2B agents you pass both. The same scoping rules apply on both ingestion paths. See Memory Scopes for the full hierarchy.

Runtime ingestion

Runtime ingestion feeds content into Synap as it is generated during live agent interactions. This is the primary ingestion path for most applications. After each conversation turn (or at the end of a conversation) your application calls sdk.memories.create() to send the turn through the pipeline. The call returns immediately, so your agent never waits for ingestion before responding to the user; fast is the natural default here.

Your agent receives a user message

Through a chat widget, API, mobile app, or other channel.

Your agent retrieves context from Synap

Before responding, the agent fetches relevant memories. See Context End to End.

Your agent generates and delivers a response

It calls an LLM with the retrieved context and conversation history, then replies to the user.

Your agent ingests the turn

After the response is delivered, the agent calls sdk.memories.create(). This returns immediately and does not block the user experience.

Synap processes in the background

The pipeline extracts, resolves, and stores structured memories, which become available for future retrieval.

The SDK call

import uuid
from maximem_synap import MaximemSynapSDK

sdk = MaximemSynapSDK(api_key="your_api_key")

result = await sdk.memories.create(
    document="User: Can you remind me what we decided about the migration timeline?\n"
             "Assistant: In our last conversation, you and the team agreed to begin the "
             "database migration on March 15th, with a two-week buffer for testing.",
    document_type="ai-chat-conversation",
    user_id="user_123",
    customer_id="acme_corp",  # B2B only; not accepted on B2C
    mode="fast",
)

print(f"Ingestion ID: {result.ingestion_id}")
# Returns immediately, processing happens in the background

import { SynapClient } from '@maximem/synap-js-sdk';

const sdk = new SynapClient({ apiKey: 'your_api_key' });

const result = await sdk.memories.create({
  document: "User: Can you remind me what we decided about the migration timeline?\nAssistant: In our last conversation, you and the team agreed to begin the database migration on March 15th, with a two-week buffer for testing.",
  document_type: 'ai-chat-conversation',
  user_id: 'user_123',
  customer_id: 'acme_corp',
  // B2B only; not accepted on B2C
  mode: 'fast',
});

console.log(`Ingestion ID: ${result.ingestion_id}`);
// Returns immediately, processing happens in the background

import { SynapClient } from '@maximem/synap-js-sdk';

const sdk = new SynapClient({ apiKey: 'your_api_key' });

const result = await sdk.memories.create({
  document: "User: Can you remind me what we decided about the migration timeline?\nAssistant: In our last conversation, you and the team agreed to begin the database migration on March 15th, with a two-week buffer for testing.",
  document_type: 'ai-chat-conversation',
  user_id: 'user_123',
  customer_id: 'acme_corp',
  // B2B only; not accepted on B2C
  mode: 'fast',
});

console.log(`Ingestion ID: ${result.ingestion_id}`);
// Returns immediately, processing happens in the background

Parameter	Required	Description
`document`	Yes	The text to ingest. For conversations, include both user and assistant messages with speaker labels.
`document_type`	Yes	The type of content. Determines how the pipeline processes the document.
`user_id`	No	The user this content belongs to. Determines user-scope storage.
`customer_id`	No	The customer organization (B2B only; not accepted on B2C). Determines customer-scope storage.
`mode`	No	`long-range` (default) or `fast`. Both run the same pipeline today.
`document_id`	No	Your own label for the document. It is not a duplicate check: the same content for the same user within 7 days returns the first ingestion.
`document_created_at`	No	Timestamp override. Defaults to the current time for runtime ingestion (usually correct).

Fitting it into the agent loop

import uuid
from maximem_synap import MaximemSynapSDK
from openai import AsyncOpenAI

sdk = MaximemSynapSDK(api_key="synap_api_key")
openai_client = AsyncOpenAI(api_key="openai_api_key")

# conversation_id must be a valid UUID
conversation_id = str(uuid.uuid4())

async def handle_message(user_message: str, user_id: str, customer_id: str, conversation_id: str):
    """Handle a single user message with memory-enabled context."""

    # 1. Retrieve relevant context from Synap
    context = await sdk.conversation.context.fetch(
        conversation_id=conversation_id,
        user_id=user_id,
        customer_id=customer_id,  # B2B only; not accepted on B2C
        search_query=[user_message],
        mode="fast",
    )

    # 2. Build the prompt with retrieved memories
    system_prompt = (
        "You are a helpful assistant. Use the following context from previous "
        "conversations to inform your response:\n\n"
        f"{context.formatted_context}\n\n"
        "If the context is not relevant, respond based on your general knowledge."
    )

    # 3. Generate the response
    response = await openai_client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": user_message},
        ],
    )
    assistant_message = response.choices[0].message.content

    # 4. Ingest the conversation turn (non-blocking)
    await sdk.memories.create(
        document=f"User: {user_message}\nAssistant: {assistant_message}",
        document_type="ai-chat-conversation",
        user_id=user_id,
        customer_id=customer_id,
        mode="fast",
    )

    return assistant_message

For the full retrieve-generate-ingest pattern, see Context End to End.

Include speaker labels in conversation content

Always format conversations with clear User: and Assistant: labels. The pipeline uses them to identify who said what, which is critical for accurate preference detection and entity attribution.

# Good: clear speaker labels
document = "User: I need the report by Friday.\nAssistant: I'll have it ready by Thursday evening."

# Bad: no speaker context
document = "I need the report by Friday. I'll have it ready by Thursday evening."

// Good: clear speaker labels
let document = "User: I need the report by Friday.\nAssistant: I'll have it ready by Thursday evening.";

// Bad: no speaker context
document = "I need the report by Friday. I'll have it ready by Thursday evening.";

// Good: clear speaker labels
let document = "User: I need the report by Friday.\nAssistant: I'll have it ready by Thursday evening.";

// Bad: no speaker context
document = "I need the report by Friday. I'll have it ready by Thursday evening.";

Use consistent user and customer IDs

Keep user_id (and customer_id on B2B) consistent across all calls for the same user and organization. Inconsistent IDs fragment the store into isolated pockets of context that cannot be retrieved together. Derive them from your auth system.

Ingest after the response is delivered

Ingest the turn after your agent has responded, not before, so the ingested content includes both the user message and the agent’s reply: complete context for future retrieval.

Handle ingestion errors gracefully

Runtime ingestion should never block or crash your agent. Wrap calls in error handling and log failures. A missed ingestion is recoverable; a crashed agent is not.

try:
    await sdk.memories.create(
        document=conversation_turn,
        document_type="ai-chat-conversation",
        user_id=user_id,
        customer_id=customer_id,
        mode="fast",
    )
except Exception as e:
    logger.warning(f"Ingestion failed, will retry: {e}")
    # Queue for retry or log for manual follow-up

try {
  await sdk.memories.create({
    document: conversation_turn,
    document_type: 'ai-chat-conversation',
    user_id: user_id,
    customer_id: customer_id,
    mode: 'fast',
  });
} catch (e) {
  console.warn(`Ingestion failed, will retry: ${e}`);
    // Queue for retry or log for manual follow-up
}

try {
  await sdk.memories.create({
    document: conversation_turn,
    document_type: 'ai-chat-conversation',
    user_id: user_id,
    customer_id: customer_id,
    mode: 'fast',
  });
} catch (e) {
  console.warn(`Ingestion failed, will retry: ${e}`);
    // Queue for retry or log for manual follow-up
}

Set document_id for idempotent retries

If you retry failed ingestion calls, set a document_id to prevent duplicate memories. Derive it from your conversation or message identifier.

Bootstrap ingestion

Bootstrap ingestion loads data you already have (past conversations, documentation, customer records) in bulk with sdk.memories.batch_create(). It shares the pipeline above, runs at BOOTSTRAP priority, defaults to long-range (which behaves the same as fast), and can run alongside your live agent. For the batch method, what to check before a large load, and a full example, see Import existing history.

Runtime vs bootstrap

Both paths share the pipeline, modes, document types, and scoping rules above. They differ in how you invoke them and which defaults fit:

	Runtime	Bootstrap
Trigger	Live agent interactions, per turn	Bulk / backfill loads
Method	`sdk.memories.create()`	`sdk.memories.batch_create()`
Default mode	`long-range`	`long-range`
Throughput	One document per call, non-blocking	Many documents per call, `BOOTSTRAP`-priority queue
Best for	Real-time conversation logging	Historical data, migrations, seeding docs and knowledge

Next steps

Retrieval Modes

How fast and accurate retrieval differ, and what the ingestion mode does.

Context End to End

The full retrieve-generate-ingest loop for memory-enabled agents.

Memory Types

What the pipeline produces: the kinds of structured memory Synap stores.

Entity Resolution

How extracted entities are matched and linked across documents.

Start here

Connecting to Synap

The memory model

Memory architecture (MACA)

How memory is processed

The ingestion pipeline

Ingestion mode

Document types

Scoping every document

Runtime ingestion

The SDK call

Fitting it into the agent loop

Bootstrap ingestion

Runtime vs bootstrap

Next steps

Retrieval Modes

Context End to End

Memory Types

Entity Resolution

​The ingestion pipeline

​Ingestion mode

​Document types

​Scoping every document

​Runtime ingestion

​The SDK call

​Fitting it into the agent loop

​Bootstrap ingestion

​Runtime vs bootstrap

​Next steps

Retrieval Modes

Context End to End

Memory Types

Entity Resolution

The ingestion pipeline

Ingestion mode

Document types

Scoping every document

Runtime ingestion

The SDK call

Fitting it into the agent loop

Bootstrap ingestion

Runtime vs bootstrap

Next steps