Agents Shouldn't Pass Context. They Should Share Memory.

Part 1 of a three-part series on the memory architecture behind our multi-agent harness. Part 2 covers how the semantic tier handles users who change their minds. Part 3 explains why we deliberately scoped down the skill tier.

Every additional agent in a chain is supposed to make the system smarter. Too often, it just makes the prompt bigger.

This is the failure mode almost every multi-agent system runs into, and it has nothing to do with model quality. It is token accumulation. Agent A finishes its work and hands everything to Agent B. Agent B adds its own output and hands all of it to Agent C. By the sixth agent in the chain, the prompt is carrying five predecessors' worth of context, needed or not. Then the same user comes back tomorrow, and the whole pile becomes the starting point for session two.

Costs climb. Latency climbs. And attention dilutes: the model is asked to find the three relevant facts inside forty thousand tokens of transcript.

We hit this building a proof of concept for a six-agent financial advisory chain: intake, transaction analysis, goal planning, market context, recommendation, and compliance, running in sequence, with the same user returning across many sessions and expecting continuity. This post describes the memory architecture we built to solve it, and the design decision everything else rests on.

The core idea: pass by reference, not by value

The instinctive way to connect agents is to pass context by value: each agent sends the next one a full copy of everything it knows. That is what makes tokens accumulate at every handoff.

Our design passes by reference. Agents do not hand anything to each other directly. Each agent writes its output to a shared memory layer, and the next agent retrieves only the slice it needs, capped to a token budget. If Agent A produced forty thousand tokens of investigation and Agent B needs the three thousand tokens of conclusion, B receives three thousand.

The consequence is worth stating plainly: token reduction becomes a property of how communication happens, not a feature bolted on afterwards and demonstrated with a benchmark. The budget bounds every prompt no matter how long the chain runs or how many sessions accumulate. In our team's experience, this reframing, from "how do we compress context" to "why are we copying context at all," was the single most consequential decision in the design.

In operational terms, passing by reference is the difference between a system whose cost grows with every conversation and one whose cost stays predictable months into a deployment. We think this belongs to a broader engineering discipline: TokenOps. Not prompt engineering, not model optimization, but designing systems whose token consumption is structurally bounded. The rest of this post is what that looks like in practice.

Three tiers, three jobs

Once we stopped copying context, the next question became obvious: where does the context live instead?

The memory layer has three persistent tiers. We refer to them as L1, L2, and L3 internally, but their real names describe what they hold.

The episodic tier (L1) is an append-only log of everything that happens in a session: every agent output, tagged with an identity tuple that records which tenant, which session, which step in the chain, which agent, and the sequence of the write. Nothing here is distilled, deduplicated, or embedded. It is the raw record and the source of truth. Writes are synchronous: an agent's turn is not done until its row is in the log.

The semantic tier (L2) holds distilled facts that persist across sessions: "this user is cautious with money," "this user wants a house in five years." Where the episodic log is keyed by when something was said, the semantic tier is keyed by what the fact is about. It holds one current belief per fact, with versioned history, and it is vector indexed so agents can retrieve facts by meaning rather than by keyword. Part 2 of this series is entirely about how this tier behaves.

The procedural tier (L3) holds reusable skills: organization-specific procedures for how to do something well, authored deliberately rather than learned by observation. It is the tier we scoped down the most, for reasons that deserve their own post. That is Part 3.

One rule ties the tiers together: facts flow upward, never back down. The episodic log is raw truth. The semantic tier is a projection distilled from it. The procedural tier is a library read alongside them. Because L2 and L3 are projections, they can always be rebuilt from L1. If an extraction bug corrupts your beliefs, you replay the log. Nothing downstream is ever the only copy of anything.

The write path: fast now, thorough later

Distilling raw output into clean facts requires an LLM call, and putting an LLM call on the hot path of every agent turn is exactly the latency this design exists to avoid. So the write path is split in two.

During the session, agents append to the episodic log synchronously. The write acknowledges as soon as the row lands. That is the whole cost an agent ever pays to write.

After the session, an asynchronous projector reads the new log entries, extracts typed fact sentences with an LLM, and merges them into the semantic tier using a search-then-decide procedure (Part 2 covers this in detail). Because this runs off the hot path, it can be slow, retried, or re-run entirely without touching agent latency. The semantic tier is a projection that is allowed to lag.

The read path: hybrid retrieval closes the gap

If the semantic tier is allowed to lag, a reader could see stale data. The fix is that no agent ever reads only one tier. Every retrieval queries the episodic log (fresh, raw, this session) and the semantic tier (distilled, cross-session) together, merges and deduplicates the results, and packs them into a fixed token budget before anything enters the prompt.

This has a property that took us a while to appreciate: the projector's lag is invisible. Anything too fresh to have been distilled into the semantic tier is, by definition, still in this session's episodic log, so the reader sees it from there. The agent is never blind, and the distillation never blocks anyone.

The budget allocator defaults to an even split between the two sources, on the logic that a typical turn needs both "what just happened" and "what do we know about this person."

Reads stay real-time throughout: log lookups are indexed table reads, and vector search over distilled beliefs is fast because the tier is small by construction.

Why the reduction is structural

Four mechanisms, all architectural rather than tuned:

Communication is by reference. No agent's prompt ever contains another agent's full output, only the retrieved slice. History is queried, not accumulated. The agent at turn ten does not carry turns one through nine verbatim. It queries recent raw events and older distilled facts and receives a budget-capped slice. Distillation happens once, at write time. Turning a thousand-token observation into a fifty-token fact costs one asynchronous LLM call. Every later read pays only the small cost. Naive context passing pays the large cost on every turn. Large blobs are stored by reference too. Tool outputs and files live in object storage; the log keeps a pointer, and the allocator inlines a blob only when it is relevant.

Informally: the naive cost per round is the sum of every agent's full history, and it grows without bound. This architecture's cost per round is the sum of each agent's budget-capped slice, and the budget is fixed. The gap between the two is the reduction, and it widens as sessions and agent counts grow.

What this looks like in a chain

In the six-agent chain, the "turn" is an agent. Each agent reads a capped slice of memory that already contains everything agents before it wrote, does its work, and appends its own output back to the log. The shared context of the chain is the log itself, not a payload passed hand to hand. When the user returns for session two, the semantic tier's cross-session beliefs feed the reads from the first turn, so the chain starts already knowing the user without replaying session one.

The same primitive covers single-agent multi-turn work, sequential chains, and parallel multi-agent rounds. That universality was a design goal: one session model, with a round counter and an agent identifier, rather than a special case per workload.

What's next

Replacing context passing with shared memory solved the scaling problem. It immediately exposed a harder one: what happens when memory itself changes? What exactly gets embedded, what happens when a user contradicts something they said last month, and how does the system avoid returning two conflicting beliefs with no way to choose?

That is Part 2.

And then there is the tier we have not really discussed. The procedural tier sounds, on first hearing, like "a library of skills the agents learn." We started there too.

Part 3 is about why that phrase promises more than the tier can deliver, and what we built instead.

Memory Architecture Series Part 1: Agents Shouldn't Pass Context. They Should Share Memory. (this post) Part 2: One Belief Per Fact: Building Agent Memory for Users Who Change Their Minds Part 3: We Scoped Down Our Agent Skill Library. You Might Not Need One at All.