One Belief Per Fact: Building Agent Memory for Users Who Change Their Minds

Part 2 of a three-part series on the memory architecture behind our multi-agent harness. Part 1 covered why agents should share memory instead of passing context. Part 3 explains why we deliberately scoped down the skill tier.

Here is a scenario that breaks most agent memory systems.

In session one, a user tells your financial advisory agents she is cautious with money and dislikes individual stocks. The system remembers it. In session two, she has changed her mind: she wants to go aggressive, individual tech stocks. The system remembers that too.

Now session three begins, and an agent asks memory: what is her risk appetite?

If your memory layer stores a row per session, it returns both answers, side by side, with no way to choose. We know because that is exactly what our first version did. This post is about how the semantic tier (L2 in our shorthand) actually works: what gets embedded, how beliefs update, and what happens on a contradiction. We will use one running example throughout, a user named Priya, drawn from the synthetic test scenario we ran the design against.

How we broke it first

The first cut of the semantic tier keyed facts by session. Every session inserted new rows, so after a handful of sessions there were six copies of Priya's risk tolerance, some of them contradicting each other, and no notion of "current." Retrieval surfaced whichever ones ranked highest, which meant a downstream agent could receive "cautious" and "aggressive" in the same context window.

Two other bugs from that first version are worth confessing, because they shaped the fixes. The transaction analysis agent invented income growth it was never told about, which led to a hard grounding rule: only figures explicitly stated in memory, otherwise answer "unknown," never estimate. And our "semantic" search was not semantic at all: the store had been created without an embedding index, so nothing was ever vectorized, and we were doing keyword matching while believing otherwise. The fix included a runtime check that the vector table is populated and that a paraphrased query actually retrieves the right belief.

The structural fix for the main bug is the subject of this post: one belief per fact, current, with the versioned history to prove it. That phrase is the whole design; everything below is what it takes to make it true.

What actually gets embedded

There are two embedding moments, and keeping them separate clears up most confusion about how retrieval works.

The write side. After a session, an asynchronous projector reads the raw episodic log and asks an LLM to break each entry into typed, self-contained fact sentences. From Priya's intake record it extracts something like:

Only one field is embedded: the text sentence. The store is configured to embed that field alone. The sentence goes to an embedding model and the resulting vector is stored in a separate vector table. Everything else, the key, the version, the flags, the provenance, is plain metadata for identity, filtering, and audit. It is never embedded.

The read side. When an agent asks "what is her appetite for investment risk?", the query sentence is embedded with the same model, and the nearest stored vectors win. The query and the stored fact share almost no vocabulary and still match, because both are compared by meaning.

One design point matters more than it first appears: identity never depends on the embedding. Which belief is which is decided by the stable key. The embedding only decides ranking at retrieval time. A weak embedding model can degrade what surfaces first; it can never corrupt the record.

Embeddings retrieve, keys identify, metadata governs. If you take one thing from this section, take that.

One current belief, a version chain behind it

The stable key is the fact type: risk_tolerance, house_goal, city. It is independent of session. That single change inverts the growth behavior of the tier: it grows with the number of distinct facts about the user, not with usage. Ten sessions restating "cautious" still produce one belief.

The update procedure is search-then-decide. When the projector extracts a new fact, it first fetches the existing belief under that key, then an LLM compares old against new and returns exactly one of three verdicts:

• SAME: no new information. Append this session to the provenance list. No new version.
• UPDATE: a compatible refinement or change. Archive the old version, write the new one.
• CONFLICT: a direct contradiction. Archive and supersede, and set the contradiction flag.

Here is Priya's flip, before and after. After session one:

risk_tolerance -> {
  text: "cautious, dislikes stocks",
  version: 1,
  current: true,
  flag: false
}

And after session two:

risk_tolerance -> {
  text: "aggressive, individual tech stocks",
  version: 2,
  current: true,
  flag: true,
  provenance: [S1, S2]
}

risk_tolerance__v1 -> {
  text: "cautious, dislikes stocks",
  version: 1,
  current: false
}

There is a history, but it is a version chain hung off one stable key, not a pile of equal rows. The current belief is the single row at the bare key. Past versions are archived with current: false and are not read by default.

Retrieval then needs two mechanisms working together: vector search finds the relevant fact by meaning, and a current: true filter picks the live version. The agent sees one clean answer, the newest belief, instead of the version-one situation where both came back with no way to choose.

The contradiction policy: flip, preserve, flag

When a CONFLICT is detected, three things happen at once, and the design is deliberately not "silently overwrite."

First, the current belief flips to the new value. Agents act on the user's most recent intent; the default read is most-recent.

Second, the old belief is preserved as history with full provenance. "What did she used to want, and when did it change?" stays answerable. Nothing is destroyed.

Third, the contradiction flag is set, which tells the system this belief flipped rather than smoothly refined. Downstream agents can key off it. In our chain, the compliance agent does exactly that: "note: your current aggressive strategy conflicts with your earlier caution and your house-fund goal."

Why resolve to most-recent instead of reconciling the two? Because deciding which of two contradictory beliefs is really true requires judgment the memory layer should not be making on its own. It would need a stance classifier and context it does not have at write time. So the layer's job is kept narrow and safe: surface the newest, preserve the old, flag the conflict. The decision about whether to honor the shift is handed to an agent or a human.

One belief per fact does not mean one truth per fact; it means one answer per question, with the disagreement on record.

The limitation we have not fixed yet

Honesty requires one more section. The projector treats each fact type as a single evolving slot, and superseding is correct for single-valued facts: risk tolerance, income, city. A new value should replace the old.

It is not automatically correct for multi-valued facts. "Buy a house" and "save for a baby" are both true at once; the second must not supersede the first. The current build handles this by having the extraction step assign distinct fact types (house_goal, baby_goal) so goals coexist, but that relies on the extraction LLM choosing distinct keys, which is a soft guarantee. If two goals collapse onto the same key, one wrongly overwrites the other.

The clean fix, documented but not yet built, is to mark certain fact types as multi-valued so they append as a set while single-valued beliefs keep the supersede behavior. The test is straightforward: add a second goal and assert both survive, while a risk-tolerance flip still supersedes.

Recap

What is embedded: only the distilled fact sentence, never the metadata. What is matched at read time: the embedded query sentence against embedded fact sentences, same model on both sides. How the tier grows: with distinct facts, not with sessions. One belief per fact, a version chain behind it, and a flag when the story changed. Embeddings retrieve, keys identify, metadata governs, and the judgment calls are deferred to something with more context than a database.

Part 3 turns to the tier we have said the least about: the procedural tier, the "library of skills the agents learn." That phrase turned out to promise a great deal more than the tier can deliver, and the reason why changed what we built.

Memory Architecture Series Part 1: Agents Shouldn't Pass Context. They Should Share Memory. Part 2: One Belief Per Fact: Building Agent Memory for Users Who Change Their Minds (this post) Part 3: We Scoped Down Our Agent Skill Library. You Might Not Need One at All.