We Scoped Down Our Agent Skill Library. You Might Not Need One at All.

Part 3 of a three-part series on the memory architecture behind our multi-agent harness. Part 1 covered why agents should share memory instead of passing context. Part 2 covered how the semantic tier handles users who change their minds.

"A library of skills the agents learn." It is a seductive phrase, and it was our original plan for the third tier of our memory architecture: an agent solves something, the system captures the skill it used, and the skill is stored for every agent to reuse. The library fills itself by watching agents work.

That plan does not survive contact with how agents are actually deployed. This post is the honest version of what the procedural tier (L3 in our shorthand) can and cannot do, why we deliberately dropped skill-learning-by-observation as a design goal rather than deferring it, and the one question that decides whether an organization should build this tier at all. By the end, the entire decision compresses into a single yes-or-no question. Spoiler on the answer: sometimes it is no, and we think saying so is a feature of the architecture rather than a gap in it.

First, what a skill actually is

Most of the confusion around skill libraries traces back to definitions, so here is the line we hold.

A tool is a raw capability: calling a PDF library, hitting an API. Tools are wired into the agent. They do not live in memory.

A fact is about a user or the world: "this client wants a house in five years." Facts belong in the semantic tier.

A skill is a procedure: "to extract totals from our vendor's invoices, the total is in the footer, dates are DD-MM-YY, validate the sum against the line items." Learned, reusable, and specific to how one organization operates.

The test for whether something belongs in the procedural tier is that it is procedural, non-trivial, reused, and specific to how you operate. If it is a raw ability, it is a tool. If it is about someone, it is a fact. And if the base model already knows it, it is noise. Keeping that line sharp is what stops the tier from becoming a dumping ground.

Where a skill library genuinely pays off

Four situations, and it is worth noticing they are not four kinds of content. They are four situations in which the same stored artifact gets used, differing only in who benefits.

• Avoiding re-derivation. An agent that re-reasons the same procedure on every run can retrieve the proven version and follow it instead. Cheaper, and it cannot get the logic wrong this time, because it is not reinventing it.
• Cross-agent transfer. A procedure worked out by one team's agent becomes available to a different agent that never solved it. This is the only case that truly needs organization-level scope; within one agent, it is just memory.
• Consistency. Shared conventions, output formats, phrasing contracts, live in one place, so every agent behaves the same way instead of each improvising and breaking whatever consumes its output.
• Onboarding. A newly deployed agent inherits the accumulated procedures immediately and starts competent rather than naive.

All four share one mechanic: an agent describes a need, and the library returns a procedure that was authored once and reused. Hold onto the word "authored." It is about to matter.

The wall: you cannot capture skills from a hosted API call

Here is where deployment reality bites. Most production agents today are calls to hosted models. The call returns output, and nothing else. There is no reasoning trace, and there is not even a discrete "skill" inside a completion to capture. The model produced tokens.

You cannot store a skill you never observed.

So any attempt to extract "the skill the agent used" from a response is reverse-engineering a plausible-looking procedure from the answer alone, with no way to tell whether it reflects what actually produced the result. Our own episodic log does not rescue this either: it records agent outputs, not reasoning traces, so mining it for procedures hits exactly the same wall.

In our assessment, this failure mode is worse than storing nothing. A confident, clean-looking skill that does not reflect reality will get retrieved, trusted, and reused, and then it will fail somewhere downstream where nobody is looking for it. An empty library is honest. A library of plausible guesses is a liability.

The conclusion we reached is worth stating as bluntly internally as we did: populating a skill library by observing running agents is a dead end for hosted models, and we dropped it as a design goal rather than deferring it. It becomes worth revisiting only if two things become true at once: faithful action traces are captured, and we own prompt construction for that agent. Neither is on the horizon for the deployments we see.

The reframe that keeps the tier usable

The black-box call only kills the tier if you think the tier's job is to observe skills from runs. It is not. There are two directions of traffic, and only one of them is affected.

Writing to the library happens by authoring. A person writes a skill file. A vetted skill is imported from a public library. An agent is explicitly asked to write one up after a success, as a deliberate act with review. In every case the skill is created on purpose as an artifact, never harvested passively from a run. Authored, not observed.

Reading from the library works fine for a black-box agent. The orchestration layer retrieves the relevant skill and injects it into the prompt before the API call. You do not need to see inside the model. You control the input to it.

That last clause is the real precondition, and it is the question we now ask before anything else in an integration. Not "can we see inside the model," because nobody can. The question is: do you control the prompt that goes into it? If the agent is custom and your platform assembles its prompt, the skill tier works. If it is a sealed third-party agent product where you only hit an endpoint, the tier has nothing to attach to for that agent, and no amount of library curation changes that.

The harder question: store skills, or just download them?

Even granting all of the above, there is a fair challenge that deserves a straight answer: why curate a library speculatively when an agent could fetch a skill from the public ecosystem at the moment it needs one?

One distraction to clear first: storage cost is a red herring. Skills are small text files; a few hundred of them amount to megabytes. The real cost of pre-storing is human effort, the selecting, reviewing, security-scanning, and maintaining of the library. So the honest comparison is curation effort versus fetching on demand.

Download on demand wins when skill needs are unpredictable and long-tailed (you would curate five hundred and use twelve), when the skills are generic (read a PDF, scrape a page, parse a CSV) and the public ecosystem already maintains them better than you will, and when nothing the agent does is specific to your organization. For a team running hosted-model agents on generic tasks, that is usually the whole picture, and the right recommendation is to skip the library and fetch.

The one question that decides it

Per deployment, everything above compresses into a single question: does this agent rely on procedures that are either unique to the organization or too sensitive to run unreviewed?

If yes, that slice justifies a small, curated procedural tier, populated by authoring and read by prompt injection.

If no, download on demand and do not build the tier. And if that answer holds for a given deployment, we would rather say so than sell a tier that sits half-empty. An architecture that tells you when one of its own components is unnecessary is, we would argue, more trustworthy for it.

That closes the series. The episodic tier is the append-only truth. The semantic tier is one belief per fact, with the history to prove it. And the procedural tier is smaller than the hype suggested, on purpose, and everything in it is authored, not observed.

Memory Architecture Series Part 1: Agents Shouldn't Pass Context. They Should Share Memory. Part 2: One Belief Per Fact: Building Agent Memory for Users Who Change Their Minds Part 3: We Scoped Down Our Agent Skill Library. You Might Not Need One at All. (this post)