Slimming Down Agent Memory: Resident vs. On-Demand

Adding memory to an agent is pure gain for the first few weeks. It remembers your preferences and last week’s traps, and you stop re-explaining yourself.

A few months later, the problems arrive. Memory files multiply, and what has to be read at every startup keeps growing. The agent “remembers” everything, but the few relevant items drown in old tasks, old status, and stale conclusions. Worse, old conclusions get treated as current facts.

I’ve written before about syncing memory across machines and agents with a git repo. This post is about something else: how to slim memory down once it has grown.

1. Two tiers: resident and on-demand

The rule is simple. There are only two tiers:

Tier	Contents	Read when
Resident	Rules, a shared summary, this machine/agent’s summary, an auto-generated recent-activity digest	Once at every startup
On demand	Direction and strategy, historical memory, project files, workflows, handoffs, inbox	Only when the current task needs them

Project context belongs to the on-demand tier; it is not a third tier. Which tier a file lives in has nothing to do with how important it is — only with whether it’s needed every time.

2. The resident tier answers only three questions

Before any lookup, the resident tier answers three questions:

1. Who am I working for: who the user is and what they mainly do.
2. What’s the current goal: current priorities and non-goals, with a review date.
3. What moved recently: which memory and project files changed in the last few days.

The third changes too fast to maintain by hand, so it is generated: at each session start, a script computes from git history which memory files changed in the last 7 days and which projects moved in the last 30, each with a one-line description, and writes them to a local file. It isn’t committed — it’s regenerated on every startup, and committing it would create a commit per session and conflicts across machines.

Beyond that, the resident tier holds only stable preferences, essential boundaries, and a few retrieval pointers.

3. What must not go into the resident tier

This matters more than what goes in. Kept out of the resident tier:

• Project backlogs and progress. They change within days; putting them in the resident tier manufactures stale facts.
• Incident narratives. An incident’s lesson can be compressed into a one-line rule in the resident tier; the narrative itself is on-demand.
• Old status. “Some branch isn’t pushed yet,” “some PR is waiting to be merged” — sentences like these are almost certainly wrong a week later.
• Generic instructions the host already provides. The tool tells the agent how to use the tool; there’s no need to copy that into memory.

One that’s easy to miss: a link is a route, not a load command. Resident files can link to topic files, but the agent shouldn’t open every link, and must never load them recursively.

4. Give the resident tier a budget

Without a budget, the resident tier will slowly put on weight. My defaults are in bytes: rules 4 KB, shared summary 4 KB, each agent’s summary 2 KB, the generated digest 2 KB — at most 12 KB per startup.

A check script runs before commits to see whether any resident file is over budget. It never truncates content — it reports, and a human decides whether to compress, move, or split. The budget is a maintenance check, not runtime permission to drop rules.

The effect is measurable. On my own store (about 180 memory files, over 300k tokens total): the old full startup loaded about 70k tokens; the tiered resident startup loads about 640; handling a specific task with one topic file comes to about 3,600.

5. The on-demand tier: scale by routing, not by reading less

The on-demand tier doesn’t scale by “reading less of one big file.” It scales by routing:

Resident summary → memory index → the one or two topic files whose descriptions match.

To make that work, big memory files are split along their existing headings into topic files, and each file gets a trigger-style description — not “about git,” but “read before git operations or parallel collaboration: pipes swallow exit codes, switching branches carries in-flight changes, no bare stash…”. A script generates the index from these descriptions. The agent reads the index, opens the one or two files whose descriptions match, and leaves the rest alone.

Splitting comes with a discipline: move text verbatim; don’t edit content during the split. Content changes are left for a later distillation, and you verify that every original heading landed in exactly one topic file.

6. Promote to resident only after repeated corrections

My criterion for promoting a rule from on-demand to resident is: repeated correction.

Take “reply in Chinese throughout; never switch to English mid-task.” It started as ordinary memory. After I corrected it three times in the same session, it was promoted to a resident rule, and now it also applies to progress updates from background tasks.

In the other direction, stale resident entries must be deleted and wrong ones fixed, with the reason for the revision recorded. One of my identity records was once wrong; the correction kept the original text for comparison and stated which sentence was void and why, so the next agent to read it wouldn’t repeat the old version.

7. An old conclusion is not a current fact

The last rule is the hardest to follow: authority doesn’t make an old factual claim current.

Every memory carries a source date and scope. When an agent reads “X is done” from three weeks ago, it should verify the current state rather than quote it. Past tasks and old handoffs aren’t current authorization either — what I asked for last week isn’t necessarily what I want this week.

Closing

The hard part of a memory system isn’t remembering. It’s what to forget, and when to recall.

My answer is two tiers: the resident tier answers only “for whom, toward what, and what moved recently,” under a budget; everything else is retrieved on demand, routed by description. Only repeatedly corrected rules get promoted; stale entries get deleted; old ones get verified.

The less an agent reads at startup, the more likely what it reads is right.