What "memory" is — the token problem, RAG and the knowledge graph

An agent's memory doesn't live in its head — it lives in your folder. The moment we call "remembering" is the moment the right file gets read behind the scenes; every flawless memory system you see on X is built on that one sentence.

Four months ago I published the memory video — the first video on my channel — and it drew far more comments than I expected. The video was short, so most of the questions went unanswered. Now I'm reopening the same topic, leaving nothing out, in a six-part series of articles alongside my new 51-minute video.

The problem itself hasn't changed in the meantime. Even on a Max plan I still run out of limit, because the moment I hand over a prompt the agent reads every file it can find — needed or not — and burns tokens. And there was this, too: no matter how big the chat window grew, nothing carried over between sessions.

This first article puts three things in place: what "remembering" actually means, the two paths and three parts underneath every memory system, and the RAG versus graph question.

The memory idea was born from a complaint, not a fascination

The biggest complaint of the past six months was tokens, limits and cost. Switching models, switching effort, compact, clear — we tried them all. Nobody showed up saying "I want to build the perfect memory system"; everybody showed up asking "why does my limit keep running out".

The cause was always the same. The agent was reading files we didn't want it to read, and it kept no information about us. Every new session it went back to the same files and read them from scratch. Second brains, vault files, graphs — everything people talk about today grew out of that pain.

There's a world of difference between the Claude Code of four months ago and today's; Anthropic ships updates almost daily. But the underlying need hasn't moved: whether the agent finds the right file still depends on the structure you build.

Before we get to the solutions, the word at the centre of all this needs pinning down: remembering.

What "remembering" really is: reading the right file

So what actually happens when an agent "remembers"? Nothing changes inside its head; behind the scenes, the right file — written earlier — is found and read. That is remembering, exactly.

The only way to add lasting knowledge to the model itself is to retrain it. You don't want to train models, and I don't want to spend hours here explaining model training. None of the systems we build involve any training; the only thing we do is set up the file structure behind the scenes.

The "the agent is improving itself" claim also falls into place from this angle: what grows isn't the model, it's the data accumulating about you. The more data the agent collects, the better it remembers — because there are more right files for it to read. There is exactly one goal: on every input, the agent reads the right files. That's it.

The first half of the skeleton — two paths: writing and reading

The post I see most on X goes: "Here's my agent system, and here's the memory architecture I built behind it." Look at the foundations and they all consist of two paths and three parts.

• The write path. After the conversation ends, the agent analyzes and synthesizes what was said and writes it to files.
• The read path. When a question comes in, the right file is found, read, and its content flows back into the conversation.

Neither works without the other. Without writing there's nothing to read; without reading, everything you wrote is dead weight. And our token trouble comes precisely from the read path: if you don't do the writing properly, the agent reads every file it has.

The second half of the skeleton — three parts: storage, map, rules file

• Storage. Your data — your markdown files. These are the shops in the metaphor.
• Map. The index file. It shows the agent which file to go to, the way Google Maps does.
• Rules file. Known as CLAUDE.md in Claude Code and AGENT.md in other CLIs. This file tells the agent how to read the map.

The same trio sits in my own vault: index.md is my map, the folders are my storage, CLAUDE.md is my rules file. Once these three are in place, the thing we call a "memory system" is built. The rest is polish.

This is not RAG

Retrieval Augmented Generation exists for the big-data problem. At the first company I worked for, we used RAG for three-dimensional coordinate files far too large for any agent to sit down and interpret. The data is split into chunks, converted into vectors by an embedding model and stored somewhere separate; when a query comes in, the nearest vectors are found and stitched together. Sometimes two, sometimes three separate models run in that chain.

The yardstick is this: if your data is too big for an agent to read file by file, you go into RAG. If you have a few hundred markdown files, you don't. On our path the retriever is the model itself: it reads the index, runs grep if it has to, opens the file; the markdown it finds lands in context, and the same model produces the answer.

The graph: nodes, edges, and the tokens that burn when an edge is missing

That famous dot cloud in Obsidian screenshots decodes simply. Every dot is a node — a file; every line is an edge — the link between two files. Without an edge, the agent cannot know it needs to go from file A to file B: it either reads a few unnecessary files on top, never finds the target at all, or hallucinates.

The numbers back this up. My vault holds 74 files and 952 wikilinks; an agent starting from the index reaches 92% of the files in two hops. Delete the square-bracket links and the number of reachable files drops from 68 to 19. The graph is not decoration; it is the token bill itself.

Try it yourself — the five-step summary

• Split your notes into one markdown file per topic.
• Put an index.md at the root and introduce every file in a single line.
• In CLAUDE.md, write down how the agent should read the map.
• Connect files that touch each other with square-bracket links.
• Ask the agent about an old piece of work and watch which files it reads.

In short: building a memory system is not making the agent smarter — it is writing your data in a shape the agent can find. The entire system I built is open source: claude-obsidian-kit. In the next part we descend into Claude Code's own memory.