How Vellum Remembers

Models alone have the memory of a goldfish. We all have the experience of reminding ChatGPT or Claude the info it missed when using it.

We built a memory system to fix that.

For AI assistants to work best, we need them to actually remember who we are and what we are working on. It took us three tries to get the Vellum Assistant’s memory to actually remember.

Here’s what we learned in the process.

Three versions, three verdicts

V1: Memory as atoms

The first version was a markdown knowledge base injected into the system prompt, with an LLM extraction pipeline on top. As you chatted, a model pulled out individual facts and stored each one as a node in a graph, and the graph had its own retriever, extraction triggers, decay logic, and consolidation pipeline. Very clever. We were very proud of it.

But it didn't work. Retrieval missed things that mattered and surfaced things that didn't. You'd have a great conversation, the system would dutifully extract a pile of nodes, and the next day a follow-up question would come back with nothing useful.

The problem was compression. Extracting atomic facts from a conversation, then trying to reconstruct relevance from those atoms, loses information at every step. A fact like "user is stressed about the launch" means something inside the conversation it came from. As a lonely node in a graph, it's a fortune cookie.

We set our sights on finding the balance of compressing information down without losing context using a pages approach.

V2: Memory as pages

We ditched the fact-atoms in V1 for a V2 system structured around .md files: real pages with sections that acted as a personal wiki the assistant maintains about your world.

At first, we used a scoring model to "spread activation." Think of every page as having a warmth score. A page warms up when it resembles what's being discussed (your message, the assistant's reply, and a now.md file holding current context) and cools down as turns pass. Warmth also spreads across links: a page about your health could warm up because a page about your medication did, even if the message only mentioned the medication.

After trial and error, an LLM router emerged as an alternative to spreading activation. It showed the model the page index plus the last turn (one forced tool call) and let it pick what was relevant. This worked better, because the model could actually reason about relevance instead of measuring word overlap.

But page-level scoring had no real signal for whether a retrieved page would help the current turn. The nascent activation vector system, while elegant, was a heuristic pretending to be a relevance model. Pages warmed up because they shared keywords with the message, even when the actual connection was thin.

V3: Agentic memory

V3 is where things finally clicked.

The whole pipeline, from "worth remembering" to "in the prompt," looks like this:

The basic idea goes like this:

• Capture: The assistant appends memories mid-conversation.
• Organize: A background agent files them into concept pages, then indexes their sections and links.
• Retrieve: Stable lanes and per-message finders build a candidate pool that errs on the side of too much.
• Inject: A gate decides whether selection is even worthwhile, then stable cards and a fresh spotlight enter the prompt.

Capture: Save first, organize later

When something seems worth remembering based on the system prompt tells the assistant what qualifies, it calls the remember tool mid-conversation. That's the entire capture step: one timestamped bullet appended to two files in the assistant's workspace.

memory/buffer.md: the working inbox, holding notes that await filing.

memory/archive/[date].md: the immutable daily journal. Never edited, never deleted.

No extraction pipeline, no graph surgery, no clever schema. A line in a file. It turns out the least lossy way to capture a thought is to just write it down.

Consolidation: Librarian background agent

Every so often, a background agent wakes up to play librarian. Crucially, it is not some stripped-down side process: it runs with the same system prompt, the same understanding of you, and the same reasoning ability as the main assistant.

It reads the buffer and, entry by entry:

• Files each note into a concept page under memory/concepts/, creating pages for new topics and editing existing pages when a topic evolves.
• Trims the buffer of everything it processed.
• Rewrites three digest views (essentials.md, threads.md, and recent.md) that feed the static info block loaded into every conversation.

Consolidation runs every 8 hours by default. It’ll run sooner if the buffer hits a size threshold or you save something yourself.

Because the librarian is the assistant, it exercises judgment rather than string matching. A buffer entry like "user mentioned they stopped taking that medication last month" gets applied as a correction to the existing health page, rather than it filed as a shiny new fact that contradicts the old one as prior memory systems would do.

Indexing: Making the pages searchable

After consolidation, the system rebuilds its search machinery: changed pages are split into sections and re-embedded (converted into vectors that capture meaning, so "my knee hurts" can find a page that says "joint pain"), deleted pages are pruned from the vector store, and the link graph, entity index, and lane caches are invalidated so the next turn rebuilds them fresh.

The important part: the index is built over real, human-readable pages. Every searchable unit is a section of a page you could open and read. This was huge for our power users that wanted greater control over their assistants’ memories without diving into raw embeddings or database rows.

Retrieval: Ten lanes cast a wide net

When you send a message, ten retrieval lanes run, split into two groups. First, four stable lanes contribute candidates no matter what the message says:

Then six per-message finders hunt based on what you actually said:

Lanes only ever add candidates; nothing at this stage can veto a page. That makes the pool recall-safe by construction: if a page could plausibly help, it's in the net. Deciding what actually gets used happens next.

The injection gate: Bouncer that saves you money

A wide net has a cost: many messages don't need memory at all. "Hey, what's up" does not require a dossier. So before spending an LLM call on selection, a gate decides whether this turn deserves one.

The gate is plain arithmetic, no model involved. It looks only at the current message's needle (keyword) and dense (meaning) scores, and it lets the turn through if any of three conditions hold:

The injection gate avoids an LLM call unless lexical or semantic retrieval produces a strong enough signal. Most turns pay no memory-selection cost.

• Dense pass: the single best meaning-match scores at or above 0.66. One clearly relevant hit is enough.
• Dense cluster: every one of the top 3 meaning-matches clears 0.6, and they sit within 0.02 of each other. No single standout, but several consistently good hits pointing the same way.
• Sparse-only strong: when meaning-based search returns nothing useful, keyword evidence alone can still pass, but the bar is higher: 0.75.

Fail all three and the gate skips the selector entirely: no LLM call, no memory block in the prompt, no added latency. Most turns take this exit.

The turns that do pass ("remember that highly sensitive thing we talked about last week?") get the full weight of retrieval and selection.

Selection: An LLM makes the final call

If the gate passes, an LLM call reads the candidate cards and picks the pages actually worth injecting. Choosing what's relevant is a judgment call, which is exactly why it's done by a model and not more arithmetic.

One nerdy but consequential detail: the selector's prompt is split into two blocks so the stable part (the candidate cards, marked with an explicit cache breakpoint) can be cached by the provider and reused across turns, while the per-turn context sits in a second block with automatic caching disabled.

Without that split, the provider would try to cache text that changes every turn, paying the cost of caching with none of the benefit.

Injection: What the Assistant finally sees

The memory that reaches the prompt comes in two layers: a stable layer that accumulates over the conversation, and a spotlight that changes every turn. Think fridge magnets versus a sticky note on the monitor.

The stable layer, aka the fridge magnets (the memory cards cards block). Pages selected earlier in the conversation render once as compact cards and are then frozen: byte-identical from turn to turn, so they sit happily in the provider's prompt cache and survive restarts. Only genuinely new selections get appended, deduped against everything already injected. Re-rendering the whole set each turn would invalidate the cache from the first memory block onward, which is quite an expensive way to say the same thing twice!

The spotlight, aka the sticky note (the memory spotlight block). Rebuilt every turn: the sections that matched the current message, plus remembered entries from the last couple of turns, deduplicated by page and heading so nothing appears twice.

The assistant reads both layers as ground truth about you and the ongoing relationship. And because frozen cards accumulate, a prune valve bounds their total footprint so the memory prefix can't grow forever.

Retrospectives catch what real-time capture misses

There are two main paths for information to enter the memory system.

Two paths into memory: consolidation files what was captured in the moment, while the retrospective forks the conversation afterward to catch what was missed and save repeated procedures as skills.

The first is consolidation, covered above: remember captures facts in the moment, and the librarian files them.

The second (my personal favorite) is the retrospective task. Real-time capture depends on the assistant noticing that something is memorable while it's busy doing its actual job, and nobody takes perfect notes mid-meeting. So afterward, the system forks the conversation and sends the fork one instruction, roughly: "read this back and save anything worth remembering that wasn't already captured." The fork inherits the conversation's compaction state, so it reads the discussion natively rather than from a summary.

When it runs: after 30 minutes of conversation or 10 messages, with a 5-minute cooldown between attempts. A backstop sweep runs every 8 hours with a 7-day look back to catch conversations that ended abnormally and never got their review.

The result: memory no longer depends on the model perfectly spotting every memorable moment in real time. There's a second pair of eyes running on the same brain.

Procedural memory turns remembering into learning

Now, here’s why I’m bullish on the retrospective task. Ready for it?

It catches procedures as well as facts.

A skill is a saved procedure: how to perform a task, step by step, with the right parameters and logic. Facts tell the assistant about you; skills tell it how to do things for you.

When the retrospective spots something the assistant did well and could plausibly do again, it saves that as a skill. Two guardrails keep the library honest:

• It must have actually happened. A skill is authored only from a procedure the assistant genuinely carried out, with real executed tool steps. No aspirational skills. (Beyond that, the retrospective prompt says to keep the bar low and author when in doubt.)
• No duplicates. Before authoring, the agent checks for similar existing skills and dedupes.

Skills then live in the same collection as concept pages, under a skills/[id] slug, and are scored by the same ten retrieval lanes, with a payload marker so the system never mistakes one for an ordinary authored page.

That placement is the point: remembering a fact and recalling a capability are the same motion. Uninstalled catalog skills are seeded into the index too, so the assistant can discover and auto-install one when your request matches what it does. And you can update and refine any of these skills yourself.

Over time, the assistant accumulates a library of procedures learned through practice, alongside facts it has been told.

The end goal here is to build AGI. No, I'm not kidding.

An assistant that keeps accumulating knowledge of how to accomplish things gets more capable over time without anyone spending billions on training runs.

The concept graph makes memory inspectable

There's a trust problem with any memory system: if you can't see what the assistant knows about you, it's a black box, and nobody fully trusts a black box with their personal context.

So we visualized the assistant’s memory for you to look at. Every concept page is a node, every authored link is a directed edge carrying its curated description, and the whole thing renders as an interactive graph. You can see which concepts connect to which, how your knowledge is structured, and what the assistant actually knows.

A live concept graph: 58 nodes across 11 lobes, color-coded by category. Edges are authored links between concept pages. Pending nodes in yellow are buffer entries awaiting consolidation.

Notes still waiting in the buffer show up as pending nodes, so a memory appears in the graph the moment it's saved, not hours later when the librarian gets around to filing it.

And you're more than a spectator: you can create, refine, or delete memories directly from this view.

That matters more than it sounds. If you don't trust that the assistant remembers correctly, you'll keep repeating yourself, and the whole system becomes pointless. The graph makes memory legible, and legibility is what builds trust.

What we learned

The tl;dr:

• Capture should be dumb. A timestamped bullet in a file is the least lossy capture mechanism there is. Every ounce of cleverness added at capture time is information lost before anything is stored. v1 had an extraction graph with its own retriever, triggers, and decay; v3 has remember appending a line to a file. Guess which one works.
• Lean into the agent loop. v1 built bespoke machinery for extracting, filing, and retrieving. v3 threw it all away and let the agent do those jobs itself: consolidation runs as the assistant with full context, and the retrospective wakes a fork of the assistant. The same model, prompt, and reasoning that carry the conversation also build the memory, and that's precisely why the memory is good.
• Procedural memory is the differentiator. Facts make an assistant context-aware; procedures make it capable of growth and execution. The same pipeline that stores what it knows can store what it can do, and that's what turns a memory system into a learning system.
• Is it done? No. The injection gate and the graph UI both have plenty of room to improve. But I'm willing to bet the architecture is right.

We want to build AI that’s pro-human and I believe memory is an integral piece to realizing that. AI that truly remembers and understands you will lead to a relationship that points it to support its user beyond its post-trained abilities.

Vellum assistants dream about ways to use your raw and unique context and creativity to build and innovate for the future. AGI will be a personal intelligence, how we build it for the benefit of humanity is the question we are trying to answer every day at Vellum.