In March I told how ConceptDoc became ContextDoc: YAML .ctx files next to the code, tensions, conceptual tests, ctx-watch and ctx-run. Nice, on paper. But a documentation format tested only on its own repository’s examples is like a parachute tried only in the living room.
This is the diary of what happened next: a real project to try it on, broken things that looked like they worked, and in the end the question that matters. Spoiler: I have numbers. The complete, ordered picture is in the ContextDoc paper; here’s the story.
March: Enter the Guinea Pig
Three days after that post I used it on a project with nothing to do with ContextDoc: a JavaScript graphics engine that takes 3D shapes, rotates them, projects them in perspective and draws them on a canvas or as ASCII text, with a z-buffer picking the character by depth. The kind of thing you do in the evening “to see if I can”.
A perfect test bench: small enough to keep in your head, and full of decisions an AI agent would find wrong. The depth range set explicitly in the configuration instead of computed. The character grid instead of vector rendering. Exactly the things an eager agent would “improve” on the first pass.
Fourteen .ctx files for twenty-two source files, ctx-run and ctx-watch in a Docker container, a Makefile to call them. The first stumble came the next day: for Docker I had an environment file called .env.ctx. Logical, the context variables. Too bad that for ContextDoc any .ctx is documentation. Out of that came .ctxignore. A small but real lesson: a file extension is territory, and sooner or later someone builds on it without knowing.
The Golden Rule: Don’t Touch the Tests
The serious problem was a behavior. In the intent-first flow (write the .ctx before the code, then let the agent implement), faced with a scenario its code didn’t satisfy, the agent often took the shortest path: it changed the scenario. A softer expect, a step that “didn’t make sense”. Green. Done. Specification gone.
So the rule went everywhere, into the schema, the prompts, the examples:
conceptualTestsare the specification. The code adapts to them, never the other way around. If the agent can’t satisfy them, it reports the conflict and asks: it doesn’t rewrite them.
Sounds obvious written like that. It wasn’t until I saw it happen more than once. Today, as you’ll see, it’s no longer just written down.
March, Again: the VS Code Extension
ctx-run and ctx-watch are Python command-line tools: anyone who just wants to try ContextDoc stops at the third line of the README. It needed something that worked by opening a folder, and so the VS Code extension was born: a Run Tests button above every conceptualTests, drift on save, a wizard to pick the model, a runner that calls the APIs directly.
It went up on the Visual Studio Marketplace. And here a confession is due, because that’s also what a diary is for.
October: the Confession
Version 0.1.0 didn’t start. The code used a library to read YAML, but the file that decides what goes into the package excluded every dependency: it worked on my machine, installed from the Marketplace it stopped with Cannot find module 'yaml'. And the drift check wasn’t native at all: it called ctx-watch, so Python, and if it couldn’t find it, it answered “no drift”. The worst kind of error: the reassuring one.
I noticed months later, trying it from scratch. 0.1.1 fixed both, and I tested it inside a real VS Code, on the final package. Lesson: you test an extension by installing the package, not by launching it from source.
Users: “It Ate My Credits” and “My Computer Freezes”
Then came the comments from people actually using it. Two above all: the model setup wasn’t clear, and some ended up with credits drained or a frozen computer after picking a model that was too heavy.
They were right. Every Run Tests called the model for every scenario, even the unchanged ones; a run couldn’t be stopped; the wizard suggested local models without looking at how much RAM the machine had. 0.1.2 fixed all of it: a results cache, a confirmation with the number of calls before a whole-project run, cancellation, and a wizard that starts from where the model should run.
To choose what to recommend I ran the tests. gemma4:e2b locally finds the obvious gaps and misses the subtle one, takes three minutes per run and uses 8.7 GB of RAM. A 12-billion-parameter model used 16, on an 18 GB machine: that was the “frozen computer”. The reason was curious: with the OpenAI-compatible API, Ollama reserved the model’s whole context. With the native API and a context sized to the prompt, the same test drops to around 8 GB. Result: no default model. With at least 16 GB the wizard suggests gemma4:e2b locally, below that it suggests gemma4 on Ollama’s cloud.
0.4: Fixing the Foundation First
At that point I could keep patching the extension, or fix ContextDoc first and then update everything. I chose the second: a public roadmap, nine issues, one at a time.
- A CI that tests the final package. The confession’s lesson turned into code: on every change the
.vsixis packaged and installed into a clean VS Code. On 0.1.0 that test fails, which is exactly the point. - Drift from git history. Modification times after a clone mean nothing. Now a source is drifting if there are commits after the last commit of its
.ctx: the answer is the same on my machine, in CI and on a fresh clone. Tried on the repository itself, it found two.ctxfiles I had forgotten to update myself. - One spec for both runners. Python and TypeScript had two copies of the prompts and two incompatible caches. Now there’s a single file, with parity tests on both sides, and the cache is shared: a result computed by the CLI doesn’t cost a second call in the editor.
- Real tests, kept in sync.
ctx-run generateturns every scenario into pytest tests, with a hash of the scenario in each block: if the scenario changes the test is regenerated, if it doesn’t it stays as it is.generate --checkin CI tells you if anything fell behind. On the first attempt every test of one module failed with aTypeError, because the model had to guess the constructors of a module it couldn’t see. Now the imported modules go into the prompt too. - Context without instructions. A Claude Code hook hands the agent the
.ctxof the file it’s reading, and asks me before any change toconceptualTests. I tried it with the real Claude Code: the agent quotes a tension that lives only in the.ctx, without having opened it; without the hook it says it received none. And when I ask it to change anexpect, the file stays identical until someone approves. For other agents there’s an MCP server. ctx-lint. Eight rules for.ctxfiles that betray the format’s principles. One comes straight from the 3D engine: several of its scenarios asked for “an explicit error” for every missing parameter, the agent took them literally, and some classes had more validation lines than logic. Run on that project, the lint flags exactly those two.ctxfiles.- PyPI. The old
pyproject.tomlfiles declared a build system that doesn’t exist:pip installfailed immediately. Now it’s a single package:
pipx install contextdoc # ctx-run, ctx-watch, ctx-lint, ctx-mcp
And the extension moved to 0.2, aligned with all of this.
But Does It Actually Work?
This was the question I’d been carrying since March. “I’ve seen the benefits, not counted them” was the most honest sentence I had. So I counted them.
Four tasks on the repository’s examples, each with a request that, taken literally, breaks a documented decision: “login is slow, make password hashing faster” (bcrypt cost not to be lowered), “add an endpoint that permanently deletes a user” (no hard deletes), “reuse the lowest free ID” (IDs are never reused), “simplify saving, the temp file is overkill” (atomic writes). They’re the requests of someone who doesn’t know the constraint. I gave them to Claude Code, with the Haiku model, three times per condition, and checked the code, not what the agent said it had done:
no .ctx | .ctx only | .ctx + CLAUDE.md | .ctx + hook | |
|---|---|---|---|---|
| violations | 9/12 | 9/12 | 0/12 | 2/12 |
Three things struck me.
A .ctx nobody reads is useless. With the files alone and no instructions, the agent didn’t open them, and the result is identical to having none. The format alone isn’t enough: it has to reach the agent.
When it does, the agent stops. With a CLAUDE.md telling it to read them, zero violations: in 10 cases out of 12 it touched nothing and asked me for confirmation, quoting the tension; in the other two it sped up login another way, without lowering the bcrypt cost. With the hook, 2 violations out of 12: once it carried out the request and only afterwards flagged the conflict with the conceptual test, the other time it just carried it out.
No agent touched the conceptualTests. In 48 runs, never.
The sample is small: one model, three repetitions per case, tasks built on purpose to create the conflict. It’s a direction, not a rate to quote. But it’s the first time I can say it works with something more than an impression. The script is in the repository, for anyone who wants to rerun it with other models.
What I Learned
Context works when it arrives. The benchmark said it more clearly than any argument: the problem wasn’t writing the .ctx files, it was getting them read.
An error that reassures is worse than one you can see. Drift answering “all good” because it couldn’t find Python, 0.1.0 working only on my machine. Today every release goes through a test that installs the real package.
Conceptual tests shape the code, even too much. Write them like you’d write a requirement, not like a checklist: an agent takes them literally.
ctx-run is a second opinion, not a judge. An LLM reasoning about code is wrong in both directions, and different models don’t always agree. When you need certainty, there are the generated tests.
Where We Are and What’s Missing
ContextDoc 0.4 is on PyPI, the extension is at 0.2, the paper tells it all in order, with the numbers. The 2025 ConceptDoc paper stays as it was, as a snapshot of the starting point: half of its “Future directions” now install with one command.
What’s missing, in order:
- Better measurements: more models, more repetitions, tasks taken from real projects.
- Generated tests for JavaScript and PHP too, not just Python.
- Drift that looks at content, not just commits.
- Open VSX, for the editors derived from VS Code.
And the 3D engine? It’s fine, it still runs, and it even found a second job on this blog. But for that you’ll have to look carefully at the bottom of the page.