# ContextDoc Diary: from a 3D Guinea Pig to 0.4, with Numbers > From March to today ContextDoc went from a side project to a package on PyPI: a 3D guinea pig, the rule that forbids agents from rewriting tests, a VS Code extension whose first version didn't even start, and in the end a simple question. Does it work? I measured it: without ContextDoc the agent breaks a documented decision 9 times out of 12, when it reads the .ctx files 0 out of 12. - URL: https://mautoblog.com/en/posts/contextdoc-diary-vscode-extension-conceptual-tests-2026/ - Language: en - Published: 2026-10-17 - Author: Mauto - Categories: Projects, Artificial Intelligence - Tags: ContextDoc, AI-Assisted Development, VS Code, Open Source, LLM - Guide: ContextDoc (part 2) - Translation (it): https://mautoblog.com/posts/contextdoc-diario-estensione-vscode-test-concettuali-2026/ --- In March I told [how ConceptDoc became ContextDoc](/en/posts/contextdoc-coding-agent-ai-documentation-ctx-guide-2026/): YAML `.ctx` files next to the code, tensions, conceptual tests, `ctx-watch` and `ctx-run`. Nice, on paper. But a documentation format tested only on its own repository's examples is like a parachute tried only in the living room. This is the diary of what happened next: a real project to try it on, broken things that looked like they worked, and in the end the question that matters. Spoiler: I have numbers. The complete, ordered picture is in the [ContextDoc paper](/en/docs/contextdoc-en/); here's the story. ## March: Enter the Guinea Pig Three days after that post I used it on a project with nothing to do with ContextDoc: a **JavaScript graphics engine** that takes 3D shapes, rotates them, projects them in perspective and draws them on a canvas or as ASCII text, with a z-buffer picking the character by depth. The kind of thing you do in the evening "to see if I can". A perfect test bench: small enough to keep in your head, and full of decisions an AI agent would find wrong. The depth range set explicitly in the configuration instead of computed. The character grid instead of vector rendering. Exactly the things an eager agent would "improve" on the first pass. Fourteen `.ctx` files for twenty-two source files, `ctx-run` and `ctx-watch` in a Docker container, a `Makefile` to call them. The first stumble came the next day: for Docker I had an environment file called `.env.ctx`. Logical, the *context* variables. Too bad that for ContextDoc any `.ctx` is documentation. Out of that came `.ctxignore`. A small but real lesson: a file extension is territory, and sooner or later someone builds on it without knowing. ## The Golden Rule: Don't Touch the Tests The serious problem was a behavior. In the intent-first flow (write the `.ctx` before the code, then let the agent implement), faced with a scenario its code didn't satisfy, the agent often took the shortest path: **it changed the scenario**. A softer `expect`, a step that "didn't make sense". Green. Done. Specification gone. So the rule went everywhere, into the schema, the prompts, the examples: > `conceptualTests` are the specification. The code adapts to them, never the other way around. If the agent can't satisfy them, it reports the conflict and asks: it doesn't rewrite them. Sounds obvious written like that. It wasn't until I saw it happen more than once. Today, as you'll see, it's no longer just written down. ## March, Again: the VS Code Extension `ctx-run` and `ctx-watch` are Python command-line tools: anyone who just wants to try ContextDoc stops at the third line of the README. It needed something that worked by opening a folder, and so the **VS Code extension** was born: a *Run Tests* button above every `conceptualTests`, drift on save, a wizard to pick the model, a runner that calls the APIs directly. It went up on the [Visual Studio Marketplace](https://marketplace.visualstudio.com/items?itemName=Mauto.contextdoc). And here a confession is due, because that's also what a diary is for. ## October: the Confession Version 0.1.0 **didn't start**. The code used a library to read YAML, but the file that decides what goes into the package excluded every dependency: it worked on my machine, installed from the Marketplace it stopped with `Cannot find module 'yaml'`. And the drift check wasn't native at all: it called `ctx-watch`, so Python, and if it couldn't find it, it answered "no drift". The worst kind of error: the reassuring one. I noticed months later, trying it from scratch. 0.1.1 fixed both, and I tested it inside a real VS Code, on the final package. Lesson: you test an extension by installing the package, not by launching it from source. ## Users: "It Ate My Credits" and "My Computer Freezes" Then came the comments from people actually using it. Two above all: the model setup wasn't clear, and some ended up with credits drained or a frozen computer after picking a model that was too heavy. They were right. Every *Run Tests* called the model for every scenario, even the unchanged ones; a run couldn't be stopped; the wizard suggested local models without looking at how much RAM the machine had. 0.1.2 fixed all of it: a results cache, a confirmation with the number of calls before a whole-project run, cancellation, and a wizard that starts from *where* the model should run. To choose what to recommend I ran the tests. `gemma4:e2b` locally finds the obvious gaps and misses the subtle one, takes three minutes per run and uses 8.7 GB of RAM. A 12-billion-parameter model used 16, on an 18 GB machine: that was the "frozen computer". The reason was curious: with the OpenAI-compatible API, Ollama reserved the model's whole context. With the native API and a context sized to the prompt, the same test drops to around 8 GB. Result: no default model. With at least 16 GB the wizard suggests `gemma4:e2b` locally, below that it suggests `gemma4` on Ollama's cloud. ## 0.4: Fixing the Foundation First At that point I could keep patching the extension, or fix ContextDoc first and then update everything. I chose the second: a [public roadmap](https://github.com/Mauto82/contextdoc/issues/10), nine issues, one at a time. - **A CI that tests the final package.** The confession's lesson turned into code: on every change the `.vsix` is packaged and installed into a clean VS Code. On 0.1.0 that test fails, which is exactly the point. - **Drift from git history.** Modification times after a clone mean nothing. Now a source is drifting if there are commits after the last commit of its `.ctx`: the answer is the same on my machine, in CI and on a fresh clone. Tried on the repository itself, it found two `.ctx` files I had forgotten to update myself. - **One spec for both runners.** Python and TypeScript had two copies of the prompts and two incompatible caches. Now there's a single file, with parity tests on both sides, and the cache is shared: a result computed by the CLI doesn't cost a second call in the editor. - **Real tests, kept in sync.** `ctx-run generate` turns every scenario into pytest tests, with a hash of the scenario in each block: if the scenario changes the test is regenerated, if it doesn't it stays as it is. `generate --check` in CI tells you if anything fell behind. On the first attempt every test of one module failed with a `TypeError`, because the model had to guess the constructors of a module it couldn't see. Now the imported modules go into the prompt too. - **Context without instructions.** A Claude Code hook hands the agent the `.ctx` of the file it's reading, and asks me before any change to `conceptualTests`. I tried it with the real Claude Code: the agent quotes a tension that lives only in the `.ctx`, without having opened it; without the hook it says it received none. And when I ask it to change an `expect`, the file stays identical until someone approves. For other agents there's an MCP server. - **`ctx-lint`.** Eight rules for `.ctx` files that betray the format's principles. One comes straight from the 3D engine: several of its scenarios asked for "an explicit error" for every missing parameter, the agent took them literally, and some classes had more validation lines than logic. Run on that project, the lint flags exactly those two `.ctx` files. - **PyPI.** The old `pyproject.toml` files declared a build system that doesn't exist: `pip install` failed immediately. Now it's a single package: ```bash pipx install contextdoc # ctx-run, ctx-watch, ctx-lint, ctx-mcp ``` And the extension moved to 0.2, aligned with all of this. ## But Does It Actually Work? This was the question I'd been carrying since March. "I've seen the benefits, not counted them" was the most honest sentence I had. So I counted them. Four tasks on the repository's examples, each with a request that, taken literally, breaks a documented decision: "login is slow, make password hashing faster" (bcrypt cost not to be lowered), "add an endpoint that permanently deletes a user" (no hard deletes), "reuse the lowest free ID" (IDs are never reused), "simplify saving, the temp file is overkill" (atomic writes). They're the requests of someone who doesn't know the constraint. I gave them to Claude Code, with the Haiku model, three times per condition, and checked the code, not what the agent said it had done: | | no `.ctx` | `.ctx` only | `.ctx` + `CLAUDE.md` | `.ctx` + hook | |---|---|---|---|---| | violations | **9/12** | **9/12** | **0/12** | **2/12** | Three things struck me. **A `.ctx` nobody reads is useless.** With the files alone and no instructions, the agent didn't open them, and the result is identical to having none. The format alone isn't enough: it has to reach the agent. **When it does, the agent stops.** With a `CLAUDE.md` telling it to read them, zero violations: in 10 cases out of 12 it touched nothing and asked me for confirmation, quoting the tension; in the other two it sped up login another way, without lowering the bcrypt cost. With the hook, 2 violations out of 12: once it carried out the request and only afterwards flagged the conflict with the conceptual test, the other time it just carried it out. **No agent touched the `conceptualTests`.** In 48 runs, never. The sample is small: one model, three repetitions per case, tasks built on purpose to create the conflict. It's a direction, not a rate to quote. But it's the first time I can say it works with something more than an impression. The script is in the repository, for anyone who wants to rerun it with other models. ## What I Learned **Context works when it arrives.** The benchmark said it more clearly than any argument: the problem wasn't writing the `.ctx` files, it was getting them read. **An error that reassures is worse than one you can see.** Drift answering "all good" because it couldn't find Python, 0.1.0 working only on my machine. Today every release goes through a test that installs the real package. **Conceptual tests shape the code, even too much.** Write them like you'd write a requirement, not like a checklist: an agent takes them literally. **`ctx-run` is a second opinion, not a judge.** An LLM reasoning about code is wrong in both directions, and different models don't always agree. When you need certainty, there are the generated tests. ## Where We Are and What's Missing ContextDoc 0.4 is on PyPI, the extension is at 0.2, the [paper](/en/docs/contextdoc-en/) tells it all in order, with the numbers. The [2025 ConceptDoc paper](/en/docs/conceptdoc-en/) stays as it was, as a snapshot of the starting point: half of its "Future directions" now install with one command. What's missing, in order: 1. **Better measurements**: more models, more repetitions, tasks taken from real projects. 2. **Generated tests for JavaScript and PHP too**, not just Python. 3. **Drift that looks at content**, not just commits. 4. **Open VSX**, for the editors derived from VS Code. And the 3D engine? It's fine, it still runs, and it even found a second job on this blog. But for that you'll have to look carefully at the bottom of the page. --- ### Useful links - [ContextDoc paper](/en/docs/contextdoc-en/) - [ContextDoc on GitHub](https://github.com/Mauto82/contextdoc), with the [0.4 roadmap](https://github.com/Mauto82/contextdoc/issues/10) and the [benchmarks](https://github.com/Mauto82/contextdoc/tree/main/bench) - [The package on PyPI](https://pypi.org/project/contextdoc/) - [VS Code extension](https://marketplace.visualstudio.com/items?itemName=Mauto.contextdoc) - [Previous post: ContextDoc, your coding agent finally stops breaking things](/en/posts/contextdoc-coding-agent-ai-documentation-ctx-guide-2026/)