In June Loom learned semantic search, with the switch off by default. With short notes Loom works great. With long documents, like PDF manuals or technical documentation hundreds of pages long, a flaw showed up that short notes never revealed.
Version 0.7 is out today, and almost all of it revolves around one sentence: the answer was on page 40.
The problem: Loom only read the beginning
When you ask a question with loom ask, Loom finds the relevant files and sends them to the LLM in a single call. To keep the context in check, each file has a budget: 8000 characters, about 2000 tokens.
With a half-page note nothing changes: the whole file fits. With a 60-page PDF, though, Loom kept the first 8000 characters and dropped the rest. Cover, table of contents, introduction. If the answer was in chapter 7, the model never saw it and answered with what it had: the table of contents.
The irony is that search worked perfectly. The right file was found, ranked first, and then reached the LLM without the part that mattered.
The fix: the opening plus the right passage
Starting with 0.7, when a file goes over budget Loom sends two pieces:
- the file’s opening (about 1200 characters): title, abstract, intro. They matter when the question is “what is this document about?”;
- the window that best answers the question, taken from anywhere in the file.
The LLM receives something like this:
Installation Manual – XYZ Server
1. Introduction
This document describes...
[...]
7.3 Connection pooling with PgBouncer
To limit database connections, set pool_mode = transaction...
[...truncated]
How does it pick the window? No embeddings and no extra calls. It slides a window over the text and scores each position by the question’s terms, with three tweaks:
- Rare terms weigh more. In a database manual the word “database” is everywhere and doesn’t help; “pgbouncer” shows up in three places and decides. It’s the same idea as BM25’s IDF, applied inside a single file.
- Crude but effective stemming: terms are cut to 6 letters, so “install” also finds “installation” and “configure” finds “configuration”. It doesn’t need to rank documents, only to land near the answer.
- A term repeated 200 times doesn’t win on its own: each term counts at most 3 times per window. A window that mentions every word of the question once beats one that repeats the same word forever.
If no window contains the question’s terms, Loom falls back to the old behaviour and sends the start of the file. It’s still one LLM call, no rescan, nothing to configure. It applies to the CLI, the GUI and the loom_ask MCP tool.
One heads-up: answers on long files may differ from before. Usually for the better, since before the model was answering from the table of contents.
loom_grep: when you need the exact word
Loom’s search (BM25 or hybrid) answers “which files are about this topic?”. But often the question is a different one: “where does this thing appear exactly?”. An invoice number, a customer’s name, a date, an error code. Ranked search can miss these: the file that mentions the invoice once, in a table, will never rank first.
That’s what the new loom_grep MCP tool is for: a regular expression over the extracted content of every indexed file (including text extracted from PDFs), or of a single file.
loom_grep(pattern="INV-2026-0[0-9]{3}", context=1)
For each matching line it returns the file, the line number, the byte offset and, if you ask, a few lines of context. No LLM call. It’s case-insensitive by default and stops at 50 matches (500 at most), so a pattern like e won’t dump your whole archive on you.
It uses Go’s RE2 engine, which runs in linear time: a badly written pattern from an agent can’t hang the process with the catastrophic backtracking that hits other regex engines.
Reading only the part you need
loom_grep makes sense together with the second addition: loom_get_file now accepts offset and length and returns a range of the file instead of the whole content.
Together, they let Claude explore a knowledge base the way it explores a codebase:
loom_grepfinds, for example, the line that mentions an error code at offset 48,210 of a 300 KB PDF;loom_get_filereads just the 4000 bytes around that offset;- Claude answers without loading the other 296 KB into context.
If you use Claude Code you’ll recognise the pattern: it’s what it does with grep and reading a file by lines. Now it works on your documents too.
excerpt_chars: the same cut over HTTP
If you use loom-http as the retrieval layer for another application, you had the same problem one step further away: /search returns each file’s full content, and your app had to cut it itself. Usually at the start.
Now you just pass excerpt_chars:
curl -s localhost:8080/search \
-d '{"query":"check-in time","limit":3,"excerpt_chars":3000}'
Long files come back cut the way ask does it, opening plus relevant passage, at about 3000 bytes, with a content_len field telling you the full length. Short files come back whole. Leave the parameter out and the response is the same as before.
I fixed a detail too: with small budgets the opening takes at most a quarter of the space. Before, a 1200-byte budget would have been all opening and no answer.
.loomignore: you decide what not to index
Loom has always skipped hidden files and folders (.git, .obsidian…). Beyond that it ignores nothing on its own, on purpose: what counts as noise (build output, old drafts, duplicate exports, the 2019 archive) is different for everyone.
So you write the rules, with the syntax you already know from Git:
# ~/loom/.loomignore
# any folder named "build", at any depth
build/
# only the top-level archive_v1 folder
/archive_v1/
# every .bak file
*.bak
drafts/**/*.old.md
If you’d rather not put a file inside a folder (an Obsidian vault, say), use the ignore list in ~/.loom/config.toml, which applies to every indexed folder:
notes_dirs = ["/Users/you/Documents/work", "/Users/you/Obsidian/vault"]
ignore = ["build/", "*.bak"]
A few practical details:
- ignored folders aren’t even walked, so scans get faster too;
- files already indexed that become ignored are removed from the index on the next scan, as if you had deleted them. Drop the rule and the next scan adds them back;
- an invalid pattern stops the scan with an error naming the file and line, without touching the index;
- negation (
!pattern) isn’t supported: it’s the one part of.gitignorethat’s missing; - in multi-corpus mode each corpus uses its own
notes/.loomignore.
The bugs you couldn’t see
Two fixes deserve a line, because they were the worst kind: the silent ones.
Saving settings from the GUI turned hybrid search off. The code that wrote config.toml always emitted the [embeddings] block commented out. Opening the GUI settings and saving, for any reason, was enough for your enabled = true to disappear. Loom fell back to plain BM25 without a word: no error, just slightly worse results on questions in another language. Now the block is written back active with your values. For the API key only the environment variable’s name is written, never the key itself.
/scan over HTTP didn’t say what went wrong (fixed in 0.6.1). It only returned counters: {"failed": 1} and good luck. Now it returns the same error list as the CLI, with file and reason, and it also reports moved files (same content, new path), which used to look like “nothing happened”. CLI, MCP and HTTP now share the same code for the report, so they can’t drift apart again.
Upgrading
brew upgrade loom
For Claude Desktop, download the new .mcpb package from the releases page. No rescan needed: relevant excerpts, loom_grep and ranges work on the index you already have. .loomignore kicks in on your next loom scan.
Wrapping up
The June updates were new features: embeddings, HTTP, multi-corpus. This one is different. It fixes things that were already there and didn’t work as they should: questions Loom should have answered and didn’t.
The rule is still the same, though: no extra infrastructure. The relevant excerpt is a loop over a string, loom_grep is Go’s standard library, .loomignore is a text file. One binary, one SQLite file, the model you already use.
Resources
- Loom repository: github.com/Mauto82/loom
- 0.7.0 changelog: CHANGELOG.md
- Homebrew tap: github.com/Mauto82/homebrew-loom
- RE2 syntax: github.com/google/re2/wiki/Syntax
- Previous posts on Loom: the introduction and optional semantic search
P.S. The GUI bug is my favourite of its kind: no error, no log, just slightly worse answers. It’s the kind of bug you blame on the embedding model for days before it occurs to you to open a config file.