hallouminate

A markdown corpus indexer for LLMs to build and query their own per-repo wikis. Hallouminate stores markdown verbatim on disk, embeds it with fastembed, indexes the embeddings in LanceDB, and exposes a small MCP surface so an LLM can author and search a per-repo knowledge base without leaving its agent loop.

The filesystem is the source of truth; LanceDB rows are derived and refreshed automatically when an LLM writes via add_markdown, or in bulk via hallouminate index. Code files (.rs, .toml, …) can also be indexed as text for semantic search, but hallouminate does no structural analysis — it’s a wiki indexer that happens to tolerate code, not a code-intelligence tool.

Why a daemon

A long-lived local daemon owns the LanceDB ground directory, per-corpus mutation locks, and config resolution. The CLI and the stdio MCP server both talk to it over a Unix domain socket — one owner, no cross-process LanceDB races. See Architecture for the full picture.

Where to go next

• Installation — install the binary and register the MCP server with your agent.
• CLI reference — every subcommand and its flags.
• MCP surface — the ten tools an LLM calls to author and search wikis.
• Configuration — the XDG baseline, repo-layer merge, and embedding-model options.
• Dogfooding — this repo maintains its own wiki with hallouminate; here’s how to read it.

License

MIT — see LICENSE.

Installation

The preferred path is npm — a prebuilt binary with no Rust toolchain, no protoc, no compile. Build from source with cargo only if your platform has no prebuilt, or you want a development checkout.

npm (recommended)

The npm package is a thin shim: its postinstall downloads the prebuilt hallouminate binary for your platform from the matching GitHub release.

npm install -g hallouminate   # persistent — puts `hallouminate` on PATH
npx hallouminate --version    # or one-off, no global install

An MCP client can also launch the server as npx -y hallouminate serve without any global install — the first run pays the binary download.

Prebuilt-binary installer

Same prebuilts without npm — downloads hallouminate for your platform and adds it to your PATH:

curl --proto '=https' --tlsv1.2 -LsSf https://github.com/paulnsorensen/hallouminate/releases/latest/download/hallouminate-installer.sh | sh

Verify it:

hallouminate --version

Prebuilt binaries are published for each release:

Re-run the one-liner any time to upgrade to the latest release. On Intel Mac, Windows, or an older glibc, build from source with cargo below.

From crates.io

Builds from source, so it works on any platform with a Rust toolchain — at the cost of compiling native dependencies (a few minutes).

Prerequisites

• A Rust toolchain (cargo) — see https://rustup.rs.
• protoc (the Protocol Buffers compiler) — the lancedb build needs it.
  • macOS: brew install protobuf
  • Debian/Ubuntu: apt install protobuf-compiler

cargo install hallouminate --locked

The binary installs to ~/.cargo/bin/hallouminate (make sure that’s on your PATH).

From source

Same prerequisites as crates.io. Clone and build:

git clone https://github.com/paulnsorensen/hallouminate.git
cd hallouminate
cargo build --release

The binary lands in target/release/hallouminate.

Install for your harness

Every integration launches the same hallouminate serve stdio server.

Plugin (recommended)

The plugin pack is the preferred integration: one install registers the MCP server (bundled .mcp.json) and the wiki skills (/hallouminate:install, wiki authoring workflows). Install it through the harness-native route:

Without the plugin: skills + MCP by hand

The plugin is a convenience wrapper — both halves can be wired manually:

• MCP — register the stdio server directly:
  • Claude Code: claude mcp add hallouminate --scope user -- hallouminate serve
  • opencode: the opencode.json snippet from the table above
  • Any other MCP client: launch hallouminate serve over stdio — or npx -y hallouminate serve to skip the PATH install entirely (first run pays the binary download)
• Skills — copy plugins/hallouminate/skills/ into your harness’s skills directory (Claude Code: ~/.claude/skills/; opencode: ~/.config/opencode/skills/). Skills are optional — the MCP tools work without them.

hallouminate serve auto-spawns the daemon if none is running, so there is no separate process to manage.

Bootstrap a config

hallouminate config init       # scaffold the XDG baseline config
hallouminate config validate   # confirm it parses

See Configuration for what goes in the config and how the XDG baseline merges with a repo-layer .hallouminate/config.toml.

CLI reference

hallouminate is a single binary. The CLI, the MCP server, and the daemon are all the same executable; CLI subcommands dial the daemon over a Unix domain socket (see Architecture).

hallouminate --version prints the version; hallouminate --help and hallouminate <command> --help print usage for any subcommand.

serve

hallouminate serve

Starts the stdio MCP server an agent connects to. It is stateless beyond its tool router and a startup-captured working directory — every tool call dials the daemon. If no daemon is running, serve spawns one.

index

hallouminate index               # rebuild every configured corpus
hallouminate index --corpus repo:hallouminate:wiki

Use this when files were touched outside hallouminate. Writes that go through add_markdown already auto-reindex just the changed file.

ground

hallouminate ground "how does the daemon work"
hallouminate ground "socket protocol" --corpus repo:hallouminate:wiki --format json-pretty

daemon

hallouminate daemon run        # run in the foreground
hallouminate daemon status     # is one running?
hallouminate daemon stop
hallouminate daemon restart

The daemon is the single owner of the LanceDB ground directory. Restart it after editing the baseline config; repo-layer edits take effect on the next request without a restart. --config PATH overrides the baseline config path.

config

hallouminate config init       # scaffold the XDG baseline config
hallouminate config show       # print the effective merged config for this cwd
hallouminate config validate   # parse + flag unknown top-level keys
hallouminate config download   # pre-fetch the configured embedding model

See Configuration.

hook

hallouminate hook install [--repo PATH]
hallouminate hook uninstall [--repo PATH]

Installs or removes a per-repo discovery hook. --repo PATH targets a repo other than the current directory.

Socket override

--socket PATH on index, ground, and the other client subcommands points at a specific daemon socket. Otherwise the socket is resolved from HALLOUMINATE_SOCKET, then $XDG_RUNTIME_DIR, then the cache dir — see Architecture.

MCP surface

hallouminate serve starts a stdio MCP server. It is stateless beyond its tool router and a startup-captured working directory; every tool call dials the local daemon over a Unix domain socket, and serve auto-spawns the daemon if none is up.

Default corpus

Read-side tools (ground, list_files, list_tree) that omit corpus default to the wiki for the repository containing the daemon’s working directory — repo:<NAME>:wiki for the deepest [[repository]] whose path is an ancestor of the cwd. When the cwd sits under no configured repo, the caller must name a corpus explicitly.

The mutating tools (add_markdown, delete_markdown) and read_markdown always require an explicit corpus, to avoid accidental writes to the wrong wiki or ambiguous reads.

The ten tools

list_corpora

Every corpus the daemon knows about — explicit [[corpus]] entries plus derived repo:NAME:wiki and repo:NAME:corpus corpora. No params. Call this first to learn what’s available.

list_files

The files currently visible in a corpus, honoring its paths/globs/exclude rules. Param: corpus (defaults to wiki-for-cwd). Returns an array of {path, absolute_path}.

list_tree

The same files as list_files, grouped into a {path, absolute_path, files, subdirs} tree. Subdirs with no markdown beneath them are pruned. Use this for progressive disclosure — navigate the wiki without reading every index.md first. Param: corpus (defaults to wiki-for-cwd).

ground

Semantic search. Embeds the query with the configured embedding model (default snowflake/snowflake-arctic-embed-s), retrieves top chunks from LanceDB, and rolls up per-file with breadcrumb context. Params: query (required), corpus, top_files, chunks_per_file, limit, snippet_chars. Returns a ripgrep-style outline in content and the full structured response in structuredContent.docs.

add_markdown

Atomic-write a markdown file to the corpus’ first configured root, then refresh just that file’s LanceDB rows. For repo:*:wiki corpora it also rebuilds the link list inside each ancestor index.md between the <!-- HALLOUMINATE:INDEX-START --> / <!-- HALLOUMINATE:INDEX-END --> markers — scaffolding a missing index.md, preserving prose outside the markers, and leaving marker-less files alone. Params: corpus, path, content, overwrite (default false). Symlinks and parent-dir escapes are rejected by the sandbox. Returns advisory lint warnings (empty-destination links, empty mermaid blocks, heading-level jumps) without blocking the write.

read_markdown

Verbatim UTF-8 contents of a file in a corpus. Params: corpus, path. Use this before add_markdown { overwrite: true } to inspect current content.

delete_markdown

Unlink a file from the corpus’ first root and prune its rows from the index. Irreversible. For repo:*:wiki corpora it also re-walks the ancestor index.mds so they no longer link to the deleted file. Params: corpus, path.

index

Bulk (re)build the LanceDB index for one or all corpora. Param: corpus (optional; omit to rebuild every configured corpus). Use this when files were touched outside hallouminate.

corpus_stats

Index health statistics for one corpus: number of indexed files, total chunk row count, newest index timestamp (last_indexed_ms, null when the corpus has never been indexed), and how many on-disk files matching the corpus globs are not yet indexed. Param: corpus (defaults to wiki-for-cwd, same resolution as list_files). structuredContent is { corpus, indexed_files, total_chunks, last_indexed_ms, unindexed_files }.

get_footnote

Resolve a single citation: the footnote target for a page’s #footnote_number. Params: corpus (defaults to wiki-for-cwd, same as ground), page (the wiki page’s relative path), footnote_number (the label after ^ — "1" for [^1], "note" for [^note]). Use this to expand one footnote without reading the whole page.

Conventions for LLM authors

Markdown is stored verbatim — hallouminate imposes no schema. The convention the indexer counts on:

• One topic per file. The chunker splits on H1/H2/H3 headings.
• First non-blank line is # Title. The H1 is the breadcrumb root for every chunk and the gloss in the parent index.md link list.
• File stem matches the slug — lowercase, kebab-case, .md.
• Idempotent writes — add_markdown rejects existing files unless overwrite: true; read_markdown first so you don’t clobber blind.

Error mapping

When the daemon is unreachable, calls return -32603 — the MCP server does not fall back to opening a local LanceDB handle, since that’s exactly the multi-process race the daemon exists to prevent.

Configuration

Config lives at $XDG_CONFIG_HOME/hallouminate/config.toml (~/.config/hallouminate/config.toml by default). Two layers combine per request: an XDG baseline loaded once at daemon boot, and a repo layer discovered fresh on every request from the client’s working directory.

hallouminate config init       # scaffold the baseline
hallouminate config show       # the effective merged config for this cwd
hallouminate config validate   # parse + flag unknown top-level keys

Sections

The XDG baseline vs the repo layer

The baseline owns explicit [[corpus]] entries, [[repository]] declarations, and the [search]/[embeddings]/[logging]/[watch]/[storage] defaults. It is loaded once at daemon startup — change it and restart the daemon. [logging] is process-wide and baseline-owned — a repo layer that sets a non-default [logging] errors, naming the repo config path.

The repo layer is <repo>/.hallouminate/config.toml, found by walking up from the cwd to the first .git boundary. It overrides scalars and adds repo-local corpora, and is re-read on every request — so repo-layer edits take effect without a daemon restart. The repo layer is required: a CLI invocation from a directory with no ancestor .hallouminate/config.toml errors out. An empty file satisfies the check.

A repo declares itself like this repo does:

[[repository]]
name = "hallouminate"
path = "."

path = "." resolves against the repo root (the parent of .hallouminate/), so the wiki lands at <repo>/.hallouminate/wiki and is searchable as repo:hallouminate:wiki from any checkout.

Merge rules

• Array entries ([[corpus]], [[repository]]) — repo entries append after baseline entries; duplicate names error.
• Scalars — the repo wins if it sets a non-default value; conflicting non-default values error and name both source paths. [logging] stays baseline-owned: a repo layer that sets a non-default [logging] is an ERROR naming the repo config path, rather than being silently ignored.

Logging

The active process log is $XDG_STATE_HOME/hallouminate/hallouminate.log (default ~/.local/state/hallouminate/hallouminate.log). Size rotation creates numbered archives hallouminate.log.1, .2, and so on, with .1 newest.

[logging]
max_file_bytes = 10485760   # 10 MiB
max_total_bytes = 104857600 # 100 MiB

Every active or archived file is capped at max_file_bytes. Retention keeps at most floor(max_total_bytes / max_file_bytes) files including the active file, so defaults retain the active file plus nine archives and never exceed 100 MiB. Both limits must be nonzero and the total must be at least the per-file limit.

HALLOUMINATE_LOG_MAX_FILE_BYTES and HALLOUMINATE_LOG_MAX_TOTAL_BYTES override the baseline limits. Invalid values stop startup and name the offending variable. Log filtering remains HALLOUMINATE_LOG, then RUST_LOG, then hallouminate=info.

Watcher failure reminders

[watch].failure_reminder_secs defaults to 60. Identical watcher reindex failures (same path and rendered error) log once, then the next occurrence at or after the interval reports how many messages were suppressed. Distinct failures remain immediate. Set the value to 0 to disable suppression, or use HALLOUMINATE_WATCH_FAILURE_REMINDER_SECS as a process override.

Embeddings

Dense embeddings are on by default, using snowflake/snowflake-arctic-embed-s. On first index hallouminate downloads that model and fuses its vector signal with lexical search.

Supported models

All embed to 384-dim vectors. Omitting embeddings.model selects the default.

Turning embeddings off

Run lexically only — full-text search + ripgrep + rerank, no embedding model downloaded (just the tokenizer used for chunking):

[embeddings]
enabled = false

Changing the embedding mode (or model) for a ground directory already indexed under a different mode trips the store’s mismatch guard on the next run. Delete the ground directory and re-index to rebuild:

rm -rf ~/.local/share/hallouminate/ground
hallouminate index

To pre-fetch the model so the first index doesn’t pay the download cost:

hallouminate config download

Paths at a glance

Architecture

Hallouminate uses a Sliced Bread layout — vertical slices with public APIs at slice boundaries, no cross-slice peeks at internals. Three top-level concerns under src/.

Layers

src/app/ — orchestration

The application layer composes domain logic with adapters. It owns the clap-derived CLI (cli.rs), the long-lived daemon/, config parsing and the XDG/repo-layer merge (config.rs), logging, and XDG path resolution. App depends on domain and adapters; it does not own pure logic.

src/domain/ — pure logic

No I/O beyond filesystem walks and hashing. Slices: corpus/ (chunker, walker, hasher, sandbox, snippet), embeddings/ (the fastembed wrapper), ground/ (search orchestration and result formatting), indexer/ (plan/apply/write), plus shared types in common.rs. Domain has no dependency on app or adapters.

src/adapters/ — external systems

lance.rs is the LanceDB vector-storage adapter; mcp/ is the rmcp-based stdio MCP server. Adapters depend on domain for types, but not on app.

The dependency direction is adapters → domain ← app → adapters: domain is the stable core, app composes adapter implementations with domain orchestration.

Why there’s a daemon

LanceDB does not support concurrent writer processes against the same table. If a CLI index and an MCP add_markdown both opened LanceDB directly, they would race on table mutations. The daemon is the single owner of the LanceDB ground directory, the per-corpus mutation locks, and the repository registry. Every other caller — CLI subcommand, MCP tool, future agent — dials the daemon over a Unix domain socket.

Socket location

The socket path resolves in this order (src/app/daemon/socket.rs):

1. HALLOUMINATE_SOCKET env var — per-process override.
2. $XDG_RUNTIME_DIR/hallouminate/daemon.sock — the default when a runtime dir exists.
3. ${XDG_CACHE_HOME:-~/.cache}/hallouminate/daemon.sock — fallback.

The daemon takes a flock on <socket>.lock to enforce single-instance ownership; a second daemon on the same socket errors out.

Wire protocol

JSON-lines over the socket: one request line in, one response line out, then the connection closes. The request carries the client’s cwd, which the daemon walks on every request to discover the active repo-layer config and merge it with the boot baseline. That’s how one daemon serves many repos with different configs.

Mutating ops (index, add_markdown, delete_markdown) take the per-corpus mutation lock and then a global write-lane semaphore, in that order. Read ops skip both and run concurrently.

Entry points

• src/main.rs — process entry; calls hallouminate::app::run().
• src/lib.rs — library facade for tests and downstream callers.
• src/app.rs — top-level run(): parses the CLI and dispatches.

A living example

This repo’s own architecture notes are also maintained as a hallouminate wiki at .hallouminate/wiki/ — see Dogfooding. The wiki entries carry file:line and commit citations that this page summarizes.

Development verification

Run local Cargo checks through the repository verification gate:

just verify

With no arguments, the gate runs cargo fmt --all --check, then holds one exclusive lease across clippy, build, and test. The leased commands run with CARGO_BUILD_JOBS=1.

Targeted checks and fixes

Use the same lease for a focused Cargo command:

just verify cargo test -p hallouminate --test it verification_gate

A targeted command must start with cargo. Caller-supplied -j, --jobs, and their attached forms are rejected before Cargo’s -- delimiter because the gate owns the job cap. All Cargo --config forms are also rejected before that delimiter because inline values and referenced files can override build.jobs.

just verify --fix holds the lease while applying machine-fixable clippy suggestions, formatting, and running the full gate. just ci and just llm remain compatibility routes to just verify and just verify --fix.

Cross-worktree lease and evidence

Linked worktrees share the paths returned by git rev-parse --git-common-dir. The gate stores its state there:

• hallouminate-verify.lock — the exclusive compiler/linker lease.
• hallouminate-verify.jsonl — append-only wait, start, and finish evidence.

Each record includes a UTC timestamp, worktree, redacted command, exact SHA-256 command digest, HEAD, content-based state digest, and PID. Cargo --token values and arguments after cargo login are omitted from the recorded command, and the log is kept owner-only at mode 0600. Start records add the wait duration and job cap; finish records add the exit status, duration, and final state digest. A prior result is reusable only when its command digest, HEAD, and state digest match the current work and its finish status is zero.

The repository lease prevents duplicate compiler/linker-heavy work for this project. It does not provide host-wide admission control or resource isolation; the execution environment must supply host-wide scheduling and cgroups or equivalent limits.

Baseline measurement

Measured on 2026-07-14 on a Mac17,9 with 24 GiB RAM, 15 logical CPUs, Rust 1.91, Python 3.14.6, and just 1.56.0. Both successful commands ran with CARGO_BUILD_JOBS=1 through the verification lease:

• A warm full just verify completed in 42.27 seconds with a maximum resident set size of 1,789,263,872 bytes (1.67 GiB).
• After cargo clean -p hallouminate removed 2.0 GiB of package artifacts, just verify cargo test --locked --no-run -p hallouminate --test it relinked the integration-test binary in 31.94 seconds with a maximum resident set size of 2,719,940,608 bytes (2.53 GiB).

Both commands exited zero. /usr/bin/time -l reported zero block input and output operations on this APFS host, so those counters do not justify a higher cap. The incident was driven by simultaneous linker I/O, and the measured package relink already used 2.53 GiB at one Cargo job. The cap stays at one because it is the only measured setting that admits no second Cargo compilation unit; raising it requires a separate measurement with explicit memory and I/O headroom.

Releasing crates

The workspace publishes in dependency order: hallouminate-domain, then hallouminate-adapters, then hallouminate-config, then hallouminate-daemon, then hallouminate.

One-time bootstrap for new crates

crates.io cannot configure a trusted publisher until a crate has its first release. Any crate that has never been published — currently hallouminate-config and hallouminate-daemon — must be bootstrapped once before the first workspace tag that ships it:

1. Create a crates.io token allowed to publish new crates.
2. Add it temporarily as the repository secret CRATES_IO_TOKEN.
3. Run the publish-crates workflow manually with bootstrap=true. The workflow walks the library crates in dependency order (hallouminate-domain, hallouminate-adapters, hallouminate-config, hallouminate-daemon), waiting for the sparse index between each, and publishes only the ones missing at the current workspace version.
4. In crates.io settings for each newly published crate (hallouminate-config, hallouminate-daemon), add this trusted publisher: repository paulnsorensen/hallouminate, workflow publish-crates.yml. (hallouminate-domain, hallouminate-adapters, and hallouminate already have theirs from the initial bootstrap.)
5. Delete the CRATES_IO_TOKEN repository secret.

After bootstrap, push the version tag normally. Tagged releases authenticate only through crates.io Trusted Publishing, check each crate’s exact version on the sparse index, and publish only missing crates in dependency order. This makes a tag safe when the bootstrap already published the new crates at that workspace version.

Dogfooding: our own wiki

Hallouminate maintains its own wiki with hallouminate. The repo declares itself as a [[repository]], so its knowledge base is searchable as the repo:hallouminate:wiki corpus from any checkout. The wiki lives at .hallouminate/wiki/ and is the canonical, durable record of how the project actually works — the source these docs are distilled from.

Two corpora, two lifecycles:

What’s in the wiki

The entries are written for an LLM working in the repo — they carry file:line and commit citations these human-facing docs summarize:

• architecture — the sliced-bread layout and dependency direction.
• mcp-surface — the ten MCP tools, params, and error mapping.
• daemon-and-cli — why there’s a daemon, the JSON-line socket protocol, the CLI surface, and the lock order.
• corpus-walker — gitignore-aware corpus walking and the explicit-root opt-in.
• config-layering — the XDG baseline plus repo-layer merge.
• wiki-conventions — how to author entries without contradicting the indexer.

Read it the way an LLM would

If you have hallouminate installed and the MCP server registered, an agent working in this repo queries the wiki directly:

list_tree   { corpus: "repo:hallouminate:wiki" }
ground      { corpus: "repo:hallouminate:wiki", query: "why is there a daemon" }
read_markdown { corpus: "repo:hallouminate:wiki", path: "daemon-and-cli.md" }

From the CLI:

hallouminate ground "socket resolution order" --corpus repo:hallouminate:wiki

Keeping it current

The repo’s AGENTS.md instructs every coding agent to refresh the wiki after a change lands on main — but only when the change altered durable knowledge (architecture, conventions, protocols, the MCP tool surface, a “why this design” decision). Routine bug fixes and transient per-task output stay out; that’s what .cheese/ is for.

Updates go through the MCP (read_markdown → add_markdown with overwrite: true), not raw file edits, so the LanceDB index and the ancestor index.md link lists stay in sync. When the wiki is edited on disk directly, re-sync with:

hallouminate index --corpus repo:hallouminate:wiki

That loop — author through the tool, search through the tool, keep the index honest — is the product proving itself on its own source.