# Filesystem awareness — the ladder to "instant, zero-lag, zero-CPU" indexing

How does the lattice learn a file changed? There are four levels of mechanism, each faster and
more complete than the last, and each costing more trust. We climb this ladder deliberately —
taking the safe rungs now and treating the kernel rung as a careful, later, tested effort.

## The four rungs

| Rung | Mechanism | Latency | Idle CPU | Scope | Needs | Risk |
|---|---|---|---|---|---|---|
| 1 | **Polling** (walk hot folders every N s) | up to ~20 s | a little | watched folders | nothing | none |
| 2 | **ReadDirectoryChangesW** (event-driven) | **~1 ms** | **0%** | watched trees | nothing (userspace) | none |
| 3 | **NTFS USN journal** | ~ms, complete | ~0% | whole volume | admin (userspace) | low |
| 4 | **Kernel minifilter driver** | instant, universal | ~0% | every I/O | signed driver + admin | higher |

### Rung 1 — Polling (what we shipped first)
A background thread walks the hot folders and notices new files on its next pass. Simple and
safe, but it has lag (a file can be up to the poll interval stale) and spends a little CPU even
when nothing changed.

### Rung 2 — ReadDirectoryChangesW — **shipped, `fswatch.py`, opt-in `LS_FSWATCH=1`**
Windows pushes a notification the instant any file under a watched tree is created / modified /
renamed / deleted. **No polling, 0% idle CPU, no admin, no kernel.** Measured on this machine:

```
event-driven detect latency: median 0.8ms (min 0.6, max 4.6) over 8 files
polling watcher (current): up to ~20000ms (the poll interval)
=> ~24,000x faster to notice a change, and 0% idle CPU
```

End-to-end proof: write a file to the Desktop → it's ingested into the live index in ~1ms →
`/api/search` finds it immediately. This is the same class of mechanism the gold-standard instant
search tool **Everything** uses. It delivers ~all of the "instant awareness" benefit with none of
the kernel risk — so it is our default target for live indexing.

### Rung 3 — NTFS USN Change Journal — **shipped, `usnwatch.py`, opt-in `LS_USN=1`**
NTFS itself keeps a journal of every change on a volume. Reading it (still userspace, but needs
admin / the backup privilege) gives **complete, whole-volume** change history — even changes that
happened while the engine was off. It's the right upgrade for "never miss a change anywhere on the
drive," and it's what **Everything** uses for whole-volume coverage. Buildable and testable without
any kernel code; gated on admin, so it's an opt-in for power users.

### Rung 4 — Kernel minifilter driver (the careful, later step)
A minifilter sits in the kernel I/O path and sees every operation on the whole system, instantly.
It's the ultimate in coverage and latency — and it's the one component where a **bug costs more**
(kernel privilege), and it requires a **signed driver** and administrator install. It is NOT a
backdoor or a spying risk (it observes "this path changed," needs signing + admin, and can't
exceed what the engine already does) — but it earns its caution.

**The process, when we build it (deliberately, not rushed):**
1. Prototype the minifilter in C against the Windows Driver Kit, in a **throwaway virtual machine**
   — never the working machine.
2. **Smoke-test** every path (attach/detach, high file-churn, denied volumes, unplug) in the VM.
3. **Benchmark** detect-latency and CPU vs Rung 2/3 — it must clearly beat ReadDirectoryChangesW
   to justify the trust cost, or we simply stay on Rung 2/3.
4. **Code-sign** the driver (Azure Trusted Signing) and ship it as an **optional installer
   component** the user ticks on — never silently installed.
5. Only then, promote it from prototype to product.

Until every one of those is green, Rungs 2 and 3 are the answer, and they're already excellent.

## The warm-cache side (the "zip / keep warm / unzip instantly" idea)

The build/artifact cache (`lattice make`, `make_cache/`) already embodies most of this:

- **Compressed at rest.** Every cached artifact is stored **zstd-compressed** on disk — the store
  stays small and nimble. (`lattice make pool` shows its size.)
- **Extracted on demand, fast.** A cache hit decompresses the artifact in milliseconds (measured:
  0.8 MB restored in ~30 ms).
- **Zipped back / evicted when cold.** `lattice make prune --days N` LRU-evicts artifacts not used
  in N days — the "cleaning service" that keeps the pool light. Cold stays compressed; unused gets
  cleared.
- **Pre-warmed dependencies.** The compiler cache (`lattice make cc`) keeps shared/dependency
  objects warm so a *new* project's first build only compiles its own new code — measured
  67.3 s → 23.2 s on a first build (see [Developing](DEVELOPING.md)).

### The "smart, self-managing" layer — the honest version
A full lattice-based neural net to orchestrate zip/unzip/prefetch is **overkill**. The value comes
from a lightweight **access-graph predictor**: record which artifacts get used *together*, and when
one is requested, pre-warm its historical neighbors (learned prefetch — the "bus"/co-access idea).
That's a small, explainable model that can't hallucinate, captures ~all of the benefit, and
self-manages the hot/cold tiers by frequency + recency + co-access. It's scoped as the next cache
enhancement; the heavy-NN version is not needed.

## Where we are

- **Rung 2 is live and benchmarked** (`fswatch.py`, `LS_FSWATCH=1`) — instant, 0% idle CPU, safe.
- **Rung 3 (USN) is live** (`usnwatch.py`, `LS_USN=1`) — whole-volume tailing WITHOUT admin (FILE_TRAVERSE volume handle + the unprivileged read FSCTL; filenames are redacted by Windows in unprivileged mode, so paths resolve via OpenFileById). Sub-second whole-drive detection, ~0 CPU.
- **Rung 4 (minifilter)** is the deliberate, VM-tested, code-signed, opt-in future — built only
  when it clearly beats Rung 2/3 in a benchmark.
- The **warm cache** already compresses, extracts fast, and LRU-evicts; **access-graph prefetch**
  is the honest "smart" enhancement on deck.
