# DRIVES-NOTES — multi-drive wiring notes (2026-07-19)

Companion to `drives.py`. Validated on this machine with `subst X: C:\sd\latticesearch\testdata`
(7 files, format byte-identical to index_meta.tsv). `index_meta_X.tsv` was left in place on
purpose: X: no longer exists, so its 7 rows are a ready-made test fixture for the
"drive missing" result state below. Delete it once wiring is tested.

## 1. TSV row format + scan3.js findings

Row format (3 columns, tab-separated, one file per line):

```
<absolute path>\t<size bytes>\t<mtime unix seconds>\n
```

- **LF-only** line endings (`\n`, never CRLF), **no BOM, no header**, file ends with a newline
  (empty drive → 0-byte file, no stray newline).
- **UTF-8** (Node WriteStream default); non-ASCII filenames pass through intact
  (verified: `café_photo_δ.txt`).
- Backslash separators, drive-absolute paths (`X:\dir\file.ext`).
- `stat` failure still emits a row with `0\t0` — MetaIndex already tolerates this.
- Row order is NOT sorted: shallow files (depth ≤ 2, walked on the main thread) first, then the
  12 worker part-files in worker order. Don't rely on ordering.

scan3.js quirks found (nothing needs changing for multi-drive):

- Root is `argv[2]` — **not** hardcoded to `C:\` (only the usage comment says C:\). No skip
  lists at all: it indexes `$Recycle.Bin`, `System Volume Information` (if readable), etc.
  Access-denied dirs are silently skipped (counted as `denied`).
- **Drive-relative trap**: Node's win32 `path.join('X:', name)` yields `X:name`
  (drive-relative). The root MUST be passed with a trailing backslash (`X:\`). drives.py
  always does this.
- Junctions/symlinks are neither followed nor indexed (`Dirent.isDirectory()`/`isFile()` are
  both false for reparse points) — good: no `C:\Users\...\Application Data` cycles.
- **`snap_progress.json` collision**: scan3 writes `snap_progress.json`, `<out>.partN` and
  `<out>.tmp` into `dirname(out)`. server.py polls `BASE\snap_progress.json` (line ~1723), so
  a naive second scan would clobber the C: scan's progress. `scan_drive()` therefore runs
  scan3 with its output inside `BASE\_drive_scan_work\<L>\` and moves the finished TSV to
  `BASE\index_meta_<L>.tsv` afterwards (`os.replace`, atomic on same volume). Per-drive
  progress is at `_drive_scan_work\<L>\snap_progress.json` if you want to surface it in the UI.
- `PRIOR_TOTAL = 1660000` is hardcoded → `pct`/`eta_s` in progress JSON are meaningless for
  small drives (pct clamps to 99.5 until done). Cosmetic only. Also `bytes` in the progress
  JSON omits the shallow main-thread files' bytes (workers only); the TSV itself is correct.
- Final merge is atomic (`fs.renameSync(tmp, out)`), so a TSV that exists is always complete.
- scan_drive runs node at BELOW_NORMAL_PRIORITY_CLASS so a drive scan won't starve live
  indexing.

## 2. Hot-adding a drive without restart — MetaIndex swap strategy

`names` is a plain module global (`names = MetaIndex()`, server.py:2831), and `MetaIndex` is
immutable after construction (blob/offsets/arrays are only ever read). That makes
swap-by-rebind the simplest correct strategy:

1. Background thread: read base lines via the existing `_read_index_lines()` (keeps the
   `.zst`-preferred logic for C:), then append lines from every `index_meta_<L>.tsv` in
   `sorted(glob('index_meta_?.tsv'))` order, and construct a complete new MetaIndex-like
   object from the combined list. (Factor `MetaIndex.__init__` to accept pre-read lines, or
   have the loader glob the per-drive files itself — either way the build happens entirely
   off-thread; the running server keeps serving from the old object.)
2. Swap: `globals()['names'] = new_index` — a single rebind, atomic under the GIL. No lock is
   strictly required for the rebind itself; requests that already grabbed the old object keep
   using it safely (it stays alive via their local reference until GC). Two hygiene rules:
   - any handler that touches `names` more than once per request should snapshot it first
     (`m = names`) so it never mixes indexes mid-request — most already effectively do;
   - do the rebind under the existing `lock` if you also mutate anything shared alongside it
     (e.g. a `drives_loaded` status dict).
3. The new object's `_build_dir_index` daemon thread repopulates `dir_files`/`media_rows`;
   `dir_ready`/`media_ready` are False for a few seconds after swap — existing code paths
   already guard on those flags.

**Memory cost**: current C: index is 1.66 M rows / 222 MB TSV. MetaIndex steady state is
roughly 400–500 B per file all-in (paths list ~170 B, lowercase blob ~115 B, offsets list
~36 B, size+mtime arrays 16 B, dir/media indexes ~100 B) → ~0.7–0.8 GB today. During a swap
**old + new coexist**, so peak ≈ 2× steady + the added drive's share. Adding a 100 k-file USB
≈ +45 MB steady, ~1.6 GB transient peak during rebuild. Fine on this box; on 8 GB machines
consider freeing the old object's `dir_files` before building the new one if it gets tight.

## 3. Removable drives on unplug

Recommendation: **keep the rows, tag the results.** When a USB drive unplugs, its rows stay
in the index; `os.stat`/preview/open fail fast (no network round-trip — a missing local
letter errors immediately).

- On each search response, compute `present = {d['letter'] for d in list_drives() if d['ready']}`
  (cache ~5 s; the ctypes calls are microseconds for local drives) and tag each hit with
  `offline: drive_of(path) not in present`. UI: dim the row + "drive not connected" chip;
  suppress open/preview actions.
- Do NOT purge offline rows — that is the whole lattice-sync USB story: the index travels
  with the machine, the content travels on the stick, and search still *finds* things that
  are currently detached ("it's on the red USB drive").
- Re-plug → rows go live again with zero work. Re-scan (`scan_drive`) only to refresh
  contents, which just overwrites `index_meta_<L>.tsv` and triggers the swap from §2.
- `is_os_hidden()` (server.py:836) stats every result path via `GetFileAttributesW`; for an
  unplugged local drive it fails fast and returns False — acceptable. See §4 for the network
  case, which is NOT fast.

## 4. Edge cases

- **BitLocker locked volumes**: `GetVolumeInformationW` and `GetDiskFreeSpaceExW` both fail →
  `list_drives()` reports the letter with `ready: False` (kind still `fixed`), and
  `scan_drive` refuses it. After unlock it flips to ready with no code changes. Treat
  "indexed but currently locked" the same as unplugged-removable (offline tag).
- **Network drives** (`kind: 'network'`): a dead/sleeping share makes every per-path Win32
  call block for seconds. Two rules: (a) **exclude network paths from `is_os_hidden` checks**
  — one stalled `GetFileAttributesW` per result row will freeze search; gate it on
  `drive_of(path)` kind, or just skip the check when kind == 'network'; (b) `list_drives()`
  itself calls `GetDiskFreeSpaceExW` on remote letters, which can also stall on a dead share
  — if you poll it per-request, poll network letters on a separate cached/timeout thread.
  Scanning network drives works (scan3 doesn't care) but is slow; treat as opt-in.
- **Drive letters changing**: removables don't always come back as the same letter.
  `list_drives()` returns the **volume serial** (`serial`, 8-hex) for exactly this: store it
  next to each per-drive index (e.g. first line comment is not possible in the TSV — use a
  sidecar `index_meta_<L>.json` with `{serial, label, scanned_at}` when you wire the server).
  On startup/replug, if letter L is absent but some ready drive has the stored serial, rename
  the TSV to the new letter (and optionally rewrite the path prefixes — cheap string replace
  on load: `line.replace('E:\\', 'F:\\', 1)` per row at concat time, no rescan needed).
- **subst / mapped aliases**: a subst'd letter inherits the underlying volume's kind
  (`fixed`) and serial — two letters with the same serial means alias, not clone; don't
  double-index the same serial unless roots differ.
- **C: guard**: `scan_drive('C')` is hard-refused because C: lives in `index_meta.tsv` (and
  its `.zst`, which `_read_index_lines()` prefers when newer — remember that when you expect
  a fresh C: rescan to show up).
