# Developing LatticeSearch

The build-fast toolchain: how the engine is run, and the tooling that makes the edit → run loop feel
instant. Everything below uses the engine's Python:

```
C:\sd\kohya_ss\venv\Scripts\python.exe
```

referred to as `<py>` from here on.

---

## Running the engine

```
<py> server.py --port 8765
```

The engine binds `127.0.0.1:8765`, loads the on-disk indexes, and serves the UI + JSON API. On start
it writes a start marker to `crash.log` and enables `faulthandler`, so any hard fault (native or
Python) leaves a trace instead of vanishing.

**Single-instance guard.** Before binding, the engine probes the port; if something already answers,
it prints `another engine is already serving on port … — exiting` and exits `3`. Two engines racing
one port previously caused a wrong-interpreter mixup — this makes a double-launch loud and safe, and
is what lets the dev-loop and watchdog restart the engine without racing.

**LITE mode.** `--lite` skips all ML model loads (SemanticIndex/DocIndex/AudIndex) — you get name
search, the instant explorer, and browse only, at ~1.27 GB RAM instead of ~3.5 GB. Good for a netbook
or a fast smoke on a low-power box.

---

## The warm model host (the hot core)

Cold-loading CLIP + e5 + CLAP dominates engine restart time (20–60 s). `model_host.py` loads those
models **once** and holds them warm forever, serving encode requests over a localhost socket in
milliseconds:

```
<py> model_host.py --port 8767
```

- Endpoints (127.0.0.1 only, `X-LS-Token` required): `/health`, `/clip_text`, `/clip_image`, `/e5`,
  `/clap_text`. CLIP and e5 load at startup; CLAP lazy-loads on first use.
- **Per-model locks** (`clip_lock`, `e5_lock`, `clap_lock`) so the engine's parallel tiers don't
  queue behind each other (one global lock cost the doc tier 150 ms+ of waiting).
- With `LS_MODEL_HOST=1` set, the engine **skips loading models** and asks the host instead. Engine
  restarts then drop from model-reload time to index-load time — measured **~60 s → ~19 s**. Any
  script or agent can also get embeddings instantly without paying a cold start.

Start the host once and leave it running; restart the engine as often as you like against it.

---

## The dev loop — auto-restart on change

```
<py> dev_reload.py          # Ctrl+C to stop
```

Watches the engine's source files (`server.py`, `content.py`, `enrich.py`, `audio.py`, `tiles.py`,
`phash.py`, `latticesync.py`, `transfer.py`, `backup.py`, `drives.py`, `memory_lattice.py`,
`clipwatch.py`, `codesym.py`). On save it kills the engine, relaunches it, waits for the port, and
prints the round-trip time. Two properties make it safe to lean on:

- **Syntax-gated.** Changed `.py` files are `ast.parse`-checked first; a file with a syntax error
  prints `SYNTAX ERROR (line N) — NOT restarting` and the good engine keeps running. You never kill a
  live engine for code that won't even import.
- **Host-aware.** If the warm model host is up, the restart is done in light mode (`LS_MODEL_HOST=1`)
  automatically — so the dev loop restarts in ~19 s, not ~60 s.

`ui.html` / `palette.html` are read per request and need no restart.

---

## Lattice Make — builds that skip what didn't change

`lattice_make.py` applies the lattice principle to builds: the win isn't hosting bytes, it's
*knowing* what changed.

- **Instant change detection.** A step's input fingerprint is `(relpath, size, mtime_ns)` for every
  input file plus the command, gathered in one filesystem sweep — **no content hashing** (the same
  stat-level signal live-watching already proves sufficient).
- **Content-addressed artifact cache.** Outputs are stored zstd-compressed by `sha256` (the CAS
  layout). An unchanged step's outputs are **restored, never rebuilt** — a cache hit is a
  fingerprint + a decompress, seconds → milliseconds (**~1000× on a hit** vs the real build).

```
<py> lattice_make.py run --step rust --cmd "cargo build --release" ^
     --cwd C:\sd\latticesearch-rs --inputs src Cargo.toml ^
     --outputs target\release\latticesearch.exe

<py> lattice_make.py list
```

On a hit you'll see e.g. `CACHE HIT — 214 inputs fingerprinted in 6ms, 1 output(s) restored (12.4MB)
in 30ms. Build SKIPPED (was 41.0s).`

---

## The Lattice Workspace — the substrate, warm and wired

The Workspace is a single session where the **whole lattice substrate is warm and wired**, so
building and working happen at lattice speed. It is the recommended environment for an agent (or a
human) doing sustained work:

```
<py> workspace.py                 # interactive REPL
<py> workspace.py -c "cmd; cmd"   # batch (scripting / benchmarking)
```

(Intended to also be reachable as `lattice workspace` once the CLI shim lands.)

Inside the workspace:

- the **warm model host is ensured up** (started if needed) — nothing cold-starts;
- **every lattice verb is a first-class command** — `search find ask grep fs glob ls read write edit
  mv cp rm remember recall clip sym star open health recover restore photos drive` — run instantly
  from RAM;
- **`build <step> --in <files> --out <files> -- <cmd>`** runs through Lattice Make (unchanged inputs
  = a cache restore, not a rebuild);
- **`!<cmd>`** runs a raw shell command, timed, with `LATTICE_URL`, `LS_MODEL_HOST=1`, and the
  mingw64 toolchain already on `PATH`;
- **everything is timed** and appended to `workspace_log.jsonl` (so runs are locatable and
  comparable).

The banner shows engine `:8765`, model-host `:8767`, and rust-name `:8766` status at a glance.

---

## The Rust name-tier sidecar

The name tier can be served by a compiled Rust sidecar (`C:\sd\latticesearch-rs`) for an 8–25×
speed-up on name searches and a far smaller footprint (an mmap byte-arena, ~150–250 MB RSS vs the
Python engine's ~1.27 GB):

```
latticesearch.exe serve --port 8766 --index C:\sd\latticesearch\index_meta.tsv
```

It's optional — the engine works without it — but worth keeping alive (the watchdog does exactly
that). Building it is a good Lattice Make step (see the `--step rust` example above). The Rust/Tauri
build is also the cross-platform production target (Windows/macOS/Linux).

**Headless Linux LITE.** `_linux_bundle/` holds a portable copy of `server.py` / `mcp_server.py` /
`lattice.py` for running a headless engine on Linux (name + index tiers), pairing with `--lite` for a
minimal, no-GPU deployment.

---

## Keeping it alive — the watchdog

```
<py> watchdog.py              # foreground
<py> watchdog.py --install    # register in HKCU Run (starts hidden at login)
<py> watchdog.py --uninstall
```

Polls `:8765` every 60 s; on **two consecutive misses** (so a restart blip isn't mistaken for a
crash) it relaunches the engine — but only after confirming no `server.py --port 8765` process
already exists, so it never double-spawns. It logs each event plus the tail of `crash.log` to
`watchdog.log`, and also keeps the Rust sidecar (`:8766`) up if the binary is present. It never kills
anything.

---

## Ops / integration tooling

- **`integrate_windows.py`** — per-user, no-admin Explorer right-click entries (`install` /
  `uninstall` / `status`, plus `sendto`): Instant Move, Fast Copy, Delete (recoverable), Find similar,
  under `HKCU\Software\Classes`. The full Ctrl+C/Ctrl+V takeover is an installer-phase item; these are
  the safe, reversible v1.

---

## STANDING LESSON — never patch code through bash heredocs

Editing source by piping a heredoc through bash on Windows **mangles backslashes and backspaces** and
silently corrupts the file. Always edit via a real script file (Write/Edit, or a `.py`/`.ps1` you run)
— never `cat <<'EOF' > file.py` style patching. This has bitten the project before; treat it as a
hard rule.
