# BLINDSPOTS — integration guide (blindspots.py)

Three indexing blind spots, attacked 2026-07-19 under a strict no-new-dependencies rule
(stdlib + what is already importable from `pylib/` + the kohya venv; nothing installed).
All code lives in `blindspots.py`; nothing else was modified. Verified interpreter:
`C:\sd\kohya_ss\venv\Scripts\python.exe` with `sys.path` += `pylib` (PIL 12.3.0, av 17.1.0
bundling FFmpeg 8.0, pypdf, zstandard...). NOT available anywhere: `pillow_heif`, `olefile`,
`extract_msg` — everything below works without them.

---

## 1) Cloud placeholders (Dropbox/OneDrive online-only)

### Measured on this machine
- Dropbox tree `C:\Users\Shadow\Dropbox`: **724,866 files, 221,651 placeholders (30.6%)**.
- Sampling bias warning: the first 5,000 files in root-first walk order contain **0**
  placeholders; a 5,000-file sample under `Dropbox\PC (2)` is **97.9%** placeholders.
  Never estimate from a walk prefix.
- **Dropbox does NOT set the Cloud-Files RECALL bits.** Its online-only signature is
  `FILE_ATTRIBUTE_OFFLINE (0x1000)` combined with `REPARSE_POINT|SPARSE|ARCHIVE`
  (observed attr sets: `0x1620` ×~4.8k/5k, `0x1420` ×24). OneDrive (unused here: 1 file)
  is the one that uses `RECALL_ON_DATA_ACCESS (0x400000)` / `RECALL_ON_OPEN (0x40000)`.
  The mask in `blindspots.py` tests all three bits, so both providers are caught.
- `GetCompressedFileSizeW` is NOT a reliable discriminator for Dropbox: a fully offline
  4-byte placeholder still reported 4 B allocated. Use the attribute bits only.

### Hydration proof (run: `python blindspots.py hydrate-proof`)
Target: 4 B placeholder, attrs `0x1620 [OFFLINE+REPARSE_POINT+SPARSE]`.
- 3× `os.stat()` + `os.listdir(parent)` → attrs unchanged (`0x1620`). **stat/scandir/
  GetFileAttributesW never hydrate. Metadata indexing is safe.**
- `open()` + `read(1)` → returned in 0.07 s, attrs flipped to `0x20` (plain ARCHIVE):
  the file was **downloaded by a 1-byte read**. Any content tier that opens placeholders
  would trigger ~222k downloads on this machine.

### API (all in `blindspots.py`)
```python
is_placeholder(path: str) -> bool          # ctypes GetFileAttributesW, never opens the file
is_placeholder_attrs(attrs: int) -> bool   # pure bit test (mask 0x00441000)
is_placeholder_st(st: os.stat_result) -> bool  # uses st.st_file_attributes — ZERO extra syscall
file_attributes(path) -> int | None
allocated_size(path) -> int | None         # GetCompressedFileSizeW (informational only)
```

### Where to hook (SKIP in content/embedding tiers ONLY — placeholders must STAY in the
filename/metadata index; names+sizes are searchable without hydration)
- `backfill.py :: candidates()` (~line 29) — it already `os.stat()`s every candidate.
  After `st = os.stat(p)` add:
  `if blindspots.is_placeholder_st(st): continue` — zero extra syscalls.
- `enrich.py :: _walk()` (~line 29) — currently uses `os.path.getsize`; either switch the
  inner loop to `os.scandir` and use `is_placeholder_st(entry.stat(follow_symlinks=False))`
  (free), or add `if blindspots.is_placeholder(p): continue` (one metadata syscall, safe).
- `content.py :: cmd_docs / cmd_videos` walkers (~lines 211, 263) — same one-liner.
- Belt-and-braces: right before any `open()` / `Image.open()` / `av.open()` in extract
  loops, since files can go online-only between enumeration and read.
- Product surface idea: count skipped placeholders per run and surface "N cloud-only files
  not embedded (would download X GB) — hydrate to include" in the UI.

---

## 2) HEIC/HEIF photos

### Census
- `.heic` / `.heif` on the indexed volume (index_meta.tsv, whole C:, L: is a projection
  of C:): **0 / 0**. Pictures/OneDrive contain none. All parser work below is therefore
  **untested on real photos** — validated against a synthetic HEIF (see self-test).
- Note: pylib Pillow 12.3.0 has **native AVIF** support (`features.check('avif') == True`)
  but **no HEIC** opener. Quick win independent of this file: add `.avif` to `IMAGE_EXTS`
  in `enrich.py` / `backfill.py` — it just works through the existing PIL path.

### What is implemented (stdlib `struct` ISO-BMFF box parser)
```python
heic_items(path) -> dict        # walks ftyp/meta: pitm, iinf/infe, iloc (v0-2, both
                                # construction methods + idat), iref, iprp/ipco/ipma
heic_exif(path) -> bytes|None   # raw TIFF blob from the 'Exif' item (spec offset field
                                # honoured, with marker-scan fallback)
heic_exif_tags(path) -> dict    # {'make','model','datetime','datetime_original','gps'}
                                # parsed with PIL's Image.Exif (already in pylib)
heic_decode(path) -> PIL.Image  # pixels via pylib av:
                                #  Path A: av.open(path) — FFmpeg 8.0 demuxes HEIF natively
                                #  Path B: manual — thumbnail/primary 'hvc1' item bytes +
                                #          'hvcC' extradata -> CodecContext('hevc')
```
- Self-test: `python blindspots.py heic-selftest` → **PASS** (synthetic HEIF with a real
  PIL-built TIFF EXIF item; exercises box walk, iloc v0, iinf v2, extraction, tag parse).
- `heic_decode` limitations (honest): untested on real files; Apple 48 MP "grid" primaries
  are derived items Path B cannot reassemble (Path A / the thumbnail item still should
  work); dimensions-without-decode (ispe) not exposed yet — trivial add if needed.

### Where to hook
- Photos EXIF (`server.py` `/api/info`, ~lines 1908–1933): the PIL branch can't open HEIC;
  add `elif ext in ('.heic', '.heif'): info.update from blindspots.heic_exif_tags(p)`
  → feeds the same `taken` / `camera` / `gps` fields with no decode and no new deps.
- Thumbnails / CLIP tier: `blindspots.heic_decode(p)` returns a PIL image usable by the
  existing thumbcache + embedding path. Gate it behind a try/except and only enable after
  validating on a real HEIC (grab one from any iPhone).
- Extension sets: add `.heic`, `.heif` to `IMAGE_EXTS` only in the tiers that route
  through `heic_decode`; EXIF-only enrichment can enable immediately.

---

## 3) Outlook PST/OST + .msg

### Census (index_meta.tsv, whole volume)
- `.pst`: **0**, `.ost`: **0**.
- `.msg`: 737 by extension, but magic-byte check shows **0 are OLE compound files** —
  all 737 are Tcl/Tk text message catalogs (`...\tcl\tk8.6\msgs\*.msg`). **Zero real
  Outlook messages on this machine.** Ship the capability for client machines; expect
  extension-only detection to lie — always verify the 8-byte magic
  `D0 CF 11 E0 A1 B1 1A E1` before routing a `.msg` to the OLE path.

### .msg — implemented despite `olefile` being absent
`olefile` is NOT in pylib or the venv, so `blindspots.py` contains a minimal MS-CFB
(OLE compound file) reader in stdlib `struct` (~90 lines: header, DIFAT, FAT, directory,
ministream + miniFAT):
```python
cfb_open(path) -> (entries, read_stream)   # generic OLE reader
msg_text(path) -> {'subject','body','sender','sender_email','to','date'}
```
Property streams read: `__substg1.0_0037001F` (subject), `1000001F` (body),
`0C1A/0C1F` (sender), `0E04` (to); `001E` ANSI variants handled; date from the
fixed-length `__properties_version1.0` stream (PR_CLIENT_SUBMIT_TIME `0x0039:0040`,
FILETIME → ISO). Known gaps (fine for search): RTF-compressed bodies (`0x1009`) are not
decompressed; embedded-message subjects could shadow if they precede top-level entries
(first-occurrence wins; top-level is first in every writer observed in the wild).
- Self-test: `python blindspots.py msg-selftest` → **PASS** (synthetic spec-shaped CFB:
  subject+sender in the ministream, >4 KB body in FAT sectors — both storage paths).
  Marked clearly: **untested on a real Outlook .msg** (none exist on this machine).

### Where to hook
- `content.py :: extract_text(path, ext)` — add `.msg` branch:
  ```python
  if ext == ".msg":
      with open(path, "rb") as f:
          if f.read(8) != b"\xd0\xcf\x11\xe0\xa1\xb1\x1a\xe1": return None  # Tcl catalog etc.
      m = blindspots.msg_text(path)
      return "\n".join(x for x in (m.get("subject"), m.get("sender"),
                                   m.get("to"), m.get("body")) if x)
  ```
  and add `.msg` to `DOC_EXTS` so `backfill.py docs` / enrich pick it up.

### PST/OST — honest feasibility (no dep-free implementation)
PST ("Personal Folders", MS-PST) is a paged B-tree database with node/block trees,
per-block encoding (NDB "permute"/"cyclic" obfuscation), and property contexts layered
on top — a correct reader is thousands of lines; a partial one silently drops mail.
Not sane under the no-new-deps rule. Recommended path, in order:
1. **Optional external tool integration** (scope for the installer): `readpst.exe`
   (libpst) or `pffexport` (libpff — also handles **OST**). Installer offers "Index
   Outlook archives" as an optional download; indexer shells out, converts to
   `.eml`/mbox in a cache dir, and the existing text tier ingests those. LGPL binaries
   stay a separate download — keeps the EULA clean.
2. **Outlook COM automation** where Outlook is installed — rejected as default path:
   needs pywin32 (new dep) or painful raw-ctypes COM, prompts security dialogs, and
   requires a configured profile.
3. **pypff / libratom Python bindings** — compiled deps, rejected under the rule.
Priority: low on THIS machine (0 files), but real for the paying audience; do the
census at install time and only offer the optional component when `.pst`/`.ost` exist.

---

## CLI recap
```
python blindspots.py placeholders  [--root PATH] [--sample N] [--all]
python blindspots.py hydrate-proof [--root PATH] [--max-size B] [--path FILE]
python blindspots.py census-index  [--index index_meta.tsv]
python blindspots.py heic-exif FILE | heic-thumb FILE OUT.png | heic-selftest
python blindspots.py msg-text FILE | msg-selftest
```
