BT411/phases/phase-04-gap-census.md

Phase 04 — The Export Gap Census (#60)

Date: 2026-08-06 · Trigger: export-gap blindness caused 4 wrong conclusions (reconstruction-gotchas §20), 2 falsely [T1]. Goal: turn "we trip over gaps when we arrive at them" into a known inventory.

Method

tools/gapcensus.py (deterministic; re-run after any re-export):

1. Index coverage — functions_index.tsv (6,267 functions with sizes) interval-merged against the PE .text extent → raw dark regions (bytes belonging to NO indexed function).
2. Export coverage — /* @ADDR */ headers in all/part_*.c vs the index.
3. Discovery — function starts inside dark regions from E8 call targets + data-section code pointers (vtables/handler tables/Performance pointers), with caller counts.
4. Annotation — pad-byte exclusion (CC/90/00), nearest TU attribution from file= tags, and visited flags (address cited anywhere in game/ context/ docs/).

Headline results

• .text = 892 KB. Index covers 87.3%. Dark: 428 regions, 105 KB raw / 90 KB real code.
• Indexed-but-unexported = 0 — the entire gap class is "Ghidra never made it a function" (consistent with #60's original find: VehicleDead was missing from the index, not filtered from the export).
• 332 function starts discovered inside dark regions (call-graph + data-pointer evidence).
• By TU family (real code): munga/ 28.5 KB (engine — source in repo, LOW priority), bt/ 24.0 KB + bt_l4/ 14.9 KB + boundary 15.6 KB = ~54 KB of GAME-side dark, munga_l4/ 9.5 KB.
• Game-side dark regions: 66 already visited by past digs, 159 never touched.

Validation

All six known dark celebrities land inside census regions: the VehicleDead handler (0x4c05c4), Searchlight ToggleLamp (0x4b838c), the TakeDamage death tail (0x4a07b8, inside the famous 0x49ffc8-0x4a1232 stretch — 89 repo citations), the master-perf region (0x4a9b5a-0x4ab188 — 69 citations; crouch/myomers/posture all came from here), the myomer integrator (0x4b8be3), the duck consumer (0x4aa011). The two most-cited dark regions are exactly the two that produced the most reconstructions — the census finds the right things.

Top uncharted leads (zero citations, real code, spot-checked)

1. 0x480a2a-0x487ca0 cluster (~9 KB across 6 regions), l4splr|btmssn boundary — head spot-check @0x486467: real enter 0x3c function driven by a global-index dispatch table ([0x4fef88] → table 0x501588) — mission-state-machine flavored. Likely the un-exported heart of BTMission (round lifecycle / mode carousel).
2. 0x4c08fc-0x4c0bc8 (613 B), btplayer, immediately before the BTPlayer ctor — real function reading player+0x24c; prime suspect for the deferred mission-review post (id 0x18) sender and/or death-cycle siblings.
3. 0x4d41ae-0x4ddc7c btl4app regions (~2.6 KB) — app/session plumbing (attract/config?).
4. 0x4af691-0x4af9cf (707 B), heat|mechmppr boundary — heat-family tail or mapper head.
5. 0x4bdd7b-0x4be078 (673 B), mechweap|btplayer boundary — weapon-family virtuals.

PART 2 — THE RE-EXPORT (same day, 2026-08-06): the dark map, cut in half

Installed (no admin, project tool convention %LOCALAPPDATA%t411-tools, beside DXSDK/cmake): JDK 21.0.12 + Ghidra 12.1.2. ⚠ Ghidra's launcher .bat expands %JAVA_HOME% UNQUOTED, so the space in the user profile ("Joe Laptop") kills it — the runner uses 8.3 SHORT paths (JOELAP~1, BT411-~1). Zips deleted after extract.

New tooling (committed, repeatable):

• reference/ghidra_scripts/ExportGaps.java — keeps ExportAll's EXACT output contract (shard format, index columns) so every tool/grep/@ADDR citation still works, and adds the gap-fill pass first: force disassembly + createFunction at (a) E8 call targets outside any function, (b) data→code pointers at a plausible prologue, (c) the census's discovered starts — iterated to a fixpoint. Logs what it created to gapfill_report.tsv.
• tools/ghidra_reexport.sh — headless runner (reprocess re-runs the script on the saved project without re-analysis). tools/gapdiff.py — scores any two censused exports.

Results (tools/gapdiff.py):

Metric	Before	After
Index coverage of .text	87.3%	93.5%
Dark REAL code	90.4 KB	40.8 KB (−49.5 KB, 54.8% recovered)
Game-side dark (bt/ + bt_l4/)	53.1 KB	21.1 KB (−32.0 KB)
Dark regions	428	321
Functions / pseudocode	6,267	6,472 (+205, 0 decompile failures)
Gap fill converged in 2 rounds (205 created: 195 from census starts, 6 call targets, 4 data
pointers) covering 56.1 KB. Ghidra 12 auto-analysis alone also found 42 more than the old export.

Every historically-bitten dark function now has pseudocode, including @0x4c05c4 VehicleDead — the function whose absence OPENED this issue — plus ToggleLamp @0x4b838c, the TakeDamage death tail, the master-perf region (crouch/myomers/posture), the myomer integrator, and the btplayer hole @0x4c0904.

Validation: the re-export CONFIRMS this week's hand reconstructions

The master-perf pseudocode matches the crouch transcription field-for-field — param_1[0xfe]=mapPosture(0x3f8), [0xe6]=duckState(0x398), [0x161]=squatCapable, [0x1e7]=myomerEffectiveness(0x79c), [0x10]=movementMode, the player+0x25c novice gate, FUN_004a7fc4(2/3)=SetLegAnimation sqd/squ, FUN_004a4c54(8)+(1)=ForceUpdate, the +0x131 stability alarm, and the duckState clear. Independent confirmation of a raw-disasm reconstruction, and of the census method that found it.

…and it caught one thing the raw-disasm pass MISSED (fixed same day)

The hand pass stopped at 0x4aa0af; the pseudocode tail carries an airborne auto-rise: if ((mode == 3 || mode == 4) && legState == 1) → SetLegAnimation(3) + ForceUpdate(8)+(1) + stability 1 — a mech that leaves the ground while parked in the squat is forced up, unconditional on duckState/squatCapable. Now in mech4.cpp; crouch re-benched un-regressed.

Promotion + archive

The re-export is now the CANONICAL reference/decomp/ (index, shards, vtables, census). The previous export is preserved verbatim at reference/decomp/archive_2025export/ so older KB citations of the form part_0NN.c:LINE stay resolvable — ⚠ line numbers and shard membership shifted in the new export (addresses did not). Cite @ADDR (stable), not part/line, going forward.

Remaining dark (40.8 KB) — the next leads

• 0x480a2a-0x487ca0 (~9 KB, l4splr|btmssn boundary) — still dark even after the fill: no call or data reference reaches it, which itself is informative (dead code? data misparsed as code? a jump-table-only region?). Worth a targeted look with the census's byte-scan.
• New from the re-export: @0x4c0904 is the MASTER BTPlayer Performance — team resolution by name ("Team ... does not exist"), the mission-end "Sending EndMission" console post, and the periodic console score heartbeat (corroborating our hand-built one). Our port attributes @0x4c083c as PlayerSimulation; that attribution needs a re-check.
• The remaining munga/munga_l4 dark (~19 KB) is low priority — we compile the real engine source.

Systemic fix status — BOTH HALVES DONE

Inventory + re-export shipped. Rule (gotcha §20) stands and is now cheap to obey: check GAP_CENSUS.md before any "absent from the export" claim; if still dark, byte-scan. Re-run after any future re-export: tools/ghidra_reexport.sh → tools/gapcensus.py → tools/gapdiff.py.