Joe DiPrimaandClaude Fable 5
f44be87ab2
Installed JDK 21 + Ghidra 12.1.2 (no admin, %LOCALAPPDATA%\bt411-tools beside DXSDK/cmake; runner uses 8.3 SHORT paths because Ghidra's .bat expands %JAVA_HOME% unquoted and the profile has a space). New tooling: reference/ghidra_scripts/ExportGaps.java -- ExportAll's exact output contract PLUS a gap-fill pass (force disassembly + createFunction at E8 call targets outside functions, data->code pointers at a plausible prologue, and the census's discovered starts; iterated to a fixpoint, logged to gapfill_report.tsv). tools/ghidra_reexport.sh (headless runner, 'reprocess' mode) and tools/gapdiff.py (score two censused exports); gapcensus.py now censuses any export dir. Results: 6267 -> 6472 functions (+205 created in 2 rounds: 195 census starts, 6 call targets, 4 data pointers; 56.1KB newly covered), ZERO decompile failures. Dark real code 90.4 -> 40.8 KB (54.8% recovered); game-side dark 53.1 -> 21.1 KB; regions 428 -> 321. EVERY historically dark function now has pseudocode -- including @0x4c05c4 VehicleDead, the absence that opened this issue. VALIDATION: the new pseudocode confirms this week's hand reconstruction of the crouch field-for-field (mapPosture/duckState/squatCapable/myomerEff/ novice gate/SetLegAnimation/ForceUpdate/stability alarm) -- and exposed one branch the raw pass missed: AIRBORNE AUTO-RISE (mode 3|4 && legState 1 -> forced squ), now implemented in mech4.cpp and re-benched un-regressed. PROMOTION: the re-export is canonical reference/decomp/; the previous export is preserved at reference/decomp/archive_2025export/ so old `part_0NN.c:LINE` citations still resolve (addresses are stable across both; line/shard membership is NOT -- cite @ADDR). New lead recorded: @0x4c0904 is the MASTER BTPlayer Performance (team resolution, EndMission console post, score heartbeat) -- our @0x4c083c PlayerSimulation attribution needs a re-check. KB: source-completeness, gotcha #20 (the rule is cheap now -- look it up), CLAUDE.md router/layout. Log: phases/phase-04-gap-census.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
114 lines
7.5 KiB
Markdown
114 lines
7.5 KiB
Markdown
# Phase 04 — The Export Gap Census (#60)
|
||
|
||
**Date:** 2026-08-06 · **Trigger:** export-gap blindness caused 4 wrong conclusions
|
||
(reconstruction-gotchas §20), 2 falsely [T1]. **Goal:** turn "we trip over gaps when we
|
||
arrive at them" into a known inventory.
|
||
|
||
## Method
|
||
`tools/gapcensus.py` (deterministic; re-run after any re-export):
|
||
1. **Index coverage** — `functions_index.tsv` (6,267 functions with sizes) interval-merged
|
||
against the PE `.text` extent → raw dark regions (bytes belonging to NO indexed function).
|
||
2. **Export coverage** — `/* @ADDR */` headers in `all/part_*.c` vs the index.
|
||
3. **Discovery** — function starts inside dark regions from E8 call targets + data-section
|
||
code pointers (vtables/handler tables/Performance pointers), with caller counts.
|
||
4. **Annotation** — pad-byte exclusion (CC/90/00), nearest TU attribution from `file=` tags,
|
||
and `visited` flags (address cited anywhere in game/ context/ docs/).
|
||
|
||
## Headline results
|
||
- `.text` = 892 KB. Index covers **87.3%**. Dark: **428 regions, 105 KB raw / 90 KB real code**.
|
||
- **Indexed-but-unexported = 0** — the entire gap class is "Ghidra never made it a function"
|
||
(consistent with #60's original find: VehicleDead was missing from the *index*, not filtered
|
||
from the export).
|
||
- 332 function starts discovered inside dark regions (call-graph + data-pointer evidence).
|
||
- **By TU family (real code):** munga/ 28.5 KB (engine — source in repo, LOW priority),
|
||
**bt/ 24.0 KB + bt_l4/ 14.9 KB + boundary 15.6 KB = ~54 KB of GAME-side dark**, munga_l4/ 9.5 KB.
|
||
- **Game-side dark regions: 66 already visited by past digs, 159 never touched.**
|
||
|
||
## Validation
|
||
All six known dark celebrities land inside census regions: the VehicleDead handler
|
||
(`0x4c05c4`), Searchlight ToggleLamp (`0x4b838c`), the TakeDamage death tail (`0x4a07b8`,
|
||
inside the famous `0x49ffc8-0x4a1232` stretch — 89 repo citations), the master-perf region
|
||
(`0x4a9b5a-0x4ab188` — 69 citations; crouch/myomers/posture all came from here), the myomer
|
||
integrator (`0x4b8be3`), the duck consumer (`0x4aa011`). The two most-cited dark regions are
|
||
exactly the two that produced the most reconstructions — the census finds the right things.
|
||
|
||
## Top uncharted leads (zero citations, real code, spot-checked)
|
||
1. **`0x480a2a-0x487ca0` cluster (~9 KB across 6 regions), l4splr|btmssn boundary** — head
|
||
spot-check @0x486467: real `enter 0x3c` function driven by a global-index dispatch table
|
||
(`[0x4fef88]` → table `0x501588`) — mission-state-machine flavored. Likely the un-exported
|
||
heart of **BTMission** (round lifecycle / mode carousel).
|
||
2. **`0x4c08fc-0x4c0bc8` (613 B), btplayer, immediately before the BTPlayer ctor** — real
|
||
function reading `player+0x24c`; prime suspect for the deferred **mission-review post
|
||
(id 0x18)** sender and/or death-cycle siblings.
|
||
3. **`0x4d41ae-0x4ddc7c` btl4app regions (~2.6 KB)** — app/session plumbing (attract/config?).
|
||
4. **`0x4af691-0x4af9cf` (707 B), heat|mechmppr boundary** — heat-family tail or mapper head.
|
||
5. **`0x4bdd7b-0x4be078` (673 B), mechweap|btplayer boundary** — weapon-family virtuals.
|
||
|
||
## PART 2 — THE RE-EXPORT (same day, 2026-08-06): the dark map, cut in half
|
||
Installed (no admin, project tool convention `%LOCALAPPDATA%t411-tools`, beside DXSDK/cmake):
|
||
**JDK 21.0.12** + **Ghidra 12.1.2**. ⚠ Ghidra's launcher `.bat` expands `%JAVA_HOME%`
|
||
UNQUOTED, so the space in the user profile ("Joe Laptop") kills it — the runner uses 8.3 SHORT
|
||
paths (`JOELAP~1`, `BT411-~1`). Zips deleted after extract.
|
||
|
||
New tooling (committed, repeatable):
|
||
- `reference/ghidra_scripts/ExportGaps.java` — keeps ExportAll's EXACT output contract (shard
|
||
format, index columns) so every tool/grep/`@ADDR` citation still works, and adds the gap-fill
|
||
pass first: force disassembly + `createFunction` at (a) E8 call targets outside any function,
|
||
(b) data→code pointers at a plausible prologue, (c) the census's discovered starts — iterated
|
||
to a fixpoint. Logs what it created to `gapfill_report.tsv`.
|
||
- `tools/ghidra_reexport.sh` — headless runner (`reprocess` re-runs the script on the saved
|
||
project without re-analysis). `tools/gapdiff.py` — scores any two censused exports.
|
||
|
||
**Results (`tools/gapdiff.py`):**
|
||
| Metric | Before | After |
|
||
|---|---|---|
|
||
| Index coverage of .text | 87.3% | **93.5%** |
|
||
| Dark REAL code | 90.4 KB | **40.8 KB** (−49.5 KB, **54.8% recovered**) |
|
||
| Game-side dark (bt/ + bt_l4/) | 53.1 KB | **21.1 KB** (−32.0 KB) |
|
||
| Dark regions | 428 | 321 |
|
||
| Functions / pseudocode | 6,267 | **6,472** (+205, **0 decompile failures**) |
|
||
Gap fill converged in 2 rounds (205 created: 195 from census starts, 6 call targets, 4 data
|
||
pointers) covering 56.1 KB. Ghidra 12 auto-analysis alone also found 42 more than the old export.
|
||
|
||
**Every historically-bitten dark function now has pseudocode**, including `@0x4c05c4`
|
||
VehicleDead — the function whose absence OPENED this issue — plus ToggleLamp `@0x4b838c`, the
|
||
TakeDamage death tail, the master-perf region (crouch/myomers/posture), the myomer integrator,
|
||
and the `btplayer` hole `@0x4c0904`.
|
||
|
||
### Validation: the re-export CONFIRMS this week's hand reconstructions
|
||
The master-perf pseudocode matches the crouch transcription field-for-field —
|
||
`param_1[0xfe]`=mapPosture(0x3f8), `[0xe6]`=duckState(0x398), `[0x161]`=squatCapable,
|
||
`[0x1e7]`=myomerEffectiveness(0x79c), `[0x10]`=movementMode, the `player+0x25c` novice gate,
|
||
`FUN_004a7fc4(2/3)`=SetLegAnimation sqd/squ, `FUN_004a4c54(8)+(1)`=ForceUpdate, the
|
||
`+0x131` stability alarm, and the duckState clear. Independent confirmation of a raw-disasm
|
||
reconstruction, and of the census method that found it.
|
||
|
||
### …and it caught one thing the raw-disasm pass MISSED (fixed same day)
|
||
The hand pass stopped at `0x4aa0af`; the pseudocode tail carries an **airborne auto-rise**:
|
||
`if ((mode == 3 || mode == 4) && legState == 1) → SetLegAnimation(3) + ForceUpdate(8)+(1) +
|
||
stability 1` — a mech that leaves the ground while parked in the squat is forced up,
|
||
unconditional on duckState/squatCapable. Now in `mech4.cpp`; crouch re-benched un-regressed.
|
||
|
||
## Promotion + archive
|
||
The re-export is now the CANONICAL `reference/decomp/` (index, shards, vtables, census). The
|
||
previous export is preserved verbatim at **`reference/decomp/archive_2025export/`** so older
|
||
KB citations of the form `part_0NN.c:LINE` stay resolvable — ⚠ **line numbers and shard
|
||
membership shifted** in the new export (addresses did not). Cite `@ADDR` (stable), not
|
||
part/line, going forward.
|
||
|
||
## Remaining dark (40.8 KB) — the next leads
|
||
- `0x480a2a-0x487ca0` (~9 KB, l4splr|btmssn boundary) — still dark even after the fill: no call
|
||
or data reference reaches it, which itself is informative (dead code? data misparsed as code?
|
||
a jump-table-only region?). Worth a targeted look with the census's byte-scan.
|
||
- New from the re-export: **`@0x4c0904` is the MASTER BTPlayer Performance** — team resolution
|
||
by name ("Team ... does not exist"), the mission-end "Sending EndMission" console post, and
|
||
the periodic console score heartbeat (corroborating our hand-built one). Our port attributes
|
||
`@0x4c083c` as PlayerSimulation; that attribution needs a re-check.
|
||
- The remaining munga/munga_l4 dark (~19 KB) is low priority — we compile the real engine source.
|
||
|
||
## Systemic fix status — BOTH HALVES DONE
|
||
Inventory + re-export shipped. Rule (gotcha §20) stands and is now cheap to obey: check
|
||
`GAP_CENSUS.md` before any "absent from the export" claim; if still dark, byte-scan.
|
||
Re-run after any future re-export: `tools/ghidra_reexport.sh` → `tools/gapcensus.py` →
|
||
`tools/gapdiff.py`.
|