BT412/docs/HARD_PROBLEMS.md

# BT Port — Hardest-Problems Front-Load Plan (scope-hardest-problems workflow + verification)

Goal: tackle the hardest, highest-LEVERAGE problems first so the rest of the port gets easier.
6 candidates were scoped by independent deep-dive agents, then the load-bearing claims were VERIFIED
against the code (several agent claims + several CLAUDE.md facts turned out wrong — see Verification).

## Ranked front-load sequence (post-verification)

| # | Problem | Verdict | Why here |
|---|---------|---------|----------|
| **1** | **P5 — Entity lifecycle / collision teardown** ★ | do-first | Hardest (4) + highest leverage (4) + hard prereq for multiplayer & multi-entity combat & real hit detection. Forces Entity/Mover base-region layout correctness that ALSO de-risks P3. |
| **2** | **P3 — Locomotion cutover** (SequenceController → world transform) | do-early (adjacent to P5) | The gait pipeline is ALREADY reconstructed in source (mech2/3/4.cpp: legAnimation@0x65c, bodyAnimation@0x6bc SequenceControllers) — this is a CUTOVER from the procedural-slide stand-in, not greenfield. Shares P5's Mech/Mover layout de-risk. Produces the speed/turn-demand consumer P2 & P6 need. |
| 3 | **P2 — Authentic input** (MechControlsMapper + RIO) | do-later | Downstream of P3 (mapper output is inert without the locomotion consumer). Software plumbing = days once P3 lands; physical RIO wiring defers to a Phase-8 session with Nick. |
| 4 | **P6 — Multiplayer** (integrate existing TCP stack) | do-later | Terminal integration. The hard part is DONE (L4NET = 3446-line WinSock TCP; master/replicant core complete). Gated on P3 + P5 + subsystem waves. Do a 2-instance smoke test early to de-risk; save fidelity for last. |
| 5 | **P1 — dpl2d reticle/PIP overlay** | opportunistic | Easy + isolated + additive (btl4vid.cpp already calls dpl2d_Circle for the PIP). Slot in whenever a visible aiming win is wanted; blocks nothing. |
| — | **P4 — Build /FORCE cleanup** | optional cosmetic | ⚠ NOT a front-load enabler. The agent's premise was REFUTED (see Verification): /FORCE hides ZERO runtime symbols. Pure cosmetic; do opportunistically, not first. |

**Prerequisite gates:** P5 gates P6 · P3 gates P2 & P6 · P5 and P3 share a hidden prereq = **Entity/Mover
base-region field-layout correctness** (do that audit once, up front, both benefit).

## Verification results (adversarial — several claims corrected)

- ✅ **P5 root cause corrected:** CLAUDE.md said "collision solids were never built." VERIFIED WRONG —
  `Mover::Mover` (RP/MUNGA/MOVER.cpp:1756) allocates `collisionLists = new BoxedSolidCollisionList[2]`
  UNCONDITIONALLY; `~Mover`:2050 deletes it. So the crash is a **clobber / dangling / teardown-order** bug.
- ✅ **P6 corrected:** CLAUDE.md §8 said "reimplement over UDP." VERIFIED WRONG — L4NET.CPP is 3446 lines of
  WinSock **TCP** (`SOCK_STREAM`×4, `ReliableMode`, zero `SOCK_DGRAM`). Already reimplemented, not greenfield.
- ❌ **P4 claim REFUTED:** agent claimed `/FORCE` swallows ~10-15 genuine runtime unresolveds (e.g.
  `Mech::WorldToLocal`). The linker emits EXACTLY 40 unresolveds = 20 `DefaultData` + 20
  `CreateStreamedSubsystem` (dead offline factory), **zero runtime symbols**. `WorldToLocal` resolves via the
  engine lib. CLAUDE.md's "dead offline-factory, cleanup TODO" characterization was correct. P4 is cosmetic.
- ✅ **P3 confirmed less-greenfield:** mech2.cpp carries the full SequenceController gait reconstruction → cutover.
- ✅ **P1 confirmed small/isolated:** dpl2d gap ≈ the reticle/PIP vector overlay only (btl4vid.cpp:563+ already
  calls dpl2d_NewDisplayList/Begin/Circle for the PIP; weapon beams are a separate 3D-renderables module;
  gauges/MFDs/radar are the separate L4GAUGE path — MUNGA_L4/L4GAUGE.cpp — already ported, OFF in BT).

## P5 — ⚠ AUDIT PIVOT: it is a TEARDOWN-SEQUENCE bug, NOT a base-region stomp (verified)

The base-region audit **disproved the base-region-stomp hypothesis** and redirected P5:
- **The enemy's engine base region is FULLY VALID at death** (`BT_ENABLE_TEARDOWN` dump): `collisionLists@0x2e4
  =0D8C8C04`, `segmentTable@0x2f0=00872CE8` (segmentCount `51`), `jointSubsystem@0x30c=00872D68` all valid;
  `lastCollisionList`/`collisionAssistant` NULL (fine). Nothing corrupts it during the enemy's life.
- **Every raw-offset base-region stomp is DEAD CODE.** `Mech::Simulate` (collision-cluster reads/writes +
  telemetry overflow past sizeof) and `FeedHeat*Gauge` (writes through `+0x2ec`) are DEFINED but NEVER
  CALLED (grep-confirmed; mech4.cpp:858 "our drivable override bypasses the unsafe Mech::Simulate"). So the
  `0x2d4-0x2f0` stomps and the `0x7e0-0x828` over-sizeof writes do not run — they are not the cause.
- **The crash is in the teardown SEQUENCE** (`FryDeathRow → ~Mech → ~JointedMover → ~Mover`): cdb_dmg5 shows
  `collisionLists`'s array already **freed** (`0xdd` cookie / `count=0xdddddddb`) by the time `~Mover`'s
  `delete[]` runs — a **double-free / destruction-order** bug (crash site varies run-to-run: `~Mover`
  collisionLists, or `~JointedMover` deleting a `0x0ccd1210` value). ~Mech (reconstructed) runs BEFORE the
  engine base dtors, so the prime suspect is a Mech member dtor / the enemy's minimal-spawn collision setup
  freeing (or aliasing) a base resource that the engine base dtor then frees again.

**Latent finding (real but not the crash cause):** compiled `sizeof(Mech)=0x638` < binary `0x854`, and
`Mech::Make` allocates the compiled size; the code that would write `0x7e0-0x828` past it is the dead
`Simulate`, so no live overflow today — but if that code is ever revived, the Mech must first be padded to 0x854.

**✅ DOUBLE-FREE PINNED (trace done):** NOT collisionLists — it's the **skeleton SEGMENT teardown**.
- `collisionLists` is LIVE at `~Mech` entry (`*cl` not `0xDD`) — ruling out the collision path.
- Real crash stack: `~JointedMover` (JMOVER.cpp:436 `SegmentTableIterator(segmentTable).DeletePlugs()`) →
  `SocketIterator::DeletePlugs` (SOCKET.cpp:157-161 `delete plug`) → `EntitySegment` scalar-deleting-dtor →
  `_free_base` → **`STATUS_HEAP_CORRUPTION` (0xC0000374)** with **`0xFEEE` freed-fill** present. So one of the
  enemy's **51 EntitySegments is freed twice**.
- Mechanism: `DeletePlugs(defeat_release_node=1)` (SOCKET.cpp:151-154) NULLs the socket's release node and
  force-`delete`s every plug, BYPASSING the ref-count release path. If the enemy's segments are ref-counted/
  shared (owned by the skeleton resource, or shallow-aliased by the minimal `Mech::Make` spawn), the normal
  release already freed them → `DeletePlugs` double-frees. (Earlier `~Mover` collisionLists `0xDD` crash was
  the same heap corruption surfacing at a different free — the segment double-free is the root.)

**Fix-trace (deep, still open):** segment-deletion trace (cdb bp on `EntitySegment::~scalar-deleting-dtor`)
shows `DeletePlugs` deletes segment #1 OK (`0d3d3f30`), then crashes on segment #2 (`0d3d5200`) —
`STATUS_HEAP_CORRUPTION` freeing its block. Only 2 of 51 deleted; **segment #2's block is invalid** and its
scalar-dtor never fired earlier, so it was freed via ANOTHER path before `~JointedMover`. Ruled out:
- `EntitySegment::~EntitySegment` (SEGMENT.cpp:116) only DeletePlugs its OWN child-index/damage/video plugs
  (integers) — it does NOT free sibling segments, so deleting #1 didn't free #2.
- The reconstructed `~Mech` body (mech.cpp:952-1008) does NOT explicitly free `subsystemArray` or segments.
- No live overflow past `sizeof(Mech)=0x638` (the over-sizeof writes are all in dead `Simulate`).

**Leading hypothesis — DUAL segment/joint DeletePlugs ownership:** the `JointSubsystem` dtor (JOINT.cpp:499-505)
`DeletePlugs()`es its **`jointTable`**, and `~JointedMover` (JMOVER.cpp:436) `DeletePlugs()`es **`segmentTable`**;
the JMOVER `#if 0` comment ("deleted by the entity subsystemArray[jointSubsystem]") flags the ownership overlap.
If the enemy's minimal `Mech::Make` spawn puts the SAME segments in both tables (or sets release-node/ref-counts
differently than a normal entity), both force-delete them → the second is the double-free.

**✅ DECISIVE TEST RUN (spawn-vs-rp-creation workflow + cdb):**
- **Dual-ownership hypothesis DISPROVEN.** Engine source confirms `segmentTable` (owns EntitySegments,
  `(this,True)`) and `jointTable` (owns Joints, `(NULL,False)`) are DISJOINT — `DeletePlugs` on one never
  touches the other (JMOVER.cpp:171/310, JOINT.cpp:501-504, SEGMENT.cpp:116-126). So there is no
  segment/joint double-delete.
- **Double-destruction DISPROVEN.** cdb bp on `JointedMover::~JointedMover` shows it runs EXACTLY ONCE for
  the enemy (`~JM this=0d04b998` == the teardown-log enemy this). Single teardown.
- **=> It is HEAP CORRUPTION of an individual segment block (Hypothesis B), during a SINGLE teardown.**
  `segmentTable.DeletePlugs` frees segment #1 OK, then segment #2's block header is invalid
  (`RtlValidateHeap ... Invalid address`). So a live write during the enemy's life corrupts one segment's
  block. **The spawn-lifecycle fix (Registry::MakeEntity / BecomeInteresting) will NOT fix this** (the
  workflow's own gate: "entered once => steps 1-5 do not fix it").
- **Source still unpinned, but these are RULED OUT:** the dead-`Simulate` raw-offset stomps (never called);
  double-destruction; subsystem writes past `sizeof(Mech)=0x638`; the segmentTable POINTER (valid at death).
  What remains: a live heap-corruptor of a segment block (candidate: the damage path mechdmg.cpp:628 which
  indexes segments while the enemy takes its 8 hits) — and it may not even be enemy-specific.

**⭐ P5 PREMISE WAS WRONG — dead mechs are SUPPOSED to stay (verified).** In the real game a killed mech does
NOT vanish: RP's death path `VTV::DeathShutdown` (VTV.cpp:1681-1691) loops subsystems calling `DeathShutdown`
(the vehicle shuts down but is NEVER removed); `CondemnToDeathRow`/entity-removal in RP is used ONLY for
transient objects (RIVET.cpp projectiles, DEMOPACK.cpp cleanup), never for a combat kill. BT death is a
STATE/VISUAL transition — `SetGraphicState(DestroyedGraphicState)` (mechdmg.cpp:355) + a death animation
(`deathAnimationLatched@0x650`, mech.hpp:505) + death effect/splash. So the mech becomes a wrecked HULK and
stays. ⇒ **Our current wreck-stays behavior IS the faithful one; `DestroyEntityMessage`-on-death is NOT a
real behavior and should never be enabled.** The teardown crash is an artifact of forcing a removal the
original never does (behind `BT_ENABLE_TEARDOWN`). There is NOTHING to fix here for faithfulness. The real
future "death" work is the OPPOSITE of removal: reconstruct the DeathShutdown sequence + collapse animation
+ destroyed skin — none of which touch the crashing teardown path.

**RECOMMENDATION: CLOSE P5 (not just park).** The wreck-stays death (explosion + stop-targeting) ships and
combat is 100% unaffected; the crash only exists behind `BT_ENABLE_TEARDOWN`. Pinning the live segment-corruptor needs
either gflags **PageHeap** (elevation — catches the overflow AT the write) or a `ba w4` drill on a segment
block, i.e. another deep session for a cosmetic "wreck vanishes" win. Better to spend the effort on P3
(locomotion cutover) and revisit this opportunistically (e.g. run once under PageHeap when elevation is available).

**(superseded) earlier next step:** hardware write-breakpoint (`ba w4`) on segment #2's block to catch the FIRST free
(which table/dtor frees it) — capture its address at spawn, set the bp, run to the free. That names the exact
first-owner and the fix (make that table release-not-delete, or de-duplicate the segment registration).
**Pragmatic alternative:** the wreck-stays behavior (explosion + stop-targeting) already ships and combat is
unaffected; full `DestroyEntityMessage` removal can stay deferred behind `BT_ENABLE_TEARDOWN` until the
segment-ownership model is reconciled (a bounded but genuinely deep engine-archaeology task).

---
### (superseded) earlier hypothesis — base-region stomp

Reproduced under cdb behind `BT_ENABLE_TEARDOWN=1` (mech4.cpp, default OFF, parallel to the working stand-in).

**Findings (all verified, not inferred):**
- The documented cause "collision solids never built" is **WRONG**. `Mover::Mover` (MOVER.cpp:1756) allocates
  `collisionLists = new BoxedSolidCollisionList[2]` unconditionally; the `[teardown]` log shows the enemy's
  `collisionLists@0x2e4 = 0x0CCBCF8C` (a valid heap ptr) at death.
- `collisionLists` sits at **Mover+0x2e4** (from `dt btl4!Mover`), inside the engine collision cluster
  `collisionVolumeCount@0x2d4 · collisionVolume@0x2d8 · collisionTemplate@0x2dc · containedByNode@0x2e0 ·
  collisionLists@0x2e4 · lastCollisionList@0x2e8 · collisionAssistant@0x2ec`. `sizeof(Mover)=0x2f0`,
  `JointedMover=0x318`, `Mech=0x638`.
- The reconstructed Mech's **declared** fields for these are safely relocated (`netOrientation@0x3ec`,
  `arrivalTime@0x500`, `torsoAimTarget@0x3e0`) — BUT `Mech::Simulate`/terrain/heat code in mech4.cpp still
  uses **stale RAW binary offsets** `this+0x2d4 / +0x2e0 / +0x2e8 / +0x2ec / +0x2f0` that land ON the engine
  collision cluster. It READS them (garbage) and DEREF-WRITES through them (e.g. mech4.cpp:1020
  `*(this+0x2ec)+0xc = heatCapacity` writes through the engine `collisionAssistant` ptr). `physicsBody/
  groundRef/groundCell` aren't even declared members — they exist ONLY as these raw offsets.
- Result: the engine base region is corrupted during the enemy's life; teardown crashes at **varying**
  base-member sites (cdb_dmg5: `~Mover`:2050 `delete[] collisionLists`, count cookie `0xdddddddb`; this run:
  `~JointedMover`:444 deleting a `0x0ccd1210` uninit member). It's the **base-region layout divergence**
  between the 1995 BT binary (raw offsets) and the 2007 RP411 engine base — the shared P3/P5 prerequisite.

**Remaining P5 fix (the real work):**
1. **Base-region audit of mech*.cpp raw offsets:** enumerate every `this+0xNN` for `0xNN < 0x318` (engine
   base), map each against the `dt btl4!Mover`/`JointedMover` layout, and convert stomping accesses to the
   correct engine field or the relocated declared member. The `0x2d4-0x2f0` set is the known-bad start;
   audit the whole base range (there are likely more, given `Mech::Simulate` is raw-offset-dense).
2. **Complete the minimal spawn** so the enemy's engine base members are all initialized (no `0xCD`).
3. **Route death through `DestroyEntityMessage → FryDeathRow`** once teardown is clean (env flag → default).

Effort: the audit is mechanical but broad (Simulate/terrain/heat are raw-offset-dense) — days-to-weeks.
Do it behind `BT_ENABLE_TEARDOWN` until green, since the shipped combat loop works by bypassing teardown.

## ✅ CLOSED — the BGF-load heap corruption (`bld08.bgf` / `Builder::~Builder` AV)

**Symptom:** mid-mission `LoadBgfFile("bld08.bgf")` → `Builder::~Builder` → `vector<float>` teardown → AV
inside `operator delete` (ntdll `RtlpFreeHeap` dereferencing `0xDDDDDDDD`), position-dependent (one combat
run crashed; walk-only and differently-routed runs were clean).

**Root cause (reconstruction TYPE CONFUSION, found by a 4-agent workflow + forensics):**
`HeatSink`'s ctor resolved its linked sink via `owner->GetSegment(heatSinkIndex)` — the Nth skeleton
**EntitySegment** (288 bytes compiled) — cast to `Subsystem*`/`HeatSink*`. The binary (`@004adda0`,
part_012.c:16999) reads **`owner->subsystemArray[heatSinkIndex]`** (the subsystem ROSTER @0x128,
bounds-checked vs subsystemCount @0x124, null-guarded before `linkedSinks.Add`). Through the bogus pointer,
every per-frame `ConductHeat` wrote `other->pendingHeat` at compiled offset 388 = **100 bytes past the
288-byte EntitySegment heap block** (and `BalanceCoolant` wrote `coolantLevel` +20 past) — thousands of
4-byte OOB writes during sustained fire, smashing NT free-list metadata adjacent to the mech's segments.
The BGF loader's big vector alloc/free churn merely DETECTED it later. Sibling bug: `PoweredSubsystem`'s
`voltageSourceIndex` (res+0xFC; raw part_013.c:1198 = the same roster lookup) was also GetSegment-resolved,
so `AttachToVoltageSource` wrote `currentTapCount` 136 bytes past the segment block at every mech spawn.

**Exoneration sweep (all CONFIRMED):** the loader itself — an exact Python mirror over all 879 content BGFs,
zero anomalies; the crashed 0x768 block = exactly the 474-float MSVC growth capacity for bld08's 472 verts
(healthy vector); every hard-coded placement-new alloc ≥ compiled sizeof (probe-compiled); Mech spawns use
`sizeof(Mech)` (immune to growth); Explosion churn is pure engine code; `Mech::Simulate` over-sizeof writes
are dead code; LoadLocomotionClips postdates the crash (timeline via /tmp mtimes) and its
`keyframeData[keyframeCount]` read is binary-faithful.

**Fix (heat.cpp + powersub.cpp):** both resolutions replaced with the binary's roster lookup via the public
`owner->GetSubsystemCount()`/`GetSubsystem(i)` (pre-checked so the engine Verify never fires; roster is
pre-zeroed so forward refs read NULL = the binary's "missing" warn path). Powersub's else-gate also fixed to
OWNER flags per raw. Payoff: the heat link now reaches a REAL sink (`heatEnergy=1.34e+07`, was ~0).

**Verification:** (1) `BT_HEAPCHECK=1` (new runtime gate, btl4main.cpp: `_CRTDBG_CHECK_ALWAYS_DF` whole-heap
validation on every alloc/free) through 100+ shots — ZERO detections (pre-fix, the first ConductHeat write
would trip it). (2) The im2 scenario re-run fast: **3601 shots + 10.6 km walked** — no AV. (3) `BT_PROBE_BGF`
(new: direct-load models at boot; `=ALL` sweeps every GEO model) — bld08 clean 25×.

**Durable lesson (systemic checklist entry):** an owner offset `+0x128` in subsystem raw decomp is the
subsystem ROSTER (`subsystemArray`), NOT the segment table — audit every `GetSegment(int)` call in
reconstructed subsystem ctors (only these two existed; both fixed).