BT412/docs/HARD_PROBLEMS.md

BT Port — Hardest-Problems Front-Load Plan (scope-hardest-problems workflow + verification)

Goal: tackle the hardest, highest-LEVERAGE problems first so the rest of the port gets easier. 6 candidates were scoped by independent deep-dive agents, then the load-bearing claims were VERIFIED against the code (several agent claims + several CLAUDE.md facts turned out wrong — see Verification).

Ranked front-load sequence (post-verification)

#	Problem	Verdict	Why here
1	P5 — Entity lifecycle / collision teardown ★	do-first	Hardest (4) + highest leverage (4) + hard prereq for multiplayer & multi-entity combat & real hit detection. Forces Entity/Mover base-region layout correctness that ALSO de-risks P3.
2	P3 — Locomotion cutover (SequenceController → world transform)	do-early (adjacent to P5)	The gait pipeline is ALREADY reconstructed in source (mech2/3/4.cpp: legAnimation@0x65c, bodyAnimation@0x6bc SequenceControllers) — this is a CUTOVER from the procedural-slide stand-in, not greenfield. Shares P5's Mech/Mover layout de-risk. Produces the speed/turn-demand consumer P2 & P6 need.
3	P2 — Authentic input (MechControlsMapper + RIO)	do-later	Downstream of P3 (mapper output is inert without the locomotion consumer). Software plumbing = days once P3 lands; physical RIO wiring defers to a Phase-8 session with Nick.
4	P6 — Multiplayer (integrate existing TCP stack)	do-later	Terminal integration. The hard part is DONE (L4NET = 3446-line WinSock TCP; master/replicant core complete). Gated on P3 + P5 + subsystem waves. Do a 2-instance smoke test early to de-risk; save fidelity for last.
5	P1 — dpl2d reticle/PIP overlay	opportunistic	Easy + isolated + additive (btl4vid.cpp already calls dpl2d_Circle for the PIP). Slot in whenever a visible aiming win is wanted; blocks nothing.
—	P4 — Build /FORCE cleanup	optional cosmetic	⚠ NOT a front-load enabler. The agent's premise was REFUTED (see Verification): /FORCE hides ZERO runtime symbols. Pure cosmetic; do opportunistically, not first.

Prerequisite gates: P5 gates P6 · P3 gates P2 & P6 · P5 and P3 share a hidden prereq = Entity/Mover base-region field-layout correctness (do that audit once, up front, both benefit).

Verification results (adversarial — several claims corrected)

• ✅ P5 root cause corrected: CLAUDE.md said "collision solids were never built." VERIFIED WRONG — Mover::Mover (RP/MUNGA/MOVER.cpp:1756) allocates collisionLists = new BoxedSolidCollisionList[2] UNCONDITIONALLY; ~Mover:2050 deletes it. So the crash is a clobber / dangling / teardown-order bug.
• ✅ P6 corrected: CLAUDE.md §8 said "reimplement over UDP." VERIFIED WRONG — L4NET.CPP is 3446 lines of WinSock TCP (SOCK_STREAM×4, ReliableMode, zero SOCK_DGRAM). Already reimplemented, not greenfield.
• ❌ P4 claim REFUTED: agent claimed /FORCE swallows ~10-15 genuine runtime unresolveds (e.g. Mech::WorldToLocal). The linker emits EXACTLY 40 unresolveds = 20 DefaultData + 20 CreateStreamedSubsystem (dead offline factory), zero runtime symbols. WorldToLocal resolves via the engine lib. CLAUDE.md's "dead offline-factory, cleanup TODO" characterization was correct. P4 is cosmetic.
• ✅ P3 confirmed less-greenfield: mech2.cpp carries the full SequenceController gait reconstruction → cutover.
• ✅ P1 confirmed small/isolated: dpl2d gap ≈ the reticle/PIP vector overlay only (btl4vid.cpp:563+ already calls dpl2d_NewDisplayList/Begin/Circle for the PIP; weapon beams are a separate 3D-renderables module; gauges/MFDs/radar are the separate L4GAUGE path — MUNGA_L4/L4GAUGE.cpp — already ported, OFF in BT).

P5 — ⚠ AUDIT PIVOT: it is a TEARDOWN-SEQUENCE bug, NOT a base-region stomp (verified)

The base-region audit disproved the base-region-stomp hypothesis and redirected P5:

• The enemy's engine base region is FULLY VALID at death (BT_ENABLE_TEARDOWN dump): collisionLists@0x2e4 =0D8C8C04, segmentTable@0x2f0=00872CE8 (segmentCount 51), jointSubsystem@0x30c=00872D68 all valid; lastCollisionList/collisionAssistant NULL (fine). Nothing corrupts it during the enemy's life.
• Every raw-offset base-region stomp is DEAD CODE. Mech::Simulate (collision-cluster reads/writes + telemetry overflow past sizeof) and FeedHeat*Gauge (writes through +0x2ec) are DEFINED but NEVER CALLED (grep-confirmed; mech4.cpp:858 "our drivable override bypasses the unsafe Mech::Simulate"). So the 0x2d4-0x2f0 stomps and the 0x7e0-0x828 over-sizeof writes do not run — they are not the cause.
• The crash is in the teardown SEQUENCE (FryDeathRow → ~Mech → ~JointedMover → ~Mover): cdb_dmg5 shows collisionLists's array already freed (0xdd cookie / count=0xdddddddb) by the time ~Mover's delete[] runs — a double-free / destruction-order bug (crash site varies run-to-run: ~Mover collisionLists, or ~JointedMover deleting a 0x0ccd1210 value). ~Mech (reconstructed) runs BEFORE the engine base dtors, so the prime suspect is a Mech member dtor / the enemy's minimal-spawn collision setup freeing (or aliasing) a base resource that the engine base dtor then frees again.

Latent finding (real but not the crash cause): compiled sizeof(Mech)=0x638 < binary 0x854, and Mech::Make allocates the compiled size; the code that would write 0x7e0-0x828 past it is the dead Simulate, so no live overflow today — but if that code is ever revived, the Mech must first be padded to 0x854.

✅ DOUBLE-FREE PINNED (trace done): NOT collisionLists — it's the skeleton SEGMENT teardown.

• collisionLists is LIVE at ~Mech entry (*cl not 0xDD) — ruling out the collision path.
• Real crash stack: ~JointedMover (JMOVER.cpp:436 SegmentTableIterator(segmentTable).DeletePlugs()) → SocketIterator::DeletePlugs (SOCKET.cpp:157-161 delete plug) → EntitySegment scalar-deleting-dtor → _free_base → STATUS_HEAP_CORRUPTION (0xC0000374) with 0xFEEE freed-fill present. So one of the enemy's 51 EntitySegments is freed twice.
• Mechanism: DeletePlugs(defeat_release_node=1) (SOCKET.cpp:151-154) NULLs the socket's release node and force-deletes every plug, BYPASSING the ref-count release path. If the enemy's segments are ref-counted/ shared (owned by the skeleton resource, or shallow-aliased by the minimal Mech::Make spawn), the normal release already freed them → DeletePlugs double-frees. (Earlier ~Mover collisionLists 0xDD crash was the same heap corruption surfacing at a different free — the segment double-free is the root.)

Fix-trace (deep, still open): segment-deletion trace (cdb bp on EntitySegment::~scalar-deleting-dtor) shows DeletePlugs deletes segment #1 OK (0d3d3f30), then crashes on segment #2 (0d3d5200) — STATUS_HEAP_CORRUPTION freeing its block. Only 2 of 51 deleted; segment #2's block is invalid and its scalar-dtor never fired earlier, so it was freed via ANOTHER path before ~JointedMover. Ruled out:

• EntitySegment::~EntitySegment (SEGMENT.cpp:116) only DeletePlugs its OWN child-index/damage/video plugs (integers) — it does NOT free sibling segments, so deleting #1 didn't free #2.
• The reconstructed ~Mech body (mech.cpp:952-1008) does NOT explicitly free subsystemArray or segments.
• No live overflow past sizeof(Mech)=0x638 (the over-sizeof writes are all in dead Simulate).

Leading hypothesis — DUAL segment/joint DeletePlugs ownership: the JointSubsystem dtor (JOINT.cpp:499-505) DeletePlugs()es its jointTable, and ~JointedMover (JMOVER.cpp:436) DeletePlugs()es segmentTable; the JMOVER #if 0 comment ("deleted by the entity subsystemArray[jointSubsystem]") flags the ownership overlap. If the enemy's minimal Mech::Make spawn puts the SAME segments in both tables (or sets release-node/ref-counts differently than a normal entity), both force-delete them → the second is the double-free.

✅ DECISIVE TEST RUN (spawn-vs-rp-creation workflow + cdb):

• Dual-ownership hypothesis DISPROVEN. Engine source confirms segmentTable (owns EntitySegments, (this,True)) and jointTable (owns Joints, (NULL,False)) are DISJOINT — DeletePlugs on one never touches the other (JMOVER.cpp:171/310, JOINT.cpp:501-504, SEGMENT.cpp:116-126). So there is no segment/joint double-delete.
• Double-destruction DISPROVEN. cdb bp on JointedMover::~JointedMover shows it runs EXACTLY ONCE for the enemy (~JM this=0d04b998 == the teardown-log enemy this). Single teardown.
• => It is HEAP CORRUPTION of an individual segment block (Hypothesis B), during a SINGLE teardown. segmentTable.DeletePlugs frees segment #1 OK, then segment #2's block header is invalid (RtlValidateHeap ... Invalid address). So a live write during the enemy's life corrupts one segment's block. The spawn-lifecycle fix (Registry::MakeEntity / BecomeInteresting) will NOT fix this (the workflow's own gate: "entered once => steps 1-5 do not fix it").
• Source still unpinned, but these are RULED OUT: the dead-Simulate raw-offset stomps (never called); double-destruction; subsystem writes past sizeof(Mech)=0x638; the segmentTable POINTER (valid at death). What remains: a live heap-corruptor of a segment block (candidate: the damage path mechdmg.cpp:628 which indexes segments while the enemy takes its 8 hits) — and it may not even be enemy-specific.

⭐ P5 PREMISE WAS WRONG — dead mechs are SUPPOSED to stay (verified). In the real game a killed mech does NOT vanish: RP's death path VTV::DeathShutdown (VTV.cpp:1681-1691) loops subsystems calling DeathShutdown (the vehicle shuts down but is NEVER removed); CondemnToDeathRow/entity-removal in RP is used ONLY for transient objects (RIVET.cpp projectiles, DEMOPACK.cpp cleanup), never for a combat kill. BT death is a STATE/VISUAL transition — SetGraphicState(DestroyedGraphicState) (mechdmg.cpp:355) + a death animation (deathAnimationLatched@0x650, mech.hpp:505) + death effect/splash. So the mech becomes a wrecked HULK and stays. ⇒ Our current wreck-stays behavior IS the faithful one; DestroyEntityMessage-on-death is NOT a real behavior and should never be enabled. The teardown crash is an artifact of forcing a removal the original never does (behind BT_ENABLE_TEARDOWN). There is NOTHING to fix here for faithfulness. The real future "death" work is the OPPOSITE of removal: reconstruct the DeathShutdown sequence + collapse animation

• destroyed skin — none of which touch the crashing teardown path.

RECOMMENDATION: CLOSE P5 (not just park). The wreck-stays death (explosion + stop-targeting) ships and combat is 100% unaffected; the crash only exists behind BT_ENABLE_TEARDOWN. Pinning the live segment-corruptor needs either gflags PageHeap (elevation — catches the overflow AT the write) or a ba w4 drill on a segment block, i.e. another deep session for a cosmetic "wreck vanishes" win. Better to spend the effort on P3 (locomotion cutover) and revisit this opportunistically (e.g. run once under PageHeap when elevation is available).

(superseded) earlier next step: hardware write-breakpoint (ba w4) on segment #2's block to catch the FIRST free (which table/dtor frees it) — capture its address at spawn, set the bp, run to the free. That names the exact first-owner and the fix (make that table release-not-delete, or de-duplicate the segment registration). Pragmatic alternative: the wreck-stays behavior (explosion + stop-targeting) already ships and combat is unaffected; full DestroyEntityMessage removal can stay deferred behind BT_ENABLE_TEARDOWN until the segment-ownership model is reconciled (a bounded but genuinely deep engine-archaeology task).

---

(superseded) earlier hypothesis — base-region stomp

Reproduced under cdb behind BT_ENABLE_TEARDOWN=1 (mech4.cpp, default OFF, parallel to the working stand-in).

Findings (all verified, not inferred):

• The documented cause "collision solids never built" is WRONG. Mover::Mover (MOVER.cpp:1756) allocates collisionLists = new BoxedSolidCollisionList[2] unconditionally; the [teardown] log shows the enemy's collisionLists@0x2e4 = 0x0CCBCF8C (a valid heap ptr) at death.
• collisionLists sits at Mover+0x2e4 (from dt btl4!Mover), inside the engine collision cluster collisionVolumeCount@0x2d4 · collisionVolume@0x2d8 · collisionTemplate@0x2dc · containedByNode@0x2e0 · collisionLists@0x2e4 · lastCollisionList@0x2e8 · collisionAssistant@0x2ec. sizeof(Mover)=0x2f0, JointedMover=0x318, Mech=0x638.
• The reconstructed Mech's declared fields for these are safely relocated (netOrientation@0x3ec, arrivalTime@0x500, torsoAimTarget@0x3e0) — BUT Mech::Simulate/terrain/heat code in mech4.cpp still uses stale RAW binary offsets this+0x2d4 / +0x2e0 / +0x2e8 / +0x2ec / +0x2f0 that land ON the engine collision cluster. It READS them (garbage) and DEREF-WRITES through them (e.g. mech4.cpp:1020 *(this+0x2ec)+0xc = heatCapacity writes through the engine collisionAssistant ptr). physicsBody/ groundRef/groundCell aren't even declared members — they exist ONLY as these raw offsets.
• Result: the engine base region is corrupted during the enemy's life; teardown crashes at varying base-member sites (cdb_dmg5: ~Mover:2050 delete[] collisionLists, count cookie 0xdddddddb; this run: ~JointedMover:444 deleting a 0x0ccd1210 uninit member). It's the base-region layout divergence between the 1995 BT binary (raw offsets) and the 2007 RP411 engine base — the shared P3/P5 prerequisite.

Remaining P5 fix (the real work):

1. Base-region audit of mech.cpp raw offsets:* enumerate every this+0xNN for 0xNN < 0x318 (engine base), map each against the dt btl4!Mover/JointedMover layout, and convert stomping accesses to the correct engine field or the relocated declared member. The 0x2d4-0x2f0 set is the known-bad start; audit the whole base range (there are likely more, given Mech::Simulate is raw-offset-dense).
2. Complete the minimal spawn so the enemy's engine base members are all initialized (no 0xCD).
3. Route death through DestroyEntityMessage → FryDeathRow once teardown is clean (env flag → default).

Effort: the audit is mechanical but broad (Simulate/terrain/heat are raw-offset-dense) — days-to-weeks. Do it behind BT_ENABLE_TEARDOWN until green, since the shipped combat loop works by bypassing teardown.

✅ CLOSED — the BGF-load heap corruption (bld08.bgf / Builder::~Builder AV)

Symptom: mid-mission LoadBgfFile("bld08.bgf") → Builder::~Builder → vector<float> teardown → AV inside operator delete (ntdll RtlpFreeHeap dereferencing 0xDDDDDDDD), position-dependent (one combat run crashed; walk-only and differently-routed runs were clean).

Root cause (reconstruction TYPE CONFUSION, found by a 4-agent workflow + forensics): HeatSink's ctor resolved its linked sink via owner->GetSegment(heatSinkIndex) — the Nth skeleton EntitySegment (288 bytes compiled) — cast to Subsystem*/HeatSink*. The binary (@004adda0, part_012.c:16999) reads owner->subsystemArray[heatSinkIndex] (the subsystem ROSTER @0x128, bounds-checked vs subsystemCount @0x124, null-guarded before linkedSinks.Add). Through the bogus pointer, every per-frame ConductHeat wrote other->pendingHeat at compiled offset 388 = 100 bytes past the 288-byte EntitySegment heap block (and BalanceCoolant wrote coolantLevel +20 past) — thousands of 4-byte OOB writes during sustained fire, smashing NT free-list metadata adjacent to the mech's segments. The BGF loader's big vector alloc/free churn merely DETECTED it later. Sibling bug: PoweredSubsystem's voltageSourceIndex (res+0xFC; raw part_013.c:1198 = the same roster lookup) was also GetSegment-resolved, so AttachToVoltageSource wrote currentTapCount 136 bytes past the segment block at every mech spawn.

Exoneration sweep (all CONFIRMED): the loader itself — an exact Python mirror over all 879 content BGFs, zero anomalies; the crashed 0x768 block = exactly the 474-float MSVC growth capacity for bld08's 472 verts (healthy vector); every hard-coded placement-new alloc ≥ compiled sizeof (probe-compiled); Mech spawns use sizeof(Mech) (immune to growth); Explosion churn is pure engine code; Mech::Simulate over-sizeof writes are dead code; LoadLocomotionClips postdates the crash (timeline via /tmp mtimes) and its keyframeData[keyframeCount] read is binary-faithful.

Fix (heat.cpp + powersub.cpp): both resolutions replaced with the binary's roster lookup via the public owner->GetSubsystemCount()/GetSubsystem(i) (pre-checked so the engine Verify never fires; roster is pre-zeroed so forward refs read NULL = the binary's "missing" warn path). Powersub's else-gate also fixed to OWNER flags per raw. Payoff: the heat link now reaches a REAL sink (heatEnergy=1.34e+07, was ~0).

Verification: (1) BT_HEAPCHECK=1 (new runtime gate, btl4main.cpp: _CRTDBG_CHECK_ALWAYS_DF whole-heap validation on every alloc/free) through 100+ shots — ZERO detections (pre-fix, the first ConductHeat write would trip it). (2) The im2 scenario re-run fast: 3601 shots + 10.6 km walked — no AV. (3) BT_PROBE_BGF (new: direct-load models at boot; =ALL sweeps every GEO model) — bld08 clean 25×.

Durable lesson (systemic checklist entry): an owner offset +0x128 in subsystem raw decomp is the subsystem ROSTER (subsystemArray), NOT the segment table — audit every GetSegment(int) call in reconstructed subsystem ctors (only these two existed; both fixed).

P7 — Heat-leaf subsystem byte-exactness (the systemic layout gap) ⭐ NEW (root-caused this session)

The reconstruction's heat-leaf branch (HeatableSubsystem:MechSubsystem -> HeatSink -> PoweredSubsystem -> the weapon/sensor leaves) is NOT byte-exact to the binary -- measured: PoweredSubsystem::auxScreenNumber compiles to 0x194 but the binary has it at 0x1DC (72 bytes short). Consequence: any code reading a heat-leaf subsystem field at a RAW binary offset above the gap gets garbage (this is what forced the aux-screen BRIDGE in vehicleSubSystems, and the Sensor RadarPercent gauge stub, and blocks authentic data on the engineering-screen cluster panels' secondary lamps). Fixing it byte-exact makes ALL raw reads correct everywhere -> a high-leverage foundational fix.

Root cause (measured + decompiled this session, authoritative):

• MechSubsystem base + HeatableSubsystem (currentTemperature@0x114 / degradationTemperature@0x118 / failureTemperature@0x11C / heatLoad@0x120) are byte-exact.
• HeatSink accumulates a +16 over-allocation from two RECONSTRUCTION LAYOUT ERRORS:
  1. HeatSink RE-DECLARES degradationTemperature/failureTemperature that already exist in HeatableSubsystem (a shadow/duplicate, +8).
  2. heatState/heatModelFlag are modeled as SEPARATE HeatSink fields (+8) but in the binary they are INSIDE the heatAlarm -- heatState is read at this[0x61]==0x184 == heatAlarm@0x170 + 0x14 (the alarm's level field). So they are modeling errors: heatState reads should be heatAlarm.GetLevel().
• Then two type-size under-allocations: SubsystemConnection is 4B but the binary's is 0xC (built by FUN_004af9cf); HeatAlarm (heat.hpp) is 8B but the binary's alarm at 0x170 is a 0x54 AlarmIndicator (ctor FUN_0041b9ec @part_002.c:4767 = base FUN_004178cc + three 0x14-byte FUN_0041c42c sub-objects at +0x18/+0x2C/+0x40, ending 0x54). HeatFilter is ALREADY 0xC (byte-exact -- NO reimplement needed; earlier fear of a behavior-risk filter change was WRONG).
• Net at PoweredSubsystem::auxScreenNumber: +8 (dup temps) +8 (spurious heatState/heatModelFlag) -8 (SubsystemConnection) -76 (HeatAlarm) -4 (a PoweredSubsystem delta) = -72.

The fix = a byte-exact re-base of the heat-leaf chain: remove the duplicate temps; remove the spurious heatState/heatModelFlag and re-point their reads to heatAlarm.GetLevel(); grow SubsystemConnection 4->0xC and HeatAlarm 8->0x54 (as shared 0x54/0xC types used everywhere the binary has them, WITHOUT breaking the already-byte-exact Watcher branch which uses separate WatcherGaugeAlarm/WatchedConnection); fix the PoweredSubsystem -4. It touches the core classes every heat/power/weapon subsystem derives from (large blast radius) + includes CODE changes (heatState), so it needs a full at-risk-read audit (raw this+0xNN reads calibrated to the CURRENT compiled layout would break) + exhaustive combat/heat verification. Safety revert point tagged pre-heatleaf-rebase. The heat-leaf-layout-audit workflow derives the full plan.

MEASUREMENT TECHNIQUE (reusable): a template<int N> struct HSOff; + friend struct XProbe; + struct XProbe { HSOff<offsetof(Class, field)> a; ... }; -- the "uses undefined struct 'HSOff'" build error reports each field's COMPILED offset in decimal. Compare to the binary offset from the ctor decomp.