RP412/tools/podium-repro/README.md

# Podium-teardown crash repro rig

For the intermittent double-delete in `JointedMover::~JointedMover` after the
winners' circle (build 4.12.217, 2026-08-11 six-player playtest). This rig
exists to catch the FIRST free in a debugger — the diagnosis must come from
the trap, not from reading code.

## Evidence already banked (2026-08-11 debug session)

The three podium dumps are `playtestlogs\20260811\rpl4crash (N)1.dmp` — note
the `1` suffix; the plain `rpl4crash (N).dmp` files are the OLD, already-fixed
`L4NetworkManager::Send` NULL-host crash. Symbols: `playtestlogs\symbols-4.12.217`,
load with `.reload /f /i rpl4opt.exe`.

All three podium dumps are byte-for-byte the same failure:

- Crash at `SocketIterator::DeletePlugs` inlined in `~JointedMover+0x68`:
  `call [edx]` where edx = the vtable dword of the plug being deleted,
  overwritten by a small float (9.75 / 12.80 / 7.52 in the three dumps) —
  freed memory reused by something that writes floats.
- The iterator lives on the stack, which IS in the minidumps:
  `numItems=15` (a full skeleton), `currentPosition=1`, so the dead plug is
  `array[0]` — segment index 0, the root segment.
- `segmentTable.socketsNode` read back == the entity pointer, and the table
  header was readable/writable — the ENTITY IS INTACT. The freed thing is
  the segment plug (or its TableEntry), not the pod.
- The frame above returns to `Application::Shutdown+0x71`, which is
  `delete viewpointEntity` (MUNGA\APP.cpp ~1038) — the FIRST entity deleted
  in Shutdown. So the dying pod is the LOCAL pod: the same entity that got
  the podium's second `MakeEntityRenderables(outsideEntity)` pass
  (`DPLRenderer::ShowViewpointFromOutside`). `videoRenderer->Shutdown()`
  runs immediately before it.
- No entries were removed from the table (15 of 15 still present), so the
  first free BYPASSED the Link/TableEntry machinery: a clean
  `delete segment` would have unhooked its TableEntry and decremented
  numItems. Whatever freed it did not run ~Plug against that table.

Nothing above says WHO freed it first. That is what this rig answers.

## What the rig is

- `setup.ps1` — builds N sandboxed installs under `%TEMP%\rp412-podium-repro`
  (junctions to `dist\{AUDIO,GAUGE,VIDEO}`, INIs/RES/DLLs copied from dist,
  `rpl4opt.exe` copied from `Release\` — build Release first).
- `runpod.cmd` — one pod under cdb. Clears `NoDefaultCurrentDirectoryInExePath`
  (Git-Bash exports it; CreateProcess then refuses CWD-relative exe names) and
  keeps the game's CWD = the pod dir (a wrong CWD produces a phantom audio-init
  crash — see project discipline notes).
- `cdbrun.txt` — cdb stdin script. At the create-process event it writes
  `ed $peb+68 02001000` (FLG_HEAP_PAGE_ALLOCS | FLG_USER_STACK_TRACE_DB), which
  turns on PAGE HEAP WITHOUT gflags/elevation — verified working on this
  machine: cdb log shows `verifier.dll` loading and
  `Page heap: pid X: page heap enabled with flags 0x2` (light page heap +
  stack traces: fills freed blocks, validates on every heap op, records
  alloc/free stacks). A double free traps at the SECOND free; the freed-fill
  pattern makes the teardown's vtable call fault deterministically instead of
  only when the heap happened to reuse the block. On any break it prints
  registers, `.exr -1`, `kb 30`, `!heap -p -a` on eax/ecx/edx, writes a FULL
  `podbreak*.dmp` in the pod dir, then quits.
- `feeder.ps1` — drives `-Races` (default 5) full races through ONE set of
  processes over the console protocol: egg -> mesh -> RunMission -> 45 s ->
  StopMission(0) (the buzzer) -> podium -> ending fade ->
  `Application::Shutdown` teardown -> back to WaitingForEgg -> next race.
  `-PodCount` (2..8) sets the size of the field. It watches the cdb logs for
  the break marker every tick, so a trap is caught in seconds rather than at
  the race timeout, and leaves everything up for post-mortem. Derived from
  `tools\two-pod-test.ps1` (proven harness).

**No human pilots are needed at any point.** Bots fill the field: pods sit on
their pads unless `RP412INPUTSCRIPT` drives them, and the race still runs to
the buzzer, still ranks players, and still places them on the stand — which is
the path under test. "More players" means more scripted instances on this one
machine, not more people.

## Run it

    powershell -NoProfile -ExecutionPolicy Bypass -File tools\podium-repro\setup.ps1
    powershell -NoProfile -ExecutionPolicy Bypass -File tools\podium-repro\feeder.ps1

800x600 game windows appear, one per pod; ~2 min per race. Watch for
`FEEDER DONE`. To reproduce the population that actually died (six placed),
run the six-pod field — pass the same count to both scripts:

    ... -File tools\podium-repro\setup.ps1  -PodCount 6
    ... -File tools\podium-repro\feeder.ps1 -PodCount 6 -Races 5

Six 32-bit instances with light page heap fit comfortably on this machine;
they will contend for CPU, which perturbs teardown timing — no bad thing for
an order-dependent bug.

Status when this rig was assembled: single-pod boot under page heap to
WaitingForEgg is VERIFIED (verifier.dll loads, "page heap enabled with flags
0x2" in the cdb log, game reaches the front end). The race loop itself has
NOT been exercised — the session that built this lost the ability to run
anything before it could. The feeder is a close copy of the proven
two-pod-test, but treat its first run as a shakedown: if the pods never reach
`WaitingForLaunch`, the mesh/egg path is the thing to debug, not the crash.

## The PEB page-heap trick is DEAD on Windows 26200 (2026-08-14)

The no-elevation trick this rig was built on no longer works, and it fails
**silently** — which is the dangerous part, because an uninstrumented run
looks exactly like an instrumented one right down to the clean exits.

Measured at the create-process event on build 26200:

    dd $peb+68 L1   ->  00000000      (before)
    ed $peb+68 02001000
    dd $peb+68 L1   ->  02001000      (the write lands)
    g
    dd $peb+68 L1   ->  00000000      (ntdll zeroed it during init)

So `NtGlobalFlag` is re-initialised after `-xe cpr` and page heap never
turns on: no `verifier.dll`, no `Page heap: pid ...` line. It was verified
working on 26100; the machine has since moved to 26200.

`feeder.ps1` now ABORTS when it cannot confirm page heap on every pod,
rather than running an uninstrumented test and reporting a clean result.
`-AllowNoPageHeap` runs anyway and says loudly in the log that the result
is not evidence of absence.

**This makes gflags (and therefore an elevated shell) the only route to a
heap instrument on this machine** — see "Full page heap" below. That is no
loss: full page heap was already the discriminating test for the overrun
reading of the dumps, and light page heap could never have caught it.

## Gotchas this rig has already paid for

- **`dist\environ.ini` ships `RP412PODIUM=0`.** The podium is OFF by default,
  so a sandbox that copies environ.ini verbatim tests NOTHING - every pod
  logs `WinnersCircle: disabled` and goes straight to the results. setup.ps1
  now appends `RP412PODIUM=1` (appended, not edited: last duplicate key wins;
  written ASCII/no-BOM, because a BOM here has invalidated runs before).
  **Check for `WinnersCircle: N placed on 8 spots` in a pod's rpl4.log - if
  it says "disabled", the run proved nothing.**
- **One mission per process is ALL you get, by design.** RESOLVED, not a
  rig fault: in APPMGR.cpp RunMissions, a finished application has
  `Application::Shutdown` called on it; Shutdown returns False, so the app is
  removed from runningApplications, and with none left RunMissions returns
  and the process exits. The teardown IS the exit. So `-Races 1` is the only
  honest setting for a console-driven pod, and repeated teardowns means
  repeated LAUNCHES, not repeated races. (This is why "race 2" never
  started: nothing was there to race.)
- **A clean exit is not a trap.** cdb's `g` returns on process exit as well
  as on a fault, so the "=== POD BREAK ===" banner prints for a normal quit.
  Detection now keys on `ExceptionAddress:` from `.exr -1`, which only
  appears for a real exception. Six "traps" were once six clean shutdowns.

- **Pilot colours must be in `colorLookUp`** (RPL4GAUG.cpp, matched on the
  first three chars): Aqua Black Blue Green Pink Purple Red White Yellow.
  Anything else → `determineEntityColor` returns its 255 fallback → the GPS
  gauge's own-pod blip adds 0xC0 → `translationTable[447]` on a 256-entry
  table → out-of-bounds read, which page heap turns into a hard AV at mission
  start. It only bites the pod that OWNS the odd colour (own blip flashes,
  others don't), so it looks like one machine at random. A six-pod run with
  "Orange" died exactly this way.
- **`!heap` is `!ext.heap`** in current debuggers — exts.dll forwards and
  prints a notice INSTEAD of running, silently costing the heap forensics.
- **Retire the survivors on a trap.** When one pod breaks, the feeder stops
  driving; without an explicit StopMission the other pods race on forever
  with the clock counting up. The feeder now buzzes then kills them, keeping
  only the trapped pod frozen.

## Reading the result

- Crash repro'd: `%TEMP%\rp412-podium-repro\pod?-cdb.log` has everything after
  `=== POD BREAK ===`. The money shot is `!heap -p -a <plug address>` — with
  page heap + stack DB the output includes the block's ALLOC stack and its
  FREE stack. The free stack that is NOT `~JointedMover` IS the first free —
  that is the bug. If the break is a verifier stop inside RtlFreeHeap
  (double free), the block address is in the verifier message/args; if it is
  an AV at `call [edx]` in `~JointedMover`, use eax/ecx (the plug) — the
  script already runs `!heap -p -a` on both. The full `podbreak*.dmp` in the
  pod dir supports any follow-up (`cdb -y C:\VWE\RP412\Release -z <dmp>`).
## Results so far

**2026-08-13, six pods, podium ON, light page heap, pods PARKED: no crash.**
All six placed on the stand (`WinnersCircle: 6 placed on 8 spots`, the fatal
night's own line), own vehicle given an exterior, full `Application::Shutdown`
teardown, six clean exits, no exception on any pod. The target path ran and
survived. What that run did NOT have: any driving - scores came out 999/1000
across the field, so nothing collided, took damage, died or respawned.

**2026-08-13, SIX pods, FULL page heap: INFRASTRUCTURE FAILURE, not a
result.** All six confirmed `flags 0x3`, meshed and ACKed, then every pod
died during LoadingMission (~7 s in) with no exception, no `Fail`, no
`rpl4-fail.log` - an allocation failure taking the process out silently.
Six 32-bit processes each carrying full page heap AND a 2400x1350 cockpit
canvas do not fit. **Do not record this as a negative.** The ceiling is
somewhere between 2 (works) and 6 (dies); 4 is the next thing to try.

Note on why a size filter does NOT rescue this: filtering page heap to
segment-sized blocks would isolate the SEGMENT on its own page, so a
neighbour could no longer reach it - the corruption would silently stop
happening instead of trapping. To trap an overrun you must guard the
CULPRIT, whose size is unknown. Unfiltered full page heap is the instrument
precisely because it guards everything: it traps an overrunning write at the
guard page AND a write through a stale pointer into a decommitted block.
Hence: reduce the field, do not filter the heap.

**2026-08-13, TWO pods, FULL page heap (gflags +hpa, `flags 0x3`), driven:
no crash.** The important half of this result is that a 32-bit rpl4opt
SURVIVES full page heap - it booted, meshed, raced 45 s, placed 2 on the
stand and tore down clean, no address-space failure. So full PH is a usable
instrument here and the six-pod run is affordable. As a crash test it is
weak on its own: two pods, and the fatal night was six.

**2026-08-13, same but DRIVEN (crashlap.txt): no crash.** Scores spread
827..1259, so the pods really did drive and score. Podium, exterior and
teardown all ran on all six; every pod exited clean. NOTE: deaths and
respawns remain UNVERIFIED - the game logs no collision, damage or death
line, so "they drove hard" is all the evidence supports. An input script
cannot press buttons (only the four analog channels), so it cannot fire a
weapon or pop a chute; if deaths turn out to matter, they need a hazardous
map or a different mechanism, not a fiercer script.

**2026-08-14, six pods, driven, podium ON — NO PAGE HEAP, and that is the
finding.** All six raced (scores 741..1010, so they really drove), all six
placed (`WinnersCircle: 6 placed on 8 spots`), full teardown, six clean
exits, no exception. Then the cdb logs turned out to carry no
`Page heap: pid` line at all: the PEB trick had stopped working (see above).
**Do not record this as a negative** — it is an uninstrumented run, which is
weaker even than the earlier light-page-heap ones. Its only value is as one
more sample of the crash failing to happen on its own.

**2026-08-14 — THE RIG CAUGHT ONE, AND THE FIX HOLDS.** `loop.ps1`, four
pods, full page heap. Launch 9 of the first batch trapped:

    ExceptionAddress: rpl4opt!Quaternion::operator=
      [inlined in Player::GoToVehicle+0xe]   reading a FREED block
    free stack: operator delete <- VTV::`scalar deleting destructor'
                                <- FryDeathRowTask::Execute

`this` was a live remote `RPPlayer`. Cause: `Player::playerVehicle` is a bare
pointer nothing ever cleared, and a replicant Player's whole performance is
`GoToVehicle`, reading it every frame - so from the fry onward it read freed
memory, normally mapped and full of garbage. Fixed in `f86b604` by clearing
it in `Entity::CondemnToDeathRow`.

Rate before the fix: 2 traps in 11 instrumented launches, about 1 in 5.
After: **20 of 20 valid samples, no fault.** At the measured rate that is
roughly a one-in-fifty coincidence, so the fix holds as well as twenty
launches can say.

A wasted batch in between is the reason `loop.ps1` now refreshes the pod
exes: it trapped at the same address on the 09:33 binary while the fix sat
in the 11:20 one. Check the `exe:` line before reading any result.

**2026-08-14, FOUR pods, FULL page heap (`flags 0x3`), driven, podium ON:
no crash. This is the first properly instrumented negative.** Everything the
diagnosis points at ran and survived: `WinnersCircle: 4 placed on 8 spots`,
`own vehicle given an exterior` (the podium's second
`MakeEntityRenderables` pass on the LOCAL pod), full `Application::Shutdown`
teardown, four clean exits, no exception on any pod. Scores 553..997 - pod D
at 553 means real collisions and damage, not a parked field.

This is the run all the earlier ones were pretending to be. Full page heap
traps a double free at the second free AND an overrun at the guard page, in
the culprit's own stack. Neither fired. So on this configuration - four pods,
loopback, one teardown - the bug does not happen.

It also settles the address-space question: 2 works, **4 works**, 6 dies. The
fatal night's six-pod population is NOT reachable under full page heap on one
machine, so that population and this instrument cannot be had at the same
time here.

What is still different from the fatal night, in the order worth attacking:
repetition (one sample of an intermittent bug is nearly nothing - loop the
4-pod full-PH run many times), then the real network (three machines, so the
podium and its teardown land at genuinely different moments per pod, instead
of loopback's near-simultaneous ~0 ms).

### What the two clean runs change

Light page heap traps a **double free** at the second free. It does NOT trap
a **buffer overrun** at the moment of the write - it only notices at the
overrun block's own free, via the fill-pattern check. The dump evidence fits
BOTH stories, and the second is arguably the better fit:

- the segment's vtable dword held a small float (9.75 / 12.80 / 7.52), which
  is what you get either from a freed block reused by float data OR from a
  neighbour writing floats past its end;
- but `numItems` was still 15 with every TableEntry intact, so nothing was
  ever unhooked - no `delete` ever ran against that segment. A proper free
  would have unhooked it. An overrun explains that with no free at all.

If it IS an overrun, these runs could not have caught it, and running the
same configuration again will not either. **Escalating to FULL page heap
(guard page immediately after every block, so the overrunning WRITE faults
with the culprit's stack) is the discriminating test.** That needs real
gflags and an elevated shell - see below.

Escalate in this order, cheapest first, and LOG which ones ran - a silent cap
reads as "covered everything":

1. **Driving, damage, deaths and respawns** (`setup.ps1 -Drive 1`, now the
   default; installs crashlap.txt as RP412INPUTSCRIPT). This is the biggest
   difference between the sterile run and the fatal night, and it is the one
   the original diagnosis flagged. Damage is also topically close to the bug:
   a destroyed segment swaps its video object (DestroyedGraphicState), and
   segments are exactly what gets double-freed.
2. **Repeated launches** - one teardown per process (see above), so loop the
   whole rig N times rather than raising -Races. The fatal night crashed on
   pods that had done one race, so a single teardown CAN do it; repetition
   just buys more samples of an intermittent event.
3. **Real network** - loopback gives ~0 ms and near-simultaneous teardown on
   every pod. The fatal night was three machines on a LAN, where the podium
   and its teardown land at genuinely different times per pod. If 1 and 2
   come up dry, this is the remaining structural difference and needs a
   second machine.

## Full page heap (the overrun trap)

The PEB trick gives LIGHT page heap, which cannot catch an overrun at the
write. Real full page heap needs gflags and an ELEVATED shell. Run the rig
with the no-PEB cdb script so only ONE mechanism is in play:

gflags is NOT on PATH - it ships beside cdb, and you want the **x86** copy
(same bitness as the game and as the x86 cdb):

    $gflags = 'C:\Program Files (x86)\Windows Kits\10\Debuggers\x86\gflags.exe'

    & $gflags -i rpl4opt.exe +hpa       # ELEVATED. full page heap
    powershell ... setup.ps1  -PodCount 2
    powershell ... feeder.ps1 -PodCount 2 -Races 1 -CdbScript cdbrun-gflags.txt
    & $gflags -i rpl4opt.exe -hpa       # ALWAYS turn it back off

`-i` takes the image NAME, never a path. The setting lives in HKLM
(Image File Execution Options), so an unelevated shell fails with an access
error rather than doing nothing quietly. `& $gflags -i rpl4opt.exe` with no
flag prints the current setting - a quick way to confirm it stuck.

If `+hpa` does not give full page heap, the explicit form is:

    & $gflags /p /enable rpl4opt.exe /full

Start at `-PodCount 2` for the FIRST full-PH attempt - not because two pods
are likelier to crash (they are not; the fatal night was six) but to find
out whether the process survives full page heap at all before spending a
six-pod run on it. Scale to 6 once it boots and races.

**Confirm the mode actually changed.** The feeder logs each pod's page-heap
line verbatim, e.g. `Page heap: pid 0x97B4: page heap enabled with flags
0x2`. The PEB trick produces `0x2`; gflags full page heap should produce a
DIFFERENT value. If it still says `0x2`, the gflags setting did not take
(wrong image name, not elevated, or 32/64-bit gflags mismatch - use the x86
gflags from the same Debuggers folder as the x86 cdb) and the run is just
the light-page-heap test again.

Full page heap gives every allocation its own page with a guard page right
after it, so a write one byte past the end faults immediately, in the
culprit's own stack - which is exactly the evidence the dumps cannot give.

**Address-space warning:** this is a 32-bit process. Full page heap costs at
least two pages per live allocation, and a game with many small objects can
exhaust the 2 GB user address space and die in allocation rather than in the
bug. Two mitigations, in order of preference:

1. Filter to the block size of interest, so only candidate blocks are
   guarded and the rest use the normal heap. EntitySegment is a few hundred
   bytes; a window either side of that is a good first cut. Check the exact
   flag spelling with `gflags /p /?` before relying on it - the size-range
   options live under `gflags /p /enable <image> /full /size <min> <max>`.
2. Drop to `-PodCount 2` while using unfiltered full page heap, and accept
   the smaller field.

If the game dies during startup or loading with an out-of-memory or
allocation failure under +hpa, that is the address space, not the bug.
Narrow the size filter or reduce the pod count and try again - and note it
in the run log so a memory failure is never mistaken for a negative result.

## When the first free is identified

- Fix OWNERSHIP; do not ship a guard that skips DeletePlugs — it would leak
  and mask the real first free.
- Re-run this rig: all races must survive WITH page heap still on.
- Then: commit (narrative style + Co-Authored-By), `stamp-version.ps1`,
  rebuild, `pack-dist.ps1 -Zip`.

## Leads from static reading (context, NOT conclusions)

- The dead plug is the ROOT segment (index 0) of the LOCAL pod, and the local
  pod is exactly the one that gets a SECOND renderable build at the podium
  (`ShowViewpointFromOutside` -> `MakeEntityRenderables(..., outsideEntity)`),
  and `videoRenderer->Shutdown()` runs just before the fatal delete.
- The renderable build path (RPL4VID.cpp `MakeEntityRenderables`) only READS
  segments; segments are built once in the JointedMover ctor. No second
  segment build happens at the podium (confirmed earlier).
- `EntitySegment` is a Plug in two sockets: the mover's `segmentTable` and the
  parent segment's `childPointerTable` — but both teardown paths looked
  self-consistent on paper. The float scribble suggests the reuser allocates
  scale/quat/matrix-sized float data (renderables are full of those).
- Whatever freed the segment did NOT unhook its links (numItems stayed 15).
  Look for frees that bypass `delete` on the plug: pool teardown
  (`TableEntryOf<V>::operator delete` deletes the SHARED per-V MemoryBlock
  pool when its global allocationCount hits zero), or a wholesale `delete` of
  something that owns segment-adjacent memory.