RP412/tools/podium-repro

Podium-teardown crash repro rig

For the intermittent double-delete in JointedMover::~JointedMover after the winners' circle (build 4.12.217, 2026-08-11 six-player playtest). This rig exists to catch the FIRST free in a debugger — the diagnosis must come from the trap, not from reading code.

Evidence already banked (2026-08-11 debug session)

The three podium dumps are playtestlogs\20260811\rpl4crash (N)1.dmp — note the 1 suffix; the plain rpl4crash (N).dmp files are the OLD, already-fixed L4NetworkManager::Send NULL-host crash. Symbols: playtestlogs\symbols-4.12.217, load with .reload /f /i rpl4opt.exe.

All three podium dumps are byte-for-byte the same failure:

• Crash at SocketIterator::DeletePlugs inlined in ~JointedMover+0x68: call [edx] where edx = the vtable dword of the plug being deleted, overwritten by a small float (9.75 / 12.80 / 7.52 in the three dumps) — freed memory reused by something that writes floats.
• The iterator lives on the stack, which IS in the minidumps: numItems=15 (a full skeleton), currentPosition=1, so the dead plug is array[0] — segment index 0, the root segment.
• segmentTable.socketsNode read back == the entity pointer, and the table header was readable/writable — the ENTITY IS INTACT. The freed thing is the segment plug (or its TableEntry), not the pod.
• The frame above returns to Application::Shutdown+0x71, which is delete viewpointEntity (MUNGA\APP.cpp ~1038) — the FIRST entity deleted in Shutdown. So the dying pod is the LOCAL pod: the same entity that got the podium's second MakeEntityRenderables(outsideEntity) pass (DPLRenderer::ShowViewpointFromOutside). videoRenderer->Shutdown() runs immediately before it.
• No entries were removed from the table (15 of 15 still present), so the first free BYPASSED the Link/TableEntry machinery: a clean delete segment would have unhooked its TableEntry and decremented numItems. Whatever freed it did not run ~Plug against that table.

Nothing above says WHO freed it first. That is what this rig answers.

What the rig is

• setup.ps1 — builds N sandboxed installs under %TEMP%\rp412-podium-repro (junctions to dist\{AUDIO,GAUGE,VIDEO}, INIs/RES/DLLs copied from dist, rpl4opt.exe copied from Release\ — build Release first).
• runpod.cmd — one pod under cdb. Clears NoDefaultCurrentDirectoryInExePath (Git-Bash exports it; CreateProcess then refuses CWD-relative exe names) and keeps the game's CWD = the pod dir (a wrong CWD produces a phantom audio-init crash — see project discipline notes).
• cdbrun.txt — cdb stdin script. At the create-process event it writes ed $peb+68 02001000 (FLG_HEAP_PAGE_ALLOCS | FLG_USER_STACK_TRACE_DB), which turns on PAGE HEAP WITHOUT gflags/elevation — verified working on this machine: cdb log shows verifier.dll loading and Page heap: pid X: page heap enabled with flags 0x2 (light page heap + stack traces: fills freed blocks, validates on every heap op, records alloc/free stacks). A double free traps at the SECOND free; the freed-fill pattern makes the teardown's vtable call fault deterministically instead of only when the heap happened to reuse the block. On any break it prints registers, .exr -1, kb 30, !heap -p -a on eax/ecx/edx, writes a FULL podbreak*.dmp in the pod dir, then quits.
• feeder.ps1 — drives -Races (default 5) full races through ONE set of processes over the console protocol: egg -> mesh -> RunMission -> 45 s -> StopMission(0) (the buzzer) -> podium -> ending fade -> Application::Shutdown teardown -> back to WaitingForEgg -> next race. -PodCount (2..8) sets the size of the field. It watches the cdb logs for the break marker every tick, so a trap is caught in seconds rather than at the race timeout, and leaves everything up for post-mortem. Derived from tools\two-pod-test.ps1 (proven harness).

No human pilots are needed at any point. Bots fill the field: pods sit on their pads unless RP412INPUTSCRIPT drives them, and the race still runs to the buzzer, still ranks players, and still places them on the stand — which is the path under test. "More players" means more scripted instances on this one machine, not more people.

Run it

powershell -NoProfile -ExecutionPolicy Bypass -File tools\podium-repro\setup.ps1
powershell -NoProfile -ExecutionPolicy Bypass -File tools\podium-repro\feeder.ps1

800x600 game windows appear, one per pod; ~2 min per race. Watch for FEEDER DONE. To reproduce the population that actually died (six placed), run the six-pod field — pass the same count to both scripts:

... -File tools\podium-repro\setup.ps1  -PodCount 6
... -File tools\podium-repro\feeder.ps1 -PodCount 6 -Races 5

Six 32-bit instances with light page heap fit comfortably on this machine; they will contend for CPU, which perturbs teardown timing — no bad thing for an order-dependent bug.

Status when this rig was assembled: single-pod boot under page heap to WaitingForEgg is VERIFIED (verifier.dll loads, "page heap enabled with flags 0x2" in the cdb log, game reaches the front end). The race loop itself has NOT been exercised — the session that built this lost the ability to run anything before it could. The feeder is a close copy of the proven two-pod-test, but treat its first run as a shakedown: if the pods never reach WaitingForLaunch, the mesh/egg path is the thing to debug, not the crash.

The PEB page-heap trick is DEAD on Windows 26200 (2026-08-14)

The no-elevation trick this rig was built on no longer works, and it fails silently — which is the dangerous part, because an uninstrumented run looks exactly like an instrumented one right down to the clean exits.

Measured at the create-process event on build 26200:

dd $peb+68 L1   ->  00000000      (before)
ed $peb+68 02001000
dd $peb+68 L1   ->  02001000      (the write lands)
g
dd $peb+68 L1   ->  00000000      (ntdll zeroed it during init)

So NtGlobalFlag is re-initialised after -xe cpr and page heap never turns on: no verifier.dll, no Page heap: pid ... line. It was verified working on 26100; the machine has since moved to 26200.

feeder.ps1 now ABORTS when it cannot confirm page heap on every pod, rather than running an uninstrumented test and reporting a clean result. -AllowNoPageHeap runs anyway and says loudly in the log that the result is not evidence of absence.

This makes gflags (and therefore an elevated shell) the only route to a heap instrument on this machine — see "Full page heap" below. That is no loss: full page heap was already the discriminating test for the overrun reading of the dumps, and light page heap could never have caught it.

Gotchas this rig has already paid for

• dist\environ.ini ships RP412PODIUM=0. The podium is OFF by default, so a sandbox that copies environ.ini verbatim tests NOTHING - every pod logs WinnersCircle: disabled and goes straight to the results. setup.ps1 now appends RP412PODIUM=1 (appended, not edited: last duplicate key wins; written ASCII/no-BOM, because a BOM here has invalidated runs before). Check for WinnersCircle: N placed on 8 spots in a pod's rpl4.log - if it says "disabled", the run proved nothing.

• One mission per process is ALL you get, by design. RESOLVED, not a rig fault: in APPMGR.cpp RunMissions, a finished application has Application::Shutdown called on it; Shutdown returns False, so the app is removed from runningApplications, and with none left RunMissions returns and the process exits. The teardown IS the exit. So -Races 1 is the only honest setting for a console-driven pod, and repeated teardowns means repeated LAUNCHES, not repeated races. (This is why "race 2" never started: nothing was there to race.)

• A clean exit is not a trap. cdb's g returns on process exit as well as on a fault, so the "=== POD BREAK ===" banner prints for a normal quit. Detection now keys on ExceptionAddress: from .exr -1, which only appears for a real exception. Six "traps" were once six clean shutdowns.

• Pilot colours must be in colorLookUp (RPL4GAUG.cpp, matched on the first three chars): Aqua Black Blue Green Pink Purple Red White Yellow. Anything else → determineEntityColor returns its 255 fallback → the GPS gauge's own-pod blip adds 0xC0 → translationTable[447] on a 256-entry table → out-of-bounds read, which page heap turns into a hard AV at mission start. It only bites the pod that OWNS the odd colour (own blip flashes, others don't), so it looks like one machine at random. A six-pod run with "Orange" died exactly this way.

• !heap is !ext.heap in current debuggers — exts.dll forwards and prints a notice INSTEAD of running, silently costing the heap forensics.

• Retire the survivors on a trap. When one pod breaks, the feeder stops driving; without an explicit StopMission the other pods race on forever with the clock counting up. The feeder now buzzes then kills them, keeping only the trapped pod frozen.

Reading the result

• Crash repro'd: %TEMP%\rp412-podium-repro\pod?-cdb.log has everything after === POD BREAK ===. The money shot is !heap -p -a <plug address> — with page heap + stack DB the output includes the block's ALLOC stack and its FREE stack. The free stack that is NOT ~JointedMover IS the first free — that is the bug. If the break is a verifier stop inside RtlFreeHeap (double free), the block address is in the verifier message/args; if it is an AV at call [edx] in ~JointedMover, use eax/ecx (the plug) — the script already runs !heap -p -a on both. The full podbreak*.dmp in the pod dir supports any follow-up (cdb -y C:\VWE\RP412\Release -z <dmp>).

Results so far

2026-08-13, six pods, podium ON, light page heap, pods PARKED: no crash. All six placed on the stand (WinnersCircle: 6 placed on 8 spots, the fatal night's own line), own vehicle given an exterior, full Application::Shutdown teardown, six clean exits, no exception on any pod. The target path ran and survived. What that run did NOT have: any driving - scores came out 999/1000 across the field, so nothing collided, took damage, died or respawned.

2026-08-13, SIX pods, FULL page heap: INFRASTRUCTURE FAILURE, not a result. All six confirmed flags 0x3, meshed and ACKed, then every pod died during LoadingMission (~7 s in) with no exception, no Fail, no rpl4-fail.log - an allocation failure taking the process out silently. Six 32-bit processes each carrying full page heap AND a 2400x1350 cockpit canvas do not fit. Do not record this as a negative. The ceiling is somewhere between 2 (works) and 6 (dies); 4 is the next thing to try.

Note on why a size filter does NOT rescue this: filtering page heap to segment-sized blocks would isolate the SEGMENT on its own page, so a neighbour could no longer reach it - the corruption would silently stop happening instead of trapping. To trap an overrun you must guard the CULPRIT, whose size is unknown. Unfiltered full page heap is the instrument precisely because it guards everything: it traps an overrunning write at the guard page AND a write through a stale pointer into a decommitted block. Hence: reduce the field, do not filter the heap.

2026-08-13, TWO pods, FULL page heap (gflags +hpa, flags 0x3), driven: no crash. The important half of this result is that a 32-bit rpl4opt SURVIVES full page heap - it booted, meshed, raced 45 s, placed 2 on the stand and tore down clean, no address-space failure. So full PH is a usable instrument here and the six-pod run is affordable. As a crash test it is weak on its own: two pods, and the fatal night was six.

2026-08-13, same but DRIVEN (crashlap.txt): no crash. Scores spread 827..1259, so the pods really did drive and score. Podium, exterior and teardown all ran on all six; every pod exited clean. NOTE: deaths and respawns remain UNVERIFIED - the game logs no collision, damage or death line, so "they drove hard" is all the evidence supports. An input script cannot press buttons (only the four analog channels), so it cannot fire a weapon or pop a chute; if deaths turn out to matter, they need a hazardous map or a different mechanism, not a fiercer script.

2026-08-14, six pods, driven, podium ON — NO PAGE HEAP, and that is the finding. All six raced (scores 741..1010, so they really drove), all six placed (WinnersCircle: 6 placed on 8 spots), full teardown, six clean exits, no exception. Then the cdb logs turned out to carry no Page heap: pid line at all: the PEB trick had stopped working (see above). Do not record this as a negative — it is an uninstrumented run, which is weaker even than the earlier light-page-heap ones. Its only value is as one more sample of the crash failing to happen on its own.

2026-08-14, FOUR pods, FULL page heap (flags 0x3), driven, podium ON: no crash. This is the first properly instrumented negative. Everything the diagnosis points at ran and survived: WinnersCircle: 4 placed on 8 spots, own vehicle given an exterior (the podium's second MakeEntityRenderables pass on the LOCAL pod), full Application::Shutdown teardown, four clean exits, no exception on any pod. Scores 553..997 - pod D at 553 means real collisions and damage, not a parked field.

This is the run all the earlier ones were pretending to be. Full page heap traps a double free at the second free AND an overrun at the guard page, in the culprit's own stack. Neither fired. So on this configuration - four pods, loopback, one teardown - the bug does not happen.

It also settles the address-space question: 2 works, 4 works, 6 dies. The fatal night's six-pod population is NOT reachable under full page heap on one machine, so that population and this instrument cannot be had at the same time here.

What is still different from the fatal night, in the order worth attacking: repetition (one sample of an intermittent bug is nearly nothing - loop the 4-pod full-PH run many times), then the real network (three machines, so the podium and its teardown land at genuinely different moments per pod, instead of loopback's near-simultaneous ~0 ms).

What the two clean runs change

Light page heap traps a double free at the second free. It does NOT trap a buffer overrun at the moment of the write - it only notices at the overrun block's own free, via the fill-pattern check. The dump evidence fits BOTH stories, and the second is arguably the better fit:

• the segment's vtable dword held a small float (9.75 / 12.80 / 7.52), which is what you get either from a freed block reused by float data OR from a neighbour writing floats past its end;
• but numItems was still 15 with every TableEntry intact, so nothing was ever unhooked - no delete ever ran against that segment. A proper free would have unhooked it. An overrun explains that with no free at all.

If it IS an overrun, these runs could not have caught it, and running the same configuration again will not either. Escalating to FULL page heap (guard page immediately after every block, so the overrunning WRITE faults with the culprit's stack) is the discriminating test. That needs real gflags and an elevated shell - see below.

Escalate in this order, cheapest first, and LOG which ones ran - a silent cap reads as "covered everything":

1. Driving, damage, deaths and respawns (setup.ps1 -Drive 1, now the default; installs crashlap.txt as RP412INPUTSCRIPT). This is the biggest difference between the sterile run and the fatal night, and it is the one the original diagnosis flagged. Damage is also topically close to the bug: a destroyed segment swaps its video object (DestroyedGraphicState), and segments are exactly what gets double-freed.
2. Repeated launches - one teardown per process (see above), so loop the whole rig N times rather than raising -Races. The fatal night crashed on pods that had done one race, so a single teardown CAN do it; repetition just buys more samples of an intermittent event.
3. Real network - loopback gives ~0 ms and near-simultaneous teardown on every pod. The fatal night was three machines on a LAN, where the podium and its teardown land at genuinely different times per pod. If 1 and 2 come up dry, this is the remaining structural difference and needs a second machine.

Full page heap (the overrun trap)

The PEB trick gives LIGHT page heap, which cannot catch an overrun at the write. Real full page heap needs gflags and an ELEVATED shell. Run the rig with the no-PEB cdb script so only ONE mechanism is in play:

gflags is NOT on PATH - it ships beside cdb, and you want the x86 copy (same bitness as the game and as the x86 cdb):

$gflags = 'C:\Program Files (x86)\Windows Kits\10\Debuggers\x86\gflags.exe'

& $gflags -i rpl4opt.exe +hpa       # ELEVATED. full page heap
powershell ... setup.ps1  -PodCount 2
powershell ... feeder.ps1 -PodCount 2 -Races 1 -CdbScript cdbrun-gflags.txt
& $gflags -i rpl4opt.exe -hpa       # ALWAYS turn it back off

-i takes the image NAME, never a path. The setting lives in HKLM (Image File Execution Options), so an unelevated shell fails with an access error rather than doing nothing quietly. & $gflags -i rpl4opt.exe with no flag prints the current setting - a quick way to confirm it stuck.

If +hpa does not give full page heap, the explicit form is:

& $gflags /p /enable rpl4opt.exe /full

Start at -PodCount 2 for the FIRST full-PH attempt - not because two pods are likelier to crash (they are not; the fatal night was six) but to find out whether the process survives full page heap at all before spending a six-pod run on it. Scale to 6 once it boots and races.

Confirm the mode actually changed. The feeder logs each pod's page-heap line verbatim, e.g. Page heap: pid 0x97B4: page heap enabled with flags 0x2. The PEB trick produces 0x2; gflags full page heap should produce a DIFFERENT value. If it still says 0x2, the gflags setting did not take (wrong image name, not elevated, or 32/64-bit gflags mismatch - use the x86 gflags from the same Debuggers folder as the x86 cdb) and the run is just the light-page-heap test again.

Full page heap gives every allocation its own page with a guard page right after it, so a write one byte past the end faults immediately, in the culprit's own stack - which is exactly the evidence the dumps cannot give.

Address-space warning: this is a 32-bit process. Full page heap costs at least two pages per live allocation, and a game with many small objects can exhaust the 2 GB user address space and die in allocation rather than in the bug. Two mitigations, in order of preference:

1. Filter to the block size of interest, so only candidate blocks are guarded and the rest use the normal heap. EntitySegment is a few hundred bytes; a window either side of that is a good first cut. Check the exact flag spelling with gflags /p /? before relying on it - the size-range options live under gflags /p /enable <image> /full /size <min> <max>.
2. Drop to -PodCount 2 while using unfiltered full page heap, and accept the smaller field.

If the game dies during startup or loading with an out-of-memory or allocation failure under +hpa, that is the address space, not the bug. Narrow the size filter or reduce the pod count and try again - and note it in the run log so a memory failure is never mistaken for a negative result.

When the first free is identified

• Fix OWNERSHIP; do not ship a guard that skips DeletePlugs — it would leak and mask the real first free.
• Re-run this rig: all races must survive WITH page heap still on.
• Then: commit (narrative style + Co-Authored-By), stamp-version.ps1, rebuild, pack-dist.ps1 -Zip.

Leads from static reading (context, NOT conclusions)

• The dead plug is the ROOT segment (index 0) of the LOCAL pod, and the local pod is exactly the one that gets a SECOND renderable build at the podium (ShowViewpointFromOutside -> MakeEntityRenderables(..., outsideEntity)), and videoRenderer->Shutdown() runs just before the fatal delete.
• The renderable build path (RPL4VID.cpp MakeEntityRenderables) only READS segments; segments are built once in the JointedMover ctor. No second segment build happens at the podium (confirmed earlier).
• EntitySegment is a Plug in two sockets: the mover's segmentTable and the parent segment's childPointerTable — but both teardown paths looked self-consistent on paper. The float scribble suggests the reuser allocates scale/quat/matrix-sized float data (renderables are full of those).
• Whatever freed the segment did NOT unhook its links (numItems stayed 15). Look for frees that bypass delete on the plug: pool teardown (TableEntryOf<V>::operator delete deletes the SHARED per-V MemoryBlock pool when its global allocationCount hits zero), or a wholesale delete of something that owns segment-adjacent memory.