pod_render_norio.conf is pod_render_rec.conf with one line changed,
serial1=disabled instead of the vrio named pipe. Same binary, same egg. With
it, our build launches and the mech walks:
[launch] state=2 minPriorityEmpty=1 ticks=85316
[queues] p0=- p1=- p2=BUSY p3=- p4=-
BTL4Application::RunMissionMessageHandler
Turning Plasma Score Display On
[sim] pos=(180.466,10,-358.703) yaw=-0.275605 spd=14.4
[sim] pos=(251.959,10,-250.498) yaw=-0.89103 spd=14.4
No fault, 290 log lines and counting. This is the first time the
reconstruction has reached a running mission on the pod.
The hang was never starvation, a deadlock, or a refill loop -- it was VOLUME,
the first candidate I listed and then talked myself out of. 530 renderer
events queue at priority 0 during load; BackgroundTasks::Execute runs exactly
one task per call round-robin over seven tasks, and the 1ms frame budget is
always blown here so no extra background passes happen. That is ~9 events per
real second, so a load legitimately takes ~60s. The fault arrived at ~40s,
before the backlog could clear. Every 'the queue never drains' reading was
really 'the process dies before it can'.
The disproof was already in hand: the post tally climbed to 530 and went FLAT,
which means nothing was refilling. p0=BUSY with nextReady=1 is equally
consistent with a large finite backlog still draining, and I read it as refill.
Why the RIO port: the pipe has no server attached, which should behave as an
unplugged cable. The emulator log shows steady serial1 RX overruns, our build
prints 'RIO never came back from test mode!', and the SHIPPED binary prints
'lost RIO analog request' and launches anyway on this identical rig. So an
unplugged RIO is survivable and our handling of it is not -- a real defect, not
a rig artifact, since a pod with an unplugged RIO cable should still boot.
The wild reference at EulerAngles::operator=(const LinearMatrix&)+0x19 is still
unexplained, but it is now reproducible on demand by enabling serial1 rather
than being a coin flip.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Per-priority occupancy, measured every second until the fault:
[queues] p0=BUSY p1=- p2=BUSY p3=- p4=- nextReady=1
That single line kills the two leading theories. p3/p4 empty means nothing
above priority 0 is competing, so the pump is free to serve it -- no
starvation. nextReady=1 means PeekAtNextEvent always has a READY event, so
priority 0 is not holding a timed event whose alarm never comes due -- no
deadlock. What remains is refill: priority 0 is replenished as fast as the
pump drains it, ~143 events per simulated second.
It is also fully deterministic. Two runs of the same binary reported
pump=352/1499 at frame 1001 and 495/2643 at frame 2002, identical to the byte.
So 'crashes about half the time' was never a race -- it was comparing runs that
differed in binary or conf.
Only InterestManager::PostRendererEvent posts at priority 0 (it maps every
renderer event there while the app is not RunningMission), so naming the
message names the flooder. Application::Post now tallies priority-0 posts by
message ID and reports the busiest three. Notably, all three producers of
NotifyOfNewInterestingEntity are entity-CREATION paths, so if that ID dominates
then something is creating entities in a loop -- which would explain the
unbounded growth behind the fault as well as the hang.
pod_render_noskl.conf now carries the same probes, so one instrumented binary
can be run with the skeleton walk on and off. Walk-off runs are the only
configuration of ours known to reach 'Turning Plasma Score Display On'.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Added BT_NO_SKL, which skips the skeleton build so ONE executable can be run
with and without it. That is the only clean way to test an intermittent
fault, and it settles attribution:
walk enabled ~50% of runs die (page fault at 66D9) at DIFFERENT points
walk disabled 0 crashes in 2 runs
shipped exe 0 crashes in 2 runs, same rig and conf
So it is our new code, not the rig, and the varying fault point points at
memory corruption surfacing later rather than a bad instruction in the walk.
Hypotheses killed by reading the private library headers: the matrix array is
the right size (dpl_MATRIX is float32[4][4]); and neither s_dplobject nor the
common dpl_node header retains an object name, weakening the dangling-string
theory (a private name cache inside the library is still possible and is the
best surviving suspect).
Also checked something that would have been much worse than a crash:
~NotationFile REWRITES its file when dirty, and these are the game's shipped
.SKL data files. MAD.SKL is untouched, and ReadSKLFile now carries the
engine's own Verify(!IsDirty()) guard.
Next tests are listed cheapest-first in the roadmap: keep the NotationFile
alive, copy the Object= names, count board allocations, and re-check whether
MakeEntityRenderables runs twice per mech.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CydandClaude Fable 5