emulator/render-bridge/first_mission_frame.png -- arena city, textured
buildings, horizon, 28fps, captured from the GL bridge while our build ran a
live mission with the mech walking. The skeleton went in with it:
[skl] video\mad.skl -> 26 nodes, 19 objects.
Corrects yesterday's RIO framing. I called serial1 an unplugged cable. Wrong
twice: VRio.App is running on this rig (the dev-rig default since 7/17), and
the emulator log recorded real byte counts on that port -- RX overruns of 471,
503, 528 -- which an unconnected pipe cannot produce. The port was LIVE and
streaming during every crashing run.
So the defect is not that we mishandle a disconnected port, it is that we
mishandle a live RIO stream, which is worse: every production pod has a real
RIO. The shipped binary on the same rig logs 'lost RIO analog request', keeps
sending characters, and runs the mission; ours logs 'RIO never came back from
test mode!' and dies partway through the load. The overrun counts say the
guest is not draining the port fast enough. L4RIO.CPP's handshake is authentic
archive code, but MechRIOMapper (BT/MECHMPPR.CPP) is ours -- that is the thread.
pod_render_norio.conf stays the way to get a running mission today, but it is a
workaround: it disables the cockpit's primary input device.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
pod_render_norio.conf is pod_render_rec.conf with one line changed,
serial1=disabled instead of the vrio named pipe. Same binary, same egg. With
it, our build launches and the mech walks:
[launch] state=2 minPriorityEmpty=1 ticks=85316
[queues] p0=- p1=- p2=BUSY p3=- p4=-
BTL4Application::RunMissionMessageHandler
Turning Plasma Score Display On
[sim] pos=(180.466,10,-358.703) yaw=-0.275605 spd=14.4
[sim] pos=(251.959,10,-250.498) yaw=-0.89103 spd=14.4
No fault, 290 log lines and counting. This is the first time the
reconstruction has reached a running mission on the pod.
The hang was never starvation, a deadlock, or a refill loop -- it was VOLUME,
the first candidate I listed and then talked myself out of. 530 renderer
events queue at priority 0 during load; BackgroundTasks::Execute runs exactly
one task per call round-robin over seven tasks, and the 1ms frame budget is
always blown here so no extra background passes happen. That is ~9 events per
real second, so a load legitimately takes ~60s. The fault arrived at ~40s,
before the backlog could clear. Every 'the queue never drains' reading was
really 'the process dies before it can'.
The disproof was already in hand: the post tally climbed to 530 and went FLAT,
which means nothing was refilling. p0=BUSY with nextReady=1 is equally
consistent with a large finite backlog still draining, and I read it as refill.
Why the RIO port: the pipe has no server attached, which should behave as an
unplugged cable. The emulator log shows steady serial1 RX overruns, our build
prints 'RIO never came back from test mode!', and the SHIPPED binary prints
'lost RIO analog request' and launches anyway on this identical rig. So an
unplugged RIO is survivable and our handling of it is not -- a real defect, not
a rig artifact, since a pod with an unplugged RIO cable should still boot.
The wild reference at EulerAngles::operator=(const LinearMatrix&)+0x19 is still
unexplained, but it is now reproducible on demand by enabling serial1 rather
than being a coin flip.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Per-priority occupancy, measured every second until the fault:
[queues] p0=BUSY p1=- p2=BUSY p3=- p4=- nextReady=1
That single line kills the two leading theories. p3/p4 empty means nothing
above priority 0 is competing, so the pump is free to serve it -- no
starvation. nextReady=1 means PeekAtNextEvent always has a READY event, so
priority 0 is not holding a timed event whose alarm never comes due -- no
deadlock. What remains is refill: priority 0 is replenished as fast as the
pump drains it, ~143 events per simulated second.
It is also fully deterministic. Two runs of the same binary reported
pump=352/1499 at frame 1001 and 495/2643 at frame 2002, identical to the byte.
So 'crashes about half the time' was never a race -- it was comparing runs that
differed in binary or conf.
Only InterestManager::PostRendererEvent posts at priority 0 (it maps every
renderer event there while the app is not RunningMission), so naming the
message names the flooder. Application::Post now tallies priority-0 posts by
message ID and reports the busiest three. Notably, all three producers of
NotifyOfNewInterestingEntity are entity-CREATION paths, so if that ID dominates
then something is creating entities in a loop -- which would explain the
unbounded growth behind the fault as well as the hang.
pod_render_noskl.conf now carries the same probes, so one instrumented binary
can be run with the skeleton walk on and off. Walk-off runs are the only
configuration of ours known to reach 'Turning Plasma Score Display On'.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CydandClaude Fable 5