emulator/render-bridge/first_mission_frame.png -- arena city, textured
buildings, horizon, 28fps, captured from the GL bridge while our build ran a
live mission with the mech walking. The skeleton went in with it:
[skl] video\mad.skl -> 26 nodes, 19 objects.
Corrects yesterday's RIO framing. I called serial1 an unplugged cable. Wrong
twice: VRio.App is running on this rig (the dev-rig default since 7/17), and
the emulator log recorded real byte counts on that port -- RX overruns of 471,
503, 528 -- which an unconnected pipe cannot produce. The port was LIVE and
streaming during every crashing run.
So the defect is not that we mishandle a disconnected port, it is that we
mishandle a live RIO stream, which is worse: every production pod has a real
RIO. The shipped binary on the same rig logs 'lost RIO analog request', keeps
sending characters, and runs the mission; ours logs 'RIO never came back from
test mode!' and dies partway through the load. The overrun counts say the
guest is not draining the port fast enough. L4RIO.CPP's handshake is authentic
archive code, but MechRIOMapper (BT/MECHMPPR.CPP) is ours -- that is the thread.
pod_render_norio.conf stays the way to get a running mission today, but it is a
workaround: it disables the cockpit's primary input device.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Per-priority occupancy, measured every second until the fault:
[queues] p0=BUSY p1=- p2=BUSY p3=- p4=- nextReady=1
That single line kills the two leading theories. p3/p4 empty means nothing
above priority 0 is competing, so the pump is free to serve it -- no
starvation. nextReady=1 means PeekAtNextEvent always has a READY event, so
priority 0 is not holding a timed event whose alarm never comes due -- no
deadlock. What remains is refill: priority 0 is replenished as fast as the
pump drains it, ~143 events per simulated second.
It is also fully deterministic. Two runs of the same binary reported
pump=352/1499 at frame 1001 and 495/2643 at frame 2002, identical to the byte.
So 'crashes about half the time' was never a race -- it was comparing runs that
differed in binary or conf.
Only InterestManager::PostRendererEvent posts at priority 0 (it maps every
renderer event there while the app is not RunningMission), so naming the
message names the flooder. Application::Post now tallies priority-0 posts by
message ID and reports the busiest three. Notably, all three producers of
NotifyOfNewInterestingEntity are entity-CREATION paths, so if that ID dominates
then something is creating entities in a loop -- which would explain the
unbounded growth behind the fault as well as the hang.
pod_render_noskl.conf now carries the same probes, so one instrumented binary
can be run with the skeleton walk on and off. Walk-off runs are the only
configuration of ours known to reach 'Turning Plasma Score Display On'.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The previous commit's attribution was wrong and is corrected in the roadmap.
WRONG: 'the crash is the skeleton walk's' rested on 0 crashes in 2 runs with
the walk disabled -- a 25% coin flip presented as evidence. The oldest
preserved dump has no [skl] line and ends on the old 'couldn't figure out how
to MakeEntityRenderables' fallback, so it predates the walk and faults
identically.
WRONG: 'it lands at different points each time'. Every dump carries the same
numbers to the byte (cr2 7000FA64, EIP 66D9). What varies is how far the log
gets, not where the fault is.
Resolving the address through btl4opt.map names it exactly:
EulerAngles::operator=(const LinearMatrix&)+0x19, a read through the matrix
reference. A binary scan finds all three call sites of that operator pass a
stack local, so the 1.79GB pointer still has no static explanation -- that is
now its own open item rather than a guess.
Two theories killed cleanly by the new BT_STACK_LOG probe and a binary scan:
ESP drift (drift=0 over 2002 frames; exactly one callee-cleans function exists
in the whole binary) and an undersized stack (our PE and the shipped one have
identical 1MB/8K geometry).
What the probe found matters more: the application is parked in state=2, which
is LoadingMission, not WaitingForLaunch. Priority 0 is where the interest
manager queues renderer events during load, the gate needs that priority empty,
and it never empties -- so the mission never launches and the renderer holds a
blank screen by design. The fault arrives ~30s into that wait. Runs that DO
launch never print a single [launch] line. So 'crashes half the time' and
'hangs during load' are one event seen twice.
A 15x disagreement between two clocks looked like a reconstruction slip --
BTL4.CPP passes GetTicksPerSecond() where ApplicationManager wants a frame
rate. Checked against the shipped binary before touching it: same instruction
sequence, same kind of static float pushed. Authentic. Documented so nobody
'fixes' it.
Also swept every subsystem DefaultData against its real C++ base. Fifteen
chain past their immediate parent, but fourteen skip only classes that add no
handlers and no attributes, and no class's attribute-ID base disagrees with its
index chain -- so there are no gap slots there. The one real defect: Generator
is a HeatSink but chained to Subsystem::MessageHandlers, so it ignored every
ToggleCooling message. Fixed (compiles next build).
Tooling: podrun.sh stages the build over BTL4REC.EXE, exports the host-side VPX
board env -- without which the run dies at the iserver handshake rather than
merely rendering nothing -- and archives every run's log, marking it -CRASH
when it faulted. Before this the only preserved dump was an accident, on a rig
where each run costs four minutes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CydandClaude Fable 5