9c959e68919b49f75f736be1d48ec9ab770d94b9
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  CydandClaude Fable 5
|
9c959e6891 |
BT410 5.3.60: first rendered frame from a running reconstructed mission
emulator/render-bridge/first_mission_frame.png -- arena city, textured buildings, horizon, 28fps, captured from the GL bridge while our build ran a live mission with the mech walking. The skeleton went in with it: [skl] video\mad.skl -> 26 nodes, 19 objects. Corrects yesterday's RIO framing. I called serial1 an unplugged cable. Wrong twice: VRio.App is running on this rig (the dev-rig default since 7/17), and the emulator log recorded real byte counts on that port -- RX overruns of 471, 503, 528 -- which an unconnected pipe cannot produce. The port was LIVE and streaming during every crashing run. So the defect is not that we mishandle a disconnected port, it is that we mishandle a live RIO stream, which is worse: every production pod has a real RIO. The shipped binary on the same rig logs 'lost RIO analog request', keeps sending characters, and runs the mission; ours logs 'RIO never came back from test mode!' and dies partway through the load. The overrun counts say the guest is not draining the port fast enough. L4RIO.CPP's handshake is authentic archive code, but MechRIOMapper (BT/MECHMPPR.CPP) is ours -- that is the thread. pod_render_norio.conf stays the way to get a running mission today, but it is a workaround: it disables the cockpit's primary input device. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
CydandClaude Fable 5
|
475ae3106e |
BT410 5.3.59: A MISSION RUNS -- the RIO serial port was blocking every load
pod_render_norio.conf is pod_render_rec.conf with one line changed, serial1=disabled instead of the vrio named pipe. Same binary, same egg. With it, our build launches and the mech walks: [launch] state=2 minPriorityEmpty=1 ticks=85316 [queues] p0=- p1=- p2=BUSY p3=- p4=- BTL4Application::RunMissionMessageHandler Turning Plasma Score Display On [sim] pos=(180.466,10,-358.703) yaw=-0.275605 spd=14.4 [sim] pos=(251.959,10,-250.498) yaw=-0.89103 spd=14.4 No fault, 290 log lines and counting. This is the first time the reconstruction has reached a running mission on the pod. The hang was never starvation, a deadlock, or a refill loop -- it was VOLUME, the first candidate I listed and then talked myself out of. 530 renderer events queue at priority 0 during load; BackgroundTasks::Execute runs exactly one task per call round-robin over seven tasks, and the 1ms frame budget is always blown here so no extra background passes happen. That is ~9 events per real second, so a load legitimately takes ~60s. The fault arrived at ~40s, before the backlog could clear. Every 'the queue never drains' reading was really 'the process dies before it can'. The disproof was already in hand: the post tally climbed to 530 and went FLAT, which means nothing was refilling. p0=BUSY with nextReady=1 is equally consistent with a large finite backlog still draining, and I read it as refill. Why the RIO port: the pipe has no server attached, which should behave as an unplugged cable. The emulator log shows steady serial1 RX overruns, our build prints 'RIO never came back from test mode!', and the SHIPPED binary prints 'lost RIO analog request' and launches anyway on this identical rig. So an unplugged RIO is survivable and our handling of it is not -- a real defect, not a rig artifact, since a pod with an unplugged RIO cable should still boot. The wild reference at EulerAngles::operator=(const LinearMatrix&)+0x19 is still unexplained, but it is now reproducible on demand by enabling serial1 rather than being a coin flip. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
CydandClaude Fable 5
|
d3b332a62a |
BT410 5.3.58: the load gate is blocked by a REFILL loop on priority 0, not starvation
Per-priority occupancy, measured every second until the fault: [queues] p0=BUSY p1=- p2=BUSY p3=- p4=- nextReady=1 That single line kills the two leading theories. p3/p4 empty means nothing above priority 0 is competing, so the pump is free to serve it -- no starvation. nextReady=1 means PeekAtNextEvent always has a READY event, so priority 0 is not holding a timed event whose alarm never comes due -- no deadlock. What remains is refill: priority 0 is replenished as fast as the pump drains it, ~143 events per simulated second. It is also fully deterministic. Two runs of the same binary reported pump=352/1499 at frame 1001 and 495/2643 at frame 2002, identical to the byte. So 'crashes about half the time' was never a race -- it was comparing runs that differed in binary or conf. Only InterestManager::PostRendererEvent posts at priority 0 (it maps every renderer event there while the app is not RunningMission), so naming the message names the flooder. Application::Post now tallies priority-0 posts by message ID and reports the busiest three. Notably, all three producers of NotifyOfNewInterestingEntity are entity-CREATION paths, so if that ID dominates then something is creating entities in a loop -- which would explain the unbounded growth behind the fault as well as the hang. pod_render_noskl.conf now carries the same probes, so one instrumented binary can be run with the skeleton walk on and off. Walk-off runs are the only configuration of ours known to reach 'Turning Plasma Score Display On'. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
CydandClaude Fable 5
|
4cf09917ce |
BT410 5.3.57: the crash is deterministic, and the real blocker is the load gate
The previous commit's attribution was wrong and is corrected in the roadmap. WRONG: 'the crash is the skeleton walk's' rested on 0 crashes in 2 runs with the walk disabled -- a 25% coin flip presented as evidence. The oldest preserved dump has no [skl] line and ends on the old 'couldn't figure out how to MakeEntityRenderables' fallback, so it predates the walk and faults identically. WRONG: 'it lands at different points each time'. Every dump carries the same numbers to the byte (cr2 7000FA64, EIP 66D9). What varies is how far the log gets, not where the fault is. Resolving the address through btl4opt.map names it exactly: EulerAngles::operator=(const LinearMatrix&)+0x19, a read through the matrix reference. A binary scan finds all three call sites of that operator pass a stack local, so the 1.79GB pointer still has no static explanation -- that is now its own open item rather than a guess. Two theories killed cleanly by the new BT_STACK_LOG probe and a binary scan: ESP drift (drift=0 over 2002 frames; exactly one callee-cleans function exists in the whole binary) and an undersized stack (our PE and the shipped one have identical 1MB/8K geometry). What the probe found matters more: the application is parked in state=2, which is LoadingMission, not WaitingForLaunch. Priority 0 is where the interest manager queues renderer events during load, the gate needs that priority empty, and it never empties -- so the mission never launches and the renderer holds a blank screen by design. The fault arrives ~30s into that wait. Runs that DO launch never print a single [launch] line. So 'crashes half the time' and 'hangs during load' are one event seen twice. A 15x disagreement between two clocks looked like a reconstruction slip -- BTL4.CPP passes GetTicksPerSecond() where ApplicationManager wants a frame rate. Checked against the shipped binary before touching it: same instruction sequence, same kind of static float pushed. Authentic. Documented so nobody 'fixes' it. Also swept every subsystem DefaultData against its real C++ base. Fifteen chain past their immediate parent, but fourteen skip only classes that add no handlers and no attributes, and no class's attribute-ID base disagrees with its index chain -- so there are no gap slots there. The one real defect: Generator is a HeatSink but chained to Subsystem::MessageHandlers, so it ignored every ToggleCooling message. Fixed (compiles next build). Tooling: podrun.sh stages the build over BTL4REC.EXE, exports the host-side VPX board env -- without which the run dies at the iserver handshake rather than merely rendering nothing -- and archives every run's log, marking it -CRASH when it faulted. Before this the only preserved dump was an accident, on a rig where each run costs four minutes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |