From 494f32efb4ab804272ca55646747b6259d3b769d Mon Sep 17 00:00:00 2001 From: Joe DiPrima Date: Thu, 30 Jul 2026 11:22:47 -0500 Subject: [PATCH] #45 ROOT CAUSE + FIX: the gauge executive starves in MP -- panels froze at healthy fps Instruments repaint via the background task pump, which authentically runs ONLY in the frame's leftover time with a floor of ONE pump per frame; the gauge renderer is 1 of ~7 round-robin tasks and each turn advances ONE gauge of ~140 active, with the rate mask advancing once per full sweep. On a busy MP mission the foreground eats the whole frame budget, the floor becomes the norm, and the whole instrument stack rotates once in MINUTES at perfect fps: comms-panel K/D stuck at 0, recharge tickers frozen, while the tallies and their replication underneath were exactly right. Measured: 4-node bench at 70-90 fps gave the PilotList ~2 Execute turns in six minutes. Fix, both env-tunable, authentic behavior restorable: BT_BG_MIN (APPMGR.cpp, default 32, 0=authentic) minimum background pumps per frame regardless of slack; BT_GAUGE_BATCH (GAUGREND.cpp, default 32, 1=authentic) gauges advanced per gauge-renderer turn (a visit is only a rate-mask check unless the gauge is due). Verified 4-node self-damage bench: sweeps 0.006/s -> 18-20/s, PilotList ~10 Exec/s, ALL FOUR panels tracked every death live (0->11) within ~1s, bg cost 2-4 ms/frame, respawn ledger clean (40 cycles, 0 swallow / 0 mismatch). Also: BT_PERF now reports gaugeTurns/sweeps/active alongside bgTasks; BT_AF_PERIOD now throttles the missile autofire group too (unthrottled spam trips the documented FailureHeat all-weapons brick, which froze run 1 of the cross-fire bench). Co-Authored-By: Claude Fable 5 --- context/gauges-hud.md | 17 ++++++++++++++++ context/open-questions.md | 12 +++++++++++ engine/MUNGA/APPMGR.cpp | 39 +++++++++++++++++++++++++++++++++++- engine/MUNGA/GAUGREND.cpp | 35 +++++++++++++++++++++++++++++++- game/reconstructed/mech4.cpp | 6 +++++- 5 files changed, 106 insertions(+), 3 deletions(-) diff --git a/context/gauges-hud.md b/context/gauges-hud.md index 7c9370e..3ecb848 100644 --- a/context/gauges-hud.md +++ b/context/gauges-hud.md @@ -682,6 +682,23 @@ BitMap-strip lamp drew `SetColor(0)` + `DrawBitMapOpaque(0,…)` = invisible. Hi Also corrected: @004c552c is OneOfSeveralStates' **Execute** override (clamp ≥0, chain base), not `BecameActive` — vtable-diffed. +## The gauge EXECUTIVE — why panels froze in MP, and the fix (#45 root, 2026-07-30) [T0 engine / T2 verified] +Instruments repaint via the BACKGROUND task pump (`APPMGR.cpp RunMissions`): after sim+render, the +loop pumps `BackgroundTasks::Execute()` (ONE task per pump, ~7 tasks round-robin: net, events, +audio, GAUGES, …) **only until the frame deadline** — the gauge renderer's turn advances **one +gauge** of the active list (`ProcessOneActiveGauge`, ~140 active), and the 16-bit `GaugeRate` +mask advances once per FULL sweep (so "rate D = every 4th frame" is really every 4th SWEEP). +On a busy MP mission the foreground eats the whole frame budget → 1 pump/frame → a full +instrument rotation took MINUTES at perfectly healthy fps: comms-panel K/D stuck at 0, recharge +tickers frozen — while the underlying tallies (and their replication) were exactly right. This +is the real root of the field "panels don't update in MP" (#45's display half). +**Fix (both env-tunable):** `BT_BG_MIN` (APPMGR.cpp, default 32, 0=authentic) guarantees a +minimum pump floor per frame; `BT_GAUGE_BATCH` (GAUGREND.cpp, default 32, 1=authentic) advances +a batch of gauges per gauge turn (a visit is only a rate-mask check unless due). Verified +4-node: sweeps 0.006/s → **18-20/s**, PilotList ~10 Exec/s, all four panels tracked every death +within ~1s, bg cost 2-4 ms/frame. Diagnostics: `BT_PERF=1` → `[perf] … bgTasks/gaugeTurns/ +sweeps/active` 1 Hz (APPMGR.cpp); `[score] panel DRAW` edge log (btl4gau3.cpp, BT_SCORE_LOG). + ## Key Relationships - Full history: `docs/GAUGE_COMPOSITE.md`; reticle recovery: `phases/phase-02-dpl2d-reticle.md`. - Uses: [[attribute-pointer]] + [[reconstruction-gotchas]]; reads [[subsystems]] state. diff --git a/context/open-questions.md b/context/open-questions.md index cbd1dbc..75301ca 100644 --- a/context/open-questions.md +++ b/context/open-questions.md @@ -761,6 +761,18 @@ register. ⚠ The audit also flags the damage-economy item as SELF-CONTRADICTOR numbers no longer point at that code). ## Multiplayer (Phase 7 / P6) +- **Cross-fire bench opens (2026-07-30, first real-weapons 4-node runs).** (a) COMBAT STALL: all + firing and dying ceased ~3 min in (run 2, throttled `BT_AF_PERIOD=7`) with everyone alive, + subsystems healthy, and fps fine — engagement/heat state needs a logging pass (`BT_GOTO_LOG` + + heat probes); run 1's max-spam stall was the documented FailureHeat all-weapons brick. + (b) NODE CRASH: one instance died silently mid-run (no SEH line, no WER record; last log line + = audio census; yesterday's WER archive holds 3× `OpenAL32.dll` abort `0x40000015` and WER + dedups repeats — suspect the audio-pool thread). procdump harness now wired into + `scratchpad/night6/mp4_cross.sh`; did NOT recur under throttled fire. (c) VERIFIED GOOD: 4 + deaths ↔ 4 kills credited to the right shooters (incl. a mutual kill), `NOCREDIT`=0, cross- + machine damage applied on victims' masters with correct attribution; one kill credit arrived + 17 s late (`SCORE type=2`) — the deferred ScoreMessage reroute path, watch it. + - **Gitea #12 MP incident (2026-07-19) — root causes found; BOTH FIXES LANDED 2026-07-19 (awaiting the human MP death-and-survive verification).** Findings [T1/T2 — see the #12 comments + `scratchpad/incident_2157/`]: diff --git a/engine/MUNGA/APPMGR.cpp b/engine/MUNGA/APPMGR.cpp index f8dfbc4..253bf3d 100644 --- a/engine/MUNGA/APPMGR.cpp +++ b/engine/MUNGA/APPMGR.cpp @@ -193,6 +193,33 @@ Background_Loop: current_application.First(); application = current_application.GetCurrent(); + // + // BT MINIMUM BACKGROUND FLOOR (#45 root cause, 2026-07-30). Authentically + // the background tasks run ONLY in the frame's leftover time, with a floor + // of a single pump per frame -- fine on a 1995 pod (one app, designed + // slack), but on a busy MP mission the foreground eats the whole frame + // budget, the floor becomes the norm, and the GAUGE renderer (1 of ~7 + // round-robin tasks, ONE gauge advanced per turn) starves: every cockpit + // instrument -- the comms-panel K/D columns, weapon recharge tickers -- + // freezes at its last paint while the underlying values (and fps!) stay + // perfectly healthy. Measured: 4-node bench at 70-90 fps gave the + // PilotList ~2 Execute turns in SIX MINUTES. Guarantee a minimum number + // of pumps per frame so a full instrument rotation completes in ~1s + // regardless of slack; the added worst-case cost is bounded and small. + // BT_BG_MIN overrides (0 = authentic slack-only behavior). + // + { + static int s_bgMin = -1; + if (s_bgMin < 0) + { + const char *e = getenv("BT_BG_MIN"); + s_bgMin = e ? atoi(e) : 32; + if (s_bgMin < 0) s_bgMin = 0; + if (s_bgMin > 256) s_bgMin = 256; + } + if (backgroundTasksRun < s_bgMin) + goto Background_Loop; + } if (t2 < end_of_frame) { @@ -219,10 +246,20 @@ Background_Loop: if (Now() - s_lastP >= 1.0f) { s_lastP = Now(); + // gauge-executive rotation health (#45): turns = one-gauge + // Updates and sweeps = full active-list passes SINCE THE LAST + // PRINT (~1s); active = the instrument roster size. A frozen + // panel shows here as sweeps=0. + extern int gBTGaugeTurns, gBTGaugeSweeps, gBTGaugeActive; DEBUG_STREAM << "[perf] frame=" << (float)(Now() - beginFrameTimestamp) << " fg=" << (float)(startBackground - beginFrameTimestamp) << " bg=" << (float)(endBackground - startBackground) - << " bgTasks=" << backgroundTasksRun << "\n" << std::flush; + << " bgTasks=" << backgroundTasksRun + << " gaugeTurns=" << gBTGaugeTurns + << " sweeps=" << gBTGaugeSweeps + << " active=" << gBTGaugeActive << "\n" << std::flush; + gBTGaugeTurns = 0; + gBTGaugeSweeps = 0; } } } diff --git a/engine/MUNGA/GAUGREND.cpp b/engine/MUNGA/GAUGREND.cpp index 4548350..56eaf30 100644 --- a/engine/MUNGA/GAUGREND.cpp +++ b/engine/MUNGA/GAUGREND.cpp @@ -3819,7 +3819,28 @@ Logical case background: { - result = ProcessOneActiveGauge(); + // BT GAUGE BATCH (#45, with the APPMGR minimum-pump floor): one gauge + // per turn is the authentic pod pacing, but a visit is just a + // rate-mask check unless the gauge is due, so on modern hardware it + // starves the rotation for no benefit -- with ~140 active instruments + // and ~4 gauge turns/frame a full sweep took ~1s and a D-rate gauge + // (the comms-panel PilotList) ran every ~4s. Advance a BATCH of + // gauges per turn so a sweep completes ~every frame and panel rows + // track the sim within a second. BT_GAUGE_BATCH=1 restores the + // authentic single-step. + static int s_batch = -1; + if (s_batch < 0) + { + const char *e = getenv("BT_GAUGE_BATCH"); + s_batch = e ? atoi(e) : 32; + if (s_batch < 1) s_batch = 1; + if (s_batch > 1024) s_batch = 1024; + } + int n = s_batch; + do + { + result = ProcessOneActiveGauge(); + } while (--n > 0 && result && taskMode == background); break; } @@ -3963,6 +3984,10 @@ void Check_Fpu(); } +// BT gauge-executive counters (#45) -- definitions; sampled + reset by the +// BT_PERF probe in APPMGR.cpp. +int gBTGaugeTurns = 0, gBTGaugeSweeps = 0, gBTGaugeActive = 0; + // //=========================================================================== // ProcessOneActiveGauge @@ -3979,8 +4004,15 @@ Logical GaugeBase *base_pointer = activeIterator->ReadAndNext(); + // BT gauge-executive counters (#45): read by the BT_PERF probe (APPMGR.cpp) + // to expose the instrument-rotation rate -- turns = single-gauge Updates, + // sweeps = full active-list passes (the rate mask advances once per sweep, + // so a "D-rate" gauge executes once per FOUR sweeps). + extern int gBTGaugeTurns, gBTGaugeSweeps, gBTGaugeActive; if (base_pointer == NULL) { + ++gBTGaugeSweeps; + gBTGaugeActive = activeIterator->GetSize(); //-------------------------------------------------- // We're done! // Bump the bit mask and index for the next pass @@ -3997,6 +4029,7 @@ Logical } else { + ++gBTGaugeTurns; //-------------------------------------------------- // Process one gauge //-------------------------------------------------- diff --git a/game/reconstructed/mech4.cpp b/game/reconstructed/mech4.cpp index dd39268..7e9dff3 100644 --- a/game/reconstructed/mech4.cpp +++ b/game/reconstructed/mech4.cpp @@ -5584,7 +5584,11 @@ void // exercise MISSILES ONLY -- isolates missile damage/splash from laser+PPC // fire (task #62 TTK verification). static const int s_afMissile = getenv("BT_AF_MISSILE") ? 1 : 0; - const int autofireMsl = (s_afMissile && targetInArc) ? s_afPhase : 0; + // afGate: BT_AF_PERIOD must throttle the missile group too -- unthrottled + // missile spam trips the documented FailureHeat permanent brick (open- + // questions "Missile launcher trips FailureHeat"), which froze every + // weapon + recharge gauge in the first 4-node cross-fire bench. + const int autofireMsl = (s_afMissile && targetInArc && afGate) ? s_afPhase : 0; int want[4]; want[0] = ((gBTLaserKey && targetInArc) || autofire) ? 1 : 0; want[1] = (gBTPinkyKey && targetInArc) ? 1 : 0; // key '4' (was unmapped)