#45 ROOT CAUSE + FIX: the gauge executive starves in MP -- panels froze at healthy fps

Instruments repaint via the background task pump, which authentically runs
ONLY in the frame's leftover time with a floor of ONE pump per frame; the
gauge renderer is 1 of ~7 round-robin tasks and each turn advances ONE gauge
of ~140 active, with the rate mask advancing once per full sweep.  On a busy
MP mission the foreground eats the whole frame budget, the floor becomes the
norm, and the whole instrument stack rotates once in MINUTES at perfect fps:
comms-panel K/D stuck at 0, recharge tickers frozen, while the tallies and
their replication underneath were exactly right.  Measured: 4-node bench at
70-90 fps gave the PilotList ~2 Execute turns in six minutes.

Fix, both env-tunable, authentic behavior restorable:
  BT_BG_MIN      (APPMGR.cpp, default 32, 0=authentic)  minimum background
                 pumps per frame regardless of slack;
  BT_GAUGE_BATCH (GAUGREND.cpp, default 32, 1=authentic) gauges advanced per
                 gauge-renderer turn (a visit is only a rate-mask check
                 unless the gauge is due).
Verified 4-node self-damage bench: sweeps 0.006/s -> 18-20/s, PilotList ~10
Exec/s, ALL FOUR panels tracked every death live (0->11) within ~1s, bg cost
2-4 ms/frame, respawn ledger clean (40 cycles, 0 swallow / 0 mismatch).

Also: BT_PERF now reports gaugeTurns/sweeps/active alongside bgTasks;
BT_AF_PERIOD now throttles the missile autofire group too (unthrottled spam
trips the documented FailureHeat all-weapons brick, which froze run 1 of the
cross-fire bench).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Joe DiPrima
2026-07-30 11:22:47 -05:00
co-authored by Claude Fable 5
parent bfa1b04990
commit 494f32efb4
5 changed files with 106 additions and 3 deletions
+17
View File
@@ -682,6 +682,23 @@ BitMap-strip lamp drew `SetColor(0)` + `DrawBitMapOpaque(0,…)` = invisible. Hi
Also corrected: @004c552c is OneOfSeveralStates' **Execute** override (clamp ≥0, chain base),
not `BecameActive` — vtable-diffed.
## The gauge EXECUTIVE — why panels froze in MP, and the fix (#45 root, 2026-07-30) [T0 engine / T2 verified]
Instruments repaint via the BACKGROUND task pump (`APPMGR.cpp RunMissions`): after sim+render, the
loop pumps `BackgroundTasks::Execute()` (ONE task per pump, ~7 tasks round-robin: net, events,
audio, GAUGES, …) **only until the frame deadline** — the gauge renderer's turn advances **one
gauge** of the active list (`ProcessOneActiveGauge`, ~140 active), and the 16-bit `GaugeRate`
mask advances once per FULL sweep (so "rate D = every 4th frame" is really every 4th SWEEP).
On a busy MP mission the foreground eats the whole frame budget → 1 pump/frame → a full
instrument rotation took MINUTES at perfectly healthy fps: comms-panel K/D stuck at 0, recharge
tickers frozen — while the underlying tallies (and their replication) were exactly right. This
is the real root of the field "panels don't update in MP" (#45's display half).
**Fix (both env-tunable):** `BT_BG_MIN` (APPMGR.cpp, default 32, 0=authentic) guarantees a
minimum pump floor per frame; `BT_GAUGE_BATCH` (GAUGREND.cpp, default 32, 1=authentic) advances
a batch of gauges per gauge turn (a visit is only a rate-mask check unless due). Verified
4-node: sweeps 0.006/s → **18-20/s**, PilotList ~10 Exec/s, all four panels tracked every death
within ~1s, bg cost 2-4 ms/frame. Diagnostics: `BT_PERF=1` → `[perf] … bgTasks/gaugeTurns/
sweeps/active` 1 Hz (APPMGR.cpp); `[score] panel DRAW` edge log (btl4gau3.cpp, BT_SCORE_LOG).
## Key Relationships
- Full history: `docs/GAUGE_COMPOSITE.md`; reticle recovery: `phases/phase-02-dpl2d-reticle.md`.
- Uses: [[attribute-pointer]] + [[reconstruction-gotchas]]; reads [[subsystems]] state.
+12
View File
@@ -761,6 +761,18 @@ register. ⚠ The audit also flags the damage-economy item as SELF-CONTRADICTOR
numbers no longer point at that code).
## Multiplayer (Phase 7 / P6)
- **Cross-fire bench opens (2026-07-30, first real-weapons 4-node runs).** (a) COMBAT STALL: all
firing and dying ceased ~3 min in (run 2, throttled `BT_AF_PERIOD=7`) with everyone alive,
subsystems healthy, and fps fine — engagement/heat state needs a logging pass (`BT_GOTO_LOG`
+ heat probes); run 1's max-spam stall was the documented FailureHeat all-weapons brick.
(b) NODE CRASH: one instance died silently mid-run (no SEH line, no WER record; last log line
= audio census; yesterday's WER archive holds 3× `OpenAL32.dll` abort `0x40000015` and WER
dedups repeats — suspect the audio-pool thread). procdump harness now wired into
`scratchpad/night6/mp4_cross.sh`; did NOT recur under throttled fire. (c) VERIFIED GOOD: 4
deaths ↔ 4 kills credited to the right shooters (incl. a mutual kill), `NOCREDIT`=0, cross-
machine damage applied on victims' masters with correct attribution; one kill credit arrived
17 s late (`SCORE type=2`) — the deferred ScoreMessage reroute path, watch it.
- **Gitea #12 MP incident (2026-07-19) — root causes found; BOTH FIXES LANDED 2026-07-19
(awaiting the human MP death-and-survive verification).** Findings [T1/T2 — see the #12
comments + `scratchpad/incident_2157/`]:
+38 -1
View File
@@ -193,6 +193,33 @@ Background_Loop:
current_application.First();
application = current_application.GetCurrent();
//
// BT MINIMUM BACKGROUND FLOOR (#45 root cause, 2026-07-30). Authentically
// the background tasks run ONLY in the frame's leftover time, with a floor
// of a single pump per frame -- fine on a 1995 pod (one app, designed
// slack), but on a busy MP mission the foreground eats the whole frame
// budget, the floor becomes the norm, and the GAUGE renderer (1 of ~7
// round-robin tasks, ONE gauge advanced per turn) starves: every cockpit
// instrument -- the comms-panel K/D columns, weapon recharge tickers --
// freezes at its last paint while the underlying values (and fps!) stay
// perfectly healthy. Measured: 4-node bench at 70-90 fps gave the
// PilotList ~2 Execute turns in SIX MINUTES. Guarantee a minimum number
// of pumps per frame so a full instrument rotation completes in ~1s
// regardless of slack; the added worst-case cost is bounded and small.
// BT_BG_MIN overrides (0 = authentic slack-only behavior).
//
{
static int s_bgMin = -1;
if (s_bgMin < 0)
{
const char *e = getenv("BT_BG_MIN");
s_bgMin = e ? atoi(e) : 32;
if (s_bgMin < 0) s_bgMin = 0;
if (s_bgMin > 256) s_bgMin = 256;
}
if (backgroundTasksRun < s_bgMin)
goto Background_Loop;
}
if (t2 < end_of_frame)
{
@@ -219,10 +246,20 @@ Background_Loop:
if (Now() - s_lastP >= 1.0f)
{
s_lastP = Now();
// gauge-executive rotation health (#45): turns = one-gauge
// Updates and sweeps = full active-list passes SINCE THE LAST
// PRINT (~1s); active = the instrument roster size. A frozen
// panel shows here as sweeps=0.
extern int gBTGaugeTurns, gBTGaugeSweeps, gBTGaugeActive;
DEBUG_STREAM << "[perf] frame=" << (float)(Now() - beginFrameTimestamp)
<< " fg=" << (float)(startBackground - beginFrameTimestamp)
<< " bg=" << (float)(endBackground - startBackground)
<< " bgTasks=" << backgroundTasksRun << "\n" << std::flush;
<< " bgTasks=" << backgroundTasksRun
<< " gaugeTurns=" << gBTGaugeTurns
<< " sweeps=" << gBTGaugeSweeps
<< " active=" << gBTGaugeActive << "\n" << std::flush;
gBTGaugeTurns = 0;
gBTGaugeSweeps = 0;
}
}
}
+34 -1
View File
@@ -3819,7 +3819,28 @@ Logical
case background:
{
result = ProcessOneActiveGauge();
// BT GAUGE BATCH (#45, with the APPMGR minimum-pump floor): one gauge
// per turn is the authentic pod pacing, but a visit is just a
// rate-mask check unless the gauge is due, so on modern hardware it
// starves the rotation for no benefit -- with ~140 active instruments
// and ~4 gauge turns/frame a full sweep took ~1s and a D-rate gauge
// (the comms-panel PilotList) ran every ~4s. Advance a BATCH of
// gauges per turn so a sweep completes ~every frame and panel rows
// track the sim within a second. BT_GAUGE_BATCH=1 restores the
// authentic single-step.
static int s_batch = -1;
if (s_batch < 0)
{
const char *e = getenv("BT_GAUGE_BATCH");
s_batch = e ? atoi(e) : 32;
if (s_batch < 1) s_batch = 1;
if (s_batch > 1024) s_batch = 1024;
}
int n = s_batch;
do
{
result = ProcessOneActiveGauge();
} while (--n > 0 && result && taskMode == background);
break;
}
@@ -3963,6 +3984,10 @@ void
Check_Fpu();
}
// BT gauge-executive counters (#45) -- definitions; sampled + reset by the
// BT_PERF probe in APPMGR.cpp.
int gBTGaugeTurns = 0, gBTGaugeSweeps = 0, gBTGaugeActive = 0;
//
//===========================================================================
// ProcessOneActiveGauge
@@ -3979,8 +4004,15 @@ Logical
GaugeBase
*base_pointer = activeIterator->ReadAndNext();
// BT gauge-executive counters (#45): read by the BT_PERF probe (APPMGR.cpp)
// to expose the instrument-rotation rate -- turns = single-gauge Updates,
// sweeps = full active-list passes (the rate mask advances once per sweep,
// so a "D-rate" gauge executes once per FOUR sweeps).
extern int gBTGaugeTurns, gBTGaugeSweeps, gBTGaugeActive;
if (base_pointer == NULL)
{
++gBTGaugeSweeps;
gBTGaugeActive = activeIterator->GetSize();
//--------------------------------------------------
// We're done!
// Bump the bit mask and index for the next pass
@@ -3997,6 +4029,7 @@ Logical
}
else
{
++gBTGaugeTurns;
//--------------------------------------------------
// Process one gauge
//--------------------------------------------------
+5 -1
View File
@@ -5584,7 +5584,11 @@ void
// exercise MISSILES ONLY -- isolates missile damage/splash from laser+PPC
// fire (task #62 TTK verification).
static const int s_afMissile = getenv("BT_AF_MISSILE") ? 1 : 0;
const int autofireMsl = (s_afMissile && targetInArc) ? s_afPhase : 0;
// afGate: BT_AF_PERIOD must throttle the missile group too -- unthrottled
// missile spam trips the documented FailureHeat permanent brick (open-
// questions "Missile launcher trips FailureHeat"), which froze every
// weapon + recharge gauge in the first 4-node cross-fire bench.
const int autofireMsl = (s_afMissile && targetInArc && afGate) ? s_afPhase : 0;
int want[4];
want[0] = ((gBTLaserKey && targetInArc) || autofire) ? 1 : 0;
want[1] = (gBTPinkyKey && targetInArc) ? 1 : 0; // key '4' (was unmapped)