From bf0555ccf1c61de1be4b5f28c397cdd3141efbf0 Mon Sep 17 00:00:00 2001 From: Joe DiPrima Date: Thu, 30 Jul 2026 01:20:13 -0500 Subject: [PATCH] #81: every MP respawn is a cross-machine round trip -- one degraded peer ghosts everyone Two-node bench with the fixed host:local labels shows each player's drop-zone request is answered by the OTHER machine's DropZone: A asks, B grants; B asks, A grants. FindGroup("DropZones") iterates replicants of remotely-mastered zones too and takes the geometrically closest, so a respawn is my request -> an arbitrary peer's DropZone -> that peer's reply -> back to me. With 5 players that is 5 round trips through arbitrary peers, and ONE degraded peer can strand everybody else's respawn. That finally explains the otherwise unexplained field datum that one machine stopped processing the death-transition stream 57% into a match and never recovered (0 explosions/wreck swaps/burials while four other machines logged 4/4/4) -- a node in that state cannot answer anyone's respawn. It also explains why solo is 100% reliable (in-process) and why a healthy 2-node bench passes. Also fixes the instrumentation before it costs a night: every [dz] line printed "entity 1" because a player's LOCAL entity id is 1 on every machine -- with five players the log would have said a respawn stalled but not WHOSE. All [dz] lines now print host:local (including the usedBy= owner of each busy slot). Doc: two candidate fixes recorded (prefer a locally-mastered DropZone / make the reply path tolerant of a deathCount that is ahead of ours), neither to be guessed at -- the [ghost] DISCARDED line's mismatch direction decides it in one line. Co-Authored-By: Claude Fable 5 --- docs/GHOST_MECH_ANALYSIS.md | 38 +++++++++++++++++++++++++++++++---- engine/MUNGA/DROPZONE.cpp | 28 +++++++++++++++++++------- scratchpad/night6/mp_ghost.sh | 31 ++++++++++++++++++++++++++++ 3 files changed, 86 insertions(+), 11 deletions(-) create mode 100644 scratchpad/night6/mp_ghost.sh diff --git a/docs/GHOST_MECH_ANALYSIS.md b/docs/GHOST_MECH_ANALYSIS.md index 1e07497..fab04bf 100644 --- a/docs/GHOST_MECH_ANALYSIS.md +++ b/docs/GHOST_MECH_ANALYSIS.md @@ -96,10 +96,40 @@ is the **Simulation-base alarm** — the same field the mech side calls `graphic nowhere to put it; the stub is harmless until we recover what reads a *Player's* alarm in the binary (likely a cockpit/HUD respawn indicator). [T1 offsets, T3 purpose] -**Revised theory of the ghost:** not drop-zone starvation (see below) and not a lost reply (see 1), -but a **`deathCount` mismatch between the player and the reply**, silently discarding a live -respawn — MP-specific in a way solo structurally cannot reproduce (solo respawn works every time; -6 of 8 MP cycles stranded), and directly coupled to the known #45 replication defect. +**4. EVERY MP RESPAWN IS A CROSS-MACHINE ROUND TRIP — and that is the single point of failure** +[T2, two-node bench `scratchpad/night6/mp_ghost.sh`]. Measured on a 2-node rig with the host:local +labels: each player's request was answered by **the OTHER machine's** DropZone. +``` +mp_a.log: [dzreq] player 2:1 asking for a drop zone ... <- A's own player asks +mp_b.log: [dz] GRANTED slot 7 to 2:1 death#1 ... <- B answered it +mp_b.log: [dzreq] player 3:1 asking for a drop zone ... <- B's own player asks +mp_a.log: [dz] GRANTED slot 0 to 3:1 death#1 ... <- A answered it +``` +`FindGroup("DropZones")` iterates **every** DropZone entity including replicants of remotely-mastered +ones and picks the geometrically closest, so a respawn is: *my request → an arbitrary peer's +DropZone → that peer's reply → back over the wire to me.* With five players that is five round trips +through arbitrary peers, and **one degraded peer can ghost everybody else.** +That dovetails exactly with the otherwise-unexplained field finding that **one machine stopped +processing the effect/death-transition stream 57 % into match 1 and never recovered** (0 explosions / +0 wreck swaps / 0 burials while the other four logged 4/4/4). A node in that state that owns the +closest drop zone cannot answer anyone's respawn — which is what "everybody ghosted" looks like. +It also explains why solo is 100 % reliable (the whole handshake is in-process) and why the +2-node bench passes (both nodes healthy, replies land, `deathCount` agrees). + +**Revised theory of the ghost (two layers):** +- *Structural:* respawn depends on a cross-machine round trip through a peer that may be lagging, + stalled, or holding a stale view of my player. Not drop-zone starvation (`slots=8`) and not a + single lost reply (the hunt retries every 2 s forever). +- *Mechanism when the reply does arrive but is stale/mismatched:* the silent discard in 2 — the + reply's `deathCount` disagrees and it is dropped without a trace, which couples this to #45. + +**Two candidate fixes, both needing the next field logs to choose between:** +(a) **Prefer a locally-mastered DropZone** when one exists, keeping the handshake in-process and +removing the remote dependency entirely — cheap and robust, but it changes which physical pad you +land on, so check it against the maps first. (b) **Make the reply path tolerant**: accept a granted +reply whose `deathCount` is *ahead* of ours (our counter is behind) instead of discarding it, and +never leave `deathPending` latched on a discard. ⚠ Neither should be guessed at blind — the +`[ghost] DROP-ZONE REPLY DISCARDED` line's mismatch direction decides it in one line of log. ## What the instrumentation already killed `slots=8` (measured, `[dz] POOL`). An 8-slot pool **cannot** be cooldown-starved by 5 players (at diff --git a/engine/MUNGA/DROPZONE.cpp b/engine/MUNGA/DROPZONE.cpp index 1120dc7..6561ac7 100644 --- a/engine/MUNGA/DROPZONE.cpp +++ b/engine/MUNGA/DROPZONE.cpp @@ -137,6 +137,13 @@ DropZone::~DropZone() } } + +// #81: print an EntityID as host:local. A player's LOCAL id is 1 on every +// machine, so printing only the local part made every one of five players read +// "entity 1" -- useless for telling WHO stalled. GetHostID() is non-const. +static inline int DZHost(const EntityID &id) { EntityID c(id); return (int)c.GetHostID(); } +static inline int DZLocal(const EntityID &id) { return (int)id; } + //############################################################################# // Dropzone assignment // @@ -223,7 +230,9 @@ void if (dzLog) { - DEBUG_STREAM << "[dz] REQUEST from entity " << (int)message->requestingEntity + DEBUG_STREAM << "[dz] REQUEST from " + << DZHost(message->requestingEntity) << ":" + << DZLocal(message->requestingEntity) << " deathCount=" << message->deathCount << " waited=" << waited << "s slots["; for (int d = 0; d < dropZoneCount; ++d) @@ -383,7 +392,8 @@ void { s_waitLastLog[wslot] = Now(); DEBUG_STREAM << (waited >= 15.0f ? "[dz] GHOST LIKELY -- " : "[dz] STALL -- ") - << "entity " << (int)message->requestingEntity + << DZHost(message->requestingEntity) << ":" + << DZLocal(message->requestingEntity) << " death#" << message->deathCount << " has waited " << waited << "s for a slot; all " << dropZoneCount << " busy ["; @@ -391,7 +401,8 @@ void { Scalar age = lastUsageTime[d].ticks ? (Scalar)(Now() - lastUsageTime[d]) : -1.0f; DEBUG_STREAM << (d ? " " : "") << d << ":usedBy=" - << (int)lastUsedBy[d] << ",age=" << age << "s"; + << DZHost(lastUsedBy[d]) << ":" << DZLocal(lastUsedBy[d]) + << ",age=" << age << "s"; } DEBUG_STREAM << "]\n" << std::flush; } @@ -399,8 +410,9 @@ void { static int s_rp = 0; if (++s_rp <= 5) - DEBUG_STREAM << "[dz] repost #" << s_rp << " (entity " - << (int)message->requestingEntity << ")\n" << std::flush; + DEBUG_STREAM << "[dz] repost #" << s_rp << " (" + << DZHost(message->requestingEntity) << ":" + << DZLocal(message->requestingEntity) << ")\n" << std::flush; } Time when=Now(); when += 0.1f; @@ -423,8 +435,10 @@ Found_One: int slot = (int)(drop_zone - dropZones); if (waited > 0.5f) ++s_stalledGrants; ++s_grants; - DEBUG_STREAM << "[dz] GRANTED slot " << slot << " to entity " - << (int)message->requestingEntity << " death#" << message->deathCount + DEBUG_STREAM << "[dz] GRANTED slot " << slot << " to " + << DZHost(message->requestingEntity) << ":" + << DZLocal(message->requestingEntity) + << " death#" << message->deathCount << " waited=" << waited << "s" << (waited >= 15.0f ? " (GHOST RECOVERED)" : "") << " [grants=" << s_grants << " stalled=" << s_stalledGrants << "]" diff --git a/scratchpad/night6/mp_ghost.sh b/scratchpad/night6/mp_ghost.sh new file mode 100644 index 0000000..e60a1ad --- /dev/null +++ b/scratchpad/night6/mp_ghost.sh @@ -0,0 +1,31 @@ +#!/usr/bin/env bash +# #81 GHOST REPRO: two nodes, BOTH die, watch the respawn handshake. +# +# Looking for, on either node: +# [dzreq] ... msgDeath= ourDeathCount= (the request side) +# [dz] GRANTED / STALL (the drop zone's answer) +# [ghost] DROP-ZONE REPLY DISCARDED (the silent-discard path) +# [respawn] ... RESET at drop zone (success) +# A node with a death cycle START and no RESET has reproduced the field ghost. +set -x +cd /c/git/bt411/content || exit 1 +rm -f mp_a.log mp_b.log +sed 's/^experience=expert/experience=novice/' MP.EGG > MPG.EGG + +# B first (listener), then A -- same order as the working mp_skate.sh +BT_LOG=mp_b.log BT_AFFINITY=0x2 BT_MP_LOG=1 BT_SCORE_LOG=1 BT_DEATH_LOG=1 \ + BT_SELF_DAMAGE=60 BT_SELF_DAMAGE_DELAY=40 \ + ../build/Release/btl4.exe -egg MPG.EGG -net 1601 & +sleep 2 +BT_LOG=mp_a.log BT_AFFINITY=0x1 BT_MP_LOG=1 BT_SCORE_LOG=1 BT_DEATH_LOG=1 \ + BT_SELF_DAMAGE=60 BT_SELF_DAMAGE_DELAY=55 \ + ../build/Release/btl4.exe -egg MPG.EGG -net 1501 & +sleep 4 +python ../tools/btconsole.py MPG.EGG 127.0.0.1:1501 127.0.0.1:1601 & +PC=$! + +sleep 175 +kill $PC 2>/dev/null +taskkill //F //IM btl4.exe 2>/dev/null +rm -f MPG.EGG +echo "=== DONE ==="