#81: every MP respawn is a cross-machine round trip -- one degraded peer ghosts everyone
Two-node bench with the fixed host:local labels shows each player's drop-zone
request is answered by the OTHER machine's DropZone: A asks, B grants; B asks, A
grants. FindGroup("DropZones") iterates replicants of remotely-mastered zones
too and takes the geometrically closest, so a respawn is
my request -> an arbitrary peer's DropZone -> that peer's reply -> back to me.
With 5 players that is 5 round trips through arbitrary peers, and ONE degraded
peer can strand everybody else's respawn. That finally explains the otherwise
unexplained field datum that one machine stopped processing the death-transition
stream 57% into a match and never recovered (0 explosions/wreck swaps/burials
while four other machines logged 4/4/4) -- a node in that state cannot answer
anyone's respawn. It also explains why solo is 100% reliable (in-process) and
why a healthy 2-node bench passes.
Also fixes the instrumentation before it costs a night: every [dz] line printed
"entity 1" because a player's LOCAL entity id is 1 on every machine -- with five
players the log would have said a respawn stalled but not WHOSE. All [dz] lines
now print host:local (including the usedBy= owner of each busy slot).
Doc: two candidate fixes recorded (prefer a locally-mastered DropZone / make the
reply path tolerant of a deathCount that is ahead of ours), neither to be guessed
at -- the [ghost] DISCARDED line's mismatch direction decides it in one line.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Joe DiPrima
co-authored by
Claude Fable 5
Claude Fable 5
parent
0dbdad17f3
commit
bf0555ccf1
@@ -96,10 +96,40 @@ is the **Simulation-base alarm** — the same field the mech side calls `graphic
|
||||
nowhere to put it; the stub is harmless until we recover what reads a *Player's* alarm in the binary
|
||||
(likely a cockpit/HUD respawn indicator). [T1 offsets, T3 purpose]
|
||||
|
||||
**Revised theory of the ghost:** not drop-zone starvation (see below) and not a lost reply (see 1),
|
||||
but a **`deathCount` mismatch between the player and the reply**, silently discarding a live
|
||||
respawn — MP-specific in a way solo structurally cannot reproduce (solo respawn works every time;
|
||||
6 of 8 MP cycles stranded), and directly coupled to the known #45 replication defect.
|
||||
**4. EVERY MP RESPAWN IS A CROSS-MACHINE ROUND TRIP — and that is the single point of failure**
|
||||
[T2, two-node bench `scratchpad/night6/mp_ghost.sh`]. Measured on a 2-node rig with the host:local
|
||||
labels: each player's request was answered by **the OTHER machine's** DropZone.
|
||||
```
|
||||
mp_a.log: [dzreq] player 2:1 asking for a drop zone ... <- A's own player asks
|
||||
mp_b.log: [dz] GRANTED slot 7 to 2:1 death#1 ... <- B answered it
|
||||
mp_b.log: [dzreq] player 3:1 asking for a drop zone ... <- B's own player asks
|
||||
mp_a.log: [dz] GRANTED slot 0 to 3:1 death#1 ... <- A answered it
|
||||
```
|
||||
`FindGroup("DropZones")` iterates **every** DropZone entity including replicants of remotely-mastered
|
||||
ones and picks the geometrically closest, so a respawn is: *my request → an arbitrary peer's
|
||||
DropZone → that peer's reply → back over the wire to me.* With five players that is five round trips
|
||||
through arbitrary peers, and **one degraded peer can ghost everybody else.**
|
||||
That dovetails exactly with the otherwise-unexplained field finding that **one machine stopped
|
||||
processing the effect/death-transition stream 57 % into match 1 and never recovered** (0 explosions /
|
||||
0 wreck swaps / 0 burials while the other four logged 4/4/4). A node in that state that owns the
|
||||
closest drop zone cannot answer anyone's respawn — which is what "everybody ghosted" looks like.
|
||||
It also explains why solo is 100 % reliable (the whole handshake is in-process) and why the
|
||||
2-node bench passes (both nodes healthy, replies land, `deathCount` agrees).
|
||||
|
||||
**Revised theory of the ghost (two layers):**
|
||||
- *Structural:* respawn depends on a cross-machine round trip through a peer that may be lagging,
|
||||
stalled, or holding a stale view of my player. Not drop-zone starvation (`slots=8`) and not a
|
||||
single lost reply (the hunt retries every 2 s forever).
|
||||
- *Mechanism when the reply does arrive but is stale/mismatched:* the silent discard in 2 — the
|
||||
reply's `deathCount` disagrees and it is dropped without a trace, which couples this to #45.
|
||||
|
||||
**Two candidate fixes, both needing the next field logs to choose between:**
|
||||
(a) **Prefer a locally-mastered DropZone** when one exists, keeping the handshake in-process and
|
||||
removing the remote dependency entirely — cheap and robust, but it changes which physical pad you
|
||||
land on, so check it against the maps first. (b) **Make the reply path tolerant**: accept a granted
|
||||
reply whose `deathCount` is *ahead* of ours (our counter is behind) instead of discarding it, and
|
||||
never leave `deathPending` latched on a discard. ⚠ Neither should be guessed at blind — the
|
||||
`[ghost] DROP-ZONE REPLY DISCARDED` line's mismatch direction decides it in one line of log.
|
||||
|
||||
## What the instrumentation already killed
|
||||
`slots=8` (measured, `[dz] POOL`). An 8-slot pool **cannot** be cooldown-starved by 5 players (at
|
||||
|
||||
@@ -137,6 +137,13 @@ DropZone::~DropZone()
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
// #81: print an EntityID as host:local. A player's LOCAL id is 1 on every
|
||||
// machine, so printing only the local part made every one of five players read
|
||||
// "entity 1" -- useless for telling WHO stalled. GetHostID() is non-const.
|
||||
static inline int DZHost(const EntityID &id) { EntityID c(id); return (int)c.GetHostID(); }
|
||||
static inline int DZLocal(const EntityID &id) { return (int)id; }
|
||||
|
||||
//#############################################################################
|
||||
// Dropzone assignment
|
||||
//
|
||||
@@ -223,7 +230,9 @@ void
|
||||
|
||||
if (dzLog)
|
||||
{
|
||||
DEBUG_STREAM << "[dz] REQUEST from entity " << (int)message->requestingEntity
|
||||
DEBUG_STREAM << "[dz] REQUEST from "
|
||||
<< DZHost(message->requestingEntity) << ":"
|
||||
<< DZLocal(message->requestingEntity)
|
||||
<< " deathCount=" << message->deathCount
|
||||
<< " waited=" << waited << "s slots[";
|
||||
for (int d = 0; d < dropZoneCount; ++d)
|
||||
@@ -383,7 +392,8 @@ void
|
||||
{
|
||||
s_waitLastLog[wslot] = Now();
|
||||
DEBUG_STREAM << (waited >= 15.0f ? "[dz] GHOST LIKELY -- " : "[dz] STALL -- ")
|
||||
<< "entity " << (int)message->requestingEntity
|
||||
<< DZHost(message->requestingEntity) << ":"
|
||||
<< DZLocal(message->requestingEntity)
|
||||
<< " death#" << message->deathCount
|
||||
<< " has waited " << waited << "s for a slot; all "
|
||||
<< dropZoneCount << " busy [";
|
||||
@@ -391,7 +401,8 @@ void
|
||||
{
|
||||
Scalar age = lastUsageTime[d].ticks ? (Scalar)(Now() - lastUsageTime[d]) : -1.0f;
|
||||
DEBUG_STREAM << (d ? " " : "") << d << ":usedBy="
|
||||
<< (int)lastUsedBy[d] << ",age=" << age << "s";
|
||||
<< DZHost(lastUsedBy[d]) << ":" << DZLocal(lastUsedBy[d])
|
||||
<< ",age=" << age << "s";
|
||||
}
|
||||
DEBUG_STREAM << "]\n" << std::flush;
|
||||
}
|
||||
@@ -399,8 +410,9 @@ void
|
||||
{
|
||||
static int s_rp = 0;
|
||||
if (++s_rp <= 5)
|
||||
DEBUG_STREAM << "[dz] repost #" << s_rp << " (entity "
|
||||
<< (int)message->requestingEntity << ")\n" << std::flush;
|
||||
DEBUG_STREAM << "[dz] repost #" << s_rp << " ("
|
||||
<< DZHost(message->requestingEntity) << ":"
|
||||
<< DZLocal(message->requestingEntity) << ")\n" << std::flush;
|
||||
}
|
||||
Time when=Now();
|
||||
when += 0.1f;
|
||||
@@ -423,8 +435,10 @@ Found_One:
|
||||
int slot = (int)(drop_zone - dropZones);
|
||||
if (waited > 0.5f) ++s_stalledGrants;
|
||||
++s_grants;
|
||||
DEBUG_STREAM << "[dz] GRANTED slot " << slot << " to entity "
|
||||
<< (int)message->requestingEntity << " death#" << message->deathCount
|
||||
DEBUG_STREAM << "[dz] GRANTED slot " << slot << " to "
|
||||
<< DZHost(message->requestingEntity) << ":"
|
||||
<< DZLocal(message->requestingEntity)
|
||||
<< " death#" << message->deathCount
|
||||
<< " waited=" << waited << "s"
|
||||
<< (waited >= 15.0f ? " (GHOST RECOVERED)" : "")
|
||||
<< " [grants=" << s_grants << " stalled=" << s_stalledGrants << "]"
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
#!/usr/bin/env bash
|
||||
# #81 GHOST REPRO: two nodes, BOTH die, watch the respawn handshake.
|
||||
#
|
||||
# Looking for, on either node:
|
||||
# [dzreq] ... msgDeath= ourDeathCount= (the request side)
|
||||
# [dz] GRANTED / STALL (the drop zone's answer)
|
||||
# [ghost] DROP-ZONE REPLY DISCARDED (the silent-discard path)
|
||||
# [respawn] ... RESET at drop zone (success)
|
||||
# A node with a death cycle START and no RESET has reproduced the field ghost.
|
||||
set -x
|
||||
cd /c/git/bt411/content || exit 1
|
||||
rm -f mp_a.log mp_b.log
|
||||
sed 's/^experience=expert/experience=novice/' MP.EGG > MPG.EGG
|
||||
|
||||
# B first (listener), then A -- same order as the working mp_skate.sh
|
||||
BT_LOG=mp_b.log BT_AFFINITY=0x2 BT_MP_LOG=1 BT_SCORE_LOG=1 BT_DEATH_LOG=1 \
|
||||
BT_SELF_DAMAGE=60 BT_SELF_DAMAGE_DELAY=40 \
|
||||
../build/Release/btl4.exe -egg MPG.EGG -net 1601 &
|
||||
sleep 2
|
||||
BT_LOG=mp_a.log BT_AFFINITY=0x1 BT_MP_LOG=1 BT_SCORE_LOG=1 BT_DEATH_LOG=1 \
|
||||
BT_SELF_DAMAGE=60 BT_SELF_DAMAGE_DELAY=55 \
|
||||
../build/Release/btl4.exe -egg MPG.EGG -net 1501 &
|
||||
sleep 4
|
||||
python ../tools/btconsole.py MPG.EGG 127.0.0.1:1501 127.0.0.1:1601 &
|
||||
PC=$!
|
||||
|
||||
sleep 175
|
||||
kill $PC 2>/dev/null
|
||||
taskkill //F //IM btl4.exe 2>/dev/null
|
||||
rm -f MPG.EGG
|
||||
echo "=== DONE ==="
|
||||
Reference in New Issue
Block a user