#81: every MP respawn is a cross-machine round trip -- one degraded peer ghosts everyone

Two-node bench with the fixed host:local labels shows each player's drop-zone
request is answered by the OTHER machine's DropZone: A asks, B grants; B asks, A
grants.  FindGroup("DropZones") iterates replicants of remotely-mastered zones
too and takes the geometrically closest, so a respawn is
  my request -> an arbitrary peer's DropZone -> that peer's reply -> back to me.
With 5 players that is 5 round trips through arbitrary peers, and ONE degraded
peer can strand everybody else's respawn.  That finally explains the otherwise
unexplained field datum that one machine stopped processing the death-transition
stream 57% into a match and never recovered (0 explosions/wreck swaps/burials
while four other machines logged 4/4/4) -- a node in that state cannot answer
anyone's respawn.  It also explains why solo is 100% reliable (in-process) and
why a healthy 2-node bench passes.

Also fixes the instrumentation before it costs a night: every [dz] line printed
"entity 1" because a player's LOCAL entity id is 1 on every machine -- with five
players the log would have said a respawn stalled but not WHOSE.  All [dz] lines
now print host:local (including the usedBy= owner of each busy slot).

Doc: two candidate fixes recorded (prefer a locally-mastered DropZone / make the
reply path tolerant of a deathCount that is ahead of ours), neither to be guessed
at -- the [ghost] DISCARDED line's mismatch direction decides it in one line.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Joe DiPrima
2026-07-30 01:20:13 -05:00
co-authored by Claude Fable 5
parent 0dbdad17f3
commit bf0555ccf1
3 changed files with 86 additions and 11 deletions
+34 -4
View File
@@ -96,10 +96,40 @@ is the **Simulation-base alarm** — the same field the mech side calls `graphic
nowhere to put it; the stub is harmless until we recover what reads a *Player's* alarm in the binary
(likely a cockpit/HUD respawn indicator). [T1 offsets, T3 purpose]
**Revised theory of the ghost:** not drop-zone starvation (see below) and not a lost reply (see 1),
but a **`deathCount` mismatch between the player and the reply**, silently discarding a live
respawn — MP-specific in a way solo structurally cannot reproduce (solo respawn works every time;
6 of 8 MP cycles stranded), and directly coupled to the known #45 replication defect.
**4. EVERY MP RESPAWN IS A CROSS-MACHINE ROUND TRIP — and that is the single point of failure**
[T2, two-node bench `scratchpad/night6/mp_ghost.sh`]. Measured on a 2-node rig with the host:local
labels: each player's request was answered by **the OTHER machine's** DropZone.
```
mp_a.log: [dzreq] player 2:1 asking for a drop zone ... <- A's own player asks
mp_b.log: [dz] GRANTED slot 7 to 2:1 death#1 ... <- B answered it
mp_b.log: [dzreq] player 3:1 asking for a drop zone ... <- B's own player asks
mp_a.log: [dz] GRANTED slot 0 to 3:1 death#1 ... <- A answered it
```
`FindGroup("DropZones")` iterates **every** DropZone entity including replicants of remotely-mastered
ones and picks the geometrically closest, so a respawn is: *my request → an arbitrary peer's
DropZone → that peer's reply → back over the wire to me.* With five players that is five round trips
through arbitrary peers, and **one degraded peer can ghost everybody else.**
That dovetails exactly with the otherwise-unexplained field finding that **one machine stopped
processing the effect/death-transition stream 57 % into match 1 and never recovered** (0 explosions /
0 wreck swaps / 0 burials while the other four logged 4/4/4). A node in that state that owns the
closest drop zone cannot answer anyone's respawn — which is what "everybody ghosted" looks like.
It also explains why solo is 100 % reliable (the whole handshake is in-process) and why the
2-node bench passes (both nodes healthy, replies land, `deathCount` agrees).
**Revised theory of the ghost (two layers):**
- *Structural:* respawn depends on a cross-machine round trip through a peer that may be lagging,
stalled, or holding a stale view of my player. Not drop-zone starvation (`slots=8`) and not a
single lost reply (the hunt retries every 2 s forever).
- *Mechanism when the reply does arrive but is stale/mismatched:* the silent discard in 2 — the
reply's `deathCount` disagrees and it is dropped without a trace, which couples this to #45.
**Two candidate fixes, both needing the next field logs to choose between:**
(a) **Prefer a locally-mastered DropZone** when one exists, keeping the handshake in-process and
removing the remote dependency entirely — cheap and robust, but it changes which physical pad you
land on, so check it against the maps first. (b) **Make the reply path tolerant**: accept a granted
reply whose `deathCount` is *ahead* of ours (our counter is behind) instead of discarding it, and
never leave `deathPending` latched on a discard. ⚠ Neither should be guessed at blind — the
`[ghost] DROP-ZONE REPLY DISCARDED` line's mismatch direction decides it in one line of log.
## What the instrumentation already killed
`slots=8` (measured, `[dz] POOL`). An 8-slot pool **cannot** be cooldown-starved by 5 players (at