Joe DiPrimaandClaude Fable 5
bf0555ccf1
Two-node bench with the fixed host:local labels shows each player's drop-zone
request is answered by the OTHER machine's DropZone: A asks, B grants; B asks, A
grants. FindGroup("DropZones") iterates replicants of remotely-mastered zones
too and takes the geometrically closest, so a respawn is
my request -> an arbitrary peer's DropZone -> that peer's reply -> back to me.
With 5 players that is 5 round trips through arbitrary peers, and ONE degraded
peer can strand everybody else's respawn. That finally explains the otherwise
unexplained field datum that one machine stopped processing the death-transition
stream 57% into a match and never recovered (0 explosions/wreck swaps/burials
while four other machines logged 4/4/4) -- a node in that state cannot answer
anyone's respawn. It also explains why solo is 100% reliable (in-process) and
why a healthy 2-node bench passes.
Also fixes the instrumentation before it costs a night: every [dz] line printed
"entity 1" because a player's LOCAL entity id is 1 on every machine -- with five
players the log would have said a respawn stalled but not WHOSE. All [dz] lines
now print host:local (including the usedBy= owner of each busy slot).
Doc: two candidate fixes recorded (prefer a locally-mastered DropZone / make the
reply path tolerant of a deathCount that is ahead of ours), neither to be guessed
at -- the [ghost] DISCARDED line's mismatch direction decides it in one line.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
179 lines
14 KiB
Markdown
179 lines
14 KiB
Markdown
# The GHOST MECH / zombie-wreck failure — field analysis 2026-07-29
|
||
|
||
**Status:** mechanism established [T1/T2]; the missing link (why the drop-zone handshake fails) is
|
||
still open. Field session: 2026-07-29 night, Steam MP, 5 players, build 4.11.642. Raw logs and the
|
||
verbatim agent findings are in `scratchpad/night6/` (uncommitted — they contain machine names and
|
||
Steam identities). Tracker: #81 (zombie wreck), #57 (deathPending latch), #45 (tally replication).
|
||
|
||
## The one-sentence version
|
||
A death whose **drop-zone reply never arrives** leaves the respawn cycle stranded, so the mech is
|
||
never reset and never repainted — and because the render-side wreck swap is **one-way**, the pilot
|
||
keeps driving a burning hulk that sinks out of the world after ~18 s and then cannot be drawn at
|
||
all. The "wreckage moving and shooting" report and the "invisible ghost" report are **the same
|
||
failure at two different ages**.
|
||
|
||
## Established from the field logs [T1]
|
||
- Exactly **8 death cycles** in the whole session. Every one has the same 3-line signature at
|
||
consecutive lines: `death cycle START` → `*** DESTROYED ***` → `WARNING: death ... SWALLOWED`.
|
||
- **Only 2 of 8 reached `RESET at drop zone`.** A stranded cycle predicts ghosting **perfectly**:
|
||
8/8 cycles, 10/10 player-matches, in both directions.
|
||
- Match 2: all four non-host players died, all four stranded, zero RESETs anywhere = "everybody
|
||
ghosted". **The host did not ghost because he never died** — his log holds no death cycle at all.
|
||
(There is nothing special about being host; an earlier reading of this was wrong.)
|
||
- Match 1: three deaths, two stranded, one completed. The completion is the control case, and it is
|
||
the "self-corrected for one player" the testers reported.
|
||
- ⚠ **The SWALLOWED warning is benign.** It fires on 8 of 8 deaths, including both clean
|
||
completions and a build-641 death, so its base rate is 100% and its correlation with ghosting is
|
||
zero. It is not even a second death — it is the same death re-entering the notify from inside the
|
||
cycle. Do not treat it as a signal (an earlier reading did).
|
||
- ⚠ **Player IDs are per-match**: node = lobby token octet + 1, host always `.1`. `player 3:1` is a
|
||
different human in every match. Any cross-machine identification must decode this first.
|
||
- ⚠ An earlier "three spurious RESETs on the host" finding was an **aggregation artifact** — one
|
||
log file spans ~18 sessions and each session legitimately logs one spawn RESET.
|
||
|
||
## The render mechanism [T1, code]
|
||
- `btl4vid.cpp:1255-1300` — the wreck sink is a **pure timer**, nothing about movement: 0.25 s
|
||
reveal, then `sink = -0.025·t²`, and at `-8.0` the hulk/debris/flames get `SetDrawObj(NULL)` plus
|
||
`[BTrender] wreck buried (sink complete)`. That threshold is reached at **t ≈ 18 s**.
|
||
- The wreck objects hang off the mech's own render tree as an offset, so **they travel with the
|
||
pilot** — hence a moving, burning, shooting wreck for the first ~18 s.
|
||
- `mech4.cpp:1827` — a healthy respawn calls `BTRebuildMechModel` on Reset precisely because "the
|
||
wreck swap is one-way on the render side". No reply → no Reset → no rebuild → the hulk sinks and
|
||
the player is invisible but still simulated and driveable.
|
||
- Peers apply the wreck swap correctly and in a bit-identical order across machines, so the ghost is
|
||
**not** an entity peers never learned about — it is an entity whose presentation is frozen dead
|
||
and then hidden.
|
||
|
||
## The stall itself [T0 WinTesla — NOT proven authentic]
|
||
`engine/MUNGA/DROPZONE.cpp`: a slot is granted only if `IsAvailable()` — busy for `DOWN_TIME = 5 s`
|
||
after use, and any other player's vehicle within 2 m marks it busy. **When no slot is free the
|
||
handler reposts the message to itself every 0.1 s at `MaxEventPriority` and never replies**, so the
|
||
requester's `deathPending` latch is never cleared.
|
||
⚠ **Provenance:** `engine/MUNGA` is **WinTesla**, the later Windows port of MUNGA taken from updated
|
||
Red Planet (which converted the Division hardware to software). **The 1995 game never ran it.** So
|
||
this behaviour is authoritative for what we compile, *not* for what the pods did. BT's own
|
||
`munga/dropzone.cpp` is **absent from the decomp export** — the 1995 assignment policy is an
|
||
unrecovered gap. Recovering it (byte-scan + windowed disasm, the technique that recovered
|
||
`FUN_004a6344`) is the open task.
|
||
|
||
## 2026-07-30 dig: the respawn handshake, walked end to end
|
||
Re-read the whole chain (death → +5 s re-post → engine hunt → DropZone grant → reply → Reset) and
|
||
**measured** the parts that were assumptions. Results, in order of importance:
|
||
|
||
**1. The engine hunt RETRIES EVERY 2 SECONDS, forever, while the mech is dead** [T0]
|
||
(`PLAYER.cpp:325-327` posts the message back to itself unconditionally; its only state-based exit is
|
||
`GetSimulationState() == DropZoneAcquiredState`, which never happens — see 3). The DropZone also has
|
||
a resend net: a repeat request from the same requester + same `deathCount` re-grants the **same**
|
||
slot even while it is busy (`DROPZONE.cpp:182-194`). **So "a transient no-slot" and "one lost reply"
|
||
are both ELIMINATED as causes** — the request keeps being re-sent and re-granted every 2 s. A
|
||
permanent strand needs a permanent reason.
|
||
|
||
**2. THE SILENT DISCARD — the one remaining unlit path, now instrumented** [T1]
|
||
`BTPlayer::DropZoneReplyMessageHandler` is `if (!playerVehicle) … else if (deathCount ==
|
||
message->deathCount) … else { return; }`. That final branch was a **bare return with no logging**:
|
||
the drop zone granted a spot and replied, the numbers disagreed, and the reply was dropped —
|
||
`deathPending` stays latched, the mech is never `Reset`, and the pilot is a permanent ghost **with
|
||
zero trace in the log**. That is exactly why 6 of 8 field cycles stranded leaving no evidence. Now
|
||
always-on (`[ghost] DROP-ZONE REPLY DISCARDED …`) and it prints the **direction** of the mismatch,
|
||
which decides the fix: `msgDeath < ours` = genuinely stale (dropping is correct); `msgDeath > ours` =
|
||
**our counter is behind and we threw away a live respawn** — which points straight at #45 (the death
|
||
tally does not replicate correctly). Deliberately NOT "recovered" yet: guessing the direction would
|
||
be a stand-in, and the wrong guess respawns a mech that is still alive.
|
||
|
||
**3. ⚠ LANDMINE — do not convert the two `Set_Alarm_Level` raw writes to `SetSimulationState()`.**
|
||
`Set_Alarm_Level` is an **empty stub** (`btstubs.cpp:87`), so the death path's
|
||
`Set_Alarm_Level(this+0x2c, 1)` and the reply path's `(this+0x2c, 2)` are **no-ops** today. Their
|
||
values decode perfectly against `Player`'s enum (`PLAYER.h:273-279`) as `DropZoneAcquiredState`(1)
|
||
and `VehicleTranslocatedState`(2) — `DropZoneAcquiredState = Entity::StateCount`, and `Entity` adds
|
||
no states, so it is **1**. That makes "obviously these should be `SetSimulationState` calls" a very
|
||
attractive and **catastrophic** fix: the engine hunt is gated on `GetSimulationState() !=
|
||
DropZoneAcquiredState`, so setting state 1 on death would stop `AssignDropZone` from ever being
|
||
dispatched and make **every** pilot ghost permanently.
|
||
Measured: our `BTPlayer::simulationState` is at **0x24**, not 0x2c, and the write leaves it at 0
|
||
(`[ghost] death raw-write(+0x2c,1): simState 0 -> 0 (hunt gate still open)`). In the binary `+0x2c`
|
||
is the **Simulation-base alarm** — the same field the mech side calls `graphicAlarm` (`@0x4ac126`:
|
||
"owner alarm+0x2C -> level 9"). Our layout models that only as a `Mech` member, so a `BTPlayer` has
|
||
nowhere to put it; the stub is harmless until we recover what reads a *Player's* alarm in the binary
|
||
(likely a cockpit/HUD respawn indicator). [T1 offsets, T3 purpose]
|
||
|
||
**4. EVERY MP RESPAWN IS A CROSS-MACHINE ROUND TRIP — and that is the single point of failure**
|
||
[T2, two-node bench `scratchpad/night6/mp_ghost.sh`]. Measured on a 2-node rig with the host:local
|
||
labels: each player's request was answered by **the OTHER machine's** DropZone.
|
||
```
|
||
mp_a.log: [dzreq] player 2:1 asking for a drop zone ... <- A's own player asks
|
||
mp_b.log: [dz] GRANTED slot 7 to 2:1 death#1 ... <- B answered it
|
||
mp_b.log: [dzreq] player 3:1 asking for a drop zone ... <- B's own player asks
|
||
mp_a.log: [dz] GRANTED slot 0 to 3:1 death#1 ... <- A answered it
|
||
```
|
||
`FindGroup("DropZones")` iterates **every** DropZone entity including replicants of remotely-mastered
|
||
ones and picks the geometrically closest, so a respawn is: *my request → an arbitrary peer's
|
||
DropZone → that peer's reply → back over the wire to me.* With five players that is five round trips
|
||
through arbitrary peers, and **one degraded peer can ghost everybody else.**
|
||
That dovetails exactly with the otherwise-unexplained field finding that **one machine stopped
|
||
processing the effect/death-transition stream 57 % into match 1 and never recovered** (0 explosions /
|
||
0 wreck swaps / 0 burials while the other four logged 4/4/4). A node in that state that owns the
|
||
closest drop zone cannot answer anyone's respawn — which is what "everybody ghosted" looks like.
|
||
It also explains why solo is 100 % reliable (the whole handshake is in-process) and why the
|
||
2-node bench passes (both nodes healthy, replies land, `deathCount` agrees).
|
||
|
||
**Revised theory of the ghost (two layers):**
|
||
- *Structural:* respawn depends on a cross-machine round trip through a peer that may be lagging,
|
||
stalled, or holding a stale view of my player. Not drop-zone starvation (`slots=8`) and not a
|
||
single lost reply (the hunt retries every 2 s forever).
|
||
- *Mechanism when the reply does arrive but is stale/mismatched:* the silent discard in 2 — the
|
||
reply's `deathCount` disagrees and it is dropped without a trace, which couples this to #45.
|
||
|
||
**Two candidate fixes, both needing the next field logs to choose between:**
|
||
(a) **Prefer a locally-mastered DropZone** when one exists, keeping the handshake in-process and
|
||
removing the remote dependency entirely — cheap and robust, but it changes which physical pad you
|
||
land on, so check it against the maps first. (b) **Make the reply path tolerant**: accept a granted
|
||
reply whose `deathCount` is *ahead* of ours (our counter is behind) instead of discarding it, and
|
||
never leave `deathPending` latched on a discard. ⚠ Neither should be guessed at blind — the
|
||
`[ghost] DROP-ZONE REPLY DISCARDED` line's mismatch direction decides it in one line of log.
|
||
|
||
## What the instrumentation already killed
|
||
`slots=8` (measured, `[dz] POOL`). An 8-slot pool **cannot** be cooldown-starved by 5 players (at
|
||
most 5 slots on cooldown at once), so the shared `dropzone=one` in `tools/eggmodel.py:42` is **not**
|
||
the bottleneck. Suspicion moves to the handshake: a lost reply, or a `deathCount` mismatch — the
|
||
engine gates the hunt on `message->deathCount == deathCount` and the DropZone's resend safety net is
|
||
keyed on `lastDeathCount == message->deathCount`, so one mismatch breaks both silently (cf. #45).
|
||
|
||
## Instrumentation shipped (`ba6756c`) — always-on, no env var needed
|
||
| line | meaning |
|
||
|---|---|
|
||
| `[dz] POOL name= slots= downTime= proximityBlock=` | once per mission: the pool size |
|
||
| `[dz] GRANTED slot N to entity E death#D waited=Xs` | one per respawn; `waited` is the headline; tags `(GHOST RECOVERED)` past 15 s |
|
||
| `[dz] STALL` / `[dz] GHOST LIKELY` | escalating, rate-limited per waiter; names **why** each slot is busy (age + last user) — this discriminates cooldown saturation from proximity blocking |
|
||
| `[dzreq] player P asking ... try=N msgDeath= ourDeathCount=` | requester half; flags `*** MISMATCH ***` |
|
||
`BT_DROPZONE_LOG=1` adds a verbose per-request slot dump for bench work.
|
||
|
||
## Next actions, in order
|
||
1. **Recover BT's 1995 drop-zone policy** from `content/BTL4OPT.EXE` (decomp gap). The key question:
|
||
when no slot was available, did BT also repost forever and never reply? If yes, the stall is
|
||
authentic and our divergence is elsewhere; if no, implement what BT did.
|
||
2. **Read the next field logs' `[dz]`/`[dzreq]` lines** — they answer cooldown-vs-proximity and
|
||
lost-reply-vs-deathCount-mismatch directly.
|
||
3. Ask the testers one question that discriminates a second defect: **did a flaming mover stay
|
||
visible for more than ~20 s?** If yes, the sink tick is not running for that entity (updater
|
||
early-return, or a render-tree rebuild resetting `wreckAge`) — a second bug on top of this one.
|
||
4. Separately filed/needed: one machine **stopped processing the effect/death-transition stream**
|
||
57 % into a match and never recovered (0 explosions / 0 wreck swaps / 0 burials while the other
|
||
four logged 4/4/4) — that machine was playing a stale world, and it is why that player reported
|
||
the whole "desync cluster".
|
||
|
||
## The desync cluster is four separate things, not one bug [T2]
|
||
- **Throttle reads zero** — local, and present in the *good* sessions too. ⚠ Unresolved nuance: the
|
||
internal `thr` field is 0 in every field `[drive]` sample while mechs moved, but reads 0.8 in solo
|
||
benches; whether that is the same thing the player saw on his gauge is **not** established.
|
||
- **No weapon audio** — local audio-source starvation caused by the **map's ambient emitters**
|
||
(`arena2_morning` ≈ 10.6 effects/s × ~2.5 voices pins the 256-source pool); thousands of
|
||
`ACQUIRE FAILED ... pool is exhausted`. Unrelated to crits or the network.
|
||
- **Weapons fire without damage** — replication staleness. The shooter resolves damage and awards
|
||
its own score locally, so a shooter always *sees* hits even when the victim applies nothing.
|
||
- **Other mechs lagging/misplaced** — the same stale-world condition as item 4 above.
|
||
|
||
## Why tonight and not before
|
||
Nothing in 642 broke multiplayer. Critical hits landing for real (first time) made deaths far more
|
||
frequent, and every death is a chance for this stall — so a rare race became a common one. Fewer
|
||
players ⇒ fewer deaths ⇒ clean matches, which is exactly what the testers observed.
|