BT411/docs/GHOST_MECH_ANALYSIS.md

# The GHOST MECH / zombie-wreck failure — field analysis 2026-07-29

**Status:** mechanism established [T1/T2]; the missing link (why the drop-zone handshake fails) is
still open. Field session: 2026-07-29 night, Steam MP, 5 players, build 4.11.642. Raw logs and the
verbatim agent findings are in `scratchpad/night6/` (uncommitted — they contain machine names and
Steam identities). Tracker: #81 (zombie wreck), #57 (deathPending latch), #45 (tally replication).

## The one-sentence version
A death whose **drop-zone reply never arrives** leaves the respawn cycle stranded, so the mech is
never reset and never repainted — and because the render-side wreck swap is **one-way**, the pilot
keeps driving a burning hulk that sinks out of the world after ~18 s and then cannot be drawn at
all. The "wreckage moving and shooting" report and the "invisible ghost" report are **the same
failure at two different ages**.

## Established from the field logs [T1]
- Exactly **8 death cycles** in the whole session. Every one has the same 3-line signature at
  consecutive lines: `death cycle START` → `*** DESTROYED ***` → `WARNING: death ... SWALLOWED`.
- **Only 2 of 8 reached `RESET at drop zone`.** A stranded cycle predicts ghosting **perfectly**:
  8/8 cycles, 10/10 player-matches, in both directions.
- Match 2: all four non-host players died, all four stranded, zero RESETs anywhere = "everybody
  ghosted". **The host did not ghost because he never died** — his log holds no death cycle at all.
  (There is nothing special about being host; an earlier reading of this was wrong.)
- Match 1: three deaths, two stranded, one completed. The completion is the control case, and it is
  the "self-corrected for one player" the testers reported.
- ⚠ **The SWALLOWED warning is benign.** It fires on 8 of 8 deaths, including both clean
  completions and a build-641 death, so its base rate is 100% and its correlation with ghosting is
  zero. It is not even a second death — it is the same death re-entering the notify from inside the
  cycle. Do not treat it as a signal (an earlier reading did).
- ⚠ **Player IDs are per-match**: node = lobby token octet + 1, host always `.1`. `player 3:1` is a
  different human in every match. Any cross-machine identification must decode this first.
- ⚠ An earlier "three spurious RESETs on the host" finding was an **aggregation artifact** — one
  log file spans ~18 sessions and each session legitimately logs one spawn RESET.

## The render mechanism [T1, code]
- `btl4vid.cpp:1255-1300` — the wreck sink is a **pure timer**, nothing about movement: 0.25 s
  reveal, then `sink = -0.025·t²`, and at `-8.0` the hulk/debris/flames get `SetDrawObj(NULL)` plus
  `[BTrender] wreck buried (sink complete)`. That threshold is reached at **t ≈ 18 s**.
- The wreck objects hang off the mech's own render tree as an offset, so **they travel with the
  pilot** — hence a moving, burning, shooting wreck for the first ~18 s.
- `mech4.cpp:1827` — a healthy respawn calls `BTRebuildMechModel` on Reset precisely because "the
  wreck swap is one-way on the render side". No reply → no Reset → no rebuild → the hulk sinks and
  the player is invisible but still simulated and driveable.
- Peers apply the wreck swap correctly and in a bit-identical order across machines, so the ghost is
  **not** an entity peers never learned about — it is an entity whose presentation is frozen dead
  and then hidden.

## The stall itself [T0 WinTesla — NOT proven authentic]
`engine/MUNGA/DROPZONE.cpp`: a slot is granted only if `IsAvailable()` — busy for `DOWN_TIME = 5 s`
after use, and any other player's vehicle within 2 m marks it busy. **When no slot is free the
handler reposts the message to itself every 0.1 s at `MaxEventPriority` and never replies**, so the
requester's `deathPending` latch is never cleared.
⚠ **Provenance:** `engine/MUNGA` is **WinTesla**, the later Windows port of MUNGA taken from updated
Red Planet (which converted the Division hardware to software). **The 1995 game never ran it.** So
this behaviour is authoritative for what we compile, *not* for what the pods did. BT's own
`munga/dropzone.cpp` is **absent from the decomp export** — the 1995 assignment policy is an
unrecovered gap. Recovering it (byte-scan + windowed disasm, the technique that recovered
`FUN_004a6344`) is the open task.

## 2026-07-30 dig: the respawn handshake, walked end to end
Re-read the whole chain (death → +5 s re-post → engine hunt → DropZone grant → reply → Reset) and
**measured** the parts that were assumptions. Results, in order of importance:

**1. The engine hunt RETRIES EVERY 2 SECONDS, forever, while the mech is dead** [T0]
(`PLAYER.cpp:325-327` posts the message back to itself unconditionally; its only state-based exit is
`GetSimulationState() == DropZoneAcquiredState`, which never happens — see 3). The DropZone also has
a resend net: a repeat request from the same requester + same `deathCount` re-grants the **same**
slot even while it is busy (`DROPZONE.cpp:182-194`). **So "a transient no-slot" and "one lost reply"
are both ELIMINATED as causes** — the request keeps being re-sent and re-granted every 2 s. A
permanent strand needs a permanent reason.

**2. THE SILENT DISCARD — the one remaining unlit path, now instrumented** [T1]
`BTPlayer::DropZoneReplyMessageHandler` is `if (!playerVehicle) … else if (deathCount ==
message->deathCount) … else { return; }`. That final branch was a **bare return with no logging**:
the drop zone granted a spot and replied, the numbers disagreed, and the reply was dropped —
`deathPending` stays latched, the mech is never `Reset`, and the pilot is a permanent ghost **with
zero trace in the log**. That is exactly why 6 of 8 field cycles stranded leaving no evidence. Now
always-on (`[ghost] DROP-ZONE REPLY DISCARDED …`) and it prints the **direction** of the mismatch,
which decides the fix: `msgDeath < ours` = genuinely stale (dropping is correct); `msgDeath > ours` =
**our counter is behind and we threw away a live respawn** — which points straight at #45 (the death
tally does not replicate correctly). Deliberately NOT "recovered" yet: guessing the direction would
be a stand-in, and the wrong guess respawns a mech that is still alive.

**3. ⚠ LANDMINE — do not convert the two `Set_Alarm_Level` raw writes to `SetSimulationState()`.**
`Set_Alarm_Level` is an **empty stub** (`btstubs.cpp:87`), so the death path's
`Set_Alarm_Level(this+0x2c, 1)` and the reply path's `(this+0x2c, 2)` are **no-ops** today. Their
values decode perfectly against `Player`'s enum (`PLAYER.h:273-279`) as `DropZoneAcquiredState`(1)
and `VehicleTranslocatedState`(2) — `DropZoneAcquiredState = Entity::StateCount`, and `Entity` adds
no states, so it is **1**. That makes "obviously these should be `SetSimulationState` calls" a very
attractive and **catastrophic** fix: the engine hunt is gated on `GetSimulationState() !=
DropZoneAcquiredState`, so setting state 1 on death would stop `AssignDropZone` from ever being
dispatched and make **every** pilot ghost permanently.
Measured: our `BTPlayer::simulationState` is at **0x24**, not 0x2c, and the write leaves it at 0
(`[ghost] death raw-write(+0x2c,1): simState 0 -> 0 (hunt gate still open)`). In the binary `+0x2c`
is the **Simulation-base alarm** — the same field the mech side calls `graphicAlarm` (`@0x4ac126`:
"owner alarm+0x2C -> level 9"). Our layout models that only as a `Mech` member, so a `BTPlayer` has
nowhere to put it; the stub is harmless until we recover what reads a *Player's* alarm in the binary
(likely a cockpit/HUD respawn indicator). [T1 offsets, T3 purpose]

**4. EVERY MP RESPAWN IS A CROSS-MACHINE ROUND TRIP — and that is the single point of failure**
[T2, two-node bench `scratchpad/night6/mp_ghost.sh`]. Measured on a 2-node rig with the host:local
labels: each player's request was answered by **the OTHER machine's** DropZone.
```
mp_a.log: [dzreq] player 2:1 asking for a drop zone ...      <- A's own player asks
mp_b.log: [dz] GRANTED slot 7 to 2:1 death#1 ...             <- B answered it
mp_b.log: [dzreq] player 3:1 asking for a drop zone ...      <- B's own player asks
mp_a.log: [dz] GRANTED slot 0 to 3:1 death#1 ...             <- A answered it
```
`FindGroup("DropZones")` iterates **every** DropZone entity including replicants of remotely-mastered
ones and picks the geometrically closest, so a respawn is: *my request → an arbitrary peer's
DropZone → that peer's reply → back over the wire to me.* With five players that is five round trips
through arbitrary peers, and **one degraded peer can ghost everybody else.**
That dovetails exactly with the otherwise-unexplained field finding that **one machine stopped
processing the effect/death-transition stream 57 % into match 1 and never recovered** (0 explosions /
0 wreck swaps / 0 burials while the other four logged 4/4/4). A node in that state that owns the
closest drop zone cannot answer anyone's respawn — which is what "everybody ghosted" looks like.
It also explains why solo is 100 % reliable (the whole handshake is in-process) and why the
2-node bench passes (both nodes healthy, replies land, `deathCount` agrees).

**Revised theory of the ghost (two layers):**
- *Structural:* respawn depends on a cross-machine round trip through a peer that may be lagging,
  stalled, or holding a stale view of my player. Not drop-zone starvation (`slots=8`) and not a
  single lost reply (the hunt retries every 2 s forever).
- *Mechanism when the reply does arrive but is stale/mismatched:* the silent discard in 2 — the
  reply's `deathCount` disagrees and it is dropped without a trace, which couples this to #45.

**Two candidate fixes, both needing the next field logs to choose between:**
(a) **Prefer a locally-mastered DropZone** when one exists, keeping the handshake in-process and
removing the remote dependency entirely — cheap and robust, but it changes which physical pad you
land on, so check it against the maps first. (b) **Make the reply path tolerant**: accept a granted
reply whose `deathCount` is *ahead* of ours (our counter is behind) instead of discarding it, and
never leave `deathPending` latched on a discard. ⚠ Neither should be guessed at blind — the
`[ghost] DROP-ZONE REPLY DISCARDED` line's mismatch direction decides it in one line of log.

## HOW THE BINARY HANDLES IT — `deathPending` is OUR INVENTION [T1, decisive]
Read BT's own respawn code (it *is* in the export, even though `munga/dropzone.cpp` is not) and
checked it against the raw binary.

**The hunt is faithful.** BT's `Player::VehicleDeadMessageHandler` = `FUN_0042db80`
(`part_003.c:12029`) gates on `message->deathCount == player->deathCount` **and**
`player+0x40 != 1` — i.e. `simulationState != DropZoneAcquiredState`, exactly WinTesla's two gates —
then runs the same closest-DropZone search skipping `"win*"` zones, dispatches `AssignDropZone`, and
**re-posts the message to itself** on a timer. So the retry loop and the cross-machine hunt are
authentic 1995 behaviour, not a WinTesla artifact.

**But the latch is not.** BT's `BTPlayer::VehicleDeadMessageHandler` = `FUN_004c012c`
(`part_013.c:10504`) ends with:
```
++deathCount;  message->deathCount = deathCount;  --lives;
updateModel |= 1;  alarm(+0x2c, 1);  save dropZoneLocation;
Post(HighPriority, self, message, Now()+delay);
*(param_1 + 0x290) = 0;      // <-- the field our reconstruction calls deathPending
```
**The binary CLEARS that field at the end of every death.** Verified against `BTL4OPT.EXE` itself:
`+0x290` is written in exactly **three** places in the whole executable
(`file_off 0x0b75fb`, `0x0bffe3`, `0x0c0a05`), and **all three store a zeroed register** —
`xor ecx,ecx` / `xor eax,eax` / `xor edx,edx` immediately before. There is **no write of 1, or of
any non-zero value, to `+0x290` anywhere in the binary.** Two of the three are constructor/reset
sweeps; the middle one is the death handler above (it sits right after the `call` to `Post` and
`add esp,0x14`, matching the decompiled tail exactly).

**So in 1995 there is no death-pending gate at all.** Our `btplayer.cpp:505` does
`deathPending = 1`, and six separate sites then have to clear it (`:382 :469 :1431 :1442 :1461
:1532`) — each one a patch for a different stranding scenario. Gitea **#57 and #55 are artifacts of
that invention**, not of the original design.

**And that is what makes a ghost PERMANENT.** In BT a failed respawn is harmless: the 2-second
re-post keeps hunting, and the next death starts a clean cycle. In ours the first failure latches
the pilot — every later death hits the dedup and is SWALLOWED ("deaths so far 1" forever), so the
pilot can never recover for the rest of the mission. That is precisely the field signature: 8 death
cycles, 6 stranded, none of them ever recovering.

**Proposed fix (faithful, and it removes the whole bug class):** match the binary — clear the field
at the end of the death handler instead of latching it, and drop the dedup gate that depends on it.
⚠ Six sites currently rely on that latch's existence, and the SWALLOWED dedup is load-bearing for
the "one death, one cycle" behaviour, so this wants a deliberate two-node bench rather than a
late-night edit. Caveat on the identification: `+0x290 ↔ deathPending` rests on our own
reconstruction's mapping (our compiled `BTPlayer` is only `0x28c` bytes, so the offsets cannot
match directly) — but the *behavioural* evidence (BT's death handler clears it; nothing in the
binary ever sets it) is read straight from the executable.

## What the instrumentation already killed
`slots=8` (measured, `[dz] POOL`). An 8-slot pool **cannot** be cooldown-starved by 5 players (at
most 5 slots on cooldown at once), so the shared `dropzone=one` in `tools/eggmodel.py:42` is **not**
the bottleneck. Suspicion moves to the handshake: a lost reply, or a `deathCount` mismatch — the
engine gates the hunt on `message->deathCount == deathCount` and the DropZone's resend safety net is
keyed on `lastDeathCount == message->deathCount`, so one mismatch breaks both silently (cf. #45).

## Instrumentation shipped (`ba6756c`) — always-on, no env var needed
| line | meaning |
|---|---|
| `[dz] POOL name= slots= downTime= proximityBlock=` | once per mission: the pool size |
| `[dz] GRANTED slot N to entity E death#D waited=Xs` | one per respawn; `waited` is the headline; tags `(GHOST RECOVERED)` past 15 s |
| `[dz] STALL` / `[dz] GHOST LIKELY` | escalating, rate-limited per waiter; names **why** each slot is busy (age + last user) — this discriminates cooldown saturation from proximity blocking |
| `[dzreq] player P asking ... try=N msgDeath= ourDeathCount=` | requester half; flags `*** MISMATCH ***` |
`BT_DROPZONE_LOG=1` adds a verbose per-request slot dump for bench work.

## Next actions, in order
1. **Recover BT's 1995 drop-zone policy** from `content/BTL4OPT.EXE` (decomp gap). The key question:
   when no slot was available, did BT also repost forever and never reply? If yes, the stall is
   authentic and our divergence is elsewhere; if no, implement what BT did.
2. **Read the next field logs' `[dz]`/`[dzreq]` lines** — they answer cooldown-vs-proximity and
   lost-reply-vs-deathCount-mismatch directly.
3. Ask the testers one question that discriminates a second defect: **did a flaming mover stay
   visible for more than ~20 s?** If yes, the sink tick is not running for that entity (updater
   early-return, or a render-tree rebuild resetting `wreckAge`) — a second bug on top of this one.
4. Separately filed/needed: one machine **stopped processing the effect/death-transition stream**
   57 % into a match and never recovered (0 explosions / 0 wreck swaps / 0 burials while the other
   four logged 4/4/4) — that machine was playing a stale world, and it is why that player reported
   the whole "desync cluster".

## The desync cluster is four separate things, not one bug [T2]
- **Throttle reads zero** — local, and present in the *good* sessions too. ⚠ Unresolved nuance: the
  internal `thr` field is 0 in every field `[drive]` sample while mechs moved, but reads 0.8 in solo
  benches; whether that is the same thing the player saw on his gauge is **not** established.
- **No weapon audio** — local audio-source starvation caused by the **map's ambient emitters**
  (`arena2_morning` ≈ 10.6 effects/s × ~2.5 voices pins the 256-source pool); thousands of
  `ACQUIRE FAILED ... pool is exhausted`. Unrelated to crits or the network.
- **Weapons fire without damage** — replication staleness. The shooter resolves damage and awards
  its own score locally, so a shooter always *sees* hits even when the victim applies nothing.
- **Other mechs lagging/misplaced** — the same stale-world condition as item 4 above.

## Why tonight and not before
Nothing in 642 broke multiplayer. Critical hits landing for real (first time) made deaths far more
frequent, and every death is a chance for this stall — so a rare race became a common one. Fewer
players ⇒ fewer deaths ⇒ clean matches, which is exactly what the testers observed.