BT411/docs/GHOST_MECH_ANALYSIS.md

The GHOST MECH / zombie-wreck failure — field analysis 2026-07-29

Status: mechanism established [T1/T2]; the missing link (why the drop-zone handshake fails) is still open. Field session: 2026-07-29 night, Steam MP, 5 players, build 4.11.642. Raw logs and the verbatim agent findings are in scratchpad/night6/ (uncommitted — they contain machine names and Steam identities). Tracker: #81 (zombie wreck), #57 (deathPending latch), #45 (tally replication).

The one-sentence version

A death whose drop-zone reply never arrives leaves the respawn cycle stranded, so the mech is never reset and never repainted — and because the render-side wreck swap is one-way, the pilot keeps driving a burning hulk that sinks out of the world after ~18 s and then cannot be drawn at all. The "wreckage moving and shooting" report and the "invisible ghost" report are the same failure at two different ages.

Established from the field logs [T1]

• Exactly 8 death cycles in the whole session. Every one has the same 3-line signature at consecutive lines: death cycle START → *** DESTROYED *** → WARNING: death ... SWALLOWED.
• Only 2 of 8 reached RESET at drop zone. A stranded cycle predicts ghosting perfectly: 8/8 cycles, 10/10 player-matches, in both directions.
• Match 2: all four non-host players died, all four stranded, zero RESETs anywhere = "everybody ghosted". The host did not ghost because he never died — his log holds no death cycle at all. (There is nothing special about being host; an earlier reading of this was wrong.)
• Match 1: three deaths, two stranded, one completed. The completion is the control case, and it is the "self-corrected for one player" the testers reported.
• ⚠ The SWALLOWED warning is benign. It fires on 8 of 8 deaths, including both clean completions and a build-641 death, so its base rate is 100% and its correlation with ghosting is zero. It is not even a second death — it is the same death re-entering the notify from inside the cycle. Do not treat it as a signal (an earlier reading did).
• ⚠ Player IDs are per-match: node = lobby token octet + 1, host always .1. player 3:1 is a different human in every match. Any cross-machine identification must decode this first.
• ⚠ An earlier "three spurious RESETs on the host" finding was an aggregation artifact — one log file spans ~18 sessions and each session legitimately logs one spawn RESET.

The render mechanism [T1, code]

• btl4vid.cpp:1255-1300 — the wreck sink is a pure timer, nothing about movement: 0.25 s reveal, then sink = -0.025·t², and at -8.0 the hulk/debris/flames get SetDrawObj(NULL) plus [BTrender] wreck buried (sink complete). That threshold is reached at t ≈ 18 s.
• The wreck objects hang off the mech's own render tree as an offset, so they travel with the pilot — hence a moving, burning, shooting wreck for the first ~18 s.
• mech4.cpp:1827 — a healthy respawn calls BTRebuildMechModel on Reset precisely because "the wreck swap is one-way on the render side". No reply → no Reset → no rebuild → the hulk sinks and the player is invisible but still simulated and driveable.
• Peers apply the wreck swap correctly and in a bit-identical order across machines, so the ghost is not an entity peers never learned about — it is an entity whose presentation is frozen dead and then hidden.

The stall itself [T0 WinTesla — NOT proven authentic]

engine/MUNGA/DROPZONE.cpp: a slot is granted only if IsAvailable() — busy for DOWN_TIME = 5 s after use, and any other player's vehicle within 2 m marks it busy. When no slot is free the handler reposts the message to itself every 0.1 s at MaxEventPriority and never replies, so the requester's deathPending latch is never cleared. ⚠ Provenance: engine/MUNGA is WinTesla, the later Windows port of MUNGA taken from updated Red Planet (which converted the Division hardware to software). The 1995 game never ran it. So this behaviour is authoritative for what we compile, not for what the pods did. BT's own munga/dropzone.cpp is absent from the decomp export — the 1995 assignment policy is an unrecovered gap. Recovering it (byte-scan + windowed disasm, the technique that recovered FUN_004a6344) is the open task.

2026-07-30 dig: the respawn handshake, walked end to end

Re-read the whole chain (death → +5 s re-post → engine hunt → DropZone grant → reply → Reset) and measured the parts that were assumptions. Results, in order of importance:

1. The engine hunt RETRIES EVERY 2 SECONDS, forever, while the mech is dead [T0] (PLAYER.cpp:325-327 posts the message back to itself unconditionally; its only state-based exit is GetSimulationState() == DropZoneAcquiredState, which never happens — see 3). The DropZone also has a resend net: a repeat request from the same requester + same deathCount re-grants the same slot even while it is busy (DROPZONE.cpp:182-194). So "a transient no-slot" and "one lost reply" are both ELIMINATED as causes — the request keeps being re-sent and re-granted every 2 s. A permanent strand needs a permanent reason.

2. THE SILENT DISCARD — the one remaining unlit path, now instrumented [T1] BTPlayer::DropZoneReplyMessageHandler is if (!playerVehicle) … else if (deathCount == message->deathCount) … else { return; }. That final branch was a bare return with no logging: the drop zone granted a spot and replied, the numbers disagreed, and the reply was dropped — deathPending stays latched, the mech is never Reset, and the pilot is a permanent ghost with zero trace in the log. That is exactly why 6 of 8 field cycles stranded leaving no evidence. Now always-on ([ghost] DROP-ZONE REPLY DISCARDED …) and it prints the direction of the mismatch, which decides the fix: msgDeath < ours = genuinely stale (dropping is correct); msgDeath > ours = our counter is behind and we threw away a live respawn — which points straight at #45 (the death tally does not replicate correctly). Deliberately NOT "recovered" yet: guessing the direction would be a stand-in, and the wrong guess respawns a mech that is still alive.

3. ⚠ LANDMINE — do not convert the two Set_Alarm_Level raw writes to SetSimulationState(). Set_Alarm_Level is an empty stub (btstubs.cpp:87), so the death path's Set_Alarm_Level(this+0x2c, 1) and the reply path's (this+0x2c, 2) are no-ops today. Their values decode perfectly against Player's enum (PLAYER.h:273-279) as DropZoneAcquiredState(1) and VehicleTranslocatedState(2) — DropZoneAcquiredState = Entity::StateCount, and Entity adds no states, so it is 1. That makes "obviously these should be SetSimulationState calls" a very attractive and catastrophic fix: the engine hunt is gated on GetSimulationState() != DropZoneAcquiredState, so setting state 1 on death would stop AssignDropZone from ever being dispatched and make every pilot ghost permanently. Measured: our BTPlayer::simulationState is at 0x24, not 0x2c, and the write leaves it at 0 ([ghost] death raw-write(+0x2c,1): simState 0 -> 0 (hunt gate still open)). In the binary +0x2c is the Simulation-base alarm — the same field the mech side calls graphicAlarm (@0x4ac126: "owner alarm+0x2C -> level 9"). Our layout models that only as a Mech member, so a BTPlayer has nowhere to put it; the stub is harmless until we recover what reads a Player's alarm in the binary (likely a cockpit/HUD respawn indicator). [T1 offsets, T3 purpose]

4. EVERY MP RESPAWN IS A CROSS-MACHINE ROUND TRIP — and that is the single point of failure [T2, two-node bench scratchpad/night6/mp_ghost.sh]. Measured on a 2-node rig with the host:local labels: each player's request was answered by the OTHER machine's DropZone.

mp_a.log: [dzreq] player 2:1 asking for a drop zone ...      <- A's own player asks
mp_b.log: [dz] GRANTED slot 7 to 2:1 death#1 ...             <- B answered it
mp_b.log: [dzreq] player 3:1 asking for a drop zone ...      <- B's own player asks
mp_a.log: [dz] GRANTED slot 0 to 3:1 death#1 ...             <- A answered it

FindGroup("DropZones") iterates every DropZone entity including replicants of remotely-mastered ones and picks the geometrically closest, so a respawn is: my request → an arbitrary peer's DropZone → that peer's reply → back over the wire to me. With five players that is five round trips through arbitrary peers, and one degraded peer can ghost everybody else. That dovetails exactly with the otherwise-unexplained field finding that one machine stopped processing the effect/death-transition stream 57 % into match 1 and never recovered (0 explosions / 0 wreck swaps / 0 burials while the other four logged 4/4/4). A node in that state that owns the closest drop zone cannot answer anyone's respawn — which is what "everybody ghosted" looks like. It also explains why solo is 100 % reliable (the whole handshake is in-process) and why the 2-node bench passes (both nodes healthy, replies land, deathCount agrees).

Revised theory of the ghost (two layers):

• Structural: respawn depends on a cross-machine round trip through a peer that may be lagging, stalled, or holding a stale view of my player. Not drop-zone starvation (slots=8) and not a single lost reply (the hunt retries every 2 s forever).
• Mechanism when the reply does arrive but is stale/mismatched: the silent discard in 2 — the reply's deathCount disagrees and it is dropped without a trace, which couples this to #45.

Two candidate fixes, both needing the next field logs to choose between: (a) Prefer a locally-mastered DropZone when one exists, keeping the handshake in-process and removing the remote dependency entirely — cheap and robust, but it changes which physical pad you land on, so check it against the maps first. (b) Make the reply path tolerant: accept a granted reply whose deathCount is ahead of ours (our counter is behind) instead of discarding it, and never leave deathPending latched on a discard. ⚠ Neither should be guessed at blind — the [ghost] DROP-ZONE REPLY DISCARDED line's mismatch direction decides it in one line of log.

What the instrumentation already killed

slots=8 (measured, [dz] POOL). An 8-slot pool cannot be cooldown-starved by 5 players (at most 5 slots on cooldown at once), so the shared dropzone=one in tools/eggmodel.py:42 is not the bottleneck. Suspicion moves to the handshake: a lost reply, or a deathCount mismatch — the engine gates the hunt on message->deathCount == deathCount and the DropZone's resend safety net is keyed on lastDeathCount == message->deathCount, so one mismatch breaks both silently (cf. #45).

Instrumentation shipped (ba6756c) — always-on, no env var needed

line	meaning
[dz] POOL name= slots= downTime= proximityBlock=	once per mission: the pool size
[dz] GRANTED slot N to entity E death#D waited=Xs	one per respawn; waited is the headline; tags (GHOST RECOVERED) past 15 s
[dz] STALL / [dz] GHOST LIKELY	escalating, rate-limited per waiter; names why each slot is busy (age + last user) — this discriminates cooldown saturation from proximity blocking
[dzreq] player P asking ... try=N msgDeath= ourDeathCount=	requester half; flags *** MISMATCH ***
BT_DROPZONE_LOG=1 adds a verbose per-request slot dump for bench work.

Next actions, in order

1. Recover BT's 1995 drop-zone policy from content/BTL4OPT.EXE (decomp gap). The key question: when no slot was available, did BT also repost forever and never reply? If yes, the stall is authentic and our divergence is elsewhere; if no, implement what BT did.
2. Read the next field logs' [dz]/[dzreq] lines — they answer cooldown-vs-proximity and lost-reply-vs-deathCount-mismatch directly.
3. Ask the testers one question that discriminates a second defect: did a flaming mover stay visible for more than ~20 s? If yes, the sink tick is not running for that entity (updater early-return, or a render-tree rebuild resetting wreckAge) — a second bug on top of this one.
4. Separately filed/needed: one machine stopped processing the effect/death-transition stream 57 % into a match and never recovered (0 explosions / 0 wreck swaps / 0 burials while the other four logged 4/4/4) — that machine was playing a stale world, and it is why that player reported the whole "desync cluster".

The desync cluster is four separate things, not one bug [T2]

• Throttle reads zero — local, and present in the good sessions too. ⚠ Unresolved nuance: the internal thr field is 0 in every field [drive] sample while mechs moved, but reads 0.8 in solo benches; whether that is the same thing the player saw on his gauge is not established.
• No weapon audio — local audio-source starvation caused by the map's ambient emitters (arena2_morning ≈ 10.6 effects/s × ~2.5 voices pins the 256-source pool); thousands of ACQUIRE FAILED ... pool is exhausted. Unrelated to crits or the network.
• Weapons fire without damage — replication staleness. The shooter resolves damage and awards its own score locally, so a shooter always sees hits even when the victim applies nothing.
• Other mechs lagging/misplaced — the same stale-world condition as item 4 above.

Why tonight and not before

Nothing in 642 broke multiplayer. Critical hits landing for real (first time) made deaths far more frequent, and every death is a chance for this stall — so a rare race became a common one. Fewer players ⇒ fewer deaths ⇒ clean matches, which is exactly what the testers observed.