BT411/docs/GHOST_MECH_ANALYSIS.md

The GHOST MECH / zombie-wreck failure — field analysis 2026-07-29

✅ FIXED 2026-07-30 — two defects that were masking each other

1. The death latch never released on failure. The binary's FUN_004c012c tail is Post(...) → *(this+0x290) = 0 → *(this+0x258) = 0. We had the Post and the suppressConsole and were missing the middle instruction, so deathPending cleared only on SUCCESS paths. One failed respawn latched the pilot for the rest of the mission — every later death hit the dedup and was SWALLOWED, so the cycle could never restart. That is what made a transient hiccup a PERMANENT ghost, and it matches the field signature exactly (8 cycles, 6 stranded, none recovering). Fixed in btplayer.cpp (death-handler tail). 2. VehicleDeadMessage was dispatched TWICE per death. BTPostKillScore (btplayer.cpp:2263) sent a second one "to credit a death" — but that message is the RESPAWN-CYCLE TRIGGER, not a scoreboard increment, and the tally is already credited by the handler's ++deathTally (:538). Both fire inside the same death transition (mech4.cpp:2006 and :2110), so they were always paired. Removed. They hid each other: the duplicate made the latch look necessary, and the latch made the duplicate invisible — every SWALLOWED warning in the field logs was simply the latch deduping our own duplicate (8 of 8 deaths, a 100 % base rate, which is why it correlated with nothing). Fixing either alone makes things visibly worse, which is why earlier passes at #57/#55 kept adding clear-sites instead of finding the root. The binary broke the tie.

Verified (4-node stress, 2026-07-30): four simultaneous networked nodes, all dying every ~35 s on staggered timers so respawn handshakes overlap (the field condition): 56 death cycles, every completed one START→RESET, 0 swallowed / 0 mismatch / 0 discarded / 0 ghost / 0 crash across all four. Scoreboard: every node's PLAYER_DEAD matchlog rows show deaths=N tally=N in lockstep, one increment per death (13/13/15/15) — the DEATHS column the comms panel draws is exact; the kill-credit path evaluated all 56 deaths and correctly declined each self-kill (NOCREDIT self=1), KILLS untouched by the fix (separate ScoreMessage path). Bench: scratchpad/night6/mp4_stress.sh

• content/MP4.EGG. Two observations from watching the 4-node run, both run down (2026-07-30): (1) "comms panel counted no deaths" — the panel machinery is CORRECT (2-node rerun with a draw-value probe: both rows on both nodes drew 0→9/0→11 live, local + replicated); the 4-node zeros were the artificial load (4 core-pinned instances → PilotList Execute starved to <0.6 Hz, panel minutes-stale). Probe stays in ([score] panel DRAW, BT_SCORE_LOG). (2) "a respawn had no blue vortex" — REAL and now fixed: the warp effect is one global slot and a PEER's un-wreck sphere (BTStartWarpEffect) could stomp the local pilot's own POV vortex mid-lifecycle; fixed by POV-priority self-skip (btl4vid.cpp; context/translocation-warp.md). Pre-fix this never showed because overlapping respawns barely existed — the ghost fix CREATED the traffic that exposed it. Verified: solo 17 consecutive death/respawn cycles, every one START→RESET, 0 swallowed / 0 mismatch / 0 crash (pre-fix this strands permanently after cycle 1). Two-node MP over a real network path with cross-machine drop-zone replies: A 11 cycles, B 12, 0 swallowed / 0 mismatch / 0 discarded / 0 crash. Bench: scratchpad/night6/mp_ghost.sh, BT_SELF_DAMAGE_REPEAT=1.

Status of the analysis below: the mechanism write-up that led to the fix. The instrumentation ([dz], [dzreq], [ghost]) stays in — it is what will prove the fix in the field, and the cross-machine dependency in §4 is still a real fragility even with the latch fixed. Field session: 2026-07-29 night, Steam MP, 5 players, build 4.11.642. Raw logs and the verbatim agent findings are in scratchpad/night6/ (uncommitted — they contain machine names and Steam identities). Tracker: #81 (zombie wreck), #57 (deathPending latch), #45 (tally replication).

The one-sentence version

A death whose drop-zone reply never arrives leaves the respawn cycle stranded, so the mech is never reset and never repainted — and because the render-side wreck swap is one-way, the pilot keeps driving a burning hulk that sinks out of the world after ~18 s and then cannot be drawn at all. The "wreckage moving and shooting" report and the "invisible ghost" report are the same failure at two different ages.

Established from the field logs [T1]

• Exactly 8 death cycles in the whole session. Every one has the same 3-line signature at consecutive lines: death cycle START → *** DESTROYED *** → WARNING: death ... SWALLOWED.
• Only 2 of 8 reached RESET at drop zone. A stranded cycle predicts ghosting perfectly: 8/8 cycles, 10/10 player-matches, in both directions.
• Match 2: all four non-host players died, all four stranded, zero RESETs anywhere = "everybody ghosted". The host did not ghost because he never died — his log holds no death cycle at all. (There is nothing special about being host; an earlier reading of this was wrong.)
• Match 1: three deaths, two stranded, one completed. The completion is the control case, and it is the "self-corrected for one player" the testers reported.
• ⚠ The SWALLOWED warning is benign. It fires on 8 of 8 deaths, including both clean completions and a build-641 death, so its base rate is 100% and its correlation with ghosting is zero. It is not even a second death — it is the same death re-entering the notify from inside the cycle. Do not treat it as a signal (an earlier reading did).
• ⚠ Player IDs are per-match: node = lobby token octet + 1, host always .1. player 3:1 is a different human in every match. Any cross-machine identification must decode this first.
• ⚠ An earlier "three spurious RESETs on the host" finding was an aggregation artifact — one log file spans ~18 sessions and each session legitimately logs one spawn RESET.

The render mechanism [T1, code]

• btl4vid.cpp:1255-1300 — the wreck sink is a pure timer, nothing about movement: 0.25 s reveal, then sink = -0.025·t², and at -8.0 the hulk/debris/flames get SetDrawObj(NULL) plus [BTrender] wreck buried (sink complete). That threshold is reached at t ≈ 18 s.
• The wreck objects hang off the mech's own render tree as an offset, so they travel with the pilot — hence a moving, burning, shooting wreck for the first ~18 s.
• mech4.cpp:1827 — a healthy respawn calls BTRebuildMechModel on Reset precisely because "the wreck swap is one-way on the render side". No reply → no Reset → no rebuild → the hulk sinks and the player is invisible but still simulated and driveable.
• Peers apply the wreck swap correctly and in a bit-identical order across machines, so the ghost is not an entity peers never learned about — it is an entity whose presentation is frozen dead and then hidden.

The stall itself [T0 WinTesla — NOT proven authentic]

engine/MUNGA/DROPZONE.cpp: a slot is granted only if IsAvailable() — busy for DOWN_TIME = 5 s after use, and any other player's vehicle within 2 m marks it busy. When no slot is free the handler reposts the message to itself every 0.1 s at MaxEventPriority and never replies, so the requester's deathPending latch is never cleared. ⚠ Provenance: engine/MUNGA is WinTesla, the later Windows port of MUNGA taken from updated Red Planet (which converted the Division hardware to software). The 1995 game never ran it. So this behaviour is authoritative for what we compile, not for what the pods did. BT's own munga/dropzone.cpp is absent from the decomp export — the 1995 assignment policy is an unrecovered gap. Recovering it (byte-scan + windowed disasm, the technique that recovered FUN_004a6344) is the open task.

2026-07-30 dig: the respawn handshake, walked end to end

Re-read the whole chain (death → +5 s re-post → engine hunt → DropZone grant → reply → Reset) and measured the parts that were assumptions. Results, in order of importance:

1. The engine hunt RETRIES EVERY 2 SECONDS, forever, while the mech is dead [T0] (PLAYER.cpp:325-327 posts the message back to itself unconditionally; its only state-based exit is GetSimulationState() == DropZoneAcquiredState, which never happens — see 3). The DropZone also has a resend net: a repeat request from the same requester + same deathCount re-grants the same slot even while it is busy (DROPZONE.cpp:182-194). So "a transient no-slot" and "one lost reply" are both ELIMINATED as causes — the request keeps being re-sent and re-granted every 2 s. A permanent strand needs a permanent reason.

2. THE SILENT DISCARD — the one remaining unlit path, now instrumented [T1] BTPlayer::DropZoneReplyMessageHandler is if (!playerVehicle) … else if (deathCount == message->deathCount) … else { return; }. That final branch was a bare return with no logging: the drop zone granted a spot and replied, the numbers disagreed, and the reply was dropped — deathPending stays latched, the mech is never Reset, and the pilot is a permanent ghost with zero trace in the log. That is exactly why 6 of 8 field cycles stranded leaving no evidence. Now always-on ([ghost] DROP-ZONE REPLY DISCARDED …) and it prints the direction of the mismatch, which decides the fix: msgDeath < ours = genuinely stale (dropping is correct); msgDeath > ours = our counter is behind and we threw away a live respawn — which points straight at #45 (the death tally does not replicate correctly). Deliberately NOT "recovered" yet: guessing the direction would be a stand-in, and the wrong guess respawns a mech that is still alive.

3. ⚠ LANDMINE — do not convert the two Set_Alarm_Level raw writes to SetSimulationState(). Set_Alarm_Level is an empty stub (btstubs.cpp:87), so the death path's Set_Alarm_Level(this+0x2c, 1) and the reply path's (this+0x2c, 2) are no-ops today. Their values decode perfectly against Player's enum (PLAYER.h:273-279) as DropZoneAcquiredState(1) and VehicleTranslocatedState(2) — DropZoneAcquiredState = Entity::StateCount, and Entity adds no states, so it is 1. That makes "obviously these should be SetSimulationState calls" a very attractive and catastrophic fix: the engine hunt is gated on GetSimulationState() != DropZoneAcquiredState, so setting state 1 on death would stop AssignDropZone from ever being dispatched and make every pilot ghost permanently. Measured: our BTPlayer::simulationState is at 0x24, not 0x2c, and the write leaves it at 0 ([ghost] death raw-write(+0x2c,1): simState 0 -> 0 (hunt gate still open)). In the binary +0x2c is the Simulation-base alarm — the same field the mech side calls graphicAlarm (@0x4ac126: "owner alarm+0x2C -> level 9"). Our layout models that only as a Mech member, so a BTPlayer has nowhere to put it; the stub is harmless until we recover what reads a Player's alarm in the binary (likely a cockpit/HUD respawn indicator). [T1 offsets, T3 purpose]

4. EVERY MP RESPAWN IS A CROSS-MACHINE ROUND TRIP — and that is the single point of failure [T2, two-node bench scratchpad/night6/mp_ghost.sh]. Measured on a 2-node rig with the host:local labels: each player's request was answered by the OTHER machine's DropZone.

mp_a.log: [dzreq] player 2:1 asking for a drop zone ...      <- A's own player asks
mp_b.log: [dz] GRANTED slot 7 to 2:1 death#1 ...             <- B answered it
mp_b.log: [dzreq] player 3:1 asking for a drop zone ...      <- B's own player asks
mp_a.log: [dz] GRANTED slot 0 to 3:1 death#1 ...             <- A answered it

FindGroup("DropZones") iterates every DropZone entity including replicants of remotely-mastered ones and picks the geometrically closest, so a respawn is: my request → an arbitrary peer's DropZone → that peer's reply → back over the wire to me. With five players that is five round trips through arbitrary peers, and one degraded peer can ghost everybody else. That dovetails exactly with the otherwise-unexplained field finding that one machine stopped processing the effect/death-transition stream 57 % into match 1 and never recovered (0 explosions / 0 wreck swaps / 0 burials while the other four logged 4/4/4). A node in that state that owns the closest drop zone cannot answer anyone's respawn — which is what "everybody ghosted" looks like. It also explains why solo is 100 % reliable (the whole handshake is in-process) and why the 2-node bench passes (both nodes healthy, replies land, deathCount agrees).

Revised theory of the ghost (two layers):

• Structural: respawn depends on a cross-machine round trip through a peer that may be lagging, stalled, or holding a stale view of my player. Not drop-zone starvation (slots=8) and not a single lost reply (the hunt retries every 2 s forever).
• Mechanism when the reply does arrive but is stale/mismatched: the silent discard in 2 — the reply's deathCount disagrees and it is dropped without a trace, which couples this to #45.

Two candidate fixes, both needing the next field logs to choose between: (a) Prefer a locally-mastered DropZone when one exists, keeping the handshake in-process and removing the remote dependency entirely — cheap and robust, but it changes which physical pad you land on, so check it against the maps first. (b) Make the reply path tolerant: accept a granted reply whose deathCount is ahead of ours (our counter is behind) instead of discarding it, and never leave deathPending latched on a discard. ⚠ Neither should be guessed at blind — the [ghost] DROP-ZONE REPLY DISCARDED line's mismatch direction decides it in one line of log.

HOW THE BINARY HANDLES IT — deathPending is OUR INVENTION [T1, decisive]

Read BT's own respawn code (it is in the export, even though munga/dropzone.cpp is not) and checked it against the raw binary.

The hunt is faithful. BT's Player::VehicleDeadMessageHandler = FUN_0042db80 (part_003.c:12029) gates on message->deathCount == player->deathCount and player+0x40 != 1 — i.e. simulationState != DropZoneAcquiredState, exactly WinTesla's two gates — then runs the same closest-DropZone search skipping "win*" zones, dispatches AssignDropZone, and re-posts the message to itself on a timer. So the retry loop and the cross-machine hunt are authentic 1995 behaviour, not a WinTesla artifact.

But the latch is not. BT's BTPlayer::VehicleDeadMessageHandler = FUN_004c012c (part_013.c:10504) ends with:

++deathCount;  message->deathCount = deathCount;  --lives;
updateModel |= 1;  alarm(+0x2c, 1);  save dropZoneLocation;
Post(HighPriority, self, message, Now()+delay);
*(param_1 + 0x290) = 0;      // <-- the field our reconstruction calls deathPending

The binary CLEARS that field at the end of every death. Verified against BTL4OPT.EXE itself: +0x290 is written in exactly three places in the whole executable (file_off 0x0b75fb, 0x0bffe3, 0x0c0a05), and all three store a zeroed register — xor ecx,ecx / xor eax,eax / xor edx,edx immediately before. There is no write of 1, or of any non-zero value, to +0x290 anywhere in the binary. Two of the three are constructor/reset sweeps; the middle one is the death handler above (it sits right after the call to Post and add esp,0x14, matching the decompiled tail exactly).

So in 1995 there is no death-pending gate at all. Our btplayer.cpp:505 does deathPending = 1, and six separate sites then have to clear it (:382 :469 :1431 :1442 :1461 :1532) — each one a patch for a different stranding scenario. Gitea #57 and #55 are artifacts of that invention, not of the original design.

And that is what makes a ghost PERMANENT. In BT a failed respawn is harmless: the 2-second re-post keeps hunting, and the next death starts a clean cycle. In ours the first failure latches the pilot — every later death hits the dedup and is SWALLOWED ("deaths so far 1" forever), so the pilot can never recover for the rest of the mission. That is precisely the field signature: 8 death cycles, 6 stranded, none of them ever recovering.

Proposed fix (faithful, and it removes the whole bug class): match the binary — clear the field at the end of the death handler instead of latching it, and drop the dedup gate that depends on it. ⚠ Six sites currently rely on that latch's existence, and the SWALLOWED dedup is load-bearing for the "one death, one cycle" behaviour, so this wants a deliberate two-node bench rather than a late-night edit. Caveat on the identification: +0x290 ↔ deathPending rests on our own reconstruction's mapping (our compiled BTPlayer is only 0x28c bytes, so the offsets cannot match directly) — but the behavioural evidence (BT's death handler clears it; nothing in the binary ever sets it) is read straight from the executable.

What the instrumentation already killed

slots=8 (measured, [dz] POOL). An 8-slot pool cannot be cooldown-starved by 5 players (at most 5 slots on cooldown at once), so the shared dropzone=one in tools/eggmodel.py:42 is not the bottleneck. Suspicion moves to the handshake: a lost reply, or a deathCount mismatch — the engine gates the hunt on message->deathCount == deathCount and the DropZone's resend safety net is keyed on lastDeathCount == message->deathCount, so one mismatch breaks both silently (cf. #45).

Instrumentation shipped (ba6756c) — always-on, no env var needed

line	meaning
[dz] POOL name= slots= downTime= proximityBlock=	once per mission: the pool size
[dz] GRANTED slot N to entity E death#D waited=Xs	one per respawn; waited is the headline; tags (GHOST RECOVERED) past 15 s
[dz] STALL / [dz] GHOST LIKELY	escalating, rate-limited per waiter; names why each slot is busy (age + last user) — this discriminates cooldown saturation from proximity blocking
[dzreq] player P asking ... try=N msgDeath= ourDeathCount=	requester half; flags *** MISMATCH ***
BT_DROPZONE_LOG=1 adds a verbose per-request slot dump for bench work.

Next actions, in order

1. Recover BT's 1995 drop-zone policy from content/BTL4OPT.EXE (decomp gap). The key question: when no slot was available, did BT also repost forever and never reply? If yes, the stall is authentic and our divergence is elsewhere; if no, implement what BT did.
2. Read the next field logs' [dz]/[dzreq] lines — they answer cooldown-vs-proximity and lost-reply-vs-deathCount-mismatch directly.
3. Ask the testers one question that discriminates a second defect: did a flaming mover stay visible for more than ~20 s? If yes, the sink tick is not running for that entity (updater early-return, or a render-tree rebuild resetting wreckAge) — a second bug on top of this one.
4. Separately filed/needed: one machine stopped processing the effect/death-transition stream 57 % into a match and never recovered (0 explosions / 0 wreck swaps / 0 burials while the other four logged 4/4/4) — that machine was playing a stale world, and it is why that player reported the whole "desync cluster".

The desync cluster is four separate things, not one bug [T2]

• Throttle reads zero — local, and present in the good sessions too. ⚠ Unresolved nuance: the internal thr field is 0 in every field [drive] sample while mechs moved, but reads 0.8 in solo benches; whether that is the same thing the player saw on his gauge is not established.
• No weapon audio — local audio-source starvation caused by the map's ambient emitters (arena2_morning ≈ 10.6 effects/s × ~2.5 voices pins the 256-source pool); thousands of ACQUIRE FAILED ... pool is exhausted. Unrelated to crits or the network.
• Weapons fire without damage — replication staleness. The shooter resolves damage and awards its own score locally, so a shooter always sees hits even when the victim applies nothing.
• Other mechs lagging/misplaced — the same stale-world condition as item 4 above.

Why tonight and not before

Nothing in 642 broke multiplayer. Critical hits landing for real (first time) made deaths far more frequent, and every death is a chance for this stall — so a rare race became a common one. Fewer players ⇒ fewer deaths ⇒ clean matches, which is exactly what the testers observed.