MP: console-death freeze NOT REPRODUCIBLE + fix the latent game-listener-close bug (task #50)
Investigated the "console death froze peer replication" open question (2026-07-14)
with a direct 2-node repro (BT_REPL_LOG on a circling peer, affinity-pinned):
- Clean relay kill -> replicant kept circling smoothly through the kill, NO
disconnect logged, node survived. Replication did NOT freeze.
- Deliberately STUCK console (scratchpad/btconsole_stuck.py: connect, start
mission, then stop recv() while holding the socket OPEN = the exact
"receive pad full -> close never seen" mode hypothesized) -> replicant kept
moving the whole run. Replication did NOT freeze.
Conclusion: peer replication is INDEPENDENT of the console (pods replicate
peer-to-peer over the GAME socket; the console is a separate egg/mission/status
channel). The original freeze was a transient -- plausibly the same single-box
packet-jitter/CPU-contention artifact root-caused in 49d73dc -- or was fixed by
later MP work.
FIX (the one real bug found): L4NetworkManager::HostDisconnectedMessageHandler's
ConsoleHostType branch closed gameListenerSocket (the GAME listener) on a CONSOLE
disconnect -- a naming bug (the comment + commented-out OpenConnection intended a
CONSOLE re-listen, which CreateConsoleHost already does). Removed. Harmless to
established peer sockets (why a live match survives console loss) but it would
have blocked a NEW peer from joining after a console cycle. Verified: 2-node
still connects (All connections completed!) + the peer circles.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
arcattack
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
d1ce99402f
commit
bc4d6e5597
+18
-13
@@ -558,19 +558,24 @@ register. ⚠ The audit also flags the damage-economy item as SELF-CONTRADICTOR
|
||||
numbers no longer point at that code).
|
||||
|
||||
## Multiplayer (Phase 7 / P6)
|
||||
- **OPEN: console death froze peer replication BOTH ways (2026-07-14) [T4 mechanism].** Live 2-node
|
||||
session: the btconsole.py emulator process died mid-match (cause unknown — its recv loop only
|
||||
caught socket.timeout; any OSError killed threads silently; NOW hardened + `-u` + console.log) and
|
||||
peer↔peer update replication froze in BOTH directions at ~that moment (each node's replicant held a
|
||||
frozen stale velocity; own sims + rendering fine). The engine NEVER logged a console disconnect —
|
||||
though `CheckBuffers` recv==0 dispatches `HostDisconnectedMessage` and the loop keeps servicing
|
||||
other hosts — so either the console host's receive pad was full (recv skipped → close never seen)
|
||||
or the dispatch doesn't reach `HostDisconnectedMessageHandler` in the port (routing gap). ⚠ Also
|
||||
found (unfired, latent): that handler's ConsoleHostType branch closes **`gameListenerSocket`** (the
|
||||
GAME listener) where the comment says it re-posts a CONSOLE listen (L4NET.CPP:969) — suspect
|
||||
`consoleListenerSocket` was intended. Diagnose on recurrence: console.log (death reason) +
|
||||
whether `[repltrn]` velocities freeze at console death. Why peer replication depends on the
|
||||
console connection at all is the core unknown.
|
||||
- **✅ Console-death peer-replication freeze — NOT REPRODUCIBLE + the one real bug FIXED
|
||||
(2026-07-15) [T2].** The 2026-07-14 report (btconsole.py died mid-match → peer↔peer replication
|
||||
froze both ways) did NOT reproduce in the current build under EITHER failure mode, tested directly
|
||||
(BT_REPL_LOG on a circling peer, affinity-pinned): (1) a clean relay kill — the replicant kept
|
||||
circling smoothly through the kill, NO disconnect logged, node survived; (2) a deliberately STUCK
|
||||
console (`scratchpad/btconsole_stuck.py` — connects, starts the mission, then stops recv()'ing while
|
||||
holding the socket OPEN = the exact "receive pad full → close never seen" mode hypothesized) — the
|
||||
replicant kept moving the entire run. So **peer replication is INDEPENDENT of the console**: the
|
||||
pods replicate peer-to-peer over the GAME socket; the console is a separate egg/mission/status
|
||||
channel. The original freeze was a transient (plausibly the same single-box packet-jitter /
|
||||
CPU-contention artifact root-caused 2026-07-15, `49d73dc`) or was fixed by later MP work.
|
||||
**The flagged latent bug IS real and is now FIXED:** `HostDisconnectedMessageHandler`'s
|
||||
ConsoleHostType branch closed **`gameListenerSocket`** (the GAME listener) on a CONSOLE disconnect —
|
||||
a naming bug; the comment + the commented-out `OpenConnection` intended a CONSOLE re-listen, which
|
||||
`CreateConsoleHost()` already does. Removed (L4NET.CPP:969). Harmless to established peer sockets
|
||||
(why a live match keeps replicating through console loss), but it would have blocked a NEW peer
|
||||
from joining after a console cycle. Verified: 2-node still connects (`All connections completed!`)
|
||||
+ the peer circles. Diag harness retained: `scratchpad/btconsole_stuck.py`.
|
||||
- ✅ Cross-pod COMBAT — DONE (tasks #46/#47: replicant targeting + damage rerouted to the owning
|
||||
master, full cross-pod kill verified). ✅ Replicant GAIT animation — DONE (task #50: replicant
|
||||
derives speedDemand from the replicated velocity, full stand→walk→run lifecycle verified).
|
||||
|
||||
@@ -966,9 +966,15 @@ void L4NetworkManager::HostDisconnectedMessageHandler(HostDisconnectedMessage* H
|
||||
//
|
||||
// Post a listen for a console so it can reconnect if it wants to.
|
||||
//
|
||||
shutdown(gameListenerSocket, SD_BOTH);
|
||||
closesocket(gameListenerSocket);
|
||||
gameListenerSocket = NULL;
|
||||
// FIX (task #50, 2026-07-15): this used to close the GAME listener here
|
||||
// (shutdown/closesocket(gameListenerSocket)) on a CONSOLE disconnect -- a
|
||||
// naming bug: the comment above + the commented-out OpenConnection below
|
||||
// intended to re-post a CONSOLE listen, which CreateConsoleHost() (below)
|
||||
// already does. Closing the game listener on console loss needlessly
|
||||
// bounced the game-accept path (harmless to already-established peer
|
||||
// sockets -- which is why a live 2-node match keeps replicating through a
|
||||
// console loss -- but wrong, and it would have blocked a NEW peer from
|
||||
// joining after a console cycle). Removed; CreateConsoleHost re-listens.
|
||||
//unsigned long console_socket = OpenConnection(
|
||||
// NETNUB_TCP_LISTEN,
|
||||
// ((L4Application *)application)->GetNetworkCommonFlatAddress(), // Local port
|
||||
|
||||
@@ -0,0 +1,95 @@
|
||||
#!/usr/bin/env python3
|
||||
"""btconsole_stuck.py -- a DELIBERATELY-STUCK console for reproducing the
|
||||
'console death froze peer replication' open question (task #50 / open-questions.md).
|
||||
|
||||
It behaves like tools/btconsole.py (connect, stream egg, send RunMission x2, drain
|
||||
the pod->console stream) BUT after STUCK_AFTER seconds it STOPS draining while
|
||||
holding the socket OPEN -- i.e. it stops calling recv(). This is the authentic
|
||||
'receive pad full -> close never seen' failure mode (NOT a clean disconnect): the
|
||||
pod keeps queueing pod->console packets, the TCP window closes, and we watch
|
||||
whether the pod's CheckBuffers loop stalls the GAME peers behind the wedged
|
||||
console host.
|
||||
|
||||
Usage: python btconsole_stuck.py <EGG> host:port host:port [STUCK_AFTER_secs]
|
||||
Env: none. STUCK_AFTER default 12s after the 2nd RunMission.
|
||||
"""
|
||||
import socket, struct, sys, time, threading, os
|
||||
|
||||
CHUNK = 1000
|
||||
PKT_FMT_HDR = "<iiii"
|
||||
PKT_FMT_MSG = "<IiIiii"
|
||||
MESSAGE_LENGTH = 1024
|
||||
EGG_MESSAGE_ID = 3
|
||||
RELIABLE_FLAG = 1
|
||||
CONSOLE_HOST_ID = 1
|
||||
APPLICATION_CLIENT_ID = 4
|
||||
RUN_MISSION_MESSAGE_ID = 5
|
||||
LAUNCH_SETTLE_SECONDS = 20.0
|
||||
LAUNCH_STEP_SECONDS = 4.0
|
||||
STUCK_AFTER = float(sys.argv[4]) if len(sys.argv) > 4 else 12.0
|
||||
|
||||
def run_mission_packet():
|
||||
pkt = struct.pack(PKT_FMT_HDR, APPLICATION_CLIENT_ID, 0, CONSOLE_HOST_ID, 0)
|
||||
pkt += struct.pack("<IiI", 12, RUN_MISSION_MESSAGE_ID, RELIABLE_FLAG)
|
||||
return pkt
|
||||
|
||||
def egg_wire_bytes(b):
|
||||
return b"".join(l + b"\0" for l in b.replace(b"\r\n", b"\n").split(b"\n"))
|
||||
|
||||
def egg_packets(egg_bytes):
|
||||
total = len(egg_bytes); seq = 0; off = 0
|
||||
while off < total:
|
||||
chunk = egg_bytes[off:off+CHUNK]
|
||||
pkt = struct.pack(PKT_FMT_HDR, 0, 0, CONSOLE_HOST_ID, 0)
|
||||
pkt += struct.pack(PKT_FMT_MSG, MESSAGE_LENGTH, EGG_MESSAGE_ID, RELIABLE_FLAG, seq, total, len(chunk))
|
||||
pkt += chunk.ljust(CHUNK, b"\0")
|
||||
yield pkt
|
||||
off += len(chunk); seq += 1
|
||||
|
||||
def serve_pod(hostport, egg_bytes):
|
||||
host, port = hostport.rsplit(":", 1); port = int(port); name = f"{host}:{port}"
|
||||
while True:
|
||||
try:
|
||||
s = socket.create_connection((host, port), timeout=5); break
|
||||
except OSError as e:
|
||||
print(f"[{name}] waiting for pod ({e})", flush=True); time.sleep(1)
|
||||
print(f"[{name}] connected; streaming egg ({len(egg_bytes)} bytes)", flush=True)
|
||||
for pkt in egg_packets(egg_bytes):
|
||||
s.sendall(pkt)
|
||||
print(f"[{name}] egg sent; launch in {LAUNCH_SETTLE_SECONDS}s", flush=True)
|
||||
s.settimeout(2.0)
|
||||
launch_at = time.time() + LAUNCH_SETTLE_SECONDS
|
||||
launches_sent = 0
|
||||
stuck_at = None
|
||||
while True:
|
||||
if launches_sent < 2 and time.time() >= launch_at:
|
||||
s.sendall(run_mission_packet()); launches_sent += 1
|
||||
launch_at = time.time() + LAUNCH_STEP_SECONDS
|
||||
print(f"[{name}] RunMission #{launches_sent} sent", flush=True)
|
||||
if launches_sent == 2:
|
||||
stuck_at = time.time() + STUCK_AFTER
|
||||
# *** THE FAILURE: once mission is running, stop draining after STUCK_AFTER ***
|
||||
if stuck_at is not None and time.time() >= stuck_at:
|
||||
print(f"[{name}] *** GOING STUCK: socket stays OPEN, no more recv() ***", flush=True)
|
||||
while True:
|
||||
time.sleep(60) # hold the socket open, never drain it again
|
||||
try:
|
||||
data = s.recv(4096)
|
||||
if not data:
|
||||
print(f"[{name}] pod closed the console socket", flush=True); return
|
||||
except socket.timeout:
|
||||
continue
|
||||
except OSError as e:
|
||||
print(f"[{name}] socket error {e!r}", flush=True); return
|
||||
|
||||
def main():
|
||||
egg_path = sys.argv[1]
|
||||
pods = sys.argv[2:4]
|
||||
with open(egg_path, "rb") as f:
|
||||
egg = egg_wire_bytes(f.read())
|
||||
ts = [threading.Thread(target=serve_pod, args=(p, egg), daemon=True) for p in pods]
|
||||
for t in ts: t.start()
|
||||
while True: time.sleep(1)
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user