diff --git a/context/operator-console.md b/context/operator-console.md index b8e8b1e..9a7ef20 100644 --- a/context/operator-console.md +++ b/context/operator-console.md @@ -6,9 +6,9 @@ source_sections: "tools/btoperator.py; tools/btconsole.py (module docstring is a related_topics: [multiplayer, steam-networking, content-archives, pod-hardware, build-and-run] key_terms: [EGG, master, replicant, relay, seat, roster] open_questions: - - "UDP endpoint map is keyed on the sender's self-declared fromHost with no check against its TCP peer -- a zombie pod could steal a live pod's downstream (found 2026-07-26, not yet fixed)" - - "egg-ACK detection is a fixed-offset parse of one recv() with no reassembly; no frame resync after a malformed frame" - - "remote-operator mode can never enable the LAUNCH button (_refresh_pod_status requires a local console_proc)" + - "MESH mode's operator buttons are inert: btconsole.main has no stdin reader, so LAUNCH/END/Re-arm are logged as sent and discarded (relay mode is unaffected). Left as-is deliberately -- mesh self-launches." + - "_log_launch_readiness can print its 'the mission will STALL -- Stop, reduce the roster' warning on a perfectly healthy launch-with-whoever, because it counts by_host against the remapped roster" + - "A straggler's late round-N FIN is indistinguishable on the wire from a pod dying mid-load, so it still triggers a ROUND ABORT (recoverable now via Re-arm, but the misclassification stands)" --- # The Operator Console + Relay @@ -205,6 +205,33 @@ ever ran live. Regression guards: `scratchpad/test_relay_rearm.py` cases 5b/5c. Half-open sockets are TCP keepalive's job; this deadline is only the backstop for platforms where the keepalive tunables do not take. +### Console-protocol reassembly [FIXED 2026-07-26] + +The pod's **egg ACK is the launch gate's only signal**, and it used to be detected by parsing a +single `recv()` at fixed offset 0 with a `len(data) >= 24` test. Two ways that lost an ACK on a real +network (loopback hid both): the 28-byte ACK arriving **split** (16 bytes then 12) was dropped by +the length test and never looked at again, and two **coalesced** messages meant only the first was +read. A missed ACK means that seat never counts, so the round can never be released — a silent, +unrecoverable wedge. `_console_read` now buffers per connection and walks every complete frame +(`16 + messageLength`), and an implausible length **drops the connection with a reason** rather than +mis-reading that pod for the rest of the night (a stream protocol cannot be resynced by guessing). +Bounds: `CONSOLE_MSG_MAX = 65536`, `CONSOLE_INBUF_MAX = 1 MiB`. + +### UDP endpoint anti-hijack [FIXED 2026-07-26] + +`from_host` in a UDP envelope is the sender's **own claim**, and the relay used it directly to +refresh that host's downstream endpoint. So any datagram claiming host N silently **stole host N's +traffic** — a zombie pod from a previous round, a stale NAT mapping, or anyone who guessed a host +id — and the victim simply stopped receiving on a channel that still looked healthy. The claim is +now bound to the identity we actually authenticated: the **IP of that host's live TCP game +connection**. The **port is deliberately not checked** (it moves on a NAT rebind, which is the whole +reason the endpoint map refreshes per datagram), and a genuine IP change cannot happen without the +TCP connection breaking and re-registering, so this never rejects a legitimate pod. Rejections are +counted (`udp_spoofed`) and logged once per offending `(host, IP)` pair. + +Limitation worth knowing: two pods on the **same machine** share an IP, so this cannot tell them +apart — same-machine trust is assumed. + Also: every pod-facing send used to flip the socket to blocking with **no timeout** on the relay's **single-threaded** selector loop, so one wedged peer could freeze the whole relay (no launch tick, no UDP fan-out, no control channel). All sends now go through `_send_all_guarded` @@ -249,11 +276,14 @@ doing nothing: LAUNCH / END MISSION / Re-arm are silently discarded in mesh mode. `Apply mission settings` is likewise inert there: the mesh console reads the egg bytes once at startup. Mesh also never prints "WAITING FOR OPERATOR", so the LAUNCH button cannot even enable. -- **REMOTE-operator mode: LAUNCH and END MISSION can never enable** — both enable conditions - require `console_proc is not None`, and the remote path never sets it. The roster lights are also - permanently empty (`SessionMonitor(True, [])` → no tags → `seated` always 0), and - `Launch local instances` is refused by the same `console_proc` guard even though the code below - it deliberately builds `BT_RELAY` from the remote host. +- **REMOTE-operator mode — FIXED 2026-07-26.** LAUNCH and END MISSION could never enable (both + conditions required `console_proc is not None`, which the remote path never sets), the roster + lights were permanently empty (`SessionMonitor(True, [])` → no tags → `seated` always 0), and + `Launch local instances` was refused by the same guard even though the code below it already + built `BT_RELAY` from the remote host. Now: all three enable conditions accept **either** channel + (`console_proc` *or* `remote_link`), and the monitor **adopts the roster** from the relay's + `roster: N pilot(s) -> hostIDs [...]: [...]` line, which the relay replays to every newly AUTHed + operator — so a remote operator gets real pilot lights and a real seat count. - **`Start session` silently overwrites the egg on disk** (including re-rasterized callsigns), with no prompt and no backup. - **`Apply` writes only the six `[mission]` keys** (map/time/weather/scenario/temperature/length). diff --git a/docs/OPERATOR_GUIDE.md b/docs/OPERATOR_GUIDE.md index 709c26d..83c8213 100644 --- a/docs/OPERATOR_GUIDE.md +++ b/docs/OPERATOR_GUIDE.md @@ -118,9 +118,11 @@ Read this once; each of these is a control that looks live but does nothing in t its rounds by itself once the pods have the mission. **If you want to control launches, use Relay mode.** -**Remote relay mode** (driving another machine's relay): LAUNCH and END MISSION **can never -enable**, the pilot lights stay blank, and `Launch local instances` is refused. You get Apply, -Re-arm and Stop. Treat remote mode as monitoring-plus-settings, not as a full console. +**Remote relay mode** (driving another machine's relay) is now a full console — LAUNCH, END MISSION, +Re-arm, Apply and Launch-local all work, and the pilot lights populate from the relay's own roster +when you connect. Two things still differ: **Stop Session only disconnects you** (the remote relay +keeps running, which is usually what you want), and **Restart Session** likewise just reconnects the +control link rather than restarting that relay. **Other sharp edges:** - **Start Session overwrites your egg file** on disk, without asking. Keep a copy of any mission you diff --git a/scratchpad/test_relay_net.py b/scratchpad/test_relay_net.py new file mode 100644 index 0000000..0613787 --- /dev/null +++ b/scratchpad/test_relay_net.py @@ -0,0 +1,183 @@ +"""Tests for the three relay/console hardening fixes (2026-07-26). + + 1. UDP anti-hijack: a datagram may only refresh the endpoint of the host whose + TCP connection it actually shares an IP with. + 2. Console reassembly: the pod's egg-ACK must be found even when it arrives + split across recv() calls, or coalesced with another message. + 3. Remote-operator roster adoption: the GUI monitor learns the roster from the + relay's replayed log line, so a remote operator gets pilot lights and a + working LAUNCH button. + +No sockets, no game, no relay process -- everything is driven through fakes. +""" +import os +import struct +import sys + +REPO = r"C:\git\bt411" +sys.path.insert(0, os.path.join(REPO, "tools")) +os.chdir(os.path.join(REPO, "content")) + +import btconsole # noqa: E402 + +EGG = os.path.join(REPO, "content", "FOGDAY.EGG") +fails = [] + + +def check(label, ok, extra=""): + print(" %-56s %s %s" % (label, "OK" if ok else "*** FAIL ***", extra)) + if not ok: + fails.append(label) + + +def new_relay(): + return btconsole.Relay(1500, EGG, "127.0.0.1", 0, manual_launch=True) + + +class FakeGameConn: + """A TCP-registered pod, as by_host holds it.""" + def __init__(self, ip, port=5000, host_id=2): + self.addr = (ip, port) + self.host_id = host_id + self.last_seen = btconsole.time.time() + self.buf = b"" + + +class FakeUdp: + """Serves queued datagrams to _udp_read, then blocks like a real socket.""" + def __init__(self, datagrams): + self.queue = list(datagrams) + self.sent = [] + def recvfrom(self, _n): + if not self.queue: + raise BlockingIOError() + return self.queue.pop(0) + def sendto(self, data, endpoint): + self.sent.append((data, endpoint)) + + +def udp_frame(route, from_host, seq=1, payload=b"\0\0\0\0"): + return struct.pack(btconsole.ENV_FMT_UDP, route, from_host, seq) + payload + + +print("=== 1. UDP ANTI-HIJACK ===") +r = new_relay() +r.by_host = {2: FakeGameConn("10.0.0.5", 5000, 2)} +r.udp_endpoint = {2: ("10.0.0.5", 6000)} +r.game_conns = list(r.by_host.values()) + +# (a) a stranger claiming host 2 must NOT move host 2's endpoint +r.udp_sock = FakeUdp([(udp_frame(btconsole.ROUTE_BCAST, 2), ("66.66.66.66", 7777))]) +r._udp_read() +check("spoofed datagram did not move the endpoint", + r.udp_endpoint[2] == ("10.0.0.5", 6000), str(r.udp_endpoint[2])) +check("spoof counted in stats", r.stats.get("udp_spoofed", 0) == 1) + +# (b) the REAL pod on a NEW port (NAT rebind) must still be honoured +r.udp_sock = FakeUdp([(udp_frame(btconsole.ROUTE_BCAST, 2), ("10.0.0.5", 9999))]) +r._udp_read() +check("NAT rebind (same IP, new port) accepted", + r.udp_endpoint[2] == ("10.0.0.5", 9999), str(r.udp_endpoint[2])) + +# (c) an unregistered host id is still dropped outright +before = dict(r.udp_endpoint) +r.udp_sock = FakeUdp([(udp_frame(btconsole.ROUTE_BCAST, 99), ("10.0.0.9", 1))]) +r._udp_read() +check("unregistered host id dropped", r.udp_endpoint == before) + + +print("\n=== 2. CONSOLE REASSEMBLY (the egg-ACK must never be missed) ===") + +ACK = (struct.pack(" 16 + 12 = 28 bytes +assert len(ACK) == 28 + + +class FakeConsoleSock: + def __init__(self, reads): + self.reads = list(reads) + def recv(self, _n): + return self.reads.pop(0) if self.reads else b"" + + +def acking_relay(): + r = new_relay() + r.console_conns = [] + r._check_launch_gate = lambda: None # isolate: we only test detection + conn = btconsole.RelayConsoleConn(FakeConsoleSock([]), ("10.0.0.5", 5001)) + r.console_conns.append(conn) + return r, conn + + +# (a) the ACK arriving in ONE piece (the old code handled only this) +r, conn = acking_relay() +conn.sock.reads = [ACK] +r._console_read(conn) +check("whole ACK detected", conn.acked is True) + +# (b) SPLIT across two reads -- the old fixed-offset parse dropped this silently, +# and a missed ACK means that seat never counts toward the launch gate. +r, conn = acking_relay() +conn.sock.reads = [ACK[:16], ACK[16:]] +r._console_read(conn) +check("split ACK: not detected on the partial half", conn.acked is False) +r._console_read(conn) +check("split ACK detected once the rest arrives", conn.acked is True) + +# (c) split at an even nastier boundary (mid-header) +r, conn = acking_relay() +conn.sock.reads = [ACK[:5], ACK[5:20], ACK[20:]] +for _ in range(3): + r._console_read(conn) +check("ACK split three ways still detected", conn.acked is True) + +# (d) two messages COALESCED into one read -- the old code read only the first +r, conn = acking_relay() +other = (struct.pack(" skip, do not fail + print(" (skipped: cannot import btoperator: %r)" % (e,)) +else: + mon = btoperator.SessionMonitor(True, []) # how remote mode starts + check("remote monitor starts with no tags", mon.tags == []) + line = ("22:27:39 [relay] roster: 2 pilot(s) -> hostIDs [2, 3]: " + "['10.99.0.1:1502', '10.99.0.2:1502']") + changed = mon.feed(line) + check("roster line adopted", mon.tags == ["10.99.0.1:1502", "10.99.0.2:1502"], + str(mon.tags)) + check("feed reported a change", changed is True) + check("every tag got a state", len(mon.state) == 2) + + # a seat line now lands on a known tag, so the pilot light works + mon.feed("[relay] PLAYER 1 SEATED (host 2) tag='10.99.0.1:1502' " + "callsign='SAURON' mech='lok2'") + check("SEATED now maps onto an adopted tag", + mon.state["10.99.0.1:1502"] == btoperator.ST_SEATED) + seated = sum(1 for s in mon.state.values() + if s in (btoperator.ST_SEATED, btoperator.ST_REGISTERED, + btoperator.ST_READY)) + check("seated count is no longer stuck at 0", seated == 1, str(seated)) + + # re-feeding the same roster must not churn state + mon.feed(line) + check("re-feeding the roster preserves known state", + mon.state["10.99.0.1:1502"] == btoperator.ST_SEATED) + +print("\n%s" % ("ALL CHECKS PASSED" if not fails else "FAILURES: %s" % fails)) +sys.exit(1 if fails else 0) diff --git a/tools/btconsole.py b/tools/btconsole.py index 6f47cc1..fc6aa50 100644 --- a/tools/btconsole.py +++ b/tools/btconsole.py @@ -308,6 +308,13 @@ VALID_CAMO_COLORS = {"Black", "Brown", "Crimson", "Green", "Grey", "Tan", # One authenticated operator at a time -- a new AUTH displaces the old. # A slow/stalled operator NEVER blocks the relay: writes are buffered per # connection and the connection is dropped if the buffer overflows. +# Console-protocol reassembly bounds (see _console_read). A pod message is a +# 16-byte header + messageLength bytes; the egg chunk is the biggest thing the +# protocol carries at 16+1024, so anything wildly past that means the stream is +# no longer frame-aligned. +CONSOLE_MSG_MAX = 65536 +CONSOLE_INBUF_MAX = 1 << 20 + CONTROL_PORT_OFFSET = 7 # control port = console port + 7 CONTROL_AUTH_TIMEOUT = 10.0 # unauthenticated sockets die after this CONTROL_OUTBUF_MAX = 262144 # slow-reader cutoff (bytes) @@ -415,6 +422,7 @@ class RelayConsoleConn: self.sock = sock self.addr = addr self.egg_sent = False + self.buf = b"" # inbound reassembly (see _console_read) self.acked = False # pod sent AcknowledgeEggFile -- a REAL pod. # LIVENESS (2026-07-26): a pod whose machine sleeps, whose Wi-Fi drops, # or which is killed without a FIN leaves a HALF-OPEN socket. With no @@ -944,11 +952,46 @@ class Relay: flush=True) self._drop_console(conn) return - if len(data) >= 24: - (clid, _gid, fh, _ts) = struct.unpack_from("console clientID={clid} fromHost={fh} msgID={mid} " - f"len={mlen}", flush=True) + # + # REASSEMBLE, then walk EVERY complete message. This used to parse one + # recv() at a fixed offset 0 and look at nothing else, which had two + # failure modes on a real network (loopback hid both): + # * the pod's 28-byte ACK arriving SPLIT (say 16 bytes then 12) was + # dropped by the `len(data) >= 24` test and never seen again -- and a + # missed ACK means that seat never counts toward the launch gate, so + # the round can never be released. A silent, unrecoverable wedge. + # * two messages coalescing into one recv() -- only the first was read. + # A frame is a 16-byte header + `messageLength` bytes (so the egg's 1040 + # = 16 + 1024, and the ACK's 28 = 16 + 12). + # + conn.buf += data + if len(conn.buf) > CONSOLE_INBUF_MAX: + print(f"[relay] console conn {conn.addr[0]}:{conn.addr[1]} dropped: " + f"inbound buffer over {CONSOLE_INBUF_MAX} bytes (desynced)", + flush=True) + self._drop_console(conn) + return + while len(conn.buf) >= 24: + (clid, _gid, fh, _ts) = struct.unpack_from(" CONSOLE_MSG_MAX: + print(f"[relay] console conn {conn.addr[0]}:{conn.addr[1]} " + f"dropped: implausible messageLength {mlen} " + f"(clientID={clid} msgID={mid}) -- stream desynced", + flush=True) + self._drop_console(conn) + return + frame = 16 + mlen + if len(conn.buf) < frame: + break # partial: wait for the rest + conn.buf = conn.buf[frame:] + print(f"[relay] pod->console clientID={clid} fromHost={fh} " + f"msgID={mid} len={mlen}", flush=True) # AcknowledgeEggFile (clientID=0 NetworkManager, msgID=4): this # connection is a REAL pod -- count it toward the launch gate. if clid == 0 and mid == 4 and not conn.acked: @@ -1546,12 +1589,39 @@ class Relay: route, from_host, _seq = struct.unpack_from(ENV_FMT_UDP, data, 0) if from_host not in self.by_host: continue # not TCP-registered: drop + live = self.by_host.get(from_host) + # + # ANTI-HIJACK (2026-07-26). `from_host` is the sender's OWN claim + # about who it is, and the next line rewrites that host's downstream + # endpoint. Unchecked, ANY datagram claiming host N silently steals + # host N's traffic -- a zombie pod process from a previous round, a + # stale NAT mapping, or a third party who guessed a host id. The + # victim then just stops receiving, on a channel that looks healthy. + # + # Bind the claim to the identity we actually authenticated: the IP of + # that host's live TCP game connection. The PORT is deliberately not + # checked -- it moves on a NAT rebind, which is the whole reason the + # endpoint map is refreshed per datagram. A genuine IP change cannot + # happen without the TCP connection breaking and being re-registered + # with the new address, so this never rejects a legitimate pod. + # + if live is not None and endpoint[0] != live.addr[0]: + self.stats["udp_spoofed"] = self.stats.get("udp_spoofed", 0) + 1 + if not hasattr(self, "_udp_spoof_warned"): + self._udp_spoof_warned = set() + key = (from_host, endpoint[0]) + if key not in self._udp_spoof_warned: + self._udp_spoof_warned.add(key) + print(f"[relay] UDP from {endpoint[0]}:{endpoint[1]} CLAIMS " + f"host {from_host}, whose TCP is {live.addr[0]} -- " + f"DROPPED (it would have stolen that pod's downstream)", + flush=True) + continue self.stats["udp_rx"] += 1 # NAT-rebind tolerant: every datagram refreshes the endpoint map. self.udp_endpoint[from_host] = endpoint # ... and counts as LIVENESS. Without this, a pod streaming updates # over UDP while its TCP sits idle looks "silent" to the reaper. - live = self.by_host.get(from_host) if live is not None: live.last_seen = time.time() if route == ROUTE_HELLO: diff --git a/tools/btoperator.py b/tools/btoperator.py index 94d256c..15e8ef6 100644 --- a/tools/btoperator.py +++ b/tools/btoperator.py @@ -75,6 +75,12 @@ class SessionMonitor: RE_RELAY_HELD = re.compile( r"launch HELD -- still loading: (.+) \((\d+)/(\d+) ready\)") RE_RELAY_RESET = re.compile(r"round RESET") + # The relay announces its roster at startup and replays that line to a + # newly AUTHed operator. A REMOTE operator has no egg loaded, so this is + # the only way it can learn the tags -- without them `state` stayed empty, + # every pilot light was invisible and `seated` was always 0. + RE_RELAY_ROSTER = re.compile( + r"roster: (\d+) pilot\(s\) -> hostIDs \[[^\]]*\]: \[(.*)\]") # A ROUND HAS ENDED / THE LAUNCHER RE-ARMED. Before these, `launched` # could only be cleared by "WAITING FOR OPERATOR" (needs every seat of the # last round to re-ACK) or "round RESET" (needs every pod gone). A games @@ -110,6 +116,14 @@ class SessionMonitor: if m: self.eggs = int(m.group(1)) changed = True + m = self.RE_RELAY_ROSTER.search(line) + if m: + tags = re.findall(r"'([^']*)'", m.group(2)) + if tags and tags != self.tags: + # keep any state we already know for a tag that survives + self.tags = tags + self.state = {t: self.state.get(t, ST_IDLE) for t in tags} + changed = True if self.RE_RELAY_WAITOP.search(line): # WAITING FOR OPERATOR = pods are ready to launch. Reset # `launched` so the LAUNCH button re-enables -- this fires @@ -1032,8 +1046,10 @@ class Operator(QMainWindow): if self.monitor.launched: head = "LAUNCHED — mission running" - if (self.console_proc and not self.end_btn.isEnabled() - and not getattr(self, "end_sent", False)): + if (not self.end_btn.isEnabled() + and not getattr(self, "end_sent", False) + and (self.console_proc is not None + or getattr(self, "remote_link", None) is not None)): self.end_btn.setEnabled(True) elif self.monitor.held_status: head = "STAGING — " + self.monitor.held_status @@ -1068,10 +1084,15 @@ class Operator(QMainWindow): combo = self.table.cellWidget(r, 2) if mech and combo is not None and combo.currentText() != mech: combo.setCurrentText(mech) + # A REMOTE operator has no console_proc -- its command channel is the + # control socket -- so requiring one made LAUNCH (and END MISSION below) + # permanently dead in remote mode. Accept either channel. + have_relay = (self.console_proc is not None + or getattr(self, "remote_link", None) is not None) self.launch_btn.setEnabled( (self.monitor.ready or (self.monitor.relay_mode and seated > 0)) and not self.monitor.launched - and self.console_proc is not None) + and have_relay) if self.monitor.stats: self.stats_label.setText(self.monitor.stats) @@ -1090,7 +1111,11 @@ class Operator(QMainWindow): QMessageBox.warning(self, "Launch", "btl4.exe not found:\n" + exe) return relay = self.mode.currentIndex() == 0 - if relay and self.console_proc is None: + # Remote mode has no console_proc but IS a live session -- and the env + # build below already points BT_RELAY at remote_link.host, so this guard + # was the only thing making the feature unreachable there. + if (relay and self.console_proc is None + and getattr(self, "remote_link", None) is None): QMessageBox.warning( self, "Launch", "The session is not running -- press Start Session first\n"