relay: LAUNCH is never silently inert again -- fix the post-round wedge (operator report)
THE SYMPTOM: "after a game ends I push LAUNCH and nothing happens until I reset
the session entirely or reboot the console." The operator also observed the
mirror image, which is the tell: with a STABLE group, relaunching worked fine
several times in a row.
ROOT CAUSE. Two gates, both all-or-nothing, and a UI latch that agreed with
them:
* btconsole.py: the manual LAUNCH branch is guarded on `launches_sent == 0`,
but a finished mission leaves it at 2. The only paths back to 0 were
_check_launch_gate (needs EVERY seat of the last round to re-ACK) and
_maybe_reset_round (needs EVERY pod gone). A games night lives between
those -- most pods rejoin, one player closes their window. Worse, the
"not all seats filled" diagnostic sits INSIDE the `launches_sent == 0`
guard, so in exactly that state NOTHING was printed.
* btoperator.py: the LAUNCH button is gated on `not monitor.launched`, and
`launched` was cleared ONLY by the two relay lines those same gates emit
("WAITING FOR OPERATOR", "round RESET"). So the button was greyed out in
precisely the wedged state -- the click was a no-op by construction.
That is why a stable group worked (everyone re-ACKs -> gate fires) and why only
a session restart recovered (fresh relay process = fresh state).
FIXES
* An explicit operator LAUNCH is now sufficient authority to start a new
round: _rearm_for_new_round() clears the finished round's state and restores
the template roster/egg, KEEPING seats/beacons/connections, so whoever is
here stays here. Triggered on a press while a round is latched.
* New `rearm`/`newround` operator command + a Re-arm button, so recovery never
needs a session restart. Proven live over the control port.
* Every operator command is logged AT RECEIPT with the state that decides its
fate, so "did it arrive or was it ignored?" is answerable from the log.
* A stale End Mission can no longer kill the next mission: _tick_stop consumed
a stop_requested set between rounds by latching it until launches_sent hit 2,
then StopMissioning the new mission in the tick it launched -- also
indistinguishable from "launch did nothing". It is now consumed + announced
while idle. The UI's end_sent likewise reset per round, not per session
(End Mission used to work exactly once a night).
* The UI clears `launched` on "StopMission sent" and on the re-arm line, so the
button returns when the round actually ends, and only greys once a command
has really been written (a dead relay now says so instead of logging
">> sent" into the void -- _console_finished leaves console_proc non-None).
HARDENING (the "reboot the console" half)
* SO_KEEPALIVE on every accepted socket + a last_seen deadline and a reaper:
nothing detected a pod that vanished WITHOUT a FIN (sleeping laptop, dropped
Wi-Fi, killed process, NAT timeout). Such a conn kept acked=True forever,
which permanently blocked the round reset and held a phantom seat. The
reaper stands down while a mission is running.
* Every pod-facing send went blocking with NO timeout on the single-threaded
selector loop, so one wedged peer could freeze the entire relay. All sends
now go through _send_all_guarded (10s timeout, drops the peer on failure).
* _send_egg no longer calls getpeername() on a possibly-dead socket.
REGRESSION I INTRODUCED AND CAUGHT ON THE RIG: the first cut re-armed whenever
launches_sent >= 1, which also matched the NORMAL state between RunMission #1 and
#2 (launch_requested deliberately stays set across the pair). That reset the
pair mid-flight and re-released eggs every tick -- a re-arm storm that kicked
every pod into an identity-resync loop. The re-arm now additionally requires
`launch_at is None`, i.e. no launch sequence in flight. Verified: 0 REJOIN lines
and exactly 1 RE-ARM line across a full 3-mission rig session.
VERIFIED
* scratchpad/test_relay_rearm.py -- 6 groups, all passing: the exact wedge
state recovers; a stale stop is consumed while idle; staging is NOT restarted
under an impatient operator; mid-pair never re-arms; the happy path is
untouched and RunMission still reaches the pods; the reaper drops a silent
conn and keeps a live one, and stands down mid-mission.
* Live 2-pod rig (scratchpad/rig_relay.ps1 + relay_ctl.py over the control
port): two clean back-to-back missions, StopMission, round RESET, a live
`rearm`, and LAUNCH-with-nobody-present now printing why instead of nothing.
NOT reproduced end-to-end: the field wedge needs 3+ pods with staggered rejoins
(this rig's pods re-exec together, so the all-gone reset always fires). The
state itself is covered by the unit test. Awaiting live confirmation on a real
games night.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
arcattack
co-authored by
Claude Opus 5
Claude Opus 5
parent
48d47ef806
commit
4736cba1ca
@@ -0,0 +1,33 @@
|
||||
"""ctl.py -- send operator commands to a running relay's control port.
|
||||
|
||||
Usage: python ctl.py <command> [more commands...]
|
||||
The shared secret is read from content/operator_secret.txt, same as the relay.
|
||||
"""
|
||||
import os
|
||||
import socket
|
||||
import sys
|
||||
import time
|
||||
|
||||
CONTENT = r"C:\git\bt411\content"
|
||||
SECRET = open(os.path.join(CONTENT, "operator_secret.txt"),
|
||||
encoding="utf-8").read().strip()
|
||||
|
||||
s = socket.create_connection(("127.0.0.1", 1507), timeout=5)
|
||||
s.settimeout(3.0)
|
||||
s.sendall(("AUTH %s\n" % SECRET).encode())
|
||||
time.sleep(0.4)
|
||||
for cmd in sys.argv[1:]:
|
||||
s.sendall((cmd + "\n").encode())
|
||||
print("sent: %s" % cmd)
|
||||
time.sleep(0.4)
|
||||
# drain whatever the relay says back
|
||||
time.sleep(0.6)
|
||||
try:
|
||||
while True:
|
||||
data = s.recv(4096)
|
||||
if not data:
|
||||
break
|
||||
sys.stdout.write(data.decode("utf-8", "replace"))
|
||||
except socket.timeout:
|
||||
pass
|
||||
s.close()
|
||||
@@ -0,0 +1,48 @@
|
||||
# rig_relay.ps1 -- end-to-end back-to-back mission regression for the relay fix.
|
||||
#
|
||||
# Starts the REAL relay (btconsole.py --relay --manual-launch) plus two real pods.
|
||||
# Operator commands are then issued over the relay's CONTROL PORT (1507) by
|
||||
# ctl.py -- the same channel a remote operator uses, which also exercises the new
|
||||
# 'rearm' command. Records only the PIDs it spawns; teardown kills exactly those.
|
||||
|
||||
$ErrorActionPreference = 'Continue'
|
||||
$repo = 'C:\git\bt411'
|
||||
$content = Join-Path $repo 'content'
|
||||
$exe = Join-Path $repo 'build\Release\btl4.exe'
|
||||
$scratch = 'C:\Users\EPILECTRIK\AppData\Local\Temp\claude\C--git-bt411\89ab96e9-6cf0-4555-939d-05bf3708745a\scratchpad'
|
||||
$pidfile = Join-Path $scratch 'rig_relay_pids.txt'
|
||||
$relaylog = Join-Path $scratch 'rig_relay.log'
|
||||
|
||||
$pre = @(Get-Process -Name btl4 -ErrorAction SilentlyContinue)
|
||||
if ($pre.Count -gt 0) { "ABORT: btl4 already running ($($pre.Id -join ','))"; exit 1 }
|
||||
|
||||
Remove-Item $relaylog -ErrorAction SilentlyContinue
|
||||
$pids = @()
|
||||
|
||||
$relay = Start-Process -FilePath 'python' `
|
||||
-ArgumentList '-u', (Join-Path $repo 'tools\btconsole.py'), '--relay', '1500',
|
||||
(Join-Path $content 'FOGDAY.EGG'), '--bind', '127.0.0.1', '--manual-launch' `
|
||||
-WorkingDirectory $content -PassThru `
|
||||
-RedirectStandardOutput $relaylog `
|
||||
-RedirectStandardError (Join-Path $scratch 'rig_relay.err')
|
||||
$pids += $relay.Id
|
||||
Start-Sleep -Seconds 3
|
||||
"relay pid=$($relay.Id) log=$relaylog"
|
||||
|
||||
$env:BT_START_INSIDE = '1'
|
||||
$env:BT_DEV_GAUGES = '1'
|
||||
$env:BT_RELAY = '127.0.0.1:1500'
|
||||
$env:BT_MATCHLOG = '0'
|
||||
|
||||
$env:BT_LOG = 'r_a.log'; $env:BT_SELF = '127.0.0.1:1502'
|
||||
$a = Start-Process -FilePath $exe -ArgumentList '-egg','FOGDAY.EGG','-net','1502' `
|
||||
-WorkingDirectory $content -PassThru
|
||||
$pids += $a.Id
|
||||
Start-Sleep -Seconds 2
|
||||
$env:BT_LOG = 'r_b.log'; $env:BT_SELF = '127.0.0.1:1602'
|
||||
$b = Start-Process -FilePath $exe -ArgumentList '-egg','FOGDAY.EGG','-net','1602' `
|
||||
-WorkingDirectory $content -PassThru
|
||||
$pids += $b.Id
|
||||
|
||||
Set-Content -Path $pidfile -Value $pids
|
||||
"pods: a=$($a.Id) b=$($b.Id) (pids recorded in $pidfile)"
|
||||
@@ -0,0 +1,156 @@
|
||||
"""Focused state-machine test for the launch re-arm fix (no sockets, no game).
|
||||
|
||||
Reproduces the operator's wedge in-process:
|
||||
a round runs (launches_sent=2), only SOME pods rejoin, LAUNCH is pressed.
|
||||
Before the fix the press was swallowed with no output; after it, the relay
|
||||
re-arms and serves the launch.
|
||||
"""
|
||||
import io
|
||||
import os
|
||||
import sys
|
||||
|
||||
REPO = r"C:\git\bt411"
|
||||
sys.path.insert(0, os.path.join(REPO, "tools"))
|
||||
os.chdir(os.path.join(REPO, "content"))
|
||||
|
||||
import btconsole # noqa: E402
|
||||
|
||||
EGG = os.path.join(REPO, "content", "FOGDAY.EGG")
|
||||
|
||||
fails = []
|
||||
|
||||
|
||||
def check(label, ok, extra=""):
|
||||
print(" %-52s %s %s" % (label, "OK" if ok else "*** FAIL ***", extra))
|
||||
if not ok:
|
||||
fails.append(label)
|
||||
|
||||
|
||||
def new_relay():
|
||||
r = btconsole.Relay(1500, EGG, "127.0.0.1", 0, manual_launch=True)
|
||||
return r
|
||||
|
||||
|
||||
class FakeSock:
|
||||
"""Records what the relay would have put on the wire."""
|
||||
def __init__(self):
|
||||
self.sent = []
|
||||
def settimeout(self, _t):
|
||||
pass
|
||||
def setblocking(self, _b):
|
||||
pass
|
||||
def sendall(self, data):
|
||||
self.sent.append(data)
|
||||
|
||||
|
||||
class FakeConn:
|
||||
"""Stands in for a pod's console connection."""
|
||||
def __init__(self, acked=True):
|
||||
self.egg_sent = True
|
||||
self.acked = acked
|
||||
self.addr = ("10.0.0.9", 5000)
|
||||
self.last_seen = btconsole.time.time()
|
||||
self.sock = FakeSock()
|
||||
|
||||
|
||||
print("=== 1. THE REPORTED WEDGE: a round ran, a partial rejoin, LAUNCH pressed ===")
|
||||
r = new_relay()
|
||||
print(" roster seats: %d" % len(r.roster))
|
||||
# A mission ran to completion.
|
||||
r.eggs_released = True
|
||||
r.launches_sent = 2
|
||||
r.mission_started_at = btconsole.time.time() - 600
|
||||
r.stop_sent = True
|
||||
# Only SOME of the last round's pods came back (fewer acked conns than seats).
|
||||
r.console_conns = [FakeConn(acked=True)]
|
||||
assert r._pods_ready() < len(r.roster), "test needs a PARTIAL rejoin"
|
||||
# The operator presses LAUNCH.
|
||||
r.launch_requested = True
|
||||
r._tick_launch()
|
||||
check("launches_sent reset from 2", r.launches_sent == 0, "-> %d" % r.launches_sent)
|
||||
check("round re-opened (eggs_released cleared)", r.eggs_released is False)
|
||||
check("roster restored to the template", len(r.roster) == len(r.orig_roster))
|
||||
check("launch request still pending (not eaten)", r.launch_requested is True)
|
||||
check("stale stop_sent cleared", r.stop_sent is False)
|
||||
|
||||
print("\n=== 2. A STALE 'End Mission' MUST NOT KILL THE NEXT MISSION ===")
|
||||
r = new_relay()
|
||||
r.launches_sent = 0 # between rounds: nothing is running
|
||||
r.stop_requested = True # operator pressed End Mission out of habit
|
||||
r._tick_stop()
|
||||
check("stop latch consumed while idle", r.stop_requested is False)
|
||||
# and it must not have sent anything / marked a stop
|
||||
check("no StopMission recorded", r.stop_sent is False)
|
||||
|
||||
print("\n=== 3. AN UN-ACKABLE SEAT: held, and the operator is told about RE-ARM ===")
|
||||
r = new_relay()
|
||||
r.eggs_released = True # eggs are out ...
|
||||
r.eggs_done_at = None # ... but a seat will never ACK, so this stays None
|
||||
r.launches_sent = 0
|
||||
r.console_conns = [FakeConn(acked=True)]
|
||||
r.launch_requested = True
|
||||
r._tick_launch()
|
||||
# Deliberately NOT auto-re-armed: during normal staging this state just means
|
||||
# "pods are still ACKing", and re-releasing eggs there would restart the handout
|
||||
# under an impatient operator. The explicit rearm command covers the stuck case.
|
||||
check("round NOT silently restarted during staging", r.eggs_released is True)
|
||||
r._rearm_for_new_round("explicit operator re-arm")
|
||||
check("explicit re-arm DOES re-open the round", r.eggs_released is False)
|
||||
|
||||
print("\n=== 3b. MID-PAIR MUST NEVER RE-ARM (the storm caught on the rig) ===")
|
||||
r = new_relay()
|
||||
r.console_conns = [FakeConn(acked=True) for _ in r.roster]
|
||||
r.eggs_released = True
|
||||
r.eggs_done_at = btconsole.time.time() - 1000
|
||||
r.launches_sent = 1 # RunMission #1 has gone out
|
||||
r.launch_at = btconsole.time.time() + 5 # #2 is timed and pending
|
||||
r.launch_requested = True # stays set across the pair
|
||||
r._tick_launch()
|
||||
check("no re-arm between RunMission #1 and #2", r.launches_sent == 1,
|
||||
"-> %d" % r.launches_sent)
|
||||
check("eggs NOT re-released mid-pair", r.eggs_released is True)
|
||||
check("pair timer still intact", r.launch_at is not None)
|
||||
|
||||
print("\n=== 4. THE HAPPY PATH IS UNCHANGED (full roster, first launch) ===")
|
||||
r = new_relay()
|
||||
r.console_conns = [FakeConn(acked=True) for _ in r.roster]
|
||||
r.eggs_released = True
|
||||
r.eggs_done_at = btconsole.time.time() - 1000 # every pod ACKed long ago
|
||||
r.launches_sent = 0
|
||||
r.launch_requested = True
|
||||
before_roster = len(r.roster)
|
||||
r._tick_launch()
|
||||
check("launch ARMED or fired", r.launch_at is not None or r.launches_sent > 0)
|
||||
check("roster untouched", len(r.roster) == before_roster)
|
||||
check("did NOT spuriously re-arm (eggs still released)", r.eggs_released is True)
|
||||
check("RunMission actually reached the pods",
|
||||
all(c.sock.sent for c in r.console_conns),
|
||||
"%d pkt(s) to pod 1" % len(r.console_conns[0].sock.sent))
|
||||
|
||||
print("\n=== 5. REAPER: a silent peer stops pinning the round open ===")
|
||||
r = new_relay()
|
||||
r.launches_sent = 0
|
||||
dead = FakeConn(acked=True)
|
||||
dead.last_seen = btconsole.time.time() - (btconsole.DEAD_PEER_SECONDS + 30)
|
||||
live = FakeConn(acked=True)
|
||||
r.console_conns = [dead, live]
|
||||
r.game_conns = []
|
||||
dropped = []
|
||||
r._drop_console = lambda c: (dropped.append(c), r.console_conns.remove(c))
|
||||
r._reap_dead_peers()
|
||||
check("the silent conn was reaped", dead in dropped)
|
||||
check("the live conn was kept", live in r.console_conns and live not in dropped)
|
||||
|
||||
print("\n=== 6. REAPER IS DISARMED WHILE A MISSION RUNS ===")
|
||||
r = new_relay()
|
||||
r.launches_sent = 2 # mission running
|
||||
quiet = FakeConn(acked=True)
|
||||
quiet.last_seen = btconsole.time.time() - 9999
|
||||
r.console_conns = [quiet]
|
||||
r.game_conns = []
|
||||
r._drop_console = lambda c: fails.append("reaped mid-mission!")
|
||||
r._reap_dead_peers()
|
||||
check("no reaping during a live mission", "reaped mid-mission!" not in fails)
|
||||
|
||||
print("\n%s" % ("ALL CHECKS PASSED" if not fails else "FAILURES: %s" % fails))
|
||||
sys.exit(1 if fails else 0)
|
||||
+214
-17
@@ -75,6 +75,61 @@ import threading
|
||||
import time
|
||||
|
||||
CHUNK = 1000
|
||||
|
||||
# LIVENESS (2026-07-26, operator report "I have to reboot the console"). Nothing
|
||||
# on the relay ever noticed a pod that vanished WITHOUT closing its TCP -- a
|
||||
# sleeping laptop, a dropped Wi-Fi link, a killed process, a NAT idle-timeout on
|
||||
# the internet relay. There was no SO_KEEPALIVE and no idle deadline anywhere,
|
||||
# so the dead connection stayed in console_conns with acked=True forever, which
|
||||
# (a) permanently blocked the round reset (it requires NO conn acked) and (b) held
|
||||
# a phantom seat in the launch count. Only restarting the relay cleared it.
|
||||
KEEPALIVE_IDLE_SECONDS = 30 # start probing after this much silence
|
||||
KEEPALIVE_INTERVAL_SECONDS = 10
|
||||
DEAD_PEER_SECONDS = 180.0 # app-level deadline: no bytes at all for this long
|
||||
SEND_TIMEOUT_SECONDS = 10.0 # a blocking sendall may not stall the whole relay
|
||||
|
||||
|
||||
def _enable_keepalive(sock):
|
||||
"""Turn on TCP keepalive so the OS reaps half-open peers for us. Best
|
||||
effort: the tunables are platform-specific and a plain SO_KEEPALIVE is still
|
||||
a large improvement over nothing."""
|
||||
try:
|
||||
sock.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
|
||||
except OSError:
|
||||
return
|
||||
try: # Windows: one ioctl sets both
|
||||
sock.ioctl(socket.SIO_KEEPALIVE_VALS,
|
||||
(1, KEEPALIVE_IDLE_SECONDS * 1000,
|
||||
KEEPALIVE_INTERVAL_SECONDS * 1000))
|
||||
except (AttributeError, OSError, ValueError):
|
||||
for opt, val in (("TCP_KEEPIDLE", KEEPALIVE_IDLE_SECONDS),
|
||||
("TCP_KEEPINTVL", KEEPALIVE_INTERVAL_SECONDS),
|
||||
("TCP_KEEPCNT", 4)):
|
||||
code = getattr(socket, opt, None)
|
||||
if code is not None:
|
||||
try:
|
||||
sock.setsockopt(socket.IPPROTO_TCP, code, val)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
def _send_all_guarded(sock, payload, what=""):
|
||||
"""Blocking sendall with a TIMEOUT. The relay is single-threaded: every
|
||||
pod-facing send used to flip the socket to blocking with no timeout, so one
|
||||
wedged peer could freeze the entire relay -- no launch tick, no UDP fan-out,
|
||||
no operator control channel -- for as long as the OS kept retransmitting.
|
||||
Raises OSError on failure/timeout; callers already handle that by dropping.
|
||||
"""
|
||||
sock.settimeout(SEND_TIMEOUT_SECONDS)
|
||||
try:
|
||||
sock.sendall(payload)
|
||||
finally:
|
||||
try:
|
||||
sock.setblocking(False)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
PKT_FMT_HDR = "<iiii" # clientID, gameID, fromHost, timeStamp
|
||||
PKT_FMT_MSG = "<IiIiii" # messageLength, messageID, messageFlags, seq, fileLen, thisLen
|
||||
MESSAGE_LENGTH = 1024 # sizeof(ReceiveEggFileMessage)
|
||||
@@ -347,6 +402,7 @@ class RelayGameConn:
|
||||
self.addr = addr
|
||||
self.buf = b""
|
||||
self.host_id = None # None until HELLO registers it
|
||||
self.last_seen = time.time() # liveness deadline (see DEAD_PEER_SECONDS)
|
||||
|
||||
def name(self):
|
||||
hid = self.host_id if self.host_id is not None else "?"
|
||||
@@ -360,6 +416,13 @@ class RelayConsoleConn:
|
||||
self.addr = addr
|
||||
self.egg_sent = False
|
||||
self.acked = False # pod sent AcknowledgeEggFile -- a REAL pod.
|
||||
# LIVENESS (2026-07-26): a pod whose machine sleeps, whose Wi-Fi drops,
|
||||
# or which is killed without a FIN leaves a HALF-OPEN socket. With no
|
||||
# keepalive and no deadline it used to sit here forever holding
|
||||
# acked=True, which permanently blocked _maybe_reset_round (it requires
|
||||
# NO conn acked) and held a phantom seat in the launch count. Only a
|
||||
# console restart cleared it -- the operator's "reboot the console".
|
||||
self.last_seen = time.time()
|
||||
# Internet hardening: the console port is exposed, and random scanners
|
||||
# connect and would otherwise count as pods. Only ACKED connections
|
||||
# count toward the launch gate; scanners never speak the protocol.
|
||||
@@ -541,6 +604,7 @@ class Relay:
|
||||
elif kind == "ctl":
|
||||
self._ctl_read(obj)
|
||||
self._tick_control()
|
||||
self._reap_dead_peers()
|
||||
self._tick_launch()
|
||||
self._tick_stop()
|
||||
self._tick_settle()
|
||||
@@ -558,6 +622,7 @@ class Relay:
|
||||
|
||||
def _accept_console(self, listener):
|
||||
sock, addr = listener.accept()
|
||||
_enable_keepalive(sock)
|
||||
sock.setblocking(False)
|
||||
conn = RelayConsoleConn(sock, addr)
|
||||
self.console_conns.append(conn)
|
||||
@@ -577,12 +642,11 @@ class Relay:
|
||||
f"present)", flush=True)
|
||||
|
||||
def _send_egg(self, conn):
|
||||
addr = conn.sock.getpeername()
|
||||
conn.sock.setblocking(True)
|
||||
addr = conn.addr # NOT getpeername(): it raises on a dead peer
|
||||
try:
|
||||
n = 0
|
||||
for pkt in egg_packets(self.egg_bytes):
|
||||
conn.sock.sendall(pkt)
|
||||
_send_all_guarded(conn.sock, pkt, "egg chunk")
|
||||
n += 1
|
||||
conn.egg_sent = True
|
||||
print(f"[relay] console conn {addr[0]}:{addr[1]}: egg sent "
|
||||
@@ -772,6 +836,36 @@ class Relay:
|
||||
f"({len(self.seat_beacons)} player(s) already waiting)",
|
||||
flush=True)
|
||||
|
||||
def _reap_dead_peers(self):
|
||||
"""Age out peers that stopped speaking entirely.
|
||||
|
||||
TCP keepalive (set on accept) handles most half-open sockets, but it is
|
||||
best-effort and platform-dependent, and a peer can also be alive at the
|
||||
TCP level while its game process is gone. A silent connection is what
|
||||
used to pin the relay: an acked console conn blocks the round reset
|
||||
forever, and a registered game conn holds its by_host seat, so BOTH
|
||||
launch re-arm paths stay unreachable and the operator has to restart.
|
||||
|
||||
Deliberately generous (DEAD_PEER_SECONDS): a pod legitimately sends
|
||||
nothing while sitting at the front end between rounds, so this must only
|
||||
catch the genuinely gone. Never applied while a mission is running --
|
||||
an in-mission pod that goes quiet is the netcode's problem, not ours.
|
||||
"""
|
||||
if self.launches_sent >= 2:
|
||||
return # mission running: leave peers alone
|
||||
now = time.time()
|
||||
for conn in list(self.console_conns):
|
||||
if now - getattr(conn, "last_seen", now) > DEAD_PEER_SECONDS:
|
||||
print(f"[relay] console conn {conn.addr[0]}:{conn.addr[1]} "
|
||||
f"REAPED -- silent for "
|
||||
f"{now - conn.last_seen:.0f}s (was acked={conn.acked}); "
|
||||
f"it can no longer hold the round open", flush=True)
|
||||
self._drop_console(conn)
|
||||
for conn in list(self.game_conns):
|
||||
if now - getattr(conn, "last_seen", now) > DEAD_PEER_SECONDS:
|
||||
self._drop_game(conn, "silent for %.0fs -- reaped"
|
||||
% (now - conn.last_seen))
|
||||
|
||||
def _pods_ready(self):
|
||||
"""REAL pods = console connections that ACKED the egg (scanners never
|
||||
speak the protocol, so they can't hold a roster slot)."""
|
||||
@@ -817,6 +911,8 @@ class Relay:
|
||||
data = conn.sock.recv(4096)
|
||||
except OSError:
|
||||
data = b""
|
||||
if data:
|
||||
conn.last_seen = time.time()
|
||||
if not data:
|
||||
print(f"[relay] console conn {conn.addr[0]}:{conn.addr[1]} closed",
|
||||
flush=True)
|
||||
@@ -868,7 +964,71 @@ class Relay:
|
||||
f"waits on the empty seat). Stop, reduce the roster to the "
|
||||
f"players present, and re-launch.", flush=True)
|
||||
|
||||
def _rearm_for_new_round(self, why):
|
||||
"""OPERATOR-DRIVEN RE-ARM (fixes the "LAUNCH does nothing after a round"
|
||||
wedge, operator report 2026-07-25).
|
||||
|
||||
The two automatic re-arm paths are both ALL-OR-NOTHING:
|
||||
`_check_launch_gate` needs EVERY seat of the last round to re-ACK, and
|
||||
`_maybe_reset_round` needs EVERY pod to be gone. A games night lives
|
||||
between those: most pods rejoin, one player closes their window or
|
||||
crashes. In that state `launches_sent` stayed latched at 2, the manual
|
||||
LAUNCH branch (guarded on `launches_sent == 0`) ignored every press, and
|
||||
because the "not all seats filled" diagnostic sits inside that same
|
||||
guard, NOTHING was printed -- the operator saw a dead button and had to
|
||||
restart the session. (The operator confirmed the mirror image: with a
|
||||
stable group, where everyone does rejoin, relaunching worked fine.)
|
||||
|
||||
So: an explicit operator LAUNCH is now itself sufficient authority to
|
||||
start a new round. Reset only the ROUND state -- seats, beacons, prefs
|
||||
and connections are left alone, so whoever is here stays here.
|
||||
"""
|
||||
print(f"[relay] RE-ARM for a new round ({why}); "
|
||||
f"previous round state cleared", flush=True)
|
||||
self.launches_sent = 0
|
||||
self.launch_at = None
|
||||
self.stop_sent = False
|
||||
# A stop the operator asked for during the LAST round must not carry
|
||||
# into this one -- see _tick_stop, where a stale latch StopMissioned
|
||||
# the next mission in the tick it launched.
|
||||
self.stop_requested = False
|
||||
self.mission_started_at = None
|
||||
self.eggs_done_at = None
|
||||
self._launch_blocked_warned = False
|
||||
# Re-open the round for joins/edits: restore the untrimmed roster and
|
||||
# the template egg, exactly as _maybe_reset_round would, so round 2+
|
||||
# does not serve last round's trimmed egg (new joiners got ROSTER FULL)
|
||||
# and between-round mission edits are picked up.
|
||||
if self.eggs_released:
|
||||
self.eggs_released = False
|
||||
self.egg_path = self.orig_egg_path
|
||||
self.egg_bytes = self.orig_egg_bytes
|
||||
self.roster = list(self.orig_roster)
|
||||
self.expected_ids = set(range(FIRST_GAME_HOST_ID,
|
||||
FIRST_GAME_HOST_ID + len(self.roster)))
|
||||
self._reload_egg_file()
|
||||
for c in self.console_conns:
|
||||
c.acked = False # last round's ACK must not count
|
||||
c.egg_sent = False # every pod needs THIS round's egg
|
||||
|
||||
def _tick_launch(self):
|
||||
# RE-ARM ON DEMAND: an operator LAUNCH while a previous round is still
|
||||
# latched (launches_sent >= 1 -- 2 after a normal round, 1 if a round
|
||||
# was aborted between the RunMission pair) is a deliberate new-round
|
||||
# request, not a no-op. This must run BEFORE the gate below, whose
|
||||
# `launches_sent == 0` guard is exactly what used to swallow the press.
|
||||
# `launch_at is not None` means a launch SEQUENCE IS IN FLIGHT: the
|
||||
# RunMission pair is timed, and `launch_requested` deliberately stays set
|
||||
# across it (it is only cleared once #2 has gone out). Re-arming on that
|
||||
# state resets launches_sent between #1 and #2, so the pair never
|
||||
# completes and the relay re-releases eggs every tick -- a re-arm storm
|
||||
# that kicks every pod into an identity-resync loop. Caught on the rig,
|
||||
# 2026-07-26. So: only re-arm when nothing is in flight.
|
||||
if (self.manual_launch and self.launch_requested
|
||||
and self.launch_at is None and self.launches_sent >= 1):
|
||||
self._rearm_for_new_round("operator LAUNCH after a finished round")
|
||||
# fall through: the request stands and is served by the gate below.
|
||||
|
||||
# Manual mode: the operator's 'launch' arms the timer, still honouring
|
||||
# the settle window (pods must reach WaitingForLaunch or the handler
|
||||
# Fail()s -- the same reason the auto timer waits).
|
||||
@@ -900,9 +1060,12 @@ class Relay:
|
||||
# roster seat has ACKed its egg) -- previously this did NOTHING
|
||||
# visible ("I pressed launch and nothing happened"). Say WHY.
|
||||
self._launch_blocked_warned = True
|
||||
print("[relay] LAUNCH pressed but NOT all seats are filled:",
|
||||
flush=True)
|
||||
print("[relay] LAUNCH pressed but NOT all seats have ACKed "
|
||||
"their egg yet -- holding:", flush=True)
|
||||
self._log_launch_readiness()
|
||||
print("[relay] if a listed seat is never coming back, press "
|
||||
"RE-ARM to re-open the round and launch with whoever is "
|
||||
"here (no session restart needed)", flush=True)
|
||||
if self.launch_at is None or self.launches_sent >= 2:
|
||||
return
|
||||
if time.time() < self.launch_at:
|
||||
@@ -936,11 +1099,11 @@ class Relay:
|
||||
pkt = run_mission_packet()
|
||||
for conn in list(self.console_conns):
|
||||
try:
|
||||
conn.sock.setblocking(True)
|
||||
conn.sock.sendall(pkt)
|
||||
conn.sock.setblocking(False)
|
||||
_send_all_guarded(conn.sock, pkt, "launch")
|
||||
except OSError as e:
|
||||
print(f"[relay] launch send failed to {conn.addr}: {e!r}", flush=True)
|
||||
print(f"[relay] launch send failed to {conn.addr}: {e!r} "
|
||||
f"-- dropping that pod", flush=True)
|
||||
self._drop_console(conn)
|
||||
self.launches_sent += 1
|
||||
self.launch_at = time.time() + LAUNCH_STEP_SECONDS
|
||||
print(f"[relay] RunMission #{self.launches_sent} sent to "
|
||||
@@ -960,7 +1123,19 @@ class Relay:
|
||||
# THE MISSION CLOCK (console-side, as in 1995): send StopMission when
|
||||
# the egg's [mission] length expires, or on the operator's 'stop'
|
||||
# command (the End Mission button).
|
||||
if self.stop_sent or self.launches_sent < 2:
|
||||
if self.launches_sent < 2:
|
||||
# NO MISSION IS RUNNING. A stop asked for now (End Mission pressed
|
||||
# between rounds, which is a natural operator reflex) used to sit
|
||||
# latched and then StopMission the NEXT mission in the very tick it
|
||||
# launched -- indistinguishable, from the operator's seat, from
|
||||
# "launch did nothing". Consume and announce it instead.
|
||||
if self.stop_requested:
|
||||
self.stop_requested = False
|
||||
print("[relay] END MISSION ignored -- no mission is running "
|
||||
"(the latch was cleared, so it cannot kill the next "
|
||||
"launch)", flush=True)
|
||||
return
|
||||
if self.stop_sent:
|
||||
return
|
||||
expired = (self.mission_length > 0
|
||||
and self.mission_started_at is not None
|
||||
@@ -971,9 +1146,7 @@ class Relay:
|
||||
pkt = stop_mission_packet()
|
||||
for conn in list(self.console_conns):
|
||||
try:
|
||||
conn.sock.setblocking(True)
|
||||
conn.sock.sendall(pkt)
|
||||
conn.sock.setblocking(False)
|
||||
_send_all_guarded(conn.sock, pkt, "stop")
|
||||
except OSError as e:
|
||||
print(f"[relay] stop send failed to {conn.addr}: {e!r}",
|
||||
flush=True)
|
||||
@@ -987,6 +1160,7 @@ class Relay:
|
||||
def _accept_game(self, listener):
|
||||
sock, addr = listener.accept()
|
||||
sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
|
||||
_enable_keepalive(sock)
|
||||
sock.setblocking(False)
|
||||
conn = RelayGameConn(sock, addr)
|
||||
self.game_conns.append(conn)
|
||||
@@ -998,6 +1172,8 @@ class Relay:
|
||||
data = conn.sock.recv(8192)
|
||||
except OSError:
|
||||
data = b""
|
||||
if data:
|
||||
conn.last_seen = time.time()
|
||||
if not data:
|
||||
self._drop_game(conn, "closed")
|
||||
return
|
||||
@@ -1238,9 +1414,7 @@ class Relay:
|
||||
|
||||
def _send_raw(self, conn, data):
|
||||
try:
|
||||
conn.sock.setblocking(True)
|
||||
conn.sock.sendall(data)
|
||||
conn.sock.setblocking(False)
|
||||
_send_all_guarded(conn.sock, data)
|
||||
self.stats["tcp_tx"] += 1
|
||||
except OSError as e:
|
||||
self._drop_game(conn, f"send failed {e!r}")
|
||||
@@ -1484,11 +1658,21 @@ class Relay:
|
||||
return
|
||||
cmd = line.split(None, 1)[0].lower()
|
||||
if cmd == "launch":
|
||||
print(f"[ctl] LAUNCH from {conn.name()}", flush=True)
|
||||
print(f"[ctl] LAUNCH from {conn.name()} "
|
||||
f"(launches_sent={self.launches_sent} "
|
||||
f"eggs_released={self.eggs_released} "
|
||||
f"pods_acked={self._pods_ready()}/{len(self.roster)})",
|
||||
flush=True)
|
||||
self.launch_requested = True
|
||||
elif cmd == "stop":
|
||||
print(f"[ctl] STOP from {conn.name()}", flush=True)
|
||||
self.stop_requested = True
|
||||
elif cmd in ("rearm", "newround"):
|
||||
# Explicit escape hatch: recover the launcher without restarting the
|
||||
# session. Before this existed, killing the relay was the only way
|
||||
# out of a latched round (operator report 2026-07-25).
|
||||
print(f"[ctl] RE-ARM from {conn.name()}", flush=True)
|
||||
self._rearm_for_new_round("operator re-arm command")
|
||||
elif cmd == "ping":
|
||||
conn.queue_out(b"[ctl] pong\n")
|
||||
elif cmd == "get":
|
||||
@@ -1620,9 +1804,22 @@ def relay_main(argv):
|
||||
for line in sys.stdin:
|
||||
command = line.strip().lower()
|
||||
if command == "launch":
|
||||
# ALWAYS record the press at RECEIPT. Every wedge in the launch
|
||||
# state machine used to be silent, so the operator could not
|
||||
# tell "the command never arrived" from "the relay ignored it".
|
||||
print(f"[relay] operator LAUNCH command received "
|
||||
f"(launches_sent={relay.launches_sent} "
|
||||
f"eggs_released={relay.eggs_released})", flush=True)
|
||||
relay.launch_requested = True
|
||||
elif command == "stop":
|
||||
print("[relay] operator STOP command received", flush=True)
|
||||
relay.stop_requested = True
|
||||
elif command in ("rearm", "newround"):
|
||||
print("[relay] operator RE-ARM command received", flush=True)
|
||||
relay._rearm_for_new_round("operator re-arm command")
|
||||
elif command:
|
||||
print(f"[relay] unknown operator command: {command!r}",
|
||||
flush=True)
|
||||
threading.Thread(target=stdin_reader, daemon=True).start()
|
||||
try:
|
||||
relay.run()
|
||||
|
||||
+73
-17
@@ -75,6 +75,15 @@ class SessionMonitor:
|
||||
RE_RELAY_HELD = re.compile(
|
||||
r"launch HELD -- still loading: (.+) \((\d+)/(\d+) ready\)")
|
||||
RE_RELAY_RESET = re.compile(r"round RESET")
|
||||
# A ROUND HAS ENDED / THE LAUNCHER RE-ARMED. Before these, `launched`
|
||||
# could only be cleared by "WAITING FOR OPERATOR" (needs every seat of the
|
||||
# last round to re-ACK) or "round RESET" (needs every pod gone). A games
|
||||
# night sits between those -- one player does not come back -- so the
|
||||
# button stayed greyed out and the operator had to restart the session
|
||||
# (report 2026-07-25). StopMission is the honest "the mission is over"
|
||||
# signal, and RE-ARM is the relay saying the launcher is ready again.
|
||||
RE_RELAY_REARM = re.compile(r"RE-ARM for a new round")
|
||||
RE_RELAY_STOPPED = re.compile(r"StopMission sent")
|
||||
RE_MESH_EGG = re.compile(r"\[([^\]]+)\] egg sent")
|
||||
RE_MESH_RUN = re.compile(r"\[([^\]]+)\] RunMission #(\d+) sent")
|
||||
|
||||
@@ -146,6 +155,13 @@ class SessionMonitor:
|
||||
self.held_status = "%s still loading (%s/%s ready)" % (
|
||||
m.group(1), m.group(2), m.group(3))
|
||||
changed = True
|
||||
if (self.RE_RELAY_REARM.search(line)
|
||||
or self.RE_RELAY_STOPPED.search(line)):
|
||||
# The round is over (or the relay re-armed): the LAUNCH button
|
||||
# must come back WITHOUT needing a full-roster re-ACK.
|
||||
self.launched = False
|
||||
self.held_status = None
|
||||
changed = True
|
||||
if self.RE_RELAY_RESET.search(line):
|
||||
# BETWEEN ROUNDS: re-arm the console for the next launch. The
|
||||
# old re-arm rode "WAITING FOR OPERATOR" which only prints on a
|
||||
@@ -454,6 +470,18 @@ class Operator(QMainWindow):
|
||||
"font-weight:bold; color:#cc5533; padding:4px 14px;")
|
||||
self.end_btn.clicked.connect(self._end_mission)
|
||||
self.end_btn.setEnabled(False)
|
||||
# ESCAPE HATCH (2026-07-26). When the launcher latched after a round,
|
||||
# restarting the whole session was the ONLY way out -- which also
|
||||
# disconnected every player who was still waiting. This asks the relay
|
||||
# to clear just the round state and keep the seats.
|
||||
self.rearm_btn = QPushButton("↻ Re-arm")
|
||||
self.rearm_btn.setToolTip(
|
||||
"Clear the finished round's state and re-open the launcher, "
|
||||
"keeping everyone who is already connected. "
|
||||
"Use this if LAUNCH looks dead after a round instead of "
|
||||
"restarting the session.")
|
||||
self.rearm_btn.clicked.connect(self._rearm_round)
|
||||
self.rearm_btn.setEnabled(False)
|
||||
self.launch_local_btn = QPushButton("Launch local instances")
|
||||
self.launch_local_btn.clicked.connect(self._launch_local)
|
||||
self.launch_local_btn.setEnabled(False)
|
||||
@@ -465,6 +493,7 @@ class Operator(QMainWindow):
|
||||
srow.addSpacing(20)
|
||||
srow.addWidget(self.launch_btn)
|
||||
srow.addWidget(self.end_btn)
|
||||
srow.addWidget(self.rearm_btn)
|
||||
srow.addSpacing(20)
|
||||
srow.addWidget(self.launch_local_btn)
|
||||
srow.addWidget(self.apply_btn)
|
||||
@@ -799,6 +828,7 @@ class Operator(QMainWindow):
|
||||
self.console_proc.start(sys.executable, ["-u"] + args)
|
||||
self.start_btn.setEnabled(False)
|
||||
self.stop_btn.setEnabled(True)
|
||||
self.rearm_btn.setEnabled(True)
|
||||
self.restart_btn.setEnabled(True)
|
||||
self.launch_local_btn.setEnabled(True)
|
||||
self.launch_btn.setEnabled(False)
|
||||
@@ -826,6 +856,7 @@ class Operator(QMainWindow):
|
||||
self.remote_link.send("get mission")
|
||||
self.start_btn.setEnabled(False)
|
||||
self.stop_btn.setEnabled(True)
|
||||
self.rearm_btn.setEnabled(True)
|
||||
self.restart_btn.setEnabled(True)
|
||||
self.launch_local_btn.setEnabled(True)
|
||||
self.launch_btn.setEnabled(False)
|
||||
@@ -840,6 +871,7 @@ class Operator(QMainWindow):
|
||||
self.remote_link = None
|
||||
self.start_btn.setEnabled(True)
|
||||
self.stop_btn.setEnabled(False)
|
||||
self.rearm_btn.setEnabled(False)
|
||||
self.restart_btn.setEnabled(False)
|
||||
self.launch_btn.setEnabled(False)
|
||||
self.end_btn.setEnabled(False)
|
||||
@@ -860,6 +892,7 @@ class Operator(QMainWindow):
|
||||
self._stop_games()
|
||||
self.start_btn.setEnabled(True)
|
||||
self.stop_btn.setEnabled(False)
|
||||
self.rearm_btn.setEnabled(False)
|
||||
self.restart_btn.setEnabled(False)
|
||||
self.launch_btn.setEnabled(False)
|
||||
self.end_btn.setEnabled(False)
|
||||
@@ -877,35 +910,52 @@ class Operator(QMainWindow):
|
||||
|
||||
def _launch_mission(self):
|
||||
"""Manual launch gate: send the relay its 'launch' command."""
|
||||
if self._relay_send("launch"):
|
||||
# Grey the button only once the command actually LEFT. The refresh
|
||||
# tick re-derives this from the relay's own reports, so if the relay
|
||||
# declines the press the button comes back rather than staying dead
|
||||
# (which is what forced a session restart before 2026-07-26).
|
||||
self.launch_btn.setEnabled(False)
|
||||
self.end_btn.setEnabled(True)
|
||||
|
||||
def _relay_send(self, command):
|
||||
"""Send an operator command and report HONESTLY whether it went.
|
||||
|
||||
The old call sites wrote to the QProcess and logged success
|
||||
unconditionally -- but _console_finished leaves console_proc non-None,
|
||||
so after the relay died every press logged ">> sent" into the void.
|
||||
"""
|
||||
if getattr(self, "remote_link", None):
|
||||
self.remote_link.send("launch")
|
||||
self.launch_btn.setEnabled(False)
|
||||
self.end_btn.setEnabled(True)
|
||||
self.log.appendPlainText(">> operator LAUNCH sent (remote)")
|
||||
elif self.console_proc:
|
||||
self.console_proc.write(b"launch\n")
|
||||
self.launch_btn.setEnabled(False)
|
||||
self.end_btn.setEnabled(True)
|
||||
self.log.appendPlainText(">> operator LAUNCH sent")
|
||||
self.remote_link.send(command)
|
||||
self.log.appendPlainText(">> operator %s sent (remote)"
|
||||
% command.upper())
|
||||
return True
|
||||
proc = self.console_proc
|
||||
if proc is None or proc.state() != QProcess.Running:
|
||||
self.log.appendPlainText(
|
||||
"!! %s NOT sent -- the console/relay is not running "
|
||||
"(press Start Session)" % command.upper())
|
||||
return False
|
||||
proc.write((command + "\n").encode())
|
||||
self.log.appendPlainText(">> operator %s sent" % command.upper())
|
||||
return True
|
||||
|
||||
def _rearm_round(self):
|
||||
"""Ask the relay to clear a finished round's state without a restart."""
|
||||
self._relay_send("rearm")
|
||||
|
||||
def _end_mission(self):
|
||||
"""End the running mission now (the relay also auto-stops at the
|
||||
egg's [mission] length -- the authentic console mission clock)."""
|
||||
if getattr(self, "remote_link", None):
|
||||
self.remote_link.send("stop")
|
||||
if self._relay_send("stop"):
|
||||
self.end_sent = True
|
||||
self.end_btn.setEnabled(False)
|
||||
self.log.appendPlainText(">> operator END MISSION sent (remote)")
|
||||
elif self.console_proc:
|
||||
self.console_proc.write(b"stop\n")
|
||||
self.end_sent = True
|
||||
self.end_btn.setEnabled(False)
|
||||
self.log.appendPlainText(">> operator END MISSION sent")
|
||||
|
||||
def _console_finished(self):
|
||||
self.log.appendPlainText("== console/relay process exited ==")
|
||||
self.start_btn.setEnabled(True)
|
||||
self.stop_btn.setEnabled(False)
|
||||
self.rearm_btn.setEnabled(False)
|
||||
|
||||
def _console_output(self):
|
||||
if not self.console_proc:
|
||||
@@ -975,6 +1025,12 @@ class Operator(QMainWindow):
|
||||
if (self.console_proc and not self.end_btn.isEnabled()
|
||||
and not getattr(self, "end_sent", False)):
|
||||
self.end_btn.setEnabled(True)
|
||||
elif getattr(self, "end_sent", False):
|
||||
# END MISSION is PER ROUND, not per session. end_sent used to be
|
||||
# cleared only by Start Session, so the button worked exactly once a
|
||||
# night; and a stale press between rounds used to kill the next
|
||||
# mission the instant it launched (the relay now refuses that too).
|
||||
self.end_sent = False
|
||||
elif self.monitor.held_status:
|
||||
head = "STAGING — " + self.monitor.held_status
|
||||
elif ready_n:
|
||||
|
||||
Reference in New Issue
Block a user