Files
BT411/scratchpad/test_relay_rearm.py
T
arcattackandClaude Opus 5 4736cba1ca relay: LAUNCH is never silently inert again -- fix the post-round wedge (operator report)
THE SYMPTOM: "after a game ends I push LAUNCH and nothing happens until I reset
the session entirely or reboot the console."  The operator also observed the
mirror image, which is the tell: with a STABLE group, relaunching worked fine
several times in a row.

ROOT CAUSE.  Two gates, both all-or-nothing, and a UI latch that agreed with
them:
  * btconsole.py: the manual LAUNCH branch is guarded on `launches_sent == 0`,
    but a finished mission leaves it at 2.  The only paths back to 0 were
    _check_launch_gate (needs EVERY seat of the last round to re-ACK) and
    _maybe_reset_round (needs EVERY pod gone).  A games night lives between
    those -- most pods rejoin, one player closes their window.  Worse, the
    "not all seats filled" diagnostic sits INSIDE the `launches_sent == 0`
    guard, so in exactly that state NOTHING was printed.
  * btoperator.py: the LAUNCH button is gated on `not monitor.launched`, and
    `launched` was cleared ONLY by the two relay lines those same gates emit
    ("WAITING FOR OPERATOR", "round RESET").  So the button was greyed out in
    precisely the wedged state -- the click was a no-op by construction.
That is why a stable group worked (everyone re-ACKs -> gate fires) and why only
a session restart recovered (fresh relay process = fresh state).

FIXES
  * An explicit operator LAUNCH is now sufficient authority to start a new
    round: _rearm_for_new_round() clears the finished round's state and restores
    the template roster/egg, KEEPING seats/beacons/connections, so whoever is
    here stays here.  Triggered on a press while a round is latched.
  * New `rearm`/`newround` operator command + a Re-arm button, so recovery never
    needs a session restart.  Proven live over the control port.
  * Every operator command is logged AT RECEIPT with the state that decides its
    fate, so "did it arrive or was it ignored?" is answerable from the log.
  * A stale End Mission can no longer kill the next mission: _tick_stop consumed
    a stop_requested set between rounds by latching it until launches_sent hit 2,
    then StopMissioning the new mission in the tick it launched -- also
    indistinguishable from "launch did nothing".  It is now consumed + announced
    while idle.  The UI's end_sent likewise reset per round, not per session
    (End Mission used to work exactly once a night).
  * The UI clears `launched` on "StopMission sent" and on the re-arm line, so the
    button returns when the round actually ends, and only greys once a command
    has really been written (a dead relay now says so instead of logging
    ">> sent" into the void -- _console_finished leaves console_proc non-None).

HARDENING (the "reboot the console" half)
  * SO_KEEPALIVE on every accepted socket + a last_seen deadline and a reaper:
    nothing detected a pod that vanished WITHOUT a FIN (sleeping laptop, dropped
    Wi-Fi, killed process, NAT timeout).  Such a conn kept acked=True forever,
    which permanently blocked the round reset and held a phantom seat.  The
    reaper stands down while a mission is running.
  * Every pod-facing send went blocking with NO timeout on the single-threaded
    selector loop, so one wedged peer could freeze the entire relay.  All sends
    now go through _send_all_guarded (10s timeout, drops the peer on failure).
  * _send_egg no longer calls getpeername() on a possibly-dead socket.

REGRESSION I INTRODUCED AND CAUGHT ON THE RIG: the first cut re-armed whenever
launches_sent >= 1, which also matched the NORMAL state between RunMission #1 and
#2 (launch_requested deliberately stays set across the pair).  That reset the
pair mid-flight and re-released eggs every tick -- a re-arm storm that kicked
every pod into an identity-resync loop.  The re-arm now additionally requires
`launch_at is None`, i.e. no launch sequence in flight.  Verified: 0 REJOIN lines
and exactly 1 RE-ARM line across a full 3-mission rig session.

VERIFIED
  * scratchpad/test_relay_rearm.py -- 6 groups, all passing: the exact wedge
    state recovers; a stale stop is consumed while idle; staging is NOT restarted
    under an impatient operator; mid-pair never re-arms; the happy path is
    untouched and RunMission still reaches the pods; the reaper drops a silent
    conn and keeps a live one, and stands down mid-mission.
  * Live 2-pod rig (scratchpad/rig_relay.ps1 + relay_ctl.py over the control
    port): two clean back-to-back missions, StopMission, round RESET, a live
    `rearm`, and LAUNCH-with-nobody-present now printing why instead of nothing.

NOT reproduced end-to-end: the field wedge needs 3+ pods with staggered rejoins
(this rig's pods re-exec together, so the all-gone reset always fires).  The
state itself is covered by the unit test.  Awaiting live confirmation on a real
games night.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 00:21:24 -05:00

157 lines
5.8 KiB
Python

"""Focused state-machine test for the launch re-arm fix (no sockets, no game).
Reproduces the operator's wedge in-process:
a round runs (launches_sent=2), only SOME pods rejoin, LAUNCH is pressed.
Before the fix the press was swallowed with no output; after it, the relay
re-arms and serves the launch.
"""
import io
import os
import sys
REPO = r"C:\git\bt411"
sys.path.insert(0, os.path.join(REPO, "tools"))
os.chdir(os.path.join(REPO, "content"))
import btconsole # noqa: E402
EGG = os.path.join(REPO, "content", "FOGDAY.EGG")
fails = []
def check(label, ok, extra=""):
print(" %-52s %s %s" % (label, "OK" if ok else "*** FAIL ***", extra))
if not ok:
fails.append(label)
def new_relay():
r = btconsole.Relay(1500, EGG, "127.0.0.1", 0, manual_launch=True)
return r
class FakeSock:
"""Records what the relay would have put on the wire."""
def __init__(self):
self.sent = []
def settimeout(self, _t):
pass
def setblocking(self, _b):
pass
def sendall(self, data):
self.sent.append(data)
class FakeConn:
"""Stands in for a pod's console connection."""
def __init__(self, acked=True):
self.egg_sent = True
self.acked = acked
self.addr = ("10.0.0.9", 5000)
self.last_seen = btconsole.time.time()
self.sock = FakeSock()
print("=== 1. THE REPORTED WEDGE: a round ran, a partial rejoin, LAUNCH pressed ===")
r = new_relay()
print(" roster seats: %d" % len(r.roster))
# A mission ran to completion.
r.eggs_released = True
r.launches_sent = 2
r.mission_started_at = btconsole.time.time() - 600
r.stop_sent = True
# Only SOME of the last round's pods came back (fewer acked conns than seats).
r.console_conns = [FakeConn(acked=True)]
assert r._pods_ready() < len(r.roster), "test needs a PARTIAL rejoin"
# The operator presses LAUNCH.
r.launch_requested = True
r._tick_launch()
check("launches_sent reset from 2", r.launches_sent == 0, "-> %d" % r.launches_sent)
check("round re-opened (eggs_released cleared)", r.eggs_released is False)
check("roster restored to the template", len(r.roster) == len(r.orig_roster))
check("launch request still pending (not eaten)", r.launch_requested is True)
check("stale stop_sent cleared", r.stop_sent is False)
print("\n=== 2. A STALE 'End Mission' MUST NOT KILL THE NEXT MISSION ===")
r = new_relay()
r.launches_sent = 0 # between rounds: nothing is running
r.stop_requested = True # operator pressed End Mission out of habit
r._tick_stop()
check("stop latch consumed while idle", r.stop_requested is False)
# and it must not have sent anything / marked a stop
check("no StopMission recorded", r.stop_sent is False)
print("\n=== 3. AN UN-ACKABLE SEAT: held, and the operator is told about RE-ARM ===")
r = new_relay()
r.eggs_released = True # eggs are out ...
r.eggs_done_at = None # ... but a seat will never ACK, so this stays None
r.launches_sent = 0
r.console_conns = [FakeConn(acked=True)]
r.launch_requested = True
r._tick_launch()
# Deliberately NOT auto-re-armed: during normal staging this state just means
# "pods are still ACKing", and re-releasing eggs there would restart the handout
# under an impatient operator. The explicit rearm command covers the stuck case.
check("round NOT silently restarted during staging", r.eggs_released is True)
r._rearm_for_new_round("explicit operator re-arm")
check("explicit re-arm DOES re-open the round", r.eggs_released is False)
print("\n=== 3b. MID-PAIR MUST NEVER RE-ARM (the storm caught on the rig) ===")
r = new_relay()
r.console_conns = [FakeConn(acked=True) for _ in r.roster]
r.eggs_released = True
r.eggs_done_at = btconsole.time.time() - 1000
r.launches_sent = 1 # RunMission #1 has gone out
r.launch_at = btconsole.time.time() + 5 # #2 is timed and pending
r.launch_requested = True # stays set across the pair
r._tick_launch()
check("no re-arm between RunMission #1 and #2", r.launches_sent == 1,
"-> %d" % r.launches_sent)
check("eggs NOT re-released mid-pair", r.eggs_released is True)
check("pair timer still intact", r.launch_at is not None)
print("\n=== 4. THE HAPPY PATH IS UNCHANGED (full roster, first launch) ===")
r = new_relay()
r.console_conns = [FakeConn(acked=True) for _ in r.roster]
r.eggs_released = True
r.eggs_done_at = btconsole.time.time() - 1000 # every pod ACKed long ago
r.launches_sent = 0
r.launch_requested = True
before_roster = len(r.roster)
r._tick_launch()
check("launch ARMED or fired", r.launch_at is not None or r.launches_sent > 0)
check("roster untouched", len(r.roster) == before_roster)
check("did NOT spuriously re-arm (eggs still released)", r.eggs_released is True)
check("RunMission actually reached the pods",
all(c.sock.sent for c in r.console_conns),
"%d pkt(s) to pod 1" % len(r.console_conns[0].sock.sent))
print("\n=== 5. REAPER: a silent peer stops pinning the round open ===")
r = new_relay()
r.launches_sent = 0
dead = FakeConn(acked=True)
dead.last_seen = btconsole.time.time() - (btconsole.DEAD_PEER_SECONDS + 30)
live = FakeConn(acked=True)
r.console_conns = [dead, live]
r.game_conns = []
dropped = []
r._drop_console = lambda c: (dropped.append(c), r.console_conns.remove(c))
r._reap_dead_peers()
check("the silent conn was reaped", dead in dropped)
check("the live conn was kept", live in r.console_conns and live not in dropped)
print("\n=== 6. REAPER IS DISARMED WHILE A MISSION RUNS ===")
r = new_relay()
r.launches_sent = 2 # mission running
quiet = FakeConn(acked=True)
quiet.last_seen = btconsole.time.time() - 9999
r.console_conns = [quiet]
r.game_conns = []
r._drop_console = lambda c: fails.append("reaped mid-mission!")
r._reap_dead_peers()
check("no reaping during a live mission", "reaped mid-mission!" not in fails)
print("\n%s" % ("ALL CHECKS PASSED" if not fails else "FAILURES: %s" % fails))
sys.exit(1 if fails else 0)