relay: LAUNCH is never silently inert again -- fix the post-round wedge (operator report)

THE SYMPTOM: "after a game ends I push LAUNCH and nothing happens until I reset
the session entirely or reboot the console."  The operator also observed the
mirror image, which is the tell: with a STABLE group, relaunching worked fine
several times in a row.

ROOT CAUSE.  Two gates, both all-or-nothing, and a UI latch that agreed with
them:
  * btconsole.py: the manual LAUNCH branch is guarded on `launches_sent == 0`,
    but a finished mission leaves it at 2.  The only paths back to 0 were
    _check_launch_gate (needs EVERY seat of the last round to re-ACK) and
    _maybe_reset_round (needs EVERY pod gone).  A games night lives between
    those -- most pods rejoin, one player closes their window.  Worse, the
    "not all seats filled" diagnostic sits INSIDE the `launches_sent == 0`
    guard, so in exactly that state NOTHING was printed.
  * btoperator.py: the LAUNCH button is gated on `not monitor.launched`, and
    `launched` was cleared ONLY by the two relay lines those same gates emit
    ("WAITING FOR OPERATOR", "round RESET").  So the button was greyed out in
    precisely the wedged state -- the click was a no-op by construction.
That is why a stable group worked (everyone re-ACKs -> gate fires) and why only
a session restart recovered (fresh relay process = fresh state).

FIXES
  * An explicit operator LAUNCH is now sufficient authority to start a new
    round: _rearm_for_new_round() clears the finished round's state and restores
    the template roster/egg, KEEPING seats/beacons/connections, so whoever is
    here stays here.  Triggered on a press while a round is latched.
  * New `rearm`/`newround` operator command + a Re-arm button, so recovery never
    needs a session restart.  Proven live over the control port.
  * Every operator command is logged AT RECEIPT with the state that decides its
    fate, so "did it arrive or was it ignored?" is answerable from the log.
  * A stale End Mission can no longer kill the next mission: _tick_stop consumed
    a stop_requested set between rounds by latching it until launches_sent hit 2,
    then StopMissioning the new mission in the tick it launched -- also
    indistinguishable from "launch did nothing".  It is now consumed + announced
    while idle.  The UI's end_sent likewise reset per round, not per session
    (End Mission used to work exactly once a night).
  * The UI clears `launched` on "StopMission sent" and on the re-arm line, so the
    button returns when the round actually ends, and only greys once a command
    has really been written (a dead relay now says so instead of logging
    ">> sent" into the void -- _console_finished leaves console_proc non-None).

HARDENING (the "reboot the console" half)
  * SO_KEEPALIVE on every accepted socket + a last_seen deadline and a reaper:
    nothing detected a pod that vanished WITHOUT a FIN (sleeping laptop, dropped
    Wi-Fi, killed process, NAT timeout).  Such a conn kept acked=True forever,
    which permanently blocked the round reset and held a phantom seat.  The
    reaper stands down while a mission is running.
  * Every pod-facing send went blocking with NO timeout on the single-threaded
    selector loop, so one wedged peer could freeze the entire relay.  All sends
    now go through _send_all_guarded (10s timeout, drops the peer on failure).
  * _send_egg no longer calls getpeername() on a possibly-dead socket.

REGRESSION I INTRODUCED AND CAUGHT ON THE RIG: the first cut re-armed whenever
launches_sent >= 1, which also matched the NORMAL state between RunMission #1 and
#2 (launch_requested deliberately stays set across the pair).  That reset the
pair mid-flight and re-released eggs every tick -- a re-arm storm that kicked
every pod into an identity-resync loop.  The re-arm now additionally requires
`launch_at is None`, i.e. no launch sequence in flight.  Verified: 0 REJOIN lines
and exactly 1 RE-ARM line across a full 3-mission rig session.

VERIFIED
  * scratchpad/test_relay_rearm.py -- 6 groups, all passing: the exact wedge
    state recovers; a stale stop is consumed while idle; staging is NOT restarted
    under an impatient operator; mid-pair never re-arms; the happy path is
    untouched and RunMission still reaches the pods; the reaper drops a silent
    conn and keeps a live one, and stands down mid-mission.
  * Live 2-pod rig (scratchpad/rig_relay.ps1 + relay_ctl.py over the control
    port): two clean back-to-back missions, StopMission, round RESET, a live
    `rearm`, and LAUNCH-with-nobody-present now printing why instead of nothing.

NOT reproduced end-to-end: the field wedge needs 3+ pods with staggered rejoins
(this rig's pods re-exec together, so the all-gone reset always fires).  The
state itself is covered by the unit test.  Awaiting live confirmation on a real
games night.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
arcattack
2026-07-26 00:21:24 -05:00
co-authored by Claude Opus 5
parent 48d47ef806
commit 4736cba1ca
5 changed files with 524 additions and 34 deletions
+73 -17
View File
@@ -75,6 +75,15 @@ class SessionMonitor:
RE_RELAY_HELD = re.compile(
r"launch HELD -- still loading: (.+) \((\d+)/(\d+) ready\)")
RE_RELAY_RESET = re.compile(r"round RESET")
# A ROUND HAS ENDED / THE LAUNCHER RE-ARMED. Before these, `launched`
# could only be cleared by "WAITING FOR OPERATOR" (needs every seat of the
# last round to re-ACK) or "round RESET" (needs every pod gone). A games
# night sits between those -- one player does not come back -- so the
# button stayed greyed out and the operator had to restart the session
# (report 2026-07-25). StopMission is the honest "the mission is over"
# signal, and RE-ARM is the relay saying the launcher is ready again.
RE_RELAY_REARM = re.compile(r"RE-ARM for a new round")
RE_RELAY_STOPPED = re.compile(r"StopMission sent")
RE_MESH_EGG = re.compile(r"\[([^\]]+)\] egg sent")
RE_MESH_RUN = re.compile(r"\[([^\]]+)\] RunMission #(\d+) sent")
@@ -146,6 +155,13 @@ class SessionMonitor:
self.held_status = "%s still loading (%s/%s ready)" % (
m.group(1), m.group(2), m.group(3))
changed = True
if (self.RE_RELAY_REARM.search(line)
or self.RE_RELAY_STOPPED.search(line)):
# The round is over (or the relay re-armed): the LAUNCH button
# must come back WITHOUT needing a full-roster re-ACK.
self.launched = False
self.held_status = None
changed = True
if self.RE_RELAY_RESET.search(line):
# BETWEEN ROUNDS: re-arm the console for the next launch. The
# old re-arm rode "WAITING FOR OPERATOR" which only prints on a
@@ -454,6 +470,18 @@ class Operator(QMainWindow):
"font-weight:bold; color:#cc5533; padding:4px 14px;")
self.end_btn.clicked.connect(self._end_mission)
self.end_btn.setEnabled(False)
# ESCAPE HATCH (2026-07-26). When the launcher latched after a round,
# restarting the whole session was the ONLY way out -- which also
# disconnected every player who was still waiting. This asks the relay
# to clear just the round state and keep the seats.
self.rearm_btn = QPushButton("↻ Re-arm")
self.rearm_btn.setToolTip(
"Clear the finished round's state and re-open the launcher, "
"keeping everyone who is already connected. "
"Use this if LAUNCH looks dead after a round instead of "
"restarting the session.")
self.rearm_btn.clicked.connect(self._rearm_round)
self.rearm_btn.setEnabled(False)
self.launch_local_btn = QPushButton("Launch local instances")
self.launch_local_btn.clicked.connect(self._launch_local)
self.launch_local_btn.setEnabled(False)
@@ -465,6 +493,7 @@ class Operator(QMainWindow):
srow.addSpacing(20)
srow.addWidget(self.launch_btn)
srow.addWidget(self.end_btn)
srow.addWidget(self.rearm_btn)
srow.addSpacing(20)
srow.addWidget(self.launch_local_btn)
srow.addWidget(self.apply_btn)
@@ -799,6 +828,7 @@ class Operator(QMainWindow):
self.console_proc.start(sys.executable, ["-u"] + args)
self.start_btn.setEnabled(False)
self.stop_btn.setEnabled(True)
self.rearm_btn.setEnabled(True)
self.restart_btn.setEnabled(True)
self.launch_local_btn.setEnabled(True)
self.launch_btn.setEnabled(False)
@@ -826,6 +856,7 @@ class Operator(QMainWindow):
self.remote_link.send("get mission")
self.start_btn.setEnabled(False)
self.stop_btn.setEnabled(True)
self.rearm_btn.setEnabled(True)
self.restart_btn.setEnabled(True)
self.launch_local_btn.setEnabled(True)
self.launch_btn.setEnabled(False)
@@ -840,6 +871,7 @@ class Operator(QMainWindow):
self.remote_link = None
self.start_btn.setEnabled(True)
self.stop_btn.setEnabled(False)
self.rearm_btn.setEnabled(False)
self.restart_btn.setEnabled(False)
self.launch_btn.setEnabled(False)
self.end_btn.setEnabled(False)
@@ -860,6 +892,7 @@ class Operator(QMainWindow):
self._stop_games()
self.start_btn.setEnabled(True)
self.stop_btn.setEnabled(False)
self.rearm_btn.setEnabled(False)
self.restart_btn.setEnabled(False)
self.launch_btn.setEnabled(False)
self.end_btn.setEnabled(False)
@@ -877,35 +910,52 @@ class Operator(QMainWindow):
def _launch_mission(self):
"""Manual launch gate: send the relay its 'launch' command."""
if self._relay_send("launch"):
# Grey the button only once the command actually LEFT. The refresh
# tick re-derives this from the relay's own reports, so if the relay
# declines the press the button comes back rather than staying dead
# (which is what forced a session restart before 2026-07-26).
self.launch_btn.setEnabled(False)
self.end_btn.setEnabled(True)
def _relay_send(self, command):
"""Send an operator command and report HONESTLY whether it went.
The old call sites wrote to the QProcess and logged success
unconditionally -- but _console_finished leaves console_proc non-None,
so after the relay died every press logged ">> sent" into the void.
"""
if getattr(self, "remote_link", None):
self.remote_link.send("launch")
self.launch_btn.setEnabled(False)
self.end_btn.setEnabled(True)
self.log.appendPlainText(">> operator LAUNCH sent (remote)")
elif self.console_proc:
self.console_proc.write(b"launch\n")
self.launch_btn.setEnabled(False)
self.end_btn.setEnabled(True)
self.log.appendPlainText(">> operator LAUNCH sent")
self.remote_link.send(command)
self.log.appendPlainText(">> operator %s sent (remote)"
% command.upper())
return True
proc = self.console_proc
if proc is None or proc.state() != QProcess.Running:
self.log.appendPlainText(
"!! %s NOT sent -- the console/relay is not running "
"(press Start Session)" % command.upper())
return False
proc.write((command + "\n").encode())
self.log.appendPlainText(">> operator %s sent" % command.upper())
return True
def _rearm_round(self):
"""Ask the relay to clear a finished round's state without a restart."""
self._relay_send("rearm")
def _end_mission(self):
"""End the running mission now (the relay also auto-stops at the
egg's [mission] length -- the authentic console mission clock)."""
if getattr(self, "remote_link", None):
self.remote_link.send("stop")
if self._relay_send("stop"):
self.end_sent = True
self.end_btn.setEnabled(False)
self.log.appendPlainText(">> operator END MISSION sent (remote)")
elif self.console_proc:
self.console_proc.write(b"stop\n")
self.end_sent = True
self.end_btn.setEnabled(False)
self.log.appendPlainText(">> operator END MISSION sent")
def _console_finished(self):
self.log.appendPlainText("== console/relay process exited ==")
self.start_btn.setEnabled(True)
self.stop_btn.setEnabled(False)
self.rearm_btn.setEnabled(False)
def _console_output(self):
if not self.console_proc:
@@ -975,6 +1025,12 @@ class Operator(QMainWindow):
if (self.console_proc and not self.end_btn.isEnabled()
and not getattr(self, "end_sent", False)):
self.end_btn.setEnabled(True)
elif getattr(self, "end_sent", False):
# END MISSION is PER ROUND, not per session. end_sent used to be
# cleared only by Start Session, so the button worked exactly once a
# night; and a stale press between rounds used to kill the next
# mission the instant it launched (the relay now refuses that too).
self.end_sent = False
elif self.monitor.held_status:
head = "STAGING — " + self.monitor.held_status
elif ready_n: