relay: LAUNCH is never silently inert again -- fix the post-round wedge (operator report)
THE SYMPTOM: "after a game ends I push LAUNCH and nothing happens until I reset
the session entirely or reboot the console." The operator also observed the
mirror image, which is the tell: with a STABLE group, relaunching worked fine
several times in a row.
ROOT CAUSE. Two gates, both all-or-nothing, and a UI latch that agreed with
them:
* btconsole.py: the manual LAUNCH branch is guarded on `launches_sent == 0`,
but a finished mission leaves it at 2. The only paths back to 0 were
_check_launch_gate (needs EVERY seat of the last round to re-ACK) and
_maybe_reset_round (needs EVERY pod gone). A games night lives between
those -- most pods rejoin, one player closes their window. Worse, the
"not all seats filled" diagnostic sits INSIDE the `launches_sent == 0`
guard, so in exactly that state NOTHING was printed.
* btoperator.py: the LAUNCH button is gated on `not monitor.launched`, and
`launched` was cleared ONLY by the two relay lines those same gates emit
("WAITING FOR OPERATOR", "round RESET"). So the button was greyed out in
precisely the wedged state -- the click was a no-op by construction.
That is why a stable group worked (everyone re-ACKs -> gate fires) and why only
a session restart recovered (fresh relay process = fresh state).
FIXES
* An explicit operator LAUNCH is now sufficient authority to start a new
round: _rearm_for_new_round() clears the finished round's state and restores
the template roster/egg, KEEPING seats/beacons/connections, so whoever is
here stays here. Triggered on a press while a round is latched.
* New `rearm`/`newround` operator command + a Re-arm button, so recovery never
needs a session restart. Proven live over the control port.
* Every operator command is logged AT RECEIPT with the state that decides its
fate, so "did it arrive or was it ignored?" is answerable from the log.
* A stale End Mission can no longer kill the next mission: _tick_stop consumed
a stop_requested set between rounds by latching it until launches_sent hit 2,
then StopMissioning the new mission in the tick it launched -- also
indistinguishable from "launch did nothing". It is now consumed + announced
while idle. The UI's end_sent likewise reset per round, not per session
(End Mission used to work exactly once a night).
* The UI clears `launched` on "StopMission sent" and on the re-arm line, so the
button returns when the round actually ends, and only greys once a command
has really been written (a dead relay now says so instead of logging
">> sent" into the void -- _console_finished leaves console_proc non-None).
HARDENING (the "reboot the console" half)
* SO_KEEPALIVE on every accepted socket + a last_seen deadline and a reaper:
nothing detected a pod that vanished WITHOUT a FIN (sleeping laptop, dropped
Wi-Fi, killed process, NAT timeout). Such a conn kept acked=True forever,
which permanently blocked the round reset and held a phantom seat. The
reaper stands down while a mission is running.
* Every pod-facing send went blocking with NO timeout on the single-threaded
selector loop, so one wedged peer could freeze the entire relay. All sends
now go through _send_all_guarded (10s timeout, drops the peer on failure).
* _send_egg no longer calls getpeername() on a possibly-dead socket.
REGRESSION I INTRODUCED AND CAUGHT ON THE RIG: the first cut re-armed whenever
launches_sent >= 1, which also matched the NORMAL state between RunMission #1 and
#2 (launch_requested deliberately stays set across the pair). That reset the
pair mid-flight and re-released eggs every tick -- a re-arm storm that kicked
every pod into an identity-resync loop. The re-arm now additionally requires
`launch_at is None`, i.e. no launch sequence in flight. Verified: 0 REJOIN lines
and exactly 1 RE-ARM line across a full 3-mission rig session.
VERIFIED
* scratchpad/test_relay_rearm.py -- 6 groups, all passing: the exact wedge
state recovers; a stale stop is consumed while idle; staging is NOT restarted
under an impatient operator; mid-pair never re-arms; the happy path is
untouched and RunMission still reaches the pods; the reaper drops a silent
conn and keeps a live one, and stands down mid-mission.
* Live 2-pod rig (scratchpad/rig_relay.ps1 + relay_ctl.py over the control
port): two clean back-to-back missions, StopMission, round RESET, a live
`rearm`, and LAUNCH-with-nobody-present now printing why instead of nothing.
NOT reproduced end-to-end: the field wedge needs 3+ pods with staggered rejoins
(this rig's pods re-exec together, so the all-gone reset always fires). The
state itself is covered by the unit test. Awaiting live confirmation on a real
games night.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
48d47ef806
commit
4736cba1ca
+73
-17
@@ -75,6 +75,15 @@ class SessionMonitor:
|
||||
RE_RELAY_HELD = re.compile(
|
||||
r"launch HELD -- still loading: (.+) \((\d+)/(\d+) ready\)")
|
||||
RE_RELAY_RESET = re.compile(r"round RESET")
|
||||
# A ROUND HAS ENDED / THE LAUNCHER RE-ARMED. Before these, `launched`
|
||||
# could only be cleared by "WAITING FOR OPERATOR" (needs every seat of the
|
||||
# last round to re-ACK) or "round RESET" (needs every pod gone). A games
|
||||
# night sits between those -- one player does not come back -- so the
|
||||
# button stayed greyed out and the operator had to restart the session
|
||||
# (report 2026-07-25). StopMission is the honest "the mission is over"
|
||||
# signal, and RE-ARM is the relay saying the launcher is ready again.
|
||||
RE_RELAY_REARM = re.compile(r"RE-ARM for a new round")
|
||||
RE_RELAY_STOPPED = re.compile(r"StopMission sent")
|
||||
RE_MESH_EGG = re.compile(r"\[([^\]]+)\] egg sent")
|
||||
RE_MESH_RUN = re.compile(r"\[([^\]]+)\] RunMission #(\d+) sent")
|
||||
|
||||
@@ -146,6 +155,13 @@ class SessionMonitor:
|
||||
self.held_status = "%s still loading (%s/%s ready)" % (
|
||||
m.group(1), m.group(2), m.group(3))
|
||||
changed = True
|
||||
if (self.RE_RELAY_REARM.search(line)
|
||||
or self.RE_RELAY_STOPPED.search(line)):
|
||||
# The round is over (or the relay re-armed): the LAUNCH button
|
||||
# must come back WITHOUT needing a full-roster re-ACK.
|
||||
self.launched = False
|
||||
self.held_status = None
|
||||
changed = True
|
||||
if self.RE_RELAY_RESET.search(line):
|
||||
# BETWEEN ROUNDS: re-arm the console for the next launch. The
|
||||
# old re-arm rode "WAITING FOR OPERATOR" which only prints on a
|
||||
@@ -454,6 +470,18 @@ class Operator(QMainWindow):
|
||||
"font-weight:bold; color:#cc5533; padding:4px 14px;")
|
||||
self.end_btn.clicked.connect(self._end_mission)
|
||||
self.end_btn.setEnabled(False)
|
||||
# ESCAPE HATCH (2026-07-26). When the launcher latched after a round,
|
||||
# restarting the whole session was the ONLY way out -- which also
|
||||
# disconnected every player who was still waiting. This asks the relay
|
||||
# to clear just the round state and keep the seats.
|
||||
self.rearm_btn = QPushButton("↻ Re-arm")
|
||||
self.rearm_btn.setToolTip(
|
||||
"Clear the finished round's state and re-open the launcher, "
|
||||
"keeping everyone who is already connected. "
|
||||
"Use this if LAUNCH looks dead after a round instead of "
|
||||
"restarting the session.")
|
||||
self.rearm_btn.clicked.connect(self._rearm_round)
|
||||
self.rearm_btn.setEnabled(False)
|
||||
self.launch_local_btn = QPushButton("Launch local instances")
|
||||
self.launch_local_btn.clicked.connect(self._launch_local)
|
||||
self.launch_local_btn.setEnabled(False)
|
||||
@@ -465,6 +493,7 @@ class Operator(QMainWindow):
|
||||
srow.addSpacing(20)
|
||||
srow.addWidget(self.launch_btn)
|
||||
srow.addWidget(self.end_btn)
|
||||
srow.addWidget(self.rearm_btn)
|
||||
srow.addSpacing(20)
|
||||
srow.addWidget(self.launch_local_btn)
|
||||
srow.addWidget(self.apply_btn)
|
||||
@@ -799,6 +828,7 @@ class Operator(QMainWindow):
|
||||
self.console_proc.start(sys.executable, ["-u"] + args)
|
||||
self.start_btn.setEnabled(False)
|
||||
self.stop_btn.setEnabled(True)
|
||||
self.rearm_btn.setEnabled(True)
|
||||
self.restart_btn.setEnabled(True)
|
||||
self.launch_local_btn.setEnabled(True)
|
||||
self.launch_btn.setEnabled(False)
|
||||
@@ -826,6 +856,7 @@ class Operator(QMainWindow):
|
||||
self.remote_link.send("get mission")
|
||||
self.start_btn.setEnabled(False)
|
||||
self.stop_btn.setEnabled(True)
|
||||
self.rearm_btn.setEnabled(True)
|
||||
self.restart_btn.setEnabled(True)
|
||||
self.launch_local_btn.setEnabled(True)
|
||||
self.launch_btn.setEnabled(False)
|
||||
@@ -840,6 +871,7 @@ class Operator(QMainWindow):
|
||||
self.remote_link = None
|
||||
self.start_btn.setEnabled(True)
|
||||
self.stop_btn.setEnabled(False)
|
||||
self.rearm_btn.setEnabled(False)
|
||||
self.restart_btn.setEnabled(False)
|
||||
self.launch_btn.setEnabled(False)
|
||||
self.end_btn.setEnabled(False)
|
||||
@@ -860,6 +892,7 @@ class Operator(QMainWindow):
|
||||
self._stop_games()
|
||||
self.start_btn.setEnabled(True)
|
||||
self.stop_btn.setEnabled(False)
|
||||
self.rearm_btn.setEnabled(False)
|
||||
self.restart_btn.setEnabled(False)
|
||||
self.launch_btn.setEnabled(False)
|
||||
self.end_btn.setEnabled(False)
|
||||
@@ -877,35 +910,52 @@ class Operator(QMainWindow):
|
||||
|
||||
def _launch_mission(self):
|
||||
"""Manual launch gate: send the relay its 'launch' command."""
|
||||
if self._relay_send("launch"):
|
||||
# Grey the button only once the command actually LEFT. The refresh
|
||||
# tick re-derives this from the relay's own reports, so if the relay
|
||||
# declines the press the button comes back rather than staying dead
|
||||
# (which is what forced a session restart before 2026-07-26).
|
||||
self.launch_btn.setEnabled(False)
|
||||
self.end_btn.setEnabled(True)
|
||||
|
||||
def _relay_send(self, command):
|
||||
"""Send an operator command and report HONESTLY whether it went.
|
||||
|
||||
The old call sites wrote to the QProcess and logged success
|
||||
unconditionally -- but _console_finished leaves console_proc non-None,
|
||||
so after the relay died every press logged ">> sent" into the void.
|
||||
"""
|
||||
if getattr(self, "remote_link", None):
|
||||
self.remote_link.send("launch")
|
||||
self.launch_btn.setEnabled(False)
|
||||
self.end_btn.setEnabled(True)
|
||||
self.log.appendPlainText(">> operator LAUNCH sent (remote)")
|
||||
elif self.console_proc:
|
||||
self.console_proc.write(b"launch\n")
|
||||
self.launch_btn.setEnabled(False)
|
||||
self.end_btn.setEnabled(True)
|
||||
self.log.appendPlainText(">> operator LAUNCH sent")
|
||||
self.remote_link.send(command)
|
||||
self.log.appendPlainText(">> operator %s sent (remote)"
|
||||
% command.upper())
|
||||
return True
|
||||
proc = self.console_proc
|
||||
if proc is None or proc.state() != QProcess.Running:
|
||||
self.log.appendPlainText(
|
||||
"!! %s NOT sent -- the console/relay is not running "
|
||||
"(press Start Session)" % command.upper())
|
||||
return False
|
||||
proc.write((command + "\n").encode())
|
||||
self.log.appendPlainText(">> operator %s sent" % command.upper())
|
||||
return True
|
||||
|
||||
def _rearm_round(self):
|
||||
"""Ask the relay to clear a finished round's state without a restart."""
|
||||
self._relay_send("rearm")
|
||||
|
||||
def _end_mission(self):
|
||||
"""End the running mission now (the relay also auto-stops at the
|
||||
egg's [mission] length -- the authentic console mission clock)."""
|
||||
if getattr(self, "remote_link", None):
|
||||
self.remote_link.send("stop")
|
||||
if self._relay_send("stop"):
|
||||
self.end_sent = True
|
||||
self.end_btn.setEnabled(False)
|
||||
self.log.appendPlainText(">> operator END MISSION sent (remote)")
|
||||
elif self.console_proc:
|
||||
self.console_proc.write(b"stop\n")
|
||||
self.end_sent = True
|
||||
self.end_btn.setEnabled(False)
|
||||
self.log.appendPlainText(">> operator END MISSION sent")
|
||||
|
||||
def _console_finished(self):
|
||||
self.log.appendPlainText("== console/relay process exited ==")
|
||||
self.start_btn.setEnabled(True)
|
||||
self.stop_btn.setEnabled(False)
|
||||
self.rearm_btn.setEnabled(False)
|
||||
|
||||
def _console_output(self):
|
||||
if not self.console_proc:
|
||||
@@ -975,6 +1025,12 @@ class Operator(QMainWindow):
|
||||
if (self.console_proc and not self.end_btn.isEnabled()
|
||||
and not getattr(self, "end_sent", False)):
|
||||
self.end_btn.setEnabled(True)
|
||||
elif getattr(self, "end_sent", False):
|
||||
# END MISSION is PER ROUND, not per session. end_sent used to be
|
||||
# cleared only by Start Session, so the button worked exactly once a
|
||||
# night; and a stale press between rounds used to kill the next
|
||||
# mission the instant it launched (the relay now refuses that too).
|
||||
self.end_sent = False
|
||||
elif self.monitor.held_status:
|
||||
head = "STAGING — " + self.monitor.held_status
|
||||
elif ready_n:
|
||||
|
||||
Reference in New Issue
Block a user