diff --git a/context/operator-console.md b/context/operator-console.md index 8c2f4c4..ead8deb 100644 --- a/context/operator-console.md +++ b/context/operator-console.md @@ -313,14 +313,21 @@ The production topology: **the relay stays parked on one machine forever** (so p Verified by reading the control handler (`tools/btconsole.py` `_ctl_command`, `_ctl_get_mission`, `_ctl_set_mission`) — this is the authoritative list, so nobody has to re-read it again. -**Park the relay** (standalone, no GUI — the control port is what makes this viable, since a -backgrounded relay has no stdin): +**Park the relay** — use the supervisor `tools/btrelay_park.py` (or double-click +`tools\park_relay.cmd`, which is also what goes in `shell:startup` for start-at-boot): ``` -cd C:\git\bt411\content -python ..\tools\btconsole.py --relay 1500 OPERATOR.EGG +python tools\btrelay_park.py --egg OPERATOR.EGG --port 1500 ``` +It runs the relay with `cwd = content\` (which fixes *which* `operator_secret.txt` is live — see +the trap below), **relaunches it on any exit**, backs off 2→5→15→30→60 s when a relay dies faster +than 20 s so a permanent fault (port taken, missing egg) cannot spin, and rotates +`content\parked_relay.log` to `.1` so the dead generation's evidence survives its replacement. +Ctrl-C stops it for good. Deliberately **not** a Windows service — session 0 would hide the very +window an operator wants to tail. A bare `python tools\btconsole.py --relay 1500 OPERATOR.EGG` from +`content\` still works for a one-off; it just has nothing watching it. + **What a remote console CAN do** (control port = console port + 7, i.e. **1507**; `AUTH `): | Command | Effect | @@ -330,6 +337,7 @@ python ..\tools\btconsole.py --relay 1500 OPERATOR.EGG | `rearm` / `newround` | the re-arm escape hatch (recover a dead LAUNCH between rounds) | | `get` | current mission settings + live seat count | | `set map=…;time=…;weather=…;scenario=…;temperature=…;length=…` | writes **the relay's own egg on the parked machine** (`CONTROL_SET_KEYS` whitelist), effective next round | +| `restart` | **exits the relay with code 42 so the supervisor relaunches it, re-reading the egg** — the only route to a roster resize or a wedge-clear from the road. Refused mid-mission (`launches_sent >= 2 and not stop_sent`): it would drop every pod out of a live round. The GUI's **Restart Session** sends this in remote mode and reconnects ~6 s later (it previously just disconnected you, which read as "Restart does nothing"). | | `ping` | liveness | | *(implicit)* | the full live log stream + the roster, re-issued from live state at every AUTH | @@ -337,10 +345,12 @@ So a remote operator can run an entire night: stage, launch, end, re-arm, change length between rounds, and watch real pilot lights. **What a remote console CANNOT do** (these need access to the parked machine itself): -- **Start the relay** — remote mode *attaches*; it cannot create a relay. -- **Restart it.** `Stop Session` only disconnects the operator; `Restart Session` just reconnects. -- **Change the seat count** — the relay refuses (`roster changed N->M seats -- IGNORED`), and - resizing needs a relay restart. +- **Start the relay** — remote mode *attaches*; it cannot create one. That is the supervisor's job. +- **Change the seat count in one step.** The running relay refuses a resize + (`roster changed N->M seats -- IGNORED`); the egg on the parked machine has to change *and* the + relay has to restart. `restart` now supplies the second half from the road — writing that egg's + roster still needs local access, since only the six mission keys travel over the wire. + (`Stop Session` remains disconnect-only, which is what you want against a parked relay.) - **Edit the roster** — callsign / mech / colour / badge / patch / experience are *not* in `CONTROL_SET_KEYS`; only the six mission keys travel over the wire. diff --git a/docs/OPERATOR_GUIDE.md b/docs/OPERATOR_GUIDE.md index a9c4de5..d6abe3a 100644 --- a/docs/OPERATOR_GUIDE.md +++ b/docs/OPERATOR_GUIDE.md @@ -136,12 +136,13 @@ The recommended setup for a travelling operator: the relay lives permanently on so **no player ever edits `join.bat`**, and you drive it from a laptop — cell internet is fine, since the console only makes an outbound connection. -**On the home machine, before you leave:** +**On the home machine, before you leave** — double-click **`tools\park_relay.cmd`** (or run +`python tools\btrelay_park.py --egg OPERATOR.EGG --port 1500`). -``` -cd C:\git\bt411\content -python ..\tools\btconsole.py --relay 1500 OPERATOR.EGG -``` +That window is the supervisor: it keeps the relay alive, **relaunching it if it ever dies** and +rotating the old log to `content\parked_relay.log.1` so you can still see what happened. Closing +the window (or Ctrl-C) stops the relay for good. To have it come back after a reboot, put a +*shortcut* to `park_relay.cmd` in your Startup folder (Win+R → `shell:startup`) — no admin needed. Leave it running. Note the port it prints for operator control (**1507** = console port + 7) and copy `content\operator_secret.txt` — that string is what any console needs to connect, and it grants @@ -152,9 +153,16 @@ full mission control, so hand it out deliberately. weather / scenario / temperature / length, applied to the egg on the home machine, effective next round) and live pilot lights. -**What still needs someone at the home machine:** starting the relay if it is not running, -restarting it, changing the **seat count**, and editing callsigns / mechs / colours — only the six -mission settings travel over the control link. +**Restart Session now really restarts the parked relay.** In remote mode the button asks the relay +to bounce itself; the supervisor brings it straight back with the egg re-read, and your console +reconnects a few seconds later. It is refused while a mission is running (it would drop everyone +mid-round) — press End Mission first. Use it to clear a wedge, or after someone has edited the egg +on the home machine. + +**What still needs someone at the home machine:** starting the supervisor in the first place, and +editing the roster itself — seat count, callsigns, mechs, colours. Only the six mission settings +travel over the control link, so a *roster* change means editing the egg there and then pressing +Restart Session from wherever you are. **Two traps worth knowing:** - **The secret depends on the relay's working directory.** It is read as a bare `operator_secret.txt` diff --git a/scratchpad/test_parked_supervisor.py b/scratchpad/test_parked_supervisor.py new file mode 100644 index 0000000..fa96cb8 --- /dev/null +++ b/scratchpad/test_parked_supervisor.py @@ -0,0 +1,193 @@ +"""Supervisor test: the PARKED relay must outlive crashes and remote restarts. + +Covers the two things a travelling operator cannot do by hand: + * the relay dies (crash / killed) -> supervisor relaunches it + * the operator sends `restart` remotely -> relay exits 42, comes back with + the egg RE-READ (roster resize) + +Also asserts the restart is REFUSED mid-mission (it would drop every pod out of +a live round) and that the log rotation keeps the dead generation's evidence. + +Spawns its own supervisor and kills only that tree. +""" +import os +import re +import shutil +import socket +import subprocess +import sys +import time + +REPO = r"C:\git\bt411" +CONTENT = os.path.join(REPO, "content") +sys.path.insert(0, os.path.join(REPO, "tools")) + +PORT = 1500 +CTL = PORT + 7 +SRC_EGG = "FOGDAY.EGG" +EGG = "_park_test.EGG" +LOG = os.path.join(CONTENT, "parked_relay.log") + +fails = [] +sup = None + + +def check(label, ok, extra=""): + print(" %-56s %s %s" % (label, "OK" if ok else "*** FAIL ***", extra)) + if not ok: + fails.append(label) + + +def log_text(path=LOG): + try: + with open(path, "r", encoding="utf-8", errors="replace") as f: + return f.read() + except OSError: + return "" + + +def wait_for(pattern, timeout, path=LOG): + end = time.time() + timeout + while time.time() < end: + if re.search(pattern, log_text(path)): + return True + time.sleep(0.25) + return False + + +def relay_pids(): + out = subprocess.run( + ["wmic", "process", "where", "name='python.exe'", + "get", "ProcessId,CommandLine"], + capture_output=True, text=True).stdout + pids = [] + for line in out.splitlines(): + if "btconsole.py" in line: + m = re.search(r"(\d+)\s*$", line.strip()) + if m: + pids.append(int(m.group(1))) + return pids + + +def ctl(command, secret): + """One control-port command, as a remote operator would send it.""" + s = socket.create_connection(("127.0.0.1", CTL), timeout=10) + try: + s.sendall(("AUTH %s\n" % secret).encode()) + time.sleep(0.6) + s.sendall((command + "\n").encode()) + time.sleep(1.2) + # The relay ACKs `restart` and then exits, so the socket may be reset + # under our recv -- that is CORRECT behaviour, not a failure. Drain + # whatever arrived and treat a reset as end-of-stream. + s.settimeout(2) + got = b"" + try: + while True: + chunk = s.recv(4096) + if not chunk: + break + got += chunk + except (socket.timeout, ConnectionResetError, OSError): + pass + return got.decode("utf-8", "replace") + finally: + s.close() + + +try: + if relay_pids(): + sys.exit("ABORT: a relay is already running -- refusing to disturb it") + + shutil.copyfile(os.path.join(CONTENT, SRC_EGG), os.path.join(CONTENT, EGG)) + with open(os.path.join(CONTENT, "operator_secret.txt"), + encoding="ascii") as f: + secret = f.read().strip() + + SUPLOG = os.path.join(os.environ.get("TEMP", "."), "park_sup.log") + suplog = open(SUPLOG, "w", encoding="utf-8") + sup = subprocess.Popen( + [sys.executable, "-u", os.path.join(REPO, "tools", "btrelay_park.py"), + "--egg", EGG, "--port", str(PORT)], + cwd=REPO, stdout=suplog, stderr=subprocess.STDOUT) + + check("supervisor brought the relay up", + wait_for(r"operator control on port %d" % CTL, 25)) + first = relay_pids() + check("relay process is running", len(first) == 1, "pids=%s" % first) + + # ---- 1. the relay DIES -> supervisor relaunches it ------------------- + subprocess.run(["taskkill", "/PID", str(first[0]), "/F"], + capture_output=True) + time.sleep(1) + back = False + for _ in range(40): + now = relay_pids() + if now and now != first: + back = True + break + time.sleep(1) + check("relay relaunched after being killed", back, + "new pids=%s" % relay_pids()) + check("the dead generation's log was kept as .1", + os.path.exists(LOG + ".1")) + check("new generation is serving again", + wait_for(r"operator control on port %d" % CTL, 25)) + + # ---- 2. remote `restart` -------------------------------------------- + before = relay_pids() + reply = ctl("restart", secret) + check("relay acknowledged the remote restart", + "restarting" in reply, reply.strip()[:60]) + swapped = False + for _ in range(40): + now = relay_pids() + if now and now != before: + swapped = True + break + time.sleep(1) + check("remote restart produced a NEW relay process", swapped, + "%s -> %s" % (before, relay_pids())) + suptext = log_text(SUPLOG) + check("supervisor called it an operator restart, NOT a crash", + "operator requested a restart" in suptext + and "relay exited 42" not in suptext, ) + check("supervisor treated the earlier KILL as a death (backoff path)", + re.search(r"relay exited \d+ after .*relaunching in \d+s", suptext) + is not None) + check("relay is serving after the remote restart", + wait_for(r"operator control on port %d" % CTL, 25)) + + # ---- 3. a roster resize really takes effect on restart --------------- + # (the thing that was impossible from the road) + doc = open(os.path.join(CONTENT, EGG), "rb").read() + check("egg is re-read on restart (roster line present in the new log)", + wait_for(r"roster: \d+ pilot\(s\)", 20)) + + # ---- 4. restart must be REFUSED mid-mission -------------------------- + # Simulate a running mission by driving the relay's own state via the + # control channel is not possible without pods, so assert the guard + # exists and is wired to launches_sent/stop_sent. + src = open(os.path.join(REPO, "tools", "btconsole.py"), + encoding="utf-8").read() + guard = re.search(r'elif cmd == "restart":(.{0,600})', src, re.S) + check("mid-mission guard present in the restart command", + bool(guard) and "launches_sent >= 2" in guard.group(1) + and "not self.stop_sent" in guard.group(1)) + +finally: + if sup is not None: + for pid in relay_pids(): + subprocess.run(["taskkill", "/PID", str(pid), "/F", "/T"], + capture_output=True) + subprocess.run(["taskkill", "/PID", str(sup.pid), "/F", "/T"], + capture_output=True) + for junk in (os.path.join(CONTENT, EGG), LOG, LOG + ".1"): + try: + os.remove(junk) + except OSError: + pass + +print() +print("ALL PASS" if not fails else "FAILURES: %s" % fails) +sys.exit(1 if fails else 0) diff --git a/tools/btconsole.py b/tools/btconsole.py index d8757f9..f11643f 100644 --- a/tools/btconsole.py +++ b/tools/btconsole.py @@ -316,6 +316,8 @@ CONSOLE_MSG_MAX = 65536 CONSOLE_INBUF_MAX = 1 << 20 CONTROL_PORT_OFFSET = 7 # control port = console port + 7 +RESTART_EXIT_CODE = 42 # relay_main's exit code for `restart` + # (btrelay_park.py relaunches on it) CONTROL_AUTH_TIMEOUT = 10.0 # unauthenticated sockets die after this CONTROL_OUTBUF_MAX = 262144 # slow-reader cutoff (bytes) CONTROL_SET_KEYS = {"map", "time", "weather", "scenario", "temperature", @@ -547,6 +549,7 @@ class Relay: "udp_dropped": 0, "udp_tcp_fallback": 0} self.stats_at = time.time() + STATS_PERIOD self.ctl_conns = [] # RelayControlConn list + self.restart_requested = False # remote `restart` (parked relay) self.ctl_secret = control_secret() # ---------------- lifecycle ---------------- @@ -617,6 +620,15 @@ class Relay: self._tick_stop() self._tick_settle() self._tick_stats() + if self.restart_requested: + # Remote `restart` (parked-relay deployment): leave the loop so + # relay_main can exit with RESTART_EXIT_CODE and the supervisor + # relaunches us with a fresh egg read. This is the ONLY way a + # travelling operator can resize the roster or recover a wedged + # relay -- the control channel cannot do either in-process. + print("[relay] RESTART requested by the operator -- closing " + "for the supervisor to relaunch", flush=True) + return def _listener(self, port): s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) @@ -1866,6 +1878,21 @@ class Relay: # out of a latched round (operator report 2026-07-25). print(f"[ctl] RE-ARM from {conn.name()}", flush=True) self._rearm_for_new_round("operator re-arm command") + elif cmd == "restart": + # Parked-relay deployment: the operator is not at the machine, so + # this is their only route to a fresh relay (roster resize, or a + # wedge nothing else clears). Refused mid-mission -- it would + # drop every pod out of a running round. + if self.launches_sent >= 2 and not self.stop_sent: + conn.queue_out(b"[ctl] restart REFUSED -- a mission is " + b"running; press End Mission first\n") + print("[ctl] restart refused (mission running)", flush=True) + else: + conn.queue_out(b"[ctl] restarting -- reconnect in a few " + b"seconds\n") + conn.flush_out() + print(f"[ctl] RESTART from {conn.name()}", flush=True) + self.restart_requested = True elif cmd == "ping": conn.queue_out(b"[ctl] pong\n") elif cmd == "get": @@ -2016,6 +2043,9 @@ def relay_main(argv): elif command in ("rearm", "newround"): print("[relay] operator RE-ARM command received", flush=True) relay._rearm_for_new_round("operator re-arm command") + elif command == "restart": + print("[relay] operator RESTART command received", flush=True) + relay.restart_requested = True elif command: print(f"[relay] unknown operator command: {command!r}", flush=True) @@ -2024,6 +2054,9 @@ def relay_main(argv): relay.run() except KeyboardInterrupt: print("[relay] shutting down") + return 0 + if relay.restart_requested: + return RESTART_EXIT_CODE # the supervisor's cue to relaunch return 0 diff --git a/tools/btoperator.py b/tools/btoperator.py index a8b4a51..fa9a0e9 100644 --- a/tools/btoperator.py +++ b/tools/btoperator.py @@ -24,7 +24,8 @@ import socket import sys import threading -from PySide6.QtCore import Qt, QProcess, QProcessEnvironment, QThread, Signal +from PySide6.QtCore import (Qt, QProcess, QProcessEnvironment, QThread, QTimer, + Signal) from PySide6.QtGui import QColor, QFont, QPalette, QAction from PySide6.QtWidgets import ( QApplication, QCheckBox, QComboBox, QFileDialog, QFormLayout, QGroupBox, @@ -966,6 +967,21 @@ class Operator(QMainWindow): def _restart_session(self): """Next round: same mission/roster, fresh session. Players just re-run their join script.""" + if getattr(self, "remote_link", None): + # PARKED RELAY: we do not own that process, so tearing down our + # own link would only disconnect us (the old behaviour, which + # looked like "Restart did nothing"). Ask the relay itself to + # restart -- its supervisor relaunches it, re-reading the egg, so + # a roster resize or a wedge is recoverable from the road. Then + # reconnect once it is back up. + host = self.f_relayhost.text().strip() + self.log.appendPlainText( + "== asking the parked relay at %s to RESTART " + "(reconnecting in a few seconds) ==" % host) + self._relay_send("restart") + self._stop_session() + QTimer.singleShot(6000, self._start_session) + return self.log.appendPlainText("== restarting session ==") self._stop_session() self._start_session() diff --git a/tools/btrelay_park.py b/tools/btrelay_park.py new file mode 100644 index 0000000..aca5a38 --- /dev/null +++ b/tools/btrelay_park.py @@ -0,0 +1,152 @@ +#!/usr/bin/env python3 +"""btrelay_park.py -- keep the relay PARKED and alive on the host machine. + +The travelling-operator deployment: the relay lives permanently on one machine +so every player's join.bat address never changes, and the operator console +dials into its control port from anywhere. For that to be true the relay has +to outlive crashes, reboots and roster changes without anyone at the keyboard -- +which is what this supervisor is for. + + python tools/btrelay_park.py [--egg OPERATOR.EGG] [--port 1500] + [--manual-launch] [--once] + +What it does: + * starts the relay with cwd = content\\ (the working directory decides which + operator_secret.txt is live -- see context/operator-console.md); + * relaunches it when it exits, whether that was a crash or the operator's + remote `restart` (RESTART_EXIT_CODE), with a short backoff so a relay that + dies instantly cannot spin the CPU; + * writes everything to content\\parked_relay.log, rotating the previous run + to .1 so a crash's evidence survives the relaunch; + * stops cleanly on Ctrl-C, taking the relay with it. + +Deliberately NOT a Windows service: a service runs in session 0 and the relay +is something the operator wants to see, tail and Ctrl-C. For start-at-boot, +put a shortcut to tools\\park_relay.cmd in the Startup folder (shell:startup) -- +no elevation, trivially removable. +""" +import argparse +import os +import subprocess +import sys +import time + +REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +CONTENT = os.path.join(REPO, "content") +RESTART_EXIT_CODE = 42 # must match tools/btconsole.py + +# A relay that dies faster than this was not really running: back off so a +# permanent fault (port in use, missing egg) cannot become a spin loop. +HEALTHY_SECONDS = 20 +BACKOFF = [2, 5, 15, 30, 60] + + +def rotate(path): + try: + if os.path.exists(path): + prev = path + ".1" + if os.path.exists(prev): + os.remove(prev) + os.replace(path, prev) + except OSError: + pass # a locked log must never stop the relay + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--egg", default="OPERATOR.EGG") + ap.add_argument("--port", type=int, default=1500) + ap.add_argument("--manual-launch", action="store_true", default=True, + help="hold the launch for the operator (the default)") + ap.add_argument("--auto-launch", dest="manual_launch", + action="store_false", + help="let the relay launch by itself once all pods stage") + ap.add_argument("--once", action="store_true", + help="do not relaunch -- run the relay a single time") + args = ap.parse_args() + + egg = args.egg if os.path.isabs(args.egg) \ + else os.path.join(CONTENT, args.egg) + if not os.path.exists(egg): + sys.exit("no such egg: %s" % egg) + + secret = os.path.join(CONTENT, "operator_secret.txt") + log_path = os.path.join(CONTENT, "parked_relay.log") + + print("=" * 68) + print(" BattleTech relay -- PARKED") + print(" egg : %s" % egg) + print(" ports : console %d | game %d | operator control %d" + % (args.port, args.port + 1, args.port + 7)) + print(" secret : %s%s" % (secret, + "" if os.path.exists(secret) + else " (will be generated)")) + print(" log : %s (previous run kept as .1)" % log_path) + print(" launch : %s" % ("operator presses LAUNCH" + if args.manual_launch else "automatic")) + print(" Ctrl-C here stops the relay for good; everything else") + print(" (crash, or the operator's remote `restart`) relaunches it.") + print("=" * 68, flush=True) + + cmd = [sys.executable, "-u", os.path.join(REPO, "tools", "btconsole.py"), + "--relay", str(args.port), egg] + if args.manual_launch: + cmd.append("--manual-launch") + + generation = 0 + strikes = 0 + while True: + generation += 1 + rotate(log_path) + started = time.time() + print("[park] generation %d starting %s" + % (generation, time.strftime("%H:%M:%S")), flush=True) + try: + with open(log_path, "w", encoding="utf-8") as log: + proc = subprocess.Popen(cmd, cwd=CONTENT, stdout=log, + stderr=subprocess.STDOUT) + code = proc.wait() + except KeyboardInterrupt: + print("\n[park] Ctrl-C -- stopping the parked relay", flush=True) + try: + proc.terminate() + except Exception: + pass + return 0 + except OSError as e: + print("[park] could not start the relay: %r" % e, flush=True) + return 1 + + lived = time.time() - started + if args.once: + print("[park] --once: relay exited %d after %.0fs" % (code, lived), + flush=True) + return code + + if code == RESTART_EXIT_CODE: + strikes = 0 # an ASKED-FOR restart is healthy + print("[park] operator requested a restart -- relaunching " + "(the egg is re-read, so roster changes take effect)", + flush=True) + time.sleep(1) + continue + + # Anything else is a death: crash, killed, or a startup failure. + if lived >= HEALTHY_SECONDS: + strikes = 0 + wait = BACKOFF[min(strikes, len(BACKOFF) - 1)] + strikes += 1 + print("[park] relay exited %d after %.0fs -- relaunching in %ds " + "(evidence in %s.1)" % (code, lived, wait, log_path), flush=True) + try: + time.sleep(wait) + except KeyboardInterrupt: + print("\n[park] Ctrl-C -- staying down", flush=True) + return 0 + + +if __name__ == "__main__": + try: + sys.exit(main()) + except KeyboardInterrupt: + sys.exit(0) diff --git a/tools/park_relay.cmd b/tools/park_relay.cmd new file mode 100644 index 0000000..47edb18 --- /dev/null +++ b/tools/park_relay.cmd @@ -0,0 +1,17 @@ +@echo off +REM =================================================================== +REM Park the BattleTech relay on this machine and keep it alive. +REM +REM Double-click to run it now, or put a SHORTCUT to this file in the +REM Startup folder (Win+R -> shell:startup) so the relay comes back by +REM itself after a reboot. No admin rights needed either way. +REM +REM The window stays open: it is the supervisor. Closing it (or Ctrl-C) +REM stops the relay for good. +REM =================================================================== +title BattleTech relay (parked) +cd /d "%~dp0.." +python "tools\btrelay_park.py" %* +echo. +echo The parked relay has stopped. +pause