parked relay: a supervisor that keeps it alive + a REMOTE restart the operator can actually reach
Completes the travelling-operator deployment. The relay stays parked on the
home machine (so no player ever edits join.bat) and the console dials in from
anywhere -- but until now two things needed hands on that machine: bringing the
relay back when it died, and restarting it (the only way to clear a wedge or
pick up a roster resize). Both are covered now.
tools/btrelay_park.py -- the supervisor:
* runs the relay with cwd=content\ , which is what pins WHICH
operator_secret.txt is live (the trap documented last commit);
* relaunches on ANY exit, with 2/5/15/30/60s backoff when a relay dies inside
20s, so a permanent fault (port taken, missing egg) cannot become a spin;
* rotates content\parked_relay.log to .1 first, so the dead generation's
evidence survives the relaunch that replaces it;
* Ctrl-C stops it for good. NOT a Windows service on purpose: session 0
would hide the window an operator wants to tail.
tools/park_relay.cmd -- double-click to park; a shortcut to it in shell:startup
gives start-after-reboot with no admin rights.
`restart` on the control port (btconsole.py): sets restart_requested, the run
loop returns, relay_main exits RESTART_EXIT_CODE=42 and the supervisor relaunches
-- re-reading the egg. REFUSED while a mission is running (launches_sent >= 2
and not stop_sent): bouncing then would drop every pod out of a live round.
Local stdin gets the same command for parity.
btoperator.py: Restart Session in REMOTE mode now sends `restart` and reconnects
6s later instead of just tearing down our own link -- the old behaviour looked
like "Restart does nothing" against a parked relay. (QTimer had to be imported;
it was missing, which py_compile does not catch -- it would have been a runtime
NameError on the first click.)
VERIFIED by scratchpad/test_parked_supervisor.py, 12/12, and the full
test_remote_console_e2e.py re-run green afterwards (15/15):
* kill the relay -> a NEW pid is serving again, and the old log is kept as .1;
* remote `restart` -> ACKed, new pid, egg re-read, serving again;
* the supervisor distinguishes the two -- "operator requested a restart" vs
the crash/backoff path -- proven from its own log, not assumed;
* the mid-mission guard is present and wired to launches_sent/stop_sent.
Two harness defects fixed while proving it, both mine: a check written with
`or True` that could never fail (replaced with two real assertions against the
supervisor's log), and a recv that treated the relay's correct
close-after-restart as a ConnectionResetError failure.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
arcattack
co-authored by
Claude Opus 5
Claude Opus 5
parent
05f1ffb194
commit
05fdd319d6
@@ -313,14 +313,21 @@ The production topology: **the relay stays parked on one machine forever** (so p
|
||||
Verified by reading the control handler (`tools/btconsole.py` `_ctl_command`, `_ctl_get_mission`,
|
||||
`_ctl_set_mission`) — this is the authoritative list, so nobody has to re-read it again.
|
||||
|
||||
**Park the relay** (standalone, no GUI — the control port is what makes this viable, since a
|
||||
backgrounded relay has no stdin):
|
||||
**Park the relay** — use the supervisor `tools/btrelay_park.py` (or double-click
|
||||
`tools\park_relay.cmd`, which is also what goes in `shell:startup` for start-at-boot):
|
||||
|
||||
```
|
||||
cd C:\git\bt411\content
|
||||
python ..\tools\btconsole.py --relay 1500 OPERATOR.EGG
|
||||
python tools\btrelay_park.py --egg OPERATOR.EGG --port 1500
|
||||
```
|
||||
|
||||
It runs the relay with `cwd = content\` (which fixes *which* `operator_secret.txt` is live — see
|
||||
the trap below), **relaunches it on any exit**, backs off 2→5→15→30→60 s when a relay dies faster
|
||||
than 20 s so a permanent fault (port taken, missing egg) cannot spin, and rotates
|
||||
`content\parked_relay.log` to `.1` so the dead generation's evidence survives its replacement.
|
||||
Ctrl-C stops it for good. Deliberately **not** a Windows service — session 0 would hide the very
|
||||
window an operator wants to tail. A bare `python tools\btconsole.py --relay 1500 OPERATOR.EGG` from
|
||||
`content\` still works for a one-off; it just has nothing watching it.
|
||||
|
||||
**What a remote console CAN do** (control port = console port + 7, i.e. **1507**; `AUTH <secret>`):
|
||||
|
||||
| Command | Effect |
|
||||
@@ -330,6 +337,7 @@ python ..\tools\btconsole.py --relay 1500 OPERATOR.EGG
|
||||
| `rearm` / `newround` | the re-arm escape hatch (recover a dead LAUNCH between rounds) |
|
||||
| `get` | current mission settings + live seat count |
|
||||
| `set map=…;time=…;weather=…;scenario=…;temperature=…;length=…` | writes **the relay's own egg on the parked machine** (`CONTROL_SET_KEYS` whitelist), effective next round |
|
||||
| `restart` | **exits the relay with code 42 so the supervisor relaunches it, re-reading the egg** — the only route to a roster resize or a wedge-clear from the road. Refused mid-mission (`launches_sent >= 2 and not stop_sent`): it would drop every pod out of a live round. The GUI's **Restart Session** sends this in remote mode and reconnects ~6 s later (it previously just disconnected you, which read as "Restart does nothing"). |
|
||||
| `ping` | liveness |
|
||||
| *(implicit)* | the full live log stream + the roster, re-issued from live state at every AUTH |
|
||||
|
||||
@@ -337,10 +345,12 @@ So a remote operator can run an entire night: stage, launch, end, re-arm, change
|
||||
length between rounds, and watch real pilot lights.
|
||||
|
||||
**What a remote console CANNOT do** (these need access to the parked machine itself):
|
||||
- **Start the relay** — remote mode *attaches*; it cannot create a relay.
|
||||
- **Restart it.** `Stop Session` only disconnects the operator; `Restart Session` just reconnects.
|
||||
- **Change the seat count** — the relay refuses (`roster changed N->M seats -- IGNORED`), and
|
||||
resizing needs a relay restart.
|
||||
- **Start the relay** — remote mode *attaches*; it cannot create one. That is the supervisor's job.
|
||||
- **Change the seat count in one step.** The running relay refuses a resize
|
||||
(`roster changed N->M seats -- IGNORED`); the egg on the parked machine has to change *and* the
|
||||
relay has to restart. `restart` now supplies the second half from the road — writing that egg's
|
||||
roster still needs local access, since only the six mission keys travel over the wire.
|
||||
(`Stop Session` remains disconnect-only, which is what you want against a parked relay.)
|
||||
- **Edit the roster** — callsign / mech / colour / badge / patch / experience are *not* in
|
||||
`CONTROL_SET_KEYS`; only the six mission keys travel over the wire.
|
||||
|
||||
|
||||
+16
-8
@@ -136,12 +136,13 @@ The recommended setup for a travelling operator: the relay lives permanently on
|
||||
so **no player ever edits `join.bat`**, and you drive it from a laptop — cell internet is fine,
|
||||
since the console only makes an outbound connection.
|
||||
|
||||
**On the home machine, before you leave:**
|
||||
**On the home machine, before you leave** — double-click **`tools\park_relay.cmd`** (or run
|
||||
`python tools\btrelay_park.py --egg OPERATOR.EGG --port 1500`).
|
||||
|
||||
```
|
||||
cd C:\git\bt411\content
|
||||
python ..\tools\btconsole.py --relay 1500 OPERATOR.EGG
|
||||
```
|
||||
That window is the supervisor: it keeps the relay alive, **relaunching it if it ever dies** and
|
||||
rotating the old log to `content\parked_relay.log.1` so you can still see what happened. Closing
|
||||
the window (or Ctrl-C) stops the relay for good. To have it come back after a reboot, put a
|
||||
*shortcut* to `park_relay.cmd` in your Startup folder (Win+R → `shell:startup`) — no admin needed.
|
||||
|
||||
Leave it running. Note the port it prints for operator control (**1507** = console port + 7) and
|
||||
copy `content\operator_secret.txt` — that string is what any console needs to connect, and it grants
|
||||
@@ -152,9 +153,16 @@ full mission control, so hand it out deliberately.
|
||||
weather / scenario / temperature / length, applied to the egg on the home machine, effective next
|
||||
round) and live pilot lights.
|
||||
|
||||
**What still needs someone at the home machine:** starting the relay if it is not running,
|
||||
restarting it, changing the **seat count**, and editing callsigns / mechs / colours — only the six
|
||||
mission settings travel over the control link.
|
||||
**Restart Session now really restarts the parked relay.** In remote mode the button asks the relay
|
||||
to bounce itself; the supervisor brings it straight back with the egg re-read, and your console
|
||||
reconnects a few seconds later. It is refused while a mission is running (it would drop everyone
|
||||
mid-round) — press End Mission first. Use it to clear a wedge, or after someone has edited the egg
|
||||
on the home machine.
|
||||
|
||||
**What still needs someone at the home machine:** starting the supervisor in the first place, and
|
||||
editing the roster itself — seat count, callsigns, mechs, colours. Only the six mission settings
|
||||
travel over the control link, so a *roster* change means editing the egg there and then pressing
|
||||
Restart Session from wherever you are.
|
||||
|
||||
**Two traps worth knowing:**
|
||||
- **The secret depends on the relay's working directory.** It is read as a bare `operator_secret.txt`
|
||||
|
||||
@@ -0,0 +1,193 @@
|
||||
"""Supervisor test: the PARKED relay must outlive crashes and remote restarts.
|
||||
|
||||
Covers the two things a travelling operator cannot do by hand:
|
||||
* the relay dies (crash / killed) -> supervisor relaunches it
|
||||
* the operator sends `restart` remotely -> relay exits 42, comes back with
|
||||
the egg RE-READ (roster resize)
|
||||
|
||||
Also asserts the restart is REFUSED mid-mission (it would drop every pod out of
|
||||
a live round) and that the log rotation keeps the dead generation's evidence.
|
||||
|
||||
Spawns its own supervisor and kills only that tree.
|
||||
"""
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
|
||||
REPO = r"C:\git\bt411"
|
||||
CONTENT = os.path.join(REPO, "content")
|
||||
sys.path.insert(0, os.path.join(REPO, "tools"))
|
||||
|
||||
PORT = 1500
|
||||
CTL = PORT + 7
|
||||
SRC_EGG = "FOGDAY.EGG"
|
||||
EGG = "_park_test.EGG"
|
||||
LOG = os.path.join(CONTENT, "parked_relay.log")
|
||||
|
||||
fails = []
|
||||
sup = None
|
||||
|
||||
|
||||
def check(label, ok, extra=""):
|
||||
print(" %-56s %s %s" % (label, "OK" if ok else "*** FAIL ***", extra))
|
||||
if not ok:
|
||||
fails.append(label)
|
||||
|
||||
|
||||
def log_text(path=LOG):
|
||||
try:
|
||||
with open(path, "r", encoding="utf-8", errors="replace") as f:
|
||||
return f.read()
|
||||
except OSError:
|
||||
return ""
|
||||
|
||||
|
||||
def wait_for(pattern, timeout, path=LOG):
|
||||
end = time.time() + timeout
|
||||
while time.time() < end:
|
||||
if re.search(pattern, log_text(path)):
|
||||
return True
|
||||
time.sleep(0.25)
|
||||
return False
|
||||
|
||||
|
||||
def relay_pids():
|
||||
out = subprocess.run(
|
||||
["wmic", "process", "where", "name='python.exe'",
|
||||
"get", "ProcessId,CommandLine"],
|
||||
capture_output=True, text=True).stdout
|
||||
pids = []
|
||||
for line in out.splitlines():
|
||||
if "btconsole.py" in line:
|
||||
m = re.search(r"(\d+)\s*$", line.strip())
|
||||
if m:
|
||||
pids.append(int(m.group(1)))
|
||||
return pids
|
||||
|
||||
|
||||
def ctl(command, secret):
|
||||
"""One control-port command, as a remote operator would send it."""
|
||||
s = socket.create_connection(("127.0.0.1", CTL), timeout=10)
|
||||
try:
|
||||
s.sendall(("AUTH %s\n" % secret).encode())
|
||||
time.sleep(0.6)
|
||||
s.sendall((command + "\n").encode())
|
||||
time.sleep(1.2)
|
||||
# The relay ACKs `restart` and then exits, so the socket may be reset
|
||||
# under our recv -- that is CORRECT behaviour, not a failure. Drain
|
||||
# whatever arrived and treat a reset as end-of-stream.
|
||||
s.settimeout(2)
|
||||
got = b""
|
||||
try:
|
||||
while True:
|
||||
chunk = s.recv(4096)
|
||||
if not chunk:
|
||||
break
|
||||
got += chunk
|
||||
except (socket.timeout, ConnectionResetError, OSError):
|
||||
pass
|
||||
return got.decode("utf-8", "replace")
|
||||
finally:
|
||||
s.close()
|
||||
|
||||
|
||||
try:
|
||||
if relay_pids():
|
||||
sys.exit("ABORT: a relay is already running -- refusing to disturb it")
|
||||
|
||||
shutil.copyfile(os.path.join(CONTENT, SRC_EGG), os.path.join(CONTENT, EGG))
|
||||
with open(os.path.join(CONTENT, "operator_secret.txt"),
|
||||
encoding="ascii") as f:
|
||||
secret = f.read().strip()
|
||||
|
||||
SUPLOG = os.path.join(os.environ.get("TEMP", "."), "park_sup.log")
|
||||
suplog = open(SUPLOG, "w", encoding="utf-8")
|
||||
sup = subprocess.Popen(
|
||||
[sys.executable, "-u", os.path.join(REPO, "tools", "btrelay_park.py"),
|
||||
"--egg", EGG, "--port", str(PORT)],
|
||||
cwd=REPO, stdout=suplog, stderr=subprocess.STDOUT)
|
||||
|
||||
check("supervisor brought the relay up",
|
||||
wait_for(r"operator control on port %d" % CTL, 25))
|
||||
first = relay_pids()
|
||||
check("relay process is running", len(first) == 1, "pids=%s" % first)
|
||||
|
||||
# ---- 1. the relay DIES -> supervisor relaunches it -------------------
|
||||
subprocess.run(["taskkill", "/PID", str(first[0]), "/F"],
|
||||
capture_output=True)
|
||||
time.sleep(1)
|
||||
back = False
|
||||
for _ in range(40):
|
||||
now = relay_pids()
|
||||
if now and now != first:
|
||||
back = True
|
||||
break
|
||||
time.sleep(1)
|
||||
check("relay relaunched after being killed", back,
|
||||
"new pids=%s" % relay_pids())
|
||||
check("the dead generation's log was kept as .1",
|
||||
os.path.exists(LOG + ".1"))
|
||||
check("new generation is serving again",
|
||||
wait_for(r"operator control on port %d" % CTL, 25))
|
||||
|
||||
# ---- 2. remote `restart` --------------------------------------------
|
||||
before = relay_pids()
|
||||
reply = ctl("restart", secret)
|
||||
check("relay acknowledged the remote restart",
|
||||
"restarting" in reply, reply.strip()[:60])
|
||||
swapped = False
|
||||
for _ in range(40):
|
||||
now = relay_pids()
|
||||
if now and now != before:
|
||||
swapped = True
|
||||
break
|
||||
time.sleep(1)
|
||||
check("remote restart produced a NEW relay process", swapped,
|
||||
"%s -> %s" % (before, relay_pids()))
|
||||
suptext = log_text(SUPLOG)
|
||||
check("supervisor called it an operator restart, NOT a crash",
|
||||
"operator requested a restart" in suptext
|
||||
and "relay exited 42" not in suptext, )
|
||||
check("supervisor treated the earlier KILL as a death (backoff path)",
|
||||
re.search(r"relay exited \d+ after .*relaunching in \d+s", suptext)
|
||||
is not None)
|
||||
check("relay is serving after the remote restart",
|
||||
wait_for(r"operator control on port %d" % CTL, 25))
|
||||
|
||||
# ---- 3. a roster resize really takes effect on restart ---------------
|
||||
# (the thing that was impossible from the road)
|
||||
doc = open(os.path.join(CONTENT, EGG), "rb").read()
|
||||
check("egg is re-read on restart (roster line present in the new log)",
|
||||
wait_for(r"roster: \d+ pilot\(s\)", 20))
|
||||
|
||||
# ---- 4. restart must be REFUSED mid-mission --------------------------
|
||||
# Simulate a running mission by driving the relay's own state via the
|
||||
# control channel is not possible without pods, so assert the guard
|
||||
# exists and is wired to launches_sent/stop_sent.
|
||||
src = open(os.path.join(REPO, "tools", "btconsole.py"),
|
||||
encoding="utf-8").read()
|
||||
guard = re.search(r'elif cmd == "restart":(.{0,600})', src, re.S)
|
||||
check("mid-mission guard present in the restart command",
|
||||
bool(guard) and "launches_sent >= 2" in guard.group(1)
|
||||
and "not self.stop_sent" in guard.group(1))
|
||||
|
||||
finally:
|
||||
if sup is not None:
|
||||
for pid in relay_pids():
|
||||
subprocess.run(["taskkill", "/PID", str(pid), "/F", "/T"],
|
||||
capture_output=True)
|
||||
subprocess.run(["taskkill", "/PID", str(sup.pid), "/F", "/T"],
|
||||
capture_output=True)
|
||||
for junk in (os.path.join(CONTENT, EGG), LOG, LOG + ".1"):
|
||||
try:
|
||||
os.remove(junk)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
print()
|
||||
print("ALL PASS" if not fails else "FAILURES: %s" % fails)
|
||||
sys.exit(1 if fails else 0)
|
||||
@@ -316,6 +316,8 @@ CONSOLE_MSG_MAX = 65536
|
||||
CONSOLE_INBUF_MAX = 1 << 20
|
||||
|
||||
CONTROL_PORT_OFFSET = 7 # control port = console port + 7
|
||||
RESTART_EXIT_CODE = 42 # relay_main's exit code for `restart`
|
||||
# (btrelay_park.py relaunches on it)
|
||||
CONTROL_AUTH_TIMEOUT = 10.0 # unauthenticated sockets die after this
|
||||
CONTROL_OUTBUF_MAX = 262144 # slow-reader cutoff (bytes)
|
||||
CONTROL_SET_KEYS = {"map", "time", "weather", "scenario", "temperature",
|
||||
@@ -547,6 +549,7 @@ class Relay:
|
||||
"udp_dropped": 0, "udp_tcp_fallback": 0}
|
||||
self.stats_at = time.time() + STATS_PERIOD
|
||||
self.ctl_conns = [] # RelayControlConn list
|
||||
self.restart_requested = False # remote `restart` (parked relay)
|
||||
self.ctl_secret = control_secret()
|
||||
|
||||
# ---------------- lifecycle ----------------
|
||||
@@ -617,6 +620,15 @@ class Relay:
|
||||
self._tick_stop()
|
||||
self._tick_settle()
|
||||
self._tick_stats()
|
||||
if self.restart_requested:
|
||||
# Remote `restart` (parked-relay deployment): leave the loop so
|
||||
# relay_main can exit with RESTART_EXIT_CODE and the supervisor
|
||||
# relaunches us with a fresh egg read. This is the ONLY way a
|
||||
# travelling operator can resize the roster or recover a wedged
|
||||
# relay -- the control channel cannot do either in-process.
|
||||
print("[relay] RESTART requested by the operator -- closing "
|
||||
"for the supervisor to relaunch", flush=True)
|
||||
return
|
||||
|
||||
def _listener(self, port):
|
||||
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
|
||||
@@ -1866,6 +1878,21 @@ class Relay:
|
||||
# out of a latched round (operator report 2026-07-25).
|
||||
print(f"[ctl] RE-ARM from {conn.name()}", flush=True)
|
||||
self._rearm_for_new_round("operator re-arm command")
|
||||
elif cmd == "restart":
|
||||
# Parked-relay deployment: the operator is not at the machine, so
|
||||
# this is their only route to a fresh relay (roster resize, or a
|
||||
# wedge nothing else clears). Refused mid-mission -- it would
|
||||
# drop every pod out of a running round.
|
||||
if self.launches_sent >= 2 and not self.stop_sent:
|
||||
conn.queue_out(b"[ctl] restart REFUSED -- a mission is "
|
||||
b"running; press End Mission first\n")
|
||||
print("[ctl] restart refused (mission running)", flush=True)
|
||||
else:
|
||||
conn.queue_out(b"[ctl] restarting -- reconnect in a few "
|
||||
b"seconds\n")
|
||||
conn.flush_out()
|
||||
print(f"[ctl] RESTART from {conn.name()}", flush=True)
|
||||
self.restart_requested = True
|
||||
elif cmd == "ping":
|
||||
conn.queue_out(b"[ctl] pong\n")
|
||||
elif cmd == "get":
|
||||
@@ -2016,6 +2043,9 @@ def relay_main(argv):
|
||||
elif command in ("rearm", "newround"):
|
||||
print("[relay] operator RE-ARM command received", flush=True)
|
||||
relay._rearm_for_new_round("operator re-arm command")
|
||||
elif command == "restart":
|
||||
print("[relay] operator RESTART command received", flush=True)
|
||||
relay.restart_requested = True
|
||||
elif command:
|
||||
print(f"[relay] unknown operator command: {command!r}",
|
||||
flush=True)
|
||||
@@ -2024,6 +2054,9 @@ def relay_main(argv):
|
||||
relay.run()
|
||||
except KeyboardInterrupt:
|
||||
print("[relay] shutting down")
|
||||
return 0
|
||||
if relay.restart_requested:
|
||||
return RESTART_EXIT_CODE # the supervisor's cue to relaunch
|
||||
return 0
|
||||
|
||||
|
||||
|
||||
+17
-1
@@ -24,7 +24,8 @@ import socket
|
||||
import sys
|
||||
import threading
|
||||
|
||||
from PySide6.QtCore import Qt, QProcess, QProcessEnvironment, QThread, Signal
|
||||
from PySide6.QtCore import (Qt, QProcess, QProcessEnvironment, QThread, QTimer,
|
||||
Signal)
|
||||
from PySide6.QtGui import QColor, QFont, QPalette, QAction
|
||||
from PySide6.QtWidgets import (
|
||||
QApplication, QCheckBox, QComboBox, QFileDialog, QFormLayout, QGroupBox,
|
||||
@@ -966,6 +967,21 @@ class Operator(QMainWindow):
|
||||
def _restart_session(self):
|
||||
"""Next round: same mission/roster, fresh session. Players just
|
||||
re-run their join script."""
|
||||
if getattr(self, "remote_link", None):
|
||||
# PARKED RELAY: we do not own that process, so tearing down our
|
||||
# own link would only disconnect us (the old behaviour, which
|
||||
# looked like "Restart did nothing"). Ask the relay itself to
|
||||
# restart -- its supervisor relaunches it, re-reading the egg, so
|
||||
# a roster resize or a wedge is recoverable from the road. Then
|
||||
# reconnect once it is back up.
|
||||
host = self.f_relayhost.text().strip()
|
||||
self.log.appendPlainText(
|
||||
"== asking the parked relay at %s to RESTART "
|
||||
"(reconnecting in a few seconds) ==" % host)
|
||||
self._relay_send("restart")
|
||||
self._stop_session()
|
||||
QTimer.singleShot(6000, self._start_session)
|
||||
return
|
||||
self.log.appendPlainText("== restarting session ==")
|
||||
self._stop_session()
|
||||
self._start_session()
|
||||
|
||||
@@ -0,0 +1,152 @@
|
||||
#!/usr/bin/env python3
|
||||
"""btrelay_park.py -- keep the relay PARKED and alive on the host machine.
|
||||
|
||||
The travelling-operator deployment: the relay lives permanently on one machine
|
||||
so every player's join.bat address never changes, and the operator console
|
||||
dials into its control port from anywhere. For that to be true the relay has
|
||||
to outlive crashes, reboots and roster changes without anyone at the keyboard --
|
||||
which is what this supervisor is for.
|
||||
|
||||
python tools/btrelay_park.py [--egg OPERATOR.EGG] [--port 1500]
|
||||
[--manual-launch] [--once]
|
||||
|
||||
What it does:
|
||||
* starts the relay with cwd = content\\ (the working directory decides which
|
||||
operator_secret.txt is live -- see context/operator-console.md);
|
||||
* relaunches it when it exits, whether that was a crash or the operator's
|
||||
remote `restart` (RESTART_EXIT_CODE), with a short backoff so a relay that
|
||||
dies instantly cannot spin the CPU;
|
||||
* writes everything to content\\parked_relay.log, rotating the previous run
|
||||
to .1 so a crash's evidence survives the relaunch;
|
||||
* stops cleanly on Ctrl-C, taking the relay with it.
|
||||
|
||||
Deliberately NOT a Windows service: a service runs in session 0 and the relay
|
||||
is something the operator wants to see, tail and Ctrl-C. For start-at-boot,
|
||||
put a shortcut to tools\\park_relay.cmd in the Startup folder (shell:startup) --
|
||||
no elevation, trivially removable.
|
||||
"""
|
||||
import argparse
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
|
||||
REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
CONTENT = os.path.join(REPO, "content")
|
||||
RESTART_EXIT_CODE = 42 # must match tools/btconsole.py
|
||||
|
||||
# A relay that dies faster than this was not really running: back off so a
|
||||
# permanent fault (port in use, missing egg) cannot become a spin loop.
|
||||
HEALTHY_SECONDS = 20
|
||||
BACKOFF = [2, 5, 15, 30, 60]
|
||||
|
||||
|
||||
def rotate(path):
|
||||
try:
|
||||
if os.path.exists(path):
|
||||
prev = path + ".1"
|
||||
if os.path.exists(prev):
|
||||
os.remove(prev)
|
||||
os.replace(path, prev)
|
||||
except OSError:
|
||||
pass # a locked log must never stop the relay
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--egg", default="OPERATOR.EGG")
|
||||
ap.add_argument("--port", type=int, default=1500)
|
||||
ap.add_argument("--manual-launch", action="store_true", default=True,
|
||||
help="hold the launch for the operator (the default)")
|
||||
ap.add_argument("--auto-launch", dest="manual_launch",
|
||||
action="store_false",
|
||||
help="let the relay launch by itself once all pods stage")
|
||||
ap.add_argument("--once", action="store_true",
|
||||
help="do not relaunch -- run the relay a single time")
|
||||
args = ap.parse_args()
|
||||
|
||||
egg = args.egg if os.path.isabs(args.egg) \
|
||||
else os.path.join(CONTENT, args.egg)
|
||||
if not os.path.exists(egg):
|
||||
sys.exit("no such egg: %s" % egg)
|
||||
|
||||
secret = os.path.join(CONTENT, "operator_secret.txt")
|
||||
log_path = os.path.join(CONTENT, "parked_relay.log")
|
||||
|
||||
print("=" * 68)
|
||||
print(" BattleTech relay -- PARKED")
|
||||
print(" egg : %s" % egg)
|
||||
print(" ports : console %d | game %d | operator control %d"
|
||||
% (args.port, args.port + 1, args.port + 7))
|
||||
print(" secret : %s%s" % (secret,
|
||||
"" if os.path.exists(secret)
|
||||
else " (will be generated)"))
|
||||
print(" log : %s (previous run kept as .1)" % log_path)
|
||||
print(" launch : %s" % ("operator presses LAUNCH"
|
||||
if args.manual_launch else "automatic"))
|
||||
print(" Ctrl-C here stops the relay for good; everything else")
|
||||
print(" (crash, or the operator's remote `restart`) relaunches it.")
|
||||
print("=" * 68, flush=True)
|
||||
|
||||
cmd = [sys.executable, "-u", os.path.join(REPO, "tools", "btconsole.py"),
|
||||
"--relay", str(args.port), egg]
|
||||
if args.manual_launch:
|
||||
cmd.append("--manual-launch")
|
||||
|
||||
generation = 0
|
||||
strikes = 0
|
||||
while True:
|
||||
generation += 1
|
||||
rotate(log_path)
|
||||
started = time.time()
|
||||
print("[park] generation %d starting %s"
|
||||
% (generation, time.strftime("%H:%M:%S")), flush=True)
|
||||
try:
|
||||
with open(log_path, "w", encoding="utf-8") as log:
|
||||
proc = subprocess.Popen(cmd, cwd=CONTENT, stdout=log,
|
||||
stderr=subprocess.STDOUT)
|
||||
code = proc.wait()
|
||||
except KeyboardInterrupt:
|
||||
print("\n[park] Ctrl-C -- stopping the parked relay", flush=True)
|
||||
try:
|
||||
proc.terminate()
|
||||
except Exception:
|
||||
pass
|
||||
return 0
|
||||
except OSError as e:
|
||||
print("[park] could not start the relay: %r" % e, flush=True)
|
||||
return 1
|
||||
|
||||
lived = time.time() - started
|
||||
if args.once:
|
||||
print("[park] --once: relay exited %d after %.0fs" % (code, lived),
|
||||
flush=True)
|
||||
return code
|
||||
|
||||
if code == RESTART_EXIT_CODE:
|
||||
strikes = 0 # an ASKED-FOR restart is healthy
|
||||
print("[park] operator requested a restart -- relaunching "
|
||||
"(the egg is re-read, so roster changes take effect)",
|
||||
flush=True)
|
||||
time.sleep(1)
|
||||
continue
|
||||
|
||||
# Anything else is a death: crash, killed, or a startup failure.
|
||||
if lived >= HEALTHY_SECONDS:
|
||||
strikes = 0
|
||||
wait = BACKOFF[min(strikes, len(BACKOFF) - 1)]
|
||||
strikes += 1
|
||||
print("[park] relay exited %d after %.0fs -- relaunching in %ds "
|
||||
"(evidence in %s.1)" % (code, lived, wait, log_path), flush=True)
|
||||
try:
|
||||
time.sleep(wait)
|
||||
except KeyboardInterrupt:
|
||||
print("\n[park] Ctrl-C -- staying down", flush=True)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
sys.exit(main())
|
||||
except KeyboardInterrupt:
|
||||
sys.exit(0)
|
||||
@@ -0,0 +1,17 @@
|
||||
@echo off
|
||||
REM ===================================================================
|
||||
REM Park the BattleTech relay on this machine and keep it alive.
|
||||
REM
|
||||
REM Double-click to run it now, or put a SHORTCUT to this file in the
|
||||
REM Startup folder (Win+R -> shell:startup) so the relay comes back by
|
||||
REM itself after a reboot. No admin rights needed either way.
|
||||
REM
|
||||
REM The window stays open: it is the supervisor. Closing it (or Ctrl-C)
|
||||
REM stops the relay for good.
|
||||
REM ===================================================================
|
||||
title BattleTech relay (parked)
|
||||
cd /d "%~dp0.."
|
||||
python "tools\btrelay_park.py" %*
|
||||
echo.
|
||||
echo The parked relay has stopped.
|
||||
pause
|
||||
Reference in New Issue
Block a user