diff --git a/CLAUDE.md b/CLAUDE.md index cfd787d..f28e365 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -74,6 +74,8 @@ precise than anything you can infer. | First-person cockpit canopy (*_cop) + the eyepoint camera | `context/cockpit-view.md` | | Cockpit gauges / MFD HUD | `context/gauges-hud.md` | | Multiplayer, replication, netcode | `context/multiplayer.md` | +| **The OPERATOR CONSOLE + RELAY** — how to launch it, ports, seats, the round/launch lifecycle, re-arm | `context/operator-console.md` | +| Running a session as sysop (operator-facing how-to) | `docs/OPERATOR_GUIDE.md` | | Pod hardware, monitors, RIO, MFD surfaces, input remap (CONTROLS.MAP/XInput) | `context/pod-hardware.md` | | Glass cockpit desktop dev layer (BT_GLASS/BT_STEAM gates, PadRIO, miniconsole, Steam) | `context/glass-cockpit.md` | | Steam internet MP (the wire seam, identity tokens, lobby) | `context/steam-networking.md` | @@ -127,6 +129,10 @@ Tag individual claims (not sections) inline: `[T1]`, `[T2]`, … A claim inherit - Issue `DestroyEntityMessage` on a mech death (the wreck STAYS; removal = the P5 teardown crash). - Conclude "gauge not built" from an early process kill — the gauge renderer builds LAZILY. - Use `DebugStream` (ReconStream, a no-op) for a log — use `DEBUG_STREAM`. +- Confuse the two console programs. **"Launch the console" = `python tools/btoperator.py`** (the + PySide6 operator GUI). `tools/btconsole.py` is the HEADLESS relay that GUI spawns — it has no + window, and backgrounded it has no stdin, so operator commands are dead. See + `context/operator-console.md`. --- diff --git a/context/multiplayer.md b/context/multiplayer.md index 213e1dd..892d770 100644 --- a/context/multiplayer.md +++ b/context/multiplayer.md @@ -581,6 +581,11 @@ phase). Plan: `~/.claude/plans/partitioned-snuggling-piglet.md`. the operator's ⏹ END MISSION button (stdin `stop`). Was half-implemented — nothing fired it in 1995 the console owned the stop. Pod runs the authentic fade → mission-loop exit → clean process exit. Verified live (40s test mission ended on the clock). +> **The console/relay now has its own topic: [[operator-console]]** (`context/operator-console.md`) +> — which of the two programs is which, ports, the seat model, the full round/launch lifecycle and +> its wedge states, plus liveness. Read that first for anything console-side; the notes below are +> the historical record of how it got built. Sysop how-to: `docs/OPERATOR_GUIDE.md`. + - **OPERATOR CONSOLE (2026-07-18)**: `tools/btoperator.py` (PySide6) — the lost 1995 operator station recreated: mission editor with ALL values validated live from BTL4.RES (`tools/eggmodel.py`: maps=type14∩26, colors/badges/patches from the type-25 vehicletable, diff --git a/context/operator-console.md b/context/operator-console.md new file mode 100644 index 0000000..b8e8b1e --- /dev/null +++ b/context/operator-console.md @@ -0,0 +1,291 @@ +--- +id: operator-console +title: "The Operator Console + Relay (btoperator.py / btconsole.py)" +status: living +source_sections: "tools/btoperator.py; tools/btconsole.py (module docstring is authoritative on the wire); context/multiplayer.md §relay/§operator console; docs/OPERATOR_GUIDE.md" +related_topics: [multiplayer, steam-networking, content-archives, pod-hardware, build-and-run] +key_terms: [EGG, master, replicant, relay, seat, roster] +open_questions: + - "UDP endpoint map is keyed on the sender's self-declared fromHost with no check against its TCP peer -- a zombie pod could steal a live pod's downstream (found 2026-07-26, not yet fixed)" + - "egg-ACK detection is a fixed-offset parse of one recv() with no reassembly; no frame resync after a malformed frame" + - "remote-operator mode can never enable the LAUNCH button (_refresh_pod_status requires a local console_proc)" +--- + +# The Operator Console + Relay + +The system-operator station: build a mission, run a session, watch pods arrive, launch rounds. +This is the piece the 1995 archive lost and we rebuilt. + +> **Sysop-facing instructions live in `docs/OPERATOR_GUIDE.md`.** This file is the +> engineering description — how it is built and where it breaks. + +--- + +## READ THIS FIRST — there are TWO programs and they are not interchangeable + +The single most common mistake (made by me, twice, after a context compaction): + +| File | What it is | How it runs | +|---|---|---| +| **`tools/btoperator.py`** | **The operator console.** A **PySide6 GUI** — the window the sysop actually uses to build missions, watch pods and press LAUNCH. | `python tools/btoperator.py` (from the repo root, no args) | +| **`tools/btconsole.py`** | **The headless console/relay.** All the wire logic: egg hand-out, seat assignment, the launch pair, the mission clock, the TCP+UDP frame router. **No UI.** | The GUI spawns it as a **QProcess**; also runnable standalone from the CLI | + +**"Launch the console" means `tools/btoperator.py`.** Backgrounding `btconsole.py` is *not* it — +it has no window, and with no stdin the operator commands are dead. The division is deliberate: +all protocol logic stays headless so the CLI, the GUI and the test rigs share one tested +implementation; the GUI is only UI + process management. + +The GUI drives the relay two ways: **stdin** (`launch`, `stop`, `rearm`) for a local child, and +the **control port** for a remote relay. Both paths reach the same handlers. + +--- + +## Topology and ports + +Pods **dial OUT** to the relay (this is what makes internet play work through NAT — no inbound +ports needed at the player's end). With console port `P` (default **1500**): + +| Port | Protocol | Purpose | +|---|---|---| +| `P` /tcp | console protocol, relay-terminated | streams the **egg** to each pod, then the **RunMission** launch pair and **StopMission** | +| `P+1` /tcp | envelope-framed game relay | the frame router between pods (`{int32 route; uint32 length}` + payload) | +| `P+1` /udp | envelope-framed unreliable channel | the 60 Hz update-record fan-out (`{int32 route; int32 fromHost; uint32 seq}` + frame) | +| `P+7` /tcp | operator control channel | `AUTH ` then `launch` / `stop` / `rearm` / `ping` / `get` / `set` (`CONTROL_PORT_OFFSET = 7`) | +| udp/15999 | LAN discovery | answers probes so pods can use `BT_RELAY=auto` | + +**Pod-side env contract:** `BT_RELAY=:

` (or `auto` for LAN discovery) and +`BT_SELF=` — or `BT_SELF=auto`, which makes the relay assign the next free +roster seat. `players/join.bat` sets these. + +### Route table (`tools/btconsole.py:276-286`) + +``` +>= 2 unicast to that hostID (relay->client: the SENDER's hostID) + -1 broadcast to every registered pod except the sender + -2 HELLO payload: int32 magic 'BTR1', int32 myHostID + -3 PEER_UP relay->client + -4 PEER_DOWN relay->client + -5 UDP HELLO-ACK (sets the client's udpUp) + -6 SEAT_REQUEST client->relay: assign me a free roster seat + -7 SEAT_ASSIGN relay->client: int32 hostID + NUL-terminated tag + -8 SEAT_FULL relay->client: no free seats + -9 MATCHLOG client->relay: upload (filename NUL + bytes); saved to + matchlogs/_, cap 8 MB + -10 READY client->relay: mission load complete (the READY light) + -11 REJOIN relay->client: abandon this round and rejoin +``` + +--- + +## The roster, seats and identity + +The **egg's `[pilots]` list is the roster** (`parse_egg_roster`, btconsole.py:372). Entry *i* +becomes **hostID `2 + i`** — `FIRST_GAME_HOST_ID = 2`, because `CONSOLE_HOST_ID = 1` is +`FirstLegalHostID` and belongs to the console itself. A pilot entry's text is its **tag** +(e.g. `10.99.0.1:1502`); the tag's **position in the received egg** is how a pod derives its own +host id, which is why a roster **trim remaps seat ids** and everything re-keys by position. + +- **Seat requests** (route -6) let a walk-up pod claim a free seat and carry its callsign + mech. +- **Reservations** last `SEAT_RESERVE_SECONDS = 60`. +- **Reclaim by identity**: a returning player (same IP + callsign) gets their old seat back + (`seat_identity`, btconsole.py:1258-1276). Watch for this in the log as + `seat RECLAIMED by identity`. + +**Launch with whoever connects:** the operator's LAUNCH can `_trim_roster` down to the seats that +actually have live beacons or claims, so a 8-seat egg runs a 3-player round. + +--- + +## The round lifecycle (the part that used to strand the operator) + +``` + session start + -> egg HELD until len(console_conns) >= len(roster) [btconsole.py:565-578] + -> _release_eggs() (or _release_eggs(present) -> _trim_roster) [:627] + -> pod ACK per pod (clientID 0, msgID 4) -> _pods_ready() + -> READY (route -10) per pod: mission load complete + -> operator LAUNCH manual mode: waits for the operator [_tick_launch] + -> RunMission #1 then #2 after LAUNCH_STEP_SECONDS = 4s launches_sent 1 -> 2 + -> mission clock the egg's [mission] length, +STOP_GRACE = 2s [_tick_stop] + -> StopMission clock expiry or operator End Mission + -> round end pods exit, join.bat relaunches them, they reconnect + -> RE-ARM so the next LAUNCH works <-- see below +``` + +Timings: `LAUNCH_SETTLE_SECONDS = 20` (egg → first launch; pods must reach WaitingForLaunch or the +handler `Fail()`s), `LAUNCH_STEP_SECONDS = 4`, `STOP_GRACE_SECONDS = 2`, +`ABORT_SETTLE_SECONDS = 8` (egg-release hold after a round abort). + +### The all-ready gate + +Firing a mission while a pod is still LOADING drowns that pod's load in the running mission's +update streams (same event queue) — the operator, who joins last, once loaded 5+ minutes. So the +RunMission pair is held until every registered pod has sent READY. Escape hatches: the holdouts +are announced every 10 s, and a **180 s cap force-fires** so one wedged pod cannot hold the night +hostage. + +### Re-arm: why LAUNCH used to do nothing after a round [FIXED 2026-07-26] + +**The failure the operator reported:** "after a game ends I push LAUNCH and nothing happens until +I reset the session or reboot the console." Their own counter-observation is the tell: *with a +stable group it worked fine repeatedly.* + +The manual LAUNCH branch is guarded on `launches_sent == 0`, but a finished mission leaves it at +**2**. The only automatic paths back to 0 were both **all-or-nothing**: + +- `_check_launch_gate` — needs **every** seat of the last round to re-ACK +- `_maybe_reset_round` — needs **every** pod to be gone + +A games night lives between those: most pods rejoin, one player closes their window. And the +"not all seats filled" diagnostic sat *inside* the `launches_sent == 0` guard, so in exactly that +state **nothing printed at all**. The GUI agreed: its LAUNCH button is gated on +`not monitor.launched`, and `launched` was cleared only by the two relay lines those same gates +emit — so the button was greyed out precisely when wedged. Only a session restart recovered, +because a fresh relay process means fresh state. + +**Now:** an explicit operator LAUNCH is itself sufficient authority to start a new round. +`_rearm_for_new_round()` clears the round state and restores the template roster/egg while +**keeping** seats, beacons and connections. Plus a `rearm`/`newround` command and a **Re-arm** +button, so recovery never needs a restart. + +**The trap to remember if you touch this:** `launch_requested` deliberately stays set across the +RunMission pair, so re-arming on `launches_sent >= 1` alone also matches the *normal* state +between #1 and #2 — that resets the pair mid-flight and re-releases eggs every tick, a **re-arm +storm** that kicks every pod into an identity-resync loop. The re-arm therefore also requires +`launch_at is None` (nothing in flight). Caught on the rig, 2026-07-26. + +### The post-abort dead lobby [FIXED 2026-07-26] + +A distinct wedge with the same symptom, **live-proven** in `content/operator_relay.log` on +2026-07-25 (≈5 minutes of a night lost). `_abort_round` sets `eggs_released = False` **before** the +surviving pods' sockets close, so every later `_maybe_reset_round` hits its own +`if not self.eggs_released: return` and **the template restore never runs**: `roster` and +`expected_ids` stay trimmed to last round's attendance and `egg_path` stays the trimmed egg. All +three automatic release paths compare against `len(self.roster)`, so with N−1 pods back nothing +releases, walk-ups get `ROSTER FULL`, and the relay sits dead. + +`_rearm_for_new_round`'s template restore is therefore **unconditional** — an earlier cut gated it +on `eggs_released`, which made Re-arm useless in the one state that most needs it. It also clears +`round_hold_until` so an abort's settle window is not inherited. + +**Aborts happen for an ordinary reason:** nothing on the wire distinguishes "a straggler's round-N +FIN arriving late" from "a pod died mid-load", so a slow or remote pod closing ~18 s after +StopMission is read as the latter. + +### The stop latch + +`_tick_stop` returns early while `launches_sent < 2`, so a `stop_requested` set **between** rounds +used to sit latched and then StopMission the next mission in the very tick it launched — which from +the operator's seat is also "launch did nothing". It is now consumed and announced while idle. + +--- + +## Liveness (the "reboot the console" half) [FIXED 2026-07-26] + +Nothing used to notice a pod that vanished **without** closing its TCP: a sleeping laptop, a +dropped Wi-Fi link, a killed process, a NAT idle-timeout. There was no `SO_KEEPALIVE` and no idle +deadline anywhere, so the dead connection sat in `console_conns` with `acked=True` **forever**, +which (a) permanently blocked `_maybe_reset_round` (it requires *no* conn acked) and (b) held a +phantom seat in the launch count. Only restarting the relay cleared it. + +Now: keepalive on every accepted socket (`_enable_keepalive`), a `last_seen` stamp on every read +**and on every inbound UDP datagram**, and `_reap_dead_peers()` with `DEAD_PEER_SECONDS = 180`. + +**The reaper is deliberately narrow, and the narrowness is load-bearing.** It runs **only in the +active staging window** — egg released, mission not yet running — and skips console pads with no egg +sent and game conns that have not HELLO'd. The first cut was gated only on "no mission running", +which armed it for the entire **between-rounds** wait, and that is a period when a pod is +**legitimately byte-silent**: its seat beacon is write-only for the process lifetime +(`L4NET.CPP`: *"the relay ignores its silence"*, and the pod sets its own keepalive on it), and its +console pad has no egg yet so it cannot ACK. A real night showed 12-minute and 5-minute gaps +between rounds, so a 180 s deadline would have dropped **healthy** players and forced their clients +to relaunch — strictly worse than the ghost the reaper exists to remove. Caught in review before it +ever ran live. Regression guards: `scratchpad/test_relay_rearm.py` cases 5b/5c. + +Half-open sockets are TCP keepalive's job; this deadline is only the backstop for platforms where +the keepalive tunables do not take. + +Also: every pod-facing send used to flip the socket to blocking with **no timeout** on the relay's +**single-threaded** selector loop, so one wedged peer could freeze the whole relay (no launch tick, +no UDP fan-out, no control channel). All sends now go through `_send_all_guarded` +(`SEND_TIMEOUT_SECONDS = 10`, drops the peer on failure). + +--- + +## Modes + +- **Relay (internet)** — `btconsole.py --relay `; pods dial out. The normal mode. +- **Mesh (LAN legacy)** — `btconsole.py ...`; the console dials OUT to each pod's + `-net` port. The authentic 1995 cabinet direction. +- **Remote relay** — the GUI drives someone else's relay over the control port instead of spawning + a child. **Known broken:** the LAUNCH button can never enable in this mode. + +**The console must STAY CONNECTED.** A disconnect trips the pod's console-loss path, which also +closes its game listener — an engine bug, not ours. + +--- + +## Logs and evidence + +| File | Notes | +|---|---| +| `content/operator_relay.log` | the relay's own stdout, teed by the GUI. **Truncated at each session start** — so a Start Session destroys the evidence of the wedge it is being used to clear. Copy it first when diagnosing. | +| `matchlogs/_` | per-pod match reports, auto-uploaded on route -9 at each clean round end (relay modes only; a mid-round crash loses it) | +| `content/join.log` | the pod's own log; **never** uploaded, only on the player's machine | + +Every relay log line is wall-clock stamped (`_StampedOut`) so post-mortems can measure delays. +The GUI's monitor regexes all use `.search()`, so the prefix is transparent to them. + +--- + +## Mode-specific traps (verified 2026-07-26 by reading `btoperator.py`) + +These are not theory — each was read out of the code, and each makes a control *look* live while +doing nothing: + +- **MESH mode: the operator buttons are inert.** `_relay_send` reports ">> operator LAUNCH sent" + whenever the child process is Running, but the **stdin command thread only exists in + `relay_main`** (btconsole.py) — the mesh path (`btconsole.main`) has no stdin reader at all. So + LAUNCH / END MISSION / Re-arm are silently discarded in mesh mode. `Apply mission settings` is + likewise inert there: the mesh console reads the egg bytes once at startup. Mesh also never + prints "WAITING FOR OPERATOR", so the LAUNCH button cannot even enable. +- **REMOTE-operator mode: LAUNCH and END MISSION can never enable** — both enable conditions + require `console_proc is not None`, and the remote path never sets it. The roster lights are also + permanently empty (`SessionMonitor(True, [])` → no tags → `seated` always 0), and + `Launch local instances` is refused by the same `console_proc` guard even though the code below + it deliberately builds `BT_RELAY` from the remote host. +- **`Start session` silently overwrites the egg on disk** (including re-rasterized callsigns), with + no prompt and no backup. +- **`Apply` writes only the six `[mission]` keys** (map/time/weather/scenario/temperature/length). + Roster edits need `Save`; a **seat-count change is refused by the relay** with + `egg file roster changed N->M seats -- IGNORED (restart the session to resize)`. +- **The Color dropdown offers every colour in the RES vehicletable**, but only + `Black/Brown/Crimson/Green/Grey/Tan/White` actually paint — anything else silently renders GRAY + (the relay warns). A "valid" mission can still produce a grey mech. +- **`operator_secret.txt` is opened by a bare relative name** by the relay, whose working directory + is inherited from whatever directory `btoperator.py` was started in (the GUI never calls + `setWorkingDirectory` on the relay child). Start from the repo root and it will *not* be the + `content\operator_secret.txt` the tooltips point at. +- **After the relay dies on its own**, `_console_finished` leaves `console_proc` non-None, so + `Launch local instances` stays enabled and will spawn clients against a dead relay. + +## Gotchas / things that look like bugs + +- **`egg HELD (n/N pods present)`** is normal staging, not an error — the egg waits so every pod's + copy carries every walk-up seat pref. +- **`*** WARNING: ... EMPTY seat(s) ... the mission will STALL`** can fire *misleadingly* after a + trim, because `_log_launch_readiness` counts `by_host` against the (remapped) roster. Do not + act on it blindly. +- **A relay-side crash leaves the GUI looking alive** — `_console_finished` does not clear + `console_proc`. Commands now report honestly instead of logging ">> sent" into the void. +- **Session restart is the big hammer**: it re-spawns the relay, so it clears *every* latch. That + is why it always "fixed" things — and why it hid the real defects for so long. +- A **`--manual-launch`** relay waits for the operator; without it the launch pair fires + automatically once all pods ACK. + +## Key Relationships + +- Uses: [[content-archives]] (the EGG is the mission + roster) · [[multiplayer]] (what the relayed + frames mean: master/replicant, dead reckoning) +- Feeds: [[multiplayer]] · [[steam-networking]] (the alternate wire seam) · [[pod-hardware]] +- Sysop instructions: `docs/OPERATOR_GUIDE.md` diff --git a/docs/OPERATOR_GUIDE.md b/docs/OPERATOR_GUIDE.md new file mode 100644 index 0000000..709c26d --- /dev/null +++ b/docs/OPERATOR_GUIDE.md @@ -0,0 +1,189 @@ +# BattleTech 4.11 — Operator Console Guide + +How to run a games night: set up a mission, get pods seated, launch rounds, and recover when +something sticks. Written for the sysop at the console, not for players. + +*Engineering detail (ports, protocol, state machine) lives in `context/operator-console.md`. +Player-facing instructions ship in the zip as `README.txt`.* + +--- + +## 1. Starting the console + +``` +cd C:\git\bt411 +python tools\btoperator.py +``` + +A window titled **BT411 Operator Console** opens. That is the console. + +> **One thing worth knowing:** there are two programs. `btoperator.py` is the window you use. +> `btconsole.py` is the headless relay it starts for you in the background — all the actual +> networking. You never launch that one directly; its output is what scrolls in the log pane at the +> bottom of the window. When this guide says "the relay said…", that is where to look. + +--- + +## 2. Set up the mission, then start the session + +Work top to bottom: + +1. **Pick the mission settings** — map, mission length, weather, experience level, and the seat + (pod) count. Every dropdown is populated live from `BTL4.RES`, so anything offered is valid. +2. **Set the seat count to roughly who you expect.** It does not have to be exact — you can launch + with fewer (see §4) — but a wildly oversized roster makes the staging messages confusing. +3. **Choose the mode**: **Relay (internet)** is normal — players dial out to you, so *they* need no + port forwarding. **Mesh (LAN)** is the original cabinet direction for a local network. +4. **Press Start Session.** The relay comes up and begins accepting pods. + +**For internet play, you must be reachable.** Forward/allow these TCP ports to this machine +(default console port 1500): + +| Port | Why | +|---|---| +| 1500/tcp | the console channel (hands out the mission egg, sends the launch) | +| 1501/tcp | game traffic between pods | +| 1501/udp | the fast update channel (position/motion) | + +Port 1507/tcp is the operator control channel — only needed if someone drives this relay remotely. +Players give their client `BT_RELAY=:1500`; `players/join.bat` handles that. + +--- + +## 3. Watch the pods arrive + +Each pilot row lights up as they progress. In order: + +**waiting** → **seated** (claimed a seat, callsign and mech shown) → **egg sent** → +**registered** → **ready** (mission loaded) → **LAUNCHED**. + +You will see this in the log and it is **normal, not an error**: + +``` +console conn 1.2.3.4:5678: egg HELD (3/8 pods present) +``` + +The mission file is deliberately held until every seat is present, so that every pod's copy +contains everyone's callsign and mech choice. If you are not waiting for the rest, just launch — +see next. + +--- + +## 4. Launch a round + +Press **🚀 LAUNCH MISSION**. + +- If everyone has loaded, the round starts within a few seconds. +- **If some seats are empty, launching is still correct** — the relay shrinks the session to the + players who are actually here and starts. You do not need to edit the roster down first. +- The relay holds the start until every pod reports **ready**, so nobody loads into a mission + already in progress. If one pod is slow you will see who, every 10 seconds; after 3 minutes it + starts anyway rather than letting one wedged client hold up the night. + +The mission then runs on its own clock (the mission length from the egg). **⏹ END MISSION** stops +it early. + +--- + +## 5. Running back-to-back rounds + +When a round ends, players' clients relaunch themselves and reconnect. Once they are back, press +**LAUNCH** again. You can change mission settings between rounds and press **Apply** first. + +### If LAUNCH looks dead — press ↻ Re-arm + +This is the fix for the most annoying failure this console had. If a round has ended and the LAUNCH +button is greyed out, or pressing it does nothing: + +> **Press ↻ Re-arm, then LAUNCH.** + +Re-arm clears the finished round's state and re-opens the launcher **while keeping everyone who is +already connected**. You do **not** need Stop Session / Start Session, which disconnects everybody. + +**Why this happens:** the console used to only consider itself ready for a new round if *every* +player from the last round came back, or if *all* of them left. In between — the normal case when +one person closes their window or their client crashes — it got stuck, and said nothing. That is +fixed (an explicit LAUNCH now re-arms by itself), and Re-arm is the belt-and-braces button. If you +ever see the old behaviour, it is a bug worth reporting. + +--- + +## 6. Known limitations — buttons that are not what they appear + +Read this once; each of these is a control that looks live but does nothing in that mode. + +**Mesh (LAN legacy) mode: the mission buttons do not work.** LAUNCH, END MISSION and Re-arm all log +">> operator … sent" and are then **silently discarded** — the mesh console has no command channel. +`Apply mission settings` is also inert (mesh reads the mission file once at startup). Mesh launches +its rounds by itself once the pods have the mission. **If you want to control launches, use Relay +mode.** + +**Remote relay mode** (driving another machine's relay): LAUNCH and END MISSION **can never +enable**, the pilot lights stay blank, and `Launch local instances` is refused. You get Apply, +Re-arm and Stop. Treat remote mode as monitoring-plus-settings, not as a full console. + +**Other sharp edges:** +- **Start Session overwrites your egg file** on disk, without asking. Keep a copy of any mission you + care about under a different name. +- **`Apply` only pushes the six mission settings** (map, time, weather, scenario, temperature, + length). Callsign / mech / colour / badge / patch / experience edits need **Save**. Changing the + **seat count** mid-session is refused outright — that needs a session restart. +- **The Colour dropdown offers colours the game cannot paint.** Only Black, Brown, Crimson, Green, + Grey, Tan and White work; anything else silently comes out grey. The relay warns in the log. +- **If the relay dies on its own**, `Launch local instances` stays clickable and will start clients + against nothing. Press Start Session first. +- **`Public host`** is used only when exporting player scripts; it does not affect what the relay + binds or advertises. + +## 7. Troubleshooting + +| What you see | What it means | What to do | +|---|---|---| +| `egg HELD (n/N pods present)` | normal staging — waiting for the rest of the seats | wait, or press LAUNCH to start with who is here | +| LAUNCH greyed out after a round | the launcher is still holding the last round | press **↻ Re-arm** | +| `LAUNCH pressed but NO players are seated yet` | nobody has claimed a seat | check the players are pointed at the right address/port | +| `LAUNCH pressed but NOT all seats have ACKed` | some pods are still taking the mission file | wait a few seconds; if one is never coming, press **Re-arm** | +| `launch HELD -- still loading: PLAYER n` | that pod has not finished loading | wait; it force-starts after 3 minutes | +| `*** WARNING: ... EMPTY seat(s) ... the mission will STALL` | **can be a false alarm** after the roster is trimmed | if the round starts and plays fine, ignore it | +| `seat RECLAIMED by identity` | a player who dropped got their seat back | nothing — this is working as intended | +| `game[...] dropped: closed` | a pod disconnected | normal at round end; mid-round it means they crashed or lost connection | +| A player gets **ROSTER FULL** | no free seat in the current round | Re-arm between rounds so the roster re-opens, then let them join | +| Console window stops responding | the relay stalled on a dead connection | Stop Session, Start Session; please keep the log (below) | +| A player says they cannot connect at all | ports, or wrong address | confirm 1500/1501 TCP + 1501 UDP reach this machine | + +### The one thing to do before restarting a session + +**Copy `content\operator_relay.log` somewhere first.** Start Session **overwrites** it — so +restarting to clear a problem also destroys the evidence of that problem. If something misbehaved, +save that file and hand it over; it is timestamped line-by-line and it is what makes these bugs +diagnosable. + +--- + +## 8. Match reports + +At the end of each clean round, every player's client **uploads its own match report to you +automatically** (relay mode only). They land in `matchlogs\` named by the sender's address. You do +not need to ask players to send anything. + +Exceptions worth knowing: a client that **crashes mid-round** loses its report, and players on +**solo** or **Steam** have no relay to upload through. In those cases ask them for +`content\matchlog_*.txt` directly. Their crash log, `content\join.log`, is **never** uploaded — ask +for it explicitly when something went wrong on their end. + +--- + +## 9. Local test instances + +**Launch local instances** starts game clients on this machine against your own session — useful +for testing alone or filling a seat. **Stop games** closes them. In relay mode the session must +already be started, or a local instance has nothing to dial. + +--- + +## 10. Which build everyone needs + +**All pods must run the same build as each other.** Build **4.11.554** changed what pods tell each +other about scores, so a mixed session shows disagreeing KILLS/DEATHS columns (nothing crashes — +the remote numbers just do not move). When you hand out a new zip, make sure everyone takes it +before the session. diff --git a/scratchpad/test_relay_rearm.py b/scratchpad/test_relay_rearm.py index cae43bd..717b0d9 100644 --- a/scratchpad/test_relay_rearm.py +++ b/scratchpad/test_relay_rearm.py @@ -127,9 +127,10 @@ check("RunMission actually reached the pods", all(c.sock.sent for c in r.console_conns), "%d pkt(s) to pod 1" % len(r.console_conns[0].sock.sent)) -print("\n=== 5. REAPER: a silent peer stops pinning the round open ===") +print("\n=== 5. REAPER: reaps a silent peer DURING STAGING only ===") r = new_relay() r.launches_sent = 0 +r.eggs_released = True # staging: the egg is out, pods should be talking dead = FakeConn(acked=True) dead.last_seen = btconsole.time.time() - (btconsole.DEAD_PEER_SECONDS + 30) live = FakeConn(acked=True) @@ -141,9 +142,43 @@ r._reap_dead_peers() check("the silent conn was reaped", dead in dropped) check("the live conn was kept", live in r.console_conns and live not in dropped) +print("\n=== 5b. REAPER MUST NOT TOUCH A WAITING POD BETWEEN ROUNDS ===") +# The regression this guards: a pod waiting for the next mission is byte-silent +# by design (its seat beacon is write-only -- L4NET.CPP "the relay ignores its +# silence" -- and its console pad has no egg to ACK). A real night showed 12- +# and 5-minute gaps between rounds, so a 180s deadline would have dropped +# HEALTHY players and forced their clients to relaunch. +r = new_relay() +r.launches_sent = 0 +r.eggs_released = False # between rounds / pre-release +waiting = FakeConn(acked=False) +waiting.egg_sent = False +waiting.last_seen = btconsole.time.time() - 900 # silent for 15 minutes +r.console_conns = [waiting] +r.game_conns = [] +reaped = [] +r._drop_console = lambda c: reaped.append(c) +r._reap_dead_peers() +check("a 15-min-silent waiting pod is NOT reaped", not reaped) + +print("\n=== 5c. REAPER SKIPS A PAD THAT WAS NEVER SENT AN EGG ===") +r = new_relay() +r.launches_sent = 0 +r.eggs_released = True +nopad = FakeConn(acked=False) +nopad.egg_sent = False # nothing was asked of it yet +nopad.last_seen = btconsole.time.time() - 900 +r.console_conns = [nopad] +r.game_conns = [] +reaped = [] +r._drop_console = lambda c: reaped.append(c) +r._reap_dead_peers() +check("no egg sent -> not reaped", not reaped) + print("\n=== 6. REAPER IS DISARMED WHILE A MISSION RUNS ===") r = new_relay() r.launches_sent = 2 # mission running +r.eggs_released = True quiet = FakeConn(acked=True) quiet.last_seen = btconsole.time.time() - 9999 r.console_conns = [quiet] diff --git a/tools/btconsole.py b/tools/btconsole.py index fb2485b..6f47cc1 100644 --- a/tools/btconsole.py +++ b/tools/btconsole.py @@ -853,17 +853,43 @@ class Relay: """ if self.launches_sent >= 2: return # mission running: leave peers alone + if not self.eggs_released: + # + # BETWEEN ROUNDS / BEFORE THE EGG GOES OUT, SILENCE IS CORRECT and + # can last a long time (a real night showed 12- and 5-minute gaps + # while the operator set up the next mission). A waiting pod has + # nothing to send: its seat beacon is deliberately write-only for the + # process lifetime -- L4NET.CPP: "the relay ignores its silence" -- + # and its console pad has no egg yet, so it cannot ACK. Reaping here + # would drop HEALTHY players and force their client to relaunch, + # which is worse than the ghost this reaper exists to remove. + # (Caught in review 2026-07-26, before it ever ran on a real night.) + # + return + # + # So the reaper only runs in the ACTIVE STAGING WINDOW: the egg is out and + # we are waiting for ACK/READY, which is exactly when a pod that has gone + # quiet is a pod that is gone. A truly half-open socket is caught sooner + # and more cheaply by TCP keepalive (set on accept, and the pod sets it on + # its own beacon too) -- this deadline is only the backstop for platforms + # where the keepalive tunables do not take. + # now = time.time() for conn in list(self.console_conns): + if not conn.egg_sent: + continue # nothing was asked of it yet if now - getattr(conn, "last_seen", now) > DEAD_PEER_SECONDS: print(f"[relay] console conn {conn.addr[0]}:{conn.addr[1]} " f"REAPED -- silent for " - f"{now - conn.last_seen:.0f}s (was acked={conn.acked}); " - f"it can no longer hold the round open", flush=True) + f"{now - conn.last_seen:.0f}s while staging " + f"(acked={conn.acked}); it can no longer hold the round " + f"open", flush=True) self._drop_console(conn) for conn in list(self.game_conns): + if conn.host_id is None: + continue # beacon / not yet HELLO'd: silent by design if now - getattr(conn, "last_seen", now) > DEAD_PEER_SECONDS: - self._drop_game(conn, "silent for %.0fs -- reaped" + self._drop_game(conn, "silent for %.0fs while staging -- reaped" % (now - conn.last_seen)) def _pods_ready(self): @@ -995,21 +1021,30 @@ class Relay: self.mission_started_at = None self.eggs_done_at = None self._launch_blocked_warned = False - # Re-open the round for joins/edits: restore the untrimmed roster and - # the template egg, exactly as _maybe_reset_round would, so round 2+ - # does not serve last round's trimmed egg (new joiners got ROSTER FULL) - # and between-round mission edits are picked up. - if self.eggs_released: - self.eggs_released = False - self.egg_path = self.orig_egg_path - self.egg_bytes = self.orig_egg_bytes - self.roster = list(self.orig_roster) - self.expected_ids = set(range(FIRST_GAME_HOST_ID, - FIRST_GAME_HOST_ID + len(self.roster))) - self._reload_egg_file() - for c in self.console_conns: - c.acked = False # last round's ACK must not count - c.egg_sent = False # every pod needs THIS round's egg + # Re-open the round for joins/edits: restore the untrimmed roster and the + # template egg, so round 2+ does not serve last round's trimmed egg (new + # joiners got ROSTER FULL) and between-round mission edits are picked up. + # + # UNCONDITIONAL, and that matters. An earlier cut gated this on + # `eggs_released`, which made it useless in the one state that most needs + # it: _abort_round clears eggs_released BEFORE the survivors' sockets + # close, so _maybe_reset_round's own `if not self.eggs_released: return` + # then skips the template restore forever. The roster stays trimmed to + # last round's attendance, the release gates (which all compare against + # len(self.roster)) can never be met, walk-ups get ROSTER FULL, and the + # relay sits dead -- live-proven on 2026-07-25, nearly 5 minutes of a + # games night lost. Re-arm has to be able to dig out of exactly that. + self.eggs_released = False + self.egg_path = self.orig_egg_path + self.egg_bytes = self.orig_egg_bytes + self.roster = list(self.orig_roster) + self.expected_ids = set(range(FIRST_GAME_HOST_ID, + FIRST_GAME_HOST_ID + len(self.roster))) + self.round_hold_until = 0 # do not inherit an abort's settle window + self._reload_egg_file() + for c in self.console_conns: + c.acked = False # last round's ACK must not count + c.egg_sent = False # every pod needs THIS round's egg def _tick_launch(self): # RE-ARM ON DEMAND: an operator LAUNCH while a previous round is still @@ -1514,6 +1549,11 @@ class Relay: self.stats["udp_rx"] += 1 # NAT-rebind tolerant: every datagram refreshes the endpoint map. self.udp_endpoint[from_host] = endpoint + # ... and counts as LIVENESS. Without this, a pod streaming updates + # over UDP while its TCP sits idle looks "silent" to the reaper. + live = self.by_host.get(from_host) + if live is not None: + live.last_seen = time.time() if route == ROUTE_HELLO: ack = struct.pack(ENV_FMT_UDP, ROUTE_UDP_ACK, CONSOLE_HOST_ID, 0) try: diff --git a/tools/btoperator.py b/tools/btoperator.py index c558601..94d256c 100644 --- a/tools/btoperator.py +++ b/tools/btoperator.py @@ -1020,17 +1020,21 @@ class Operator(QMainWindow): seated = sum(1 for s in self.monitor.state.values() if s in (ST_SEATED, ST_REGISTERED, ST_READY)) ready_n = sum(1 for s in self.monitor.state.values() if s == ST_READY) + # END MISSION is PER ROUND, not per session. `end_sent` used to be + # cleared only by Start Session, so the button worked exactly once a + # night; and a stale press between rounds used to kill the next mission + # the instant it launched (the relay now refuses that too). This MUST + # sit outside the head-assignment chain below: as an `elif` it swallowed + # a branch and left `head` unbound -> UnboundLocalError on the first + # refresh after End Mission (review catch, 2026-07-26). + if not self.monitor.launched and getattr(self, "end_sent", False): + self.end_sent = False + if self.monitor.launched: head = "LAUNCHED — mission running" if (self.console_proc and not self.end_btn.isEnabled() and not getattr(self, "end_sent", False)): self.end_btn.setEnabled(True) - elif getattr(self, "end_sent", False): - # END MISSION is PER ROUND, not per session. end_sent used to be - # cleared only by Start Session, so the button worked exactly once a - # night; and a stale press between rounds used to kill the next - # mission the instant it launched (the relay now refuses that too). - self.end_sent = False elif self.monitor.held_status: head = "STAGING — " + self.monitor.held_status elif ready_n: