238 lines
14 KiB
Markdown
238 lines
14 KiB
Markdown
# Phase 0 — Lab proof plan
|
||
|
||
Drafted 2026-07-10. Goal: prove every load-bearing SiteLink assumption **with zero
|
||
cockpit hardware and zero travel**, using VMs that simulate two sites + the hub.
|
||
Everything marked "verify" in [BRAINSTORM.md](BRAINSTORM.md) gets an experiment here;
|
||
each experiment names its pass criteria and the open question it closes.
|
||
|
||
## 1. Objectives / exit criteria
|
||
|
||
Phase 0 is done when we can state, with captures/logs in hand:
|
||
|
||
1. A TeslaConsole at site A commands a pod (vPOD) at site B over routed WireGuard
|
||
(RPC 53290) — including under realistic WAN latency. *(OQ: console over WAN)*
|
||
2. A Firestorm client at site B joins a session hosted at site A (or the hub) by
|
||
directed IP, across subnets, no broadcast required — and we know the exact
|
||
port set on the wire. *(OQ 1, 2)*
|
||
3. We have a measured latency/jitter/loss **cliff table** for Firestorm, and know
|
||
what a mid-match tunnel drop does. *(OQ 4, 10)*
|
||
4. BT411's console can push an egg to pods on two subnets and the pod mesh forms
|
||
(or we know exactly why not). *(OQ 3)*
|
||
5. The hub concept is rehearsed: camera-ship-as-host confirmed (or refuted, with
|
||
the dedicated-server fallback exercised), output captured and streamed.
|
||
*(OQ 5 + hub/Live Cam design)*
|
||
6. An event-day dry run passes end to end: collect `<site>.siteconfig`s →
|
||
`SiteConfigMerge` exits 0 → central console sees and commands both sites' pods.
|
||
|
||
Deferred past Phase 0: TeslaRel410/NetNub cross-subnet testing (OQ 11) — waits for
|
||
the emulator's own networking phase (its PLAN.md Phase 6).
|
||
|
||
## 2. Lab topology
|
||
|
||
Mirrors the production design 1:1, just smaller: two sites of the *minimum* real bay
|
||
shape (console + 2 cockpits) plus the hub.
|
||
|
||
```
|
||
"WAN" (host-only net, 192.168.77.0/24)
|
||
netem applied here = simulated internet
|
||
┌──────────────────┬──────────────────┬─────────────────┐
|
||
│ │ │
|
||
┌────┴────┐ ┌────┴────┐ ┌────┴────┐
|
||
│ gw-a │ │ hub-gw │ │ gw-b │
|
||
│ Linux │◄──WG──►│ Linux │◄──WG──►│ Linux │ WG overlay 10.255.0.0/24
|
||
└────┬────┘ └────┬────┘ └────┬────┘ hub-and-spoke (as production)
|
||
│ │ │
|
||
site A LAN hub LAN 10.0.0.0/24 site B LAN
|
||
10.0.1.0/24 ┌──────────────┐ 10.0.2.0/24
|
||
┌────────────┐ │ hub-fs (Win) │ ┌────────────┐
|
||
│ console-a │ │ FS host / LC │ │ console-b │
|
||
│ pod-a1 │ │ + OBS stream │ │ pod-b1 │
|
||
│ pod-a2 │ └──────────────┘ │ pod-b2 │
|
||
└────────────┘ └────────────┘
|
||
```
|
||
|
||
### Lab IP conventions (strawman — align with real bays before Phase 1)
|
||
|
||
| Host | Address | Notes |
|
||
|------|---------|-------|
|
||
| Site gateways | `10.0.<site>.254` | Linux, WireGuard + nftables + netem |
|
||
| Cockpits | `10.0.<site>.1–.8`, `.11–.18` | per the historical seat map (`ctcl-game.ini`) |
|
||
| Camera / Live Cam | `10.0.<site>.9` | historical |
|
||
| Mission Review | `10.0.<site>.10` | proposed |
|
||
| Site console | `10.0.<site>.100` | proposed |
|
||
| Hub WG | `10.0.0.1` | hub-gw |
|
||
| Hub FS host / LC / MR | `10.0.0.20` | hub-fs |
|
||
| Hub services (PDF share, later) | `10.0.0.10` | |
|
||
| WG overlay | `10.255.0.0/24` | hub `.1`, gw-a `.2`, gw-b `.3` |
|
||
|
||
**Open item for Phase 1:** confirm the canonical last-octet map for console/MR
|
||
against how real bays are actually numbered; the pod seat octets are historical
|
||
fact, the rest above is proposal.
|
||
|
||
## 3. VM inventory
|
||
|
||
### Full lab (9 VMs)
|
||
|
||
| VM | OS | vCPU / RAM / disk | Role |
|
||
|----|----|-------------------|------|
|
||
| gw-a, gw-b, hub-gw | Debian 12 (or Alpine) | 1 / 512 MB / 4 GB | routing, WireGuard, nftables, netem |
|
||
| console-a, console-b | Windows 10 22H2 | 2 / 4 GB / 60 GB | TeslaConsole, SiteConfigMerge, btconsole.py |
|
||
| pod-a1, pod-a2, pod-b1 (+pod-b2 optional) | Windows 10 22H2 | 2 / 4 GB / 60 GB | TeslaLauncher + vPOD; MW4 deploy for match tests |
|
||
| hub-fs | Windows 10 22H2 | 4 / 8 GB / 80 GB, **3D accel ON** | FS host/camera ship, OBS, later MR instance |
|
||
|
||
### Minimum viable lab (5 VMs) — start here
|
||
|
||
gw-a, gw-b (one of them doubling as hub-gw), console-a, pod-b1, hub-fs. That's
|
||
enough for experiments E1–E3 and E6; grow toward the full set as experiments demand.
|
||
|
||
## 4. VM management recommendations
|
||
|
||
**Hypervisor.** The dev box runs Windows 11 Home (no Hyper-V); a Pro upgrade is
|
||
*available but not required* — it buys a second valid stack, not a better version of
|
||
the first. **Pick exactly one stack; don't mix** (enabling Hyper-V forces VMware
|
||
through the Windows Hypervisor Platform with a performance penalty).
|
||
|
||
- **Stack A — VMware Workstation Pro 17 on Home (default, no upgrade needed):** free
|
||
for personal use since 2024, and the **best old-DirectX 3D path of any desktop
|
||
hypervisor** — the only stack with a real shot at MW4 rendering *inside* a VM.
|
||
Proper snapshot trees, **linked clones** (one Windows gold image, thin per-VM
|
||
deltas), per-VMnet host-only networks mapping exactly to the topology above
|
||
(VMnet2 = site A, VMnet3 = site B, VMnet4 = hub LAN, VMnet5 = WAN transit;
|
||
**DHCP off** everywhere — static IPs per plan, mirroring the bays' air-gapped
|
||
discipline).
|
||
- **Stack B — Win11 Pro + Hyper-V for infrastructure, physical boxes for game
|
||
clients:** Hyper-V is hopeless for 1999 DirectDraw guests (RemoteFX vGPU removed;
|
||
GPU-P targets modern DX12), but it is *excellent* for everything that isn't the
|
||
game — gateways, consoles, vPODs: native internal switches, PowerShell
|
||
automation, checkpoints, VMs auto-start as a service, zero third-party installs.
|
||
Since "games on physical hosts" is already this plan's fallback (§7), Stack B is
|
||
simply that fallback embraced from day one. Choose it if an all-Microsoft,
|
||
scriptable lab appeals more than the chance of games-in-VMs.
|
||
- Pro perk either way: the lab host becomes an **RDP host** — handy for remote
|
||
lab access. Not a reason to upgrade by itself.
|
||
- **Persistent lab (recommended once Phase 0 proves out): Proxmox VE on a spare
|
||
box.** Pod operators tend to have spare hardware; a single Proxmox host gives the
|
||
lab a permanent home with a web UI, scheduled snapshots/backups, Linux bridges
|
||
per site LAN, and it can later graduate into the *real* hub staging machine.
|
||
16 GB RAM hosts the minimum lab; 32 GB hosts the full set comfortably.
|
||
- VirtualBox works as a fallback but its 3D path for 1999-era DirectDraw is the
|
||
weakest of the three — expect to lean harder on the physical-host fallback for
|
||
game VMs.
|
||
|
||
**Images and clones.**
|
||
- Build **one Windows 10 gold image** (22H2, VMware Tools, updates frozen, Defender
|
||
real-time off *in the lab only*, Windows Firewall configured per experiment —
|
||
never just disabled, since firewall behavior is part of what Phase 0 tests).
|
||
Clone everything Windows from it (linked clones). Keep one **Linux gold** the
|
||
same way.
|
||
- Windows licensing for the lab: 90-day Enterprise eval ISOs are fine; snapshots
|
||
+ re-arm cover Phase 0's lifetime. Nothing in the lab needs activation.
|
||
- Name VMs exactly as in §3 (`gw-a`, `pod-b1`, …) — captures, logs, and notes all
|
||
key off those names.
|
||
|
||
**Snapshots as method.** Take a `baseline` snapshot of every VM the moment its role
|
||
software is installed and verified idle-healthy. **Snapshot before every experiment**
|
||
(`E3-pre`), and roll back rather than un-configuring. The experiment log (§6) records
|
||
which snapshot each result came from. Export the gold images as OVAs to backup
|
||
storage once — everything else is reproducible from them.
|
||
|
||
**Config as code, in this repo.** Everything text lands under `lab/`:
|
||
`lab/gw/` (wg configs with lab keys, nftables rules, `wan.sh`), `lab/checklists/`,
|
||
`lab/results/` (experiment logs + pcap summaries; raw pcaps stay out of git, keep
|
||
them on the lab host). Lab WireGuard keys are throwaway and may be committed;
|
||
production keys never (per `.gitignore` policy).
|
||
|
||
**netem control.** One knob, on both site gateways' WAN egress (half the RTT each
|
||
side, symmetric):
|
||
|
||
```sh
|
||
# lab/gw/wan.sh — usage: wan.sh <delay> <jitter> <loss> e.g. wan.sh 30ms 5ms 0.1%
|
||
tc qdisc replace dev eth0 root netem delay $1 $2 distribution normal loss $3
|
||
```
|
||
|
||
Standard sweep points for every latency-sensitive experiment:
|
||
LAN-like (0/0/0) → regional (15ms/2ms/0) → cross-country (40ms/5ms/0.1%) →
|
||
bad day (80ms/15ms/0.5%) → hostile (150ms/30ms/1%) → find the cliff.
|
||
|
||
## 5. Experiments
|
||
|
||
Each: goal → method → pass criteria. Run in order; later ones reuse earlier setup.
|
||
|
||
- **E1 — Routed fabric + console RPC over WAN.** Bring up hub-and-spoke WireGuard,
|
||
static routes, nftables allowlist per the ecosystem port map. From console-a,
|
||
provision nothing (provisioning stays local by design) but command a **vPOD** on
|
||
pod-b1: status, Install Product, launch. Sweep netem.
|
||
*Pass:* RPC 53290 works at every sweep point up to "bad day"; note where timeouts
|
||
start (feeds the WAN-tolerant-timeout to-do).
|
||
- **E2 — Broadcast locality sanity.** Confirm SecureConfig UDP beacons and DirectPlay
|
||
LAN browse do *not* cross the tunnel (expected, by design) and that nothing else
|
||
in the console/pod bring-up secretly depends on broadcast.
|
||
*Pass:* pod provisioned locally works remotely thereafter; no cross-site flow ever
|
||
relied on broadcast (pcap-verified).
|
||
- **E3 — Firestorm directed join across subnets.** MW4 host on console-a (or hub-fs),
|
||
client on pod-b1. Set the `DirectPlayPort` registry value; join by IP
|
||
(`TryToJoinASpecificGame` path via the ConLobby/CTCL flow, not the LAN browser).
|
||
Wireshark both gateways.
|
||
*Pass:* client joins and plays; complete port matrix documented (the firewall
|
||
allowlist becomes fact instead of DX7 documentation); confirms whether the fixed
|
||
port carries all session traffic. **Closes OQ 1 + 2.**
|
||
- **E4 — Firestorm latency cliff.** With E3 running, walk the netem sweep during
|
||
actual play (movement + weapons). Record subjective playability + any desync or
|
||
disconnect per point.
|
||
*Pass:* a written cliff table ("clean ≤ X ms RTT, degraded at Y, breaks at Z").
|
||
**Closes OQ 4 for FS.**
|
||
- **E5 — WAN-drop behavior.** Kill the tunnel mid-match (down the WG interface);
|
||
restore after 10s / 60s / 5min.
|
||
*Pass:* documented behavior per flow (game session, console RPC, vPOD state) and
|
||
confirmation each site's bay-local operation is unaffected. **Closes OQ 10.**
|
||
- **E6 — MTU/fragmentation.** DF-bit probing host-to-host across the tunnel
|
||
(WG MTU 1420); watch E3's DirectPlay UDP for fragmentation; test an MSS clamp on
|
||
the gateways.
|
||
*Pass:* no silent blackholing; a stated MTU/clamp recommendation for production
|
||
gateways.
|
||
- **E7 — BT411 cross-subnet egg push.** `btconsole.py MP.EGG 10.0.1.x:1501
|
||
10.0.2.x:1501` with btl4 instances at both sites. Inspect how the `[pilots]` mesh
|
||
addresses peers (read `L4NET.CPP` alongside the capture). Netem sweep. Also
|
||
observe the console-disconnect quirk over a flaky tunnel.
|
||
*Pass:* mesh forms across subnets (or root cause written up); latency tolerance
|
||
noted. **Closes OQ 3.**
|
||
- **E8 — Hub host + broadcast rehearsal.** On hub-fs: run the FS host in camera-ship
|
||
role (validate camera-ship-as-DirectPlay-host against the CTCL flow); capture
|
||
with OBS → SRT to both consoles as stand-in "Live Cam screens". Build/run check
|
||
of `mw4dedicatedui` as the headless fallback. If VM rendering is unusable, rerun
|
||
on a physical host (see risks).
|
||
*Pass:* one match hosted at the hub with both sites joined, stream watched at
|
||
both consoles. **Closes OQ 5 and validates the hub/Live Cam design.**
|
||
- **E9 — Event-day dry run.** Simulate the full authority-handover ceremony:
|
||
`console-a` and `console-b` each export `<site>.siteconfig` → transfer to hub →
|
||
`SiteConfigMerge merge` (must exit 0) → central console (hub or console-a wearing
|
||
the hat) loads `master.siteconfig` and commands vPODs at **both** sites → restore
|
||
site configs afterward.
|
||
*Pass:* scripted checklist completes without manual surgery; becomes the seed of
|
||
the production event-day runbook.
|
||
|
||
## 6. Results discipline
|
||
|
||
One markdown file per experiment in `lab/results/` (`E3-fs-directed-join.md`):
|
||
date, snapshot names, netem settings, what happened, pcap filenames (pcaps stay on
|
||
the lab host), verdict against pass criteria. Findings that change the design get
|
||
folded back into BRAINSTORM/ECOSYSTEM in the same commit — the docs stay truthful.
|
||
|
||
## 7. Risks / known unknowns
|
||
|
||
| Risk | Mitigation |
|
||
|------|------------|
|
||
| MW4 rendering inside VMs (DX7/DirectDraw + `DWM8And16BitMitigation` shim, keyed on exe path) | VMware 3D accel + DDrawCompat if needed; windowed mode; **fallback: run game instances on physical hosts** while gateways/consoles stay virtual — the network fabric under test doesn't care where the game runs |
|
||
| vPOD fidelity limits | vPOD impersonates launcher + Munga control, **not** DirectPlay gameplay — E3/E4/E8 need real MW4 instances; don't over-conclude from vPOD-only runs |
|
||
| Windows Firewall/Defender masking network results | Firewall rules are explicit per experiment, never blanket-off; Defender off only in lab gold image, noted in every result |
|
||
| netem placement asymmetry | Apply on both gateways' egress; sanity-check RTT with ping before each run |
|
||
| CTCL flow surprises (host IP propagation) | E3 exercises the *real* ConLobby/CTCL join, not just raw DirectPlay — that's the point |
|
||
|
||
## 8. Sequencing / effort (evenings-and-weekends scale)
|
||
|
||
- **Weekend 1:** gold images, minimum lab (5 VMs), E1 + E2.
|
||
- **Week 2:** E3 + E6 (join + ports + MTU), start E4.
|
||
- **Week 3:** E4 + E5 (cliff table, drop behavior).
|
||
- **Week 4:** E7 (BT411) and E8 (hub/stream rehearsal).
|
||
- **Wrap:** E9 dry run, fold results into the docs, go/no-go for Phase 1.
|