# Network testing — WAN emulation, telemetry, and the stall test

How to make the loopback two-pod harness behave like the internet, and
how to read what the new instrumentation says about it. Added with the
first networking measurement+correctness build (2026-08-13); the plan it
serves is `~/.claude/plans` material, summarized here so the procedure
outlives the session that wrote it.

Scope honestly stated: everything below exercises the **plain TCP
transport** on loopback. There is no SDR emulation — claims about the
Steam path rest on live telemetry (`RP412NETSTATS=1`) gathered during
real Steam races ([STEAM-3-MACHINE-TEST.md](STEAM-3-MACHINE-TEST.md)).

## The instrumentation (what a race now writes)

| Switch | Default | What appears in rpl4.log |
|--------|---------|--------------------------|
| `RP412NETLOG` | on | `NetLog:` lines — mission t0 tick at the green light; per-pod race totals at the stop (updates, interval med/p95, widest gap, long/queued counts, snaps, correction mean/worst); per-peer netclock offset and window-step counts; the console's per-pod RunMission send ticks |
| `RP412NETSTATS` | off | `NetStats:` lines — 5 s per-connection transport rollups (messages, bytes, wire writes, partials, would-blocks, retry-queue high water); on Steam also ping/quality/pending/unacked per connection and a route description at teardown |
| `RP412NETSENDQ` | on | set `=0` to restore fire-and-forget sends (still logs what it would have lost) — the A/B for the send queue |

Offline: `rpl4opt -spoolstats SPOOLS\<name>.spl` reads any Live Cam or
pod recording and prints per-host one-way-delay-above-minimum and
per-entity sender-vs-arrival gap decompositions, writing the raw rows to
`<name>.spl.netstats.csv` beside it. Every spool ever recorded is
retroactively a latency dataset; the RECORDING toggle on the setup
screen is how new ones get made.

## WAN emulation with clumsy

[clumsy](https://jagt.github.io/clumsy/) (WinDivert underneath) is the
one Windows emulator that captures loopback traffic, which is what the
two-pod harness runs on. Run it as administrator, set the filter, pick a
profile, **then** start `tools\two-pod-test.ps1`.

Filter — the game mesh only (console channels stay clean):

    tcp and (tcp.DstPort == 1502 or tcp.SrcPort == 1502 or tcp.DstPort == 1602 or tcp.SrcPort == 1602)

Add `1501`/`1601` terms to also stress the console marshaling (egg feed,
state polls, the stop).

Profiles (stock clumsy has fixed lag, not jitter — say which was used
when reporting numbers):

| Profile | clumsy settings | What it stands in for |
|---------|-----------------|----------------------|
| **Steady internet** | Lag 80 ms, inbound + outbound | a good SDR route (~160 ms RTT) |
| **Asymmetric** | Lag 150 ms, inbound only | one slow direction, the shape NETCLOCK's min-filter has to survive |
| **Disorderly** | Lag 30 ms both + Out-of-order 2% | reorder torture — TCP re-orders below the engine, so the engine-visible symptom is added jitter, not reordering; the stale counter should stay 0 |

What to look at afterward: both pods' `rpl4.log` (the harness leaves
them in its scratch `podA\` / `podB\` folders) — the `NetLog:` race
summary should show interval medians tracking the imposed lag pattern,
snaps staying rare, and corrections staying under ~0.5 m mean. Compare
against a clean run of the same script; that pair of summaries *is* the
result.

## The stall-recovery test (the send queue's reason to exist)

A peer stalled in its 10–30 s mission load stops reading; before the
send queue, every machine still racing silently dropped — or worse,
half-sent and permanently sheared — whatever it sent them. To reproduce
the stall deliberately:

    # freeze pod B for 20 s mid-race, then thaw it
    $p = Get-Process rpl4opt | Sort-Object StartTime | Select-Object -Last 1
    # (suspend/resume via SysInternals: pssuspend $p.Id ; sleep 20 ; pssuspend -r $p.Id)

Expected with the queue (default): pod A's `NetStats` rollup shows
would-blocks and a rising retry queue during the freeze, draining after
the thaw; the race continues; no `framing broken` line ever appears.
Expected with `RP412NETSENDQ=0`: pod A logs `legacy mode lost a
message` / `sheared the stream` — the old behavior, now at least named.
A `Receive: host N framing broken ... dropping the connection` line
means the receive-side validation caught a sheared stream and dropped
the peer rather than crashing — correct behavior, but on the queue path
it should never be needed.

## Version discipline

The wire format is unchanged by all of this. A mixed run — one pod on
the previous build, one on this — must interoperate, and doing that once
per build is part of the checklist: it is the proof the format did not
drift.