CydandClaude Fable 5
7d485c9672
Cyd asked for an analysis of the networking stack and what would make the simulation feel better over the internet. The analysis found something more urgent than latency: the transport has been losing data silently since the arcade, and nothing in the game could see it happen. Every send result was discarded - L4NET, the console, all of it. On the 1ms arcade LAN the socket buffer never filled, so it never mattered. Over the internet it matters twice. A peer stalled in its own 10-30 second mission load stops reading, its window closes, and our nonblocking send starts answering would-block, which threw the message away; or worse, answering a PARTIAL count, and since framing on that stream is recovered purely from each message's length prefix, the bytes that never followed sheared it for good. Both are reachable in an ordinary race, because every race has a load in it. So sends go through a bounded per-connection queue now. What the wire will not take is kept, byte-exact, and retried at three flush points - before the render (the present blocks on vsync, and this frame's state should be travelling while it does), at the top of the receive pump, and before a connect sequence. Nothing is ever dropped from the middle: these are reliable ordered messages carrying entity creation, damage and race control, so a queue that overflows its 256K declares the connection dead and lets the disconnect path run rather than quietly desyncing the stream. RP412NETSENDQ=0 restores the old behaviour and still logs what it would have lost, which is the honest way to A/B it. On Steam the same queue finally surfaces k_EResultLimitExceeded, which the old code collapsed into -1 and discarded - that was backpressure, unlogged. The receive side gained the check the release build never had. The length prefix is untrusted input; Verify() compiles away in release, so a corrupt one went to memmove as a negative, or copied 4096 bytes of assembled packet into a 1600-byte stack buffer, or named a size the pad could never complete and wedged the connection forever. It is now validated against the same bounds the sender works to, and a stream that fails them is dropped like any other lost peer. And a fry that never ends: drop zones are map entities dealt round-robin at load, ownership transfer is not implemented, so a leaver's pads stay in the DropZones group. The respawn request dispatched to one goes to a host that is gone - dropped at the send, the 'no host N in the table' path - and the two-second retry re-dispatches to the same dead owner forever. The pad scan now skips zones whose owner has left, and re-validates one assigned earlier before reusing it. The rest is measurement, because the symptoms this work exists to chase are all reported in prose and none of them are in any log. Sixteen logs from the six-player night contain zero player-facing latency lines. A race now ends with a NetLog summary: per remote pod, how many updates arrived and how evenly (median and p95 out of a log2 histogram), the widest gap, how many gaps were long enough to mean a quiet sender versus short enough to mean OUR loop stalled, how often its motion snapped instead of blending, and how far arriving updates moved it. Per peer, whether the clock alignment ever had to step mid-race - which is the input for deciding if it needs slewing, rather than guessing. The mission t0 tick goes in the log too, alongside the console's per-pod RunMission send ticks, because nothing has ever measured how far apart the machines actually start; the clockwork doors inherit that skew directly. RP412NETSTATS adds the transport's own view - per connection: messages, bytes, wire writes, partials, refusals, how much sat queued - and on Steam the first read this codebase has ever taken of GetConnectionRealTime Status. Ping, quality, pending and unacked bytes, and one route description per connection at teardown. The API was vendored and never called; there was no RTT number anywhere in the game. Finally, rpl4opt -spoolstats reads any recording offline. The data was already in every spool ever made and nothing read it that way: the recorder restamps each packet with local arrival time while the update records inside keep the sender's sim-grid stamp, so the difference is clock offset plus one-way delay, and the same running-minimum estimator the game runs live separates them. It prints delay above the per-host minimum, and decomposes each entity's gaps into sender pacing versus delivery jitter - which no live counter can do. It lives in the game exe rather than RPL4TOOL because the tool is deliberately not /Zp1 and would misread every struct in the file. Verified on the two-pod loopback harness: mesh up, egg fed, 60s raced, stopped on command, scores collected, and both summaries reading exactly what a pair of PARKED pods should read - heartbeat cadence, one snap per heartbeat, sub-quarter-metre corrections, no clock steps. The t0 ticks and the netclock offsets agree with each other to the two seconds the pods launched apart. The latency tier is deliberately NOT here. TCP_NODELAY, the Steam NoNagle flag, per-frame coalescing and the pre-sim receive drain are all scoped and all wait on this build's numbers, because the point of shipping measurement first is to find out whether the thing we would fix is the thing that hurts. Nagle is still on. Interest management is still inert. The wire format is untouched, so this build and the last one still race each other. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
86 lines
4.7 KiB
Markdown
86 lines
4.7 KiB
Markdown
# Network testing — WAN emulation, telemetry, and the stall test
|
||
|
||
How to make the loopback two-pod harness behave like the internet, and
|
||
how to read what the new instrumentation says about it. Added with the
|
||
first networking measurement+correctness build (2026-08-13); the plan it
|
||
serves is `~/.claude/plans` material, summarized here so the procedure
|
||
outlives the session that wrote it.
|
||
|
||
Scope honestly stated: everything below exercises the **plain TCP
|
||
transport** on loopback. There is no SDR emulation — claims about the
|
||
Steam path rest on live telemetry (`RP412NETSTATS=1`) gathered during
|
||
real Steam races ([STEAM-3-MACHINE-TEST.md](STEAM-3-MACHINE-TEST.md)).
|
||
|
||
## The instrumentation (what a race now writes)
|
||
|
||
| Switch | Default | What appears in rpl4.log |
|
||
|--------|---------|--------------------------|
|
||
| `RP412NETLOG` | on | `NetLog:` lines — mission t0 tick at the green light; per-pod race totals at the stop (updates, interval med/p95, widest gap, long/queued counts, snaps, correction mean/worst); per-peer netclock offset and window-step counts; the console's per-pod RunMission send ticks |
|
||
| `RP412NETSTATS` | off | `NetStats:` lines — 5 s per-connection transport rollups (messages, bytes, wire writes, partials, would-blocks, retry-queue high water); on Steam also ping/quality/pending/unacked per connection and a route description at teardown |
|
||
| `RP412NETSENDQ` | on | set `=0` to restore fire-and-forget sends (still logs what it would have lost) — the A/B for the send queue |
|
||
|
||
Offline: `rpl4opt -spoolstats SPOOLS\<name>.spl` reads any Live Cam or
|
||
pod recording and prints per-host one-way-delay-above-minimum and
|
||
per-entity sender-vs-arrival gap decompositions, writing the raw rows to
|
||
`<name>.spl.netstats.csv` beside it. Every spool ever recorded is
|
||
retroactively a latency dataset; the RECORDING toggle on the setup
|
||
screen is how new ones get made.
|
||
|
||
## WAN emulation with clumsy
|
||
|
||
[clumsy](https://jagt.github.io/clumsy/) (WinDivert underneath) is the
|
||
one Windows emulator that captures loopback traffic, which is what the
|
||
two-pod harness runs on. Run it as administrator, set the filter, pick a
|
||
profile, **then** start `tools\two-pod-test.ps1`.
|
||
|
||
Filter — the game mesh only (console channels stay clean):
|
||
|
||
tcp and (tcp.DstPort == 1502 or tcp.SrcPort == 1502 or tcp.DstPort == 1602 or tcp.SrcPort == 1602)
|
||
|
||
Add `1501`/`1601` terms to also stress the console marshaling (egg feed,
|
||
state polls, the stop).
|
||
|
||
Profiles (stock clumsy has fixed lag, not jitter — say which was used
|
||
when reporting numbers):
|
||
|
||
| Profile | clumsy settings | What it stands in for |
|
||
|---------|-----------------|----------------------|
|
||
| **Steady internet** | Lag 80 ms, inbound + outbound | a good SDR route (~160 ms RTT) |
|
||
| **Asymmetric** | Lag 150 ms, inbound only | one slow direction, the shape NETCLOCK's min-filter has to survive |
|
||
| **Disorderly** | Lag 30 ms both + Out-of-order 2% | reorder torture — TCP re-orders below the engine, so the engine-visible symptom is added jitter, not reordering; the stale counter should stay 0 |
|
||
|
||
What to look at afterward: both pods' `rpl4.log` (the harness leaves
|
||
them in its scratch `podA\` / `podB\` folders) — the `NetLog:` race
|
||
summary should show interval medians tracking the imposed lag pattern,
|
||
snaps staying rare, and corrections staying under ~0.5 m mean. Compare
|
||
against a clean run of the same script; that pair of summaries *is* the
|
||
result.
|
||
|
||
## The stall-recovery test (the send queue's reason to exist)
|
||
|
||
A peer stalled in its 10–30 s mission load stops reading; before the
|
||
send queue, every machine still racing silently dropped — or worse,
|
||
half-sent and permanently sheared — whatever it sent them. To reproduce
|
||
the stall deliberately:
|
||
|
||
# freeze pod B for 20 s mid-race, then thaw it
|
||
$p = Get-Process rpl4opt | Sort-Object StartTime | Select-Object -Last 1
|
||
# (suspend/resume via SysInternals: pssuspend $p.Id ; sleep 20 ; pssuspend -r $p.Id)
|
||
|
||
Expected with the queue (default): pod A's `NetStats` rollup shows
|
||
would-blocks and a rising retry queue during the freeze, draining after
|
||
the thaw; the race continues; no `framing broken` line ever appears.
|
||
Expected with `RP412NETSENDQ=0`: pod A logs `legacy mode lost a
|
||
message` / `sheared the stream` — the old behavior, now at least named.
|
||
A `Receive: host N framing broken ... dropping the connection` line
|
||
means the receive-side validation caught a sheared stream and dropped
|
||
the peer rather than crashing — correct behavior, but on the queue path
|
||
it should never be needed.
|
||
|
||
## Version discipline
|
||
|
||
The wire format is unchanged by all of this. A mixed run — one pod on
|
||
the previous build, one on this — must interoperate, and doing that once
|
||
per build is part of the checklist: it is the proof the format did not
|
||
drift.
|