Cyd asked for an analysis of the networking stack and what would make the simulation feel better over the internet. The analysis found something more urgent than latency: the transport has been losing data silently since the arcade, and nothing in the game could see it happen. Every send result was discarded - L4NET, the console, all of it. On the 1ms arcade LAN the socket buffer never filled, so it never mattered. Over the internet it matters twice. A peer stalled in its own 10-30 second mission load stops reading, its window closes, and our nonblocking send starts answering would-block, which threw the message away; or worse, answering a PARTIAL count, and since framing on that stream is recovered purely from each message's length prefix, the bytes that never followed sheared it for good. Both are reachable in an ordinary race, because every race has a load in it. So sends go through a bounded per-connection queue now. What the wire will not take is kept, byte-exact, and retried at three flush points - before the render (the present blocks on vsync, and this frame's state should be travelling while it does), at the top of the receive pump, and before a connect sequence. Nothing is ever dropped from the middle: these are reliable ordered messages carrying entity creation, damage and race control, so a queue that overflows its 256K declares the connection dead and lets the disconnect path run rather than quietly desyncing the stream. RP412NETSENDQ=0 restores the old behaviour and still logs what it would have lost, which is the honest way to A/B it. On Steam the same queue finally surfaces k_EResultLimitExceeded, which the old code collapsed into -1 and discarded - that was backpressure, unlogged. The receive side gained the check the release build never had. The length prefix is untrusted input; Verify() compiles away in release, so a corrupt one went to memmove as a negative, or copied 4096 bytes of assembled packet into a 1600-byte stack buffer, or named a size the pad could never complete and wedged the connection forever. It is now validated against the same bounds the sender works to, and a stream that fails them is dropped like any other lost peer. And a fry that never ends: drop zones are map entities dealt round-robin at load, ownership transfer is not implemented, so a leaver's pads stay in the DropZones group. The respawn request dispatched to one goes to a host that is gone - dropped at the send, the 'no host N in the table' path - and the two-second retry re-dispatches to the same dead owner forever. The pad scan now skips zones whose owner has left, and re-validates one assigned earlier before reusing it. The rest is measurement, because the symptoms this work exists to chase are all reported in prose and none of them are in any log. Sixteen logs from the six-player night contain zero player-facing latency lines. A race now ends with a NetLog summary: per remote pod, how many updates arrived and how evenly (median and p95 out of a log2 histogram), the widest gap, how many gaps were long enough to mean a quiet sender versus short enough to mean OUR loop stalled, how often its motion snapped instead of blending, and how far arriving updates moved it. Per peer, whether the clock alignment ever had to step mid-race - which is the input for deciding if it needs slewing, rather than guessing. The mission t0 tick goes in the log too, alongside the console's per-pod RunMission send ticks, because nothing has ever measured how far apart the machines actually start; the clockwork doors inherit that skew directly. RP412NETSTATS adds the transport's own view - per connection: messages, bytes, wire writes, partials, refusals, how much sat queued - and on Steam the first read this codebase has ever taken of GetConnectionRealTime Status. Ping, quality, pending and unacked bytes, and one route description per connection at teardown. The API was vendored and never called; there was no RTT number anywhere in the game. Finally, rpl4opt -spoolstats reads any recording offline. The data was already in every spool ever made and nothing read it that way: the recorder restamps each packet with local arrival time while the update records inside keep the sender's sim-grid stamp, so the difference is clock offset plus one-way delay, and the same running-minimum estimator the game runs live separates them. It prints delay above the per-host minimum, and decomposes each entity's gaps into sender pacing versus delivery jitter - which no live counter can do. It lives in the game exe rather than RPL4TOOL because the tool is deliberately not /Zp1 and would misread every struct in the file. Verified on the two-pod loopback harness: mesh up, egg fed, 60s raced, stopped on command, scores collected, and both summaries reading exactly what a pair of PARKED pods should read - heartbeat cadence, one snap per heartbeat, sub-quarter-metre corrections, no clock steps. The t0 ticks and the netclock offsets agree with each other to the two seconds the pods launched apart. The latency tier is deliberately NOT here. TCP_NODELAY, the Steam NoNagle flag, per-frame coalescing and the pre-sim receive drain are all scoped and all wait on this build's numbers, because the point of shipping measurement first is to find out whether the thing we would fix is the thing that hurts. Nagle is still on. Interest management is still inert. The wire format is untouched, so this build and the last one still race each other. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
269 lines
8.9 KiB
C++
269 lines
8.9 KiB
C++
//===========================================================================//
|
|
// File: l4nettransport.h //
|
|
// Project: MUNGA Brick: Network Transport Seam //
|
|
// Contents: The wire interface under the network manager //
|
|
//---------------------------------------------------------------------------//
|
|
// Copyright (C) 1994-1995, Virtual World Entertainment, Inc. //
|
|
// PROPRIETARY AND CONFIDENTIAL //
|
|
//===========================================================================//
|
|
|
|
#pragma once
|
|
|
|
#include "..\munga\style.h"
|
|
#include <Winsock2.h>
|
|
|
|
//########################################################################
|
|
// NetTransport - the seam between the network manager's mesh/console
|
|
// logic and the wire (docs/RP412-FRONTEND-DESIGN.md section 3). Mirrors
|
|
// the RIOBase pattern: L4NET keeps hosts, message queues, and the
|
|
// deterministic mesh ordering; only connect/listen/send/recv live
|
|
// behind this interface.
|
|
//
|
|
// Implementations:
|
|
// WinsockNetTransport (l4nettransport.cpp) - TCP; the arcade/LAN wire
|
|
// SteamNetTransport (future) - ISteamNetworkingSockets
|
|
// P2P over SDR
|
|
//
|
|
// Addresses stay SOCKADDR_IN-shaped on purpose: Steam's FakeIP system
|
|
// hands out fake IPv4 addresses for exactly this kind of engine, so
|
|
// the [pilots] list keeps working as ip[:port] strings in both worlds.
|
|
//########################################################################
|
|
|
|
class NetTransport
|
|
{
|
|
public:
|
|
// Opaque connection handle. SOCKET for Winsock (SOCKET is UINT_PTR),
|
|
// HSteamNetConnection for Steam - both fit; callers must not
|
|
// interpret it.
|
|
typedef UINT_PTR Connection;
|
|
static const Connection InvalidConnection = (Connection) ~0; // == INVALID_SOCKET
|
|
|
|
// Receive() results when no payload came back
|
|
enum
|
|
{
|
|
ReceiveDisconnected = 0, // orderly close or connection reset
|
|
ReceiveNoData = -1 // nothing pending (connections are nonblocking)
|
|
};
|
|
|
|
virtual ~NetTransport()
|
|
{
|
|
}
|
|
|
|
//---------------------------------------------------------------
|
|
// Lifecycle
|
|
//---------------------------------------------------------------
|
|
virtual Logical
|
|
Startup() = 0;
|
|
virtual void
|
|
Cleanup() = 0;
|
|
|
|
//---------------------------------------------------------------
|
|
// Addressing: the local interface list (the mesh identifies
|
|
// "which [pilots] entry is me" against it) and numeric ip[:port]
|
|
// parsing. Port 0 in the result means "caller applies default".
|
|
//---------------------------------------------------------------
|
|
virtual int
|
|
GetLocalAddresses(
|
|
unsigned long *addresses,
|
|
int max_count
|
|
) = 0;
|
|
virtual Logical
|
|
Resolve(
|
|
const char *host_name,
|
|
SOCKADDR_IN *address
|
|
) = 0;
|
|
|
|
//---------------------------------------------------------------
|
|
// Connections (the deterministic mesh + the console channel).
|
|
// Connect blocks until the remote end accepts (the mesh relies
|
|
// on retry-until-up ordering), then goes nonblocking. Listeners
|
|
// are nonblocking from the start; Accept polls one.
|
|
//---------------------------------------------------------------
|
|
virtual Connection
|
|
Connect(
|
|
const SOCKADDR_IN *remote,
|
|
int local_port
|
|
) = 0;
|
|
virtual Connection
|
|
Listen(
|
|
int local_port,
|
|
int backlog
|
|
) = 0;
|
|
virtual Connection
|
|
Accept(Connection listener) = 0;
|
|
virtual void
|
|
Close(Connection connection) = 0;
|
|
|
|
//---------------------------------------------------------------
|
|
// Data plane (nonblocking)
|
|
//---------------------------------------------------------------
|
|
virtual int
|
|
Send(
|
|
Connection connection,
|
|
const void *data,
|
|
int size
|
|
) = 0;
|
|
virtual int
|
|
Receive(
|
|
Connection connection,
|
|
void *buffer,
|
|
int size
|
|
) = 0;
|
|
|
|
// Retry anything a connection could not take when Send was called.
|
|
// The engine calls this at its flush points (before the render, at
|
|
// the top of the receive pump, before a connect sequence); default
|
|
// is a no-op so a transport without queues needs nothing.
|
|
virtual void
|
|
FlushSends()
|
|
{
|
|
}
|
|
|
|
// remote endpoint of a live connection (mesh identity checks)
|
|
virtual Logical
|
|
GetRemoteAddress(
|
|
Connection connection,
|
|
SOCKADDR_IN *address
|
|
) = 0;
|
|
};
|
|
|
|
// The process-wide transport. Defaults to Winsock TCP; a Steam build
|
|
// installs its transport BEFORE the network manager comes up.
|
|
NetTransport *
|
|
NetTransport_Get();
|
|
void
|
|
NetTransport_Set(NetTransport *transport);
|
|
|
|
//########################################################################
|
|
// Outbound send queue (RP412NETSENDQ).
|
|
//
|
|
// Historically every send result was discarded. On the 1 ms arcade LAN
|
|
// the socket buffer never filled, so nothing was ever lost - but over
|
|
// the internet a peer stalled in its 10-30 s mission load stops reading,
|
|
// its window closes, and our nonblocking send starts returning
|
|
// would-block (message silently vanished) or a PARTIAL count. A partial
|
|
// send is the worse of the two: framing on the stream is recovered
|
|
// purely from each message's length prefix, so the bytes that never
|
|
// followed shear the stream for good.
|
|
//
|
|
// The queue closes both holes: a connection that cannot take the bytes
|
|
// now keeps them, byte-exact, and FlushSends() retries at the engine's
|
|
// flush points. Nothing is ever dropped from the middle of the stream -
|
|
// these are reliable, ordered, non-idempotent messages, so the only
|
|
// legal overflow response is to declare the connection dead and let the
|
|
// normal disconnect path run (Receive() reports it).
|
|
//
|
|
// RP412NETSENDQ=0 restores the old direct sends but keeps counting and
|
|
// logging the losses it would have prevented.
|
|
//########################################################################
|
|
|
|
Logical
|
|
NetTransport_SendQueueEnabled(); // RP412NETSENDQ, default on
|
|
int
|
|
NetTransport_SendQueueCapBytes(); // RP412NETQUEUEKB * 1024, default 256K
|
|
Logical
|
|
NetTransport_StatsEnabled(); // RP412NETSTATS, default off: 5 s
|
|
// per-connection rollups (and, on
|
|
// Steam, a 1 Hz ping/quality poll)
|
|
// printed from FlushSends
|
|
|
|
class NetSendQueue
|
|
{
|
|
public:
|
|
NetTransport::Connection
|
|
connection; // InvalidConnection marks a free slot
|
|
unsigned char
|
|
*buffer; // allocated on first append
|
|
int
|
|
head, // first unsent byte
|
|
tail; // one past the last queued byte
|
|
Logical
|
|
dead; // overflow or hard error; Receive() turns this
|
|
// into ReceiveDisconnected
|
|
|
|
// telemetry, printed under RP412NETSTATS
|
|
unsigned long
|
|
sentMessages, // engine Send() calls accepted
|
|
sentBytes,
|
|
wireWrites, // actual writes on the wire
|
|
partialWrites, // wire writes that took only part of the bytes
|
|
wouldBlocks, // wire writes refused outright
|
|
queuedHighWater; // worst PendingBytes() seen
|
|
|
|
int
|
|
PendingBytes() const
|
|
{
|
|
return tail - head;
|
|
}
|
|
};
|
|
|
|
class NetSendQueueTable
|
|
{
|
|
public:
|
|
enum { MaxQueues = 24 }; // 8 pods' mesh + console + listeners + slack
|
|
|
|
NetSendQueueTable();
|
|
~NetSendQueueTable();
|
|
|
|
NetSendQueue *
|
|
Find(NetTransport::Connection connection);
|
|
NetSendQueue *
|
|
FindOrAdopt(NetTransport::Connection connection);
|
|
void
|
|
Release(NetTransport::Connection connection);
|
|
void
|
|
ReleaseAll();
|
|
|
|
// Append bytes to a queue. False = overflow: the pending bytes are
|
|
// freed and the queue is marked dead (see the block comment above).
|
|
Logical
|
|
Append(
|
|
NetSendQueue *queue,
|
|
const void *data,
|
|
int size);
|
|
|
|
NetSendQueue
|
|
queues[MaxQueues];
|
|
};
|
|
|
|
//########################################################################
|
|
// Waiting for a peer without hanging the window.
|
|
//
|
|
// Connect retries until the peer answers, because the egg-ACK ordering
|
|
// means it may still be booting - that part is deliberate. What was not
|
|
// deliberate is that the wait slept without pumping messages, so Windows
|
|
// saw a process that had stopped answering and painted the whole thing
|
|
// "Not responding" for up to two minutes. It happens before the engine
|
|
// block, so there is no render loop keeping the window alive either.
|
|
//
|
|
// Sleep through this instead. It pumps, so the window keeps drawing and
|
|
// can be moved; it watches for a cancel; and it is re-entrancy guarded,
|
|
// because dispatching a message can run application code that reaches a
|
|
// connect of its own.
|
|
//########################################################################
|
|
|
|
enum NetWaitResult
|
|
{
|
|
NetWaitContinue, // carry on waiting
|
|
NetWaitCancelled, // the user pressed escape
|
|
NetWaitQuit // WM_QUIT arrived; the caller must unwind
|
|
};
|
|
|
|
NetWaitResult
|
|
NetTransport_PumpAndSleep(int milliseconds);
|
|
|
|
// How long Connect keeps redialling, in seconds. RP412CONNECTWAIT
|
|
// overrides; see the definition for why the default came down from 120.
|
|
int
|
|
NetTransport_ConnectWaitSeconds();
|
|
|
|
// Progress, shown in the window title while a connect is outstanding.
|
|
// Pass NULL to put the original title back.
|
|
void
|
|
NetTransport_SetWaitProgress(const char *text);
|
|
|
|
// Forget a previous escape, so one race's cancel cannot cancel the next.
|
|
// Call before starting a connect sequence.
|
|
void
|
|
NetTransport_ClearWaitCancel();
|