Cyd asked for an analysis of the networking stack and what would make the simulation feel better over the internet. The analysis found something more urgent than latency: the transport has been losing data silently since the arcade, and nothing in the game could see it happen. Every send result was discarded - L4NET, the console, all of it. On the 1ms arcade LAN the socket buffer never filled, so it never mattered. Over the internet it matters twice. A peer stalled in its own 10-30 second mission load stops reading, its window closes, and our nonblocking send starts answering would-block, which threw the message away; or worse, answering a PARTIAL count, and since framing on that stream is recovered purely from each message's length prefix, the bytes that never followed sheared it for good. Both are reachable in an ordinary race, because every race has a load in it. So sends go through a bounded per-connection queue now. What the wire will not take is kept, byte-exact, and retried at three flush points - before the render (the present blocks on vsync, and this frame's state should be travelling while it does), at the top of the receive pump, and before a connect sequence. Nothing is ever dropped from the middle: these are reliable ordered messages carrying entity creation, damage and race control, so a queue that overflows its 256K declares the connection dead and lets the disconnect path run rather than quietly desyncing the stream. RP412NETSENDQ=0 restores the old behaviour and still logs what it would have lost, which is the honest way to A/B it. On Steam the same queue finally surfaces k_EResultLimitExceeded, which the old code collapsed into -1 and discarded - that was backpressure, unlogged. The receive side gained the check the release build never had. The length prefix is untrusted input; Verify() compiles away in release, so a corrupt one went to memmove as a negative, or copied 4096 bytes of assembled packet into a 1600-byte stack buffer, or named a size the pad could never complete and wedged the connection forever. It is now validated against the same bounds the sender works to, and a stream that fails them is dropped like any other lost peer. And a fry that never ends: drop zones are map entities dealt round-robin at load, ownership transfer is not implemented, so a leaver's pads stay in the DropZones group. The respawn request dispatched to one goes to a host that is gone - dropped at the send, the 'no host N in the table' path - and the two-second retry re-dispatches to the same dead owner forever. The pad scan now skips zones whose owner has left, and re-validates one assigned earlier before reusing it. The rest is measurement, because the symptoms this work exists to chase are all reported in prose and none of them are in any log. Sixteen logs from the six-player night contain zero player-facing latency lines. A race now ends with a NetLog summary: per remote pod, how many updates arrived and how evenly (median and p95 out of a log2 histogram), the widest gap, how many gaps were long enough to mean a quiet sender versus short enough to mean OUR loop stalled, how often its motion snapped instead of blending, and how far arriving updates moved it. Per peer, whether the clock alignment ever had to step mid-race - which is the input for deciding if it needs slewing, rather than guessing. The mission t0 tick goes in the log too, alongside the console's per-pod RunMission send ticks, because nothing has ever measured how far apart the machines actually start; the clockwork doors inherit that skew directly. RP412NETSTATS adds the transport's own view - per connection: messages, bytes, wire writes, partials, refusals, how much sat queued - and on Steam the first read this codebase has ever taken of GetConnectionRealTime Status. Ping, quality, pending and unacked bytes, and one route description per connection at teardown. The API was vendored and never called; there was no RTT number anywhere in the game. Finally, rpl4opt -spoolstats reads any recording offline. The data was already in every spool ever made and nothing read it that way: the recorder restamps each packet with local arrival time while the update records inside keep the sender's sim-grid stamp, so the difference is clock offset plus one-way delay, and the same running-minimum estimator the game runs live separates them. It prints delay above the per-host minimum, and decomposes each entity's gaps into sender pacing versus delivery jitter - which no live counter can do. It lives in the game exe rather than RPL4TOOL because the tool is deliberately not /Zp1 and would misread every struct in the file. Verified on the two-pod loopback harness: mesh up, egg fed, 60s raced, stopped on command, scores collected, and both summaries reading exactly what a pair of PARKED pods should read - heartbeat cadence, one snap per heartbeat, sub-quarter-metre corrections, no clock steps. The t0 ticks and the netclock offsets agree with each other to the two seconds the pods launched apart. The latency tier is deliberately NOT here. TCP_NODELAY, the Steam NoNagle flag, per-frame coalescing and the pre-sim receive drain are all scoped and all wait on this build's numbers, because the point of shipping measurement first is to find out whether the thing we would fix is the thing that hurts. Nagle is still on. Interest management is still inert. The wire format is untouched, so this build and the last one still race each other. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
432 lines
11 KiB
C++
432 lines
11 KiB
C++
//===========================================================================//
|
|
// File: l4net.hh //
|
|
// Project: MUNGA Brick: Network Manager //
|
|
// Contents: Interface specification for network brick //
|
|
//---------------------------------------------------------------------------//
|
|
// Date Who Modification //
|
|
// -------- --- ---------------------------------------------------------- //
|
|
// 03/02/95 GAC Initial coding. //
|
|
//---------------------------------------------------------------------------//
|
|
// Copyright (C) 1994-1995, Virtual World Entertainment, Inc. //
|
|
// PROPRIETARY AND CONFIDENTIAL //
|
|
//===========================================================================//
|
|
|
|
#pragma once
|
|
|
|
#include "l4host.h"
|
|
|
|
#include "..\munga\network.h"
|
|
#include "..\munga\hostmgr.h"
|
|
|
|
//WinSock support :ADB 01/06/07
|
|
//#include "..\munga\netnub.h"
|
|
#define NETNUB_TCP_OPEN 3 // Opens a TCP stream to another computer
|
|
#define NETNUB_TCP_LISTEN 4 // Queue's a TCP listen that we can accept a connection on
|
|
#define MULTIPLE_SEND_PACKET_MAX 10
|
|
#define MAX_RECEIVE_DATA_SIZE 1600
|
|
#include <Winsock2.h>
|
|
#include <Ws2tcpip.h>
|
|
|
|
class NotationFile;
|
|
class L4NetworkManager__ReceiveEggFileMessage;
|
|
class L4NetworkManager__AcknowledgeEggFileMessage;
|
|
class L4NetworkManager__HostConnectedMessage;
|
|
class L4NetworkManager__HostDisconnectedMessage;
|
|
|
|
class NetNub
|
|
{
|
|
friend int
|
|
Netnub_Open_File(
|
|
const char* filename,
|
|
int access,
|
|
unsigned int model
|
|
);
|
|
friend int
|
|
Netnub_Write_File(
|
|
int handle,
|
|
void *buffer,
|
|
size_t length
|
|
);
|
|
friend int
|
|
Netnub_Close_File(int handle);
|
|
|
|
public:
|
|
//WinSock support :ADB 01/06/07
|
|
//static void SendCommand();
|
|
};
|
|
|
|
//extern Netcom_Ptr
|
|
// Net_Common_Ptr;
|
|
|
|
//~~~~~~~~~~~~~~~~~~~~~ MessageQueue__SendRequest ~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
class MessageQueue__SendRequest:
|
|
public Plug
|
|
{
|
|
friend class HostMessageBuffer__MessageQueue;
|
|
friend class L4NetworkManager__MessageBuffer;
|
|
|
|
//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
// Private methods
|
|
//
|
|
private:
|
|
MessageQueue__SendRequest(
|
|
NetworkClient::ClientID client_ID,
|
|
Receiver::Message *message
|
|
);
|
|
~MessageQueue__SendRequest();
|
|
|
|
Logical
|
|
TestInstance() const;
|
|
|
|
//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
// Private data
|
|
//
|
|
private:
|
|
NetworkClient::ClientID
|
|
clientID;
|
|
Receiver::Message
|
|
*messageToSend;
|
|
};
|
|
|
|
//~~~~~~~~~~~~~~~~~~~ HostMessageBuffer__MessageQueue ~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
class HostMessageBuffer__MessageQueue:
|
|
public Node
|
|
{
|
|
friend class L4NetworkManager__MessageBuffer;
|
|
|
|
//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
// Private methods
|
|
//
|
|
private:
|
|
HostMessageBuffer__MessageQueue(HostID host_ID);
|
|
~HostMessageBuffer__MessageQueue();
|
|
|
|
Logical
|
|
TestInstance() const;
|
|
|
|
void
|
|
AddSendRequest(
|
|
NetworkClient::ClientID client_ID,
|
|
Receiver::Message *message
|
|
);
|
|
|
|
//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
// Private data
|
|
//
|
|
private:
|
|
HostID
|
|
hostID;
|
|
|
|
typedef MessageQueue__SendRequest
|
|
SendRequest;
|
|
ChainOf<SendRequest*>
|
|
sendRequestSocket;
|
|
};
|
|
|
|
//~~~~~~~~~~~~~~~~~~ L4NetworkManager__MessageBuffer ~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
class L4NetworkManager__MessageBuffer:
|
|
public Node
|
|
{
|
|
//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
// Public methods
|
|
//
|
|
public:
|
|
L4NetworkManager__MessageBuffer(L4NetworkManager *network_manager);
|
|
~L4NetworkManager__MessageBuffer();
|
|
|
|
Logical
|
|
TestInstance() const;
|
|
|
|
Logical
|
|
IsEmpty();
|
|
void
|
|
AddSendRequest(
|
|
HostID host_ID,
|
|
NetworkClient::ClientID client_ID,
|
|
Receiver::Message *message
|
|
);
|
|
void
|
|
AttemptToSend();
|
|
|
|
//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
// Private data
|
|
//
|
|
private:
|
|
L4NetworkManager
|
|
*networkManager;
|
|
typedef HostMessageBuffer__MessageQueue
|
|
MessageQueue;
|
|
TableOf<MessageQueue*, HostID>
|
|
messageQueueSocket;
|
|
IteratorPosition
|
|
currentQueueIndex;
|
|
CollectionSize
|
|
bufferSize;
|
|
#ifdef LAB_ONLY
|
|
CollectionSize
|
|
messageCount,
|
|
maxBufferSize;
|
|
#endif
|
|
};
|
|
|
|
inline Logical
|
|
L4NetworkManager__MessageBuffer::IsEmpty()
|
|
{
|
|
Check(this);
|
|
return (bufferSize == 0);
|
|
}
|
|
|
|
//~~~~~~~~~~~~~~~~~~~~~~~~~~ l4Network manager~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
class L4NetworkManager: public NetworkManager
|
|
{
|
|
friend class L4NetworkManager__MessageBuffer;
|
|
|
|
//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
// Constructor, Destructor, Testing
|
|
//
|
|
public:
|
|
L4NetworkManager();
|
|
~L4NetworkManager();
|
|
|
|
static Logical TestClass();
|
|
Logical TestInstance() const;
|
|
|
|
//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
// Network message methods
|
|
//
|
|
public:
|
|
void Send(
|
|
Message *what,
|
|
ClientID to,
|
|
HostID host_ID);
|
|
void ExclusiveBroadcast(
|
|
Message *what,
|
|
ClientID to);
|
|
|
|
void StartConnecting(Mission *mission);
|
|
|
|
// forwards to the wire transport's queue drain (l4nettransport.h)
|
|
void FlushSends();
|
|
|
|
Logical Shutdown();
|
|
|
|
Logical CheckBuffers(NetworkPacket *packet);
|
|
|
|
void RemovePacket(NetworkPacket *packet);
|
|
|
|
Logical ExecuteBackground();
|
|
|
|
void Marker(char *marker_text);
|
|
|
|
void Mode(NetworkMode myMode);
|
|
|
|
void CreateConsoleHost();
|
|
|
|
//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
// Network maintenance support
|
|
//
|
|
public:
|
|
int netPlayerCount;
|
|
|
|
enum NetworkState
|
|
{
|
|
NormalState,
|
|
ConsoleOnly
|
|
};
|
|
|
|
enum NetworkStartupMode
|
|
{
|
|
SlaveMode,
|
|
MasterMode
|
|
};
|
|
|
|
bool CheckSocket(SOCKET socket, SOCKADDR_IN *remoteEndpoint);
|
|
bool ResolveAddress(CString host_name, SOCKADDR_IN *address);
|
|
//WinSock support :ADB 01/06/07
|
|
NetworkAddress* GetMyAddress();
|
|
|
|
//WinSock support :ADB 01/06/07
|
|
SOCKET OpenConnection(
|
|
int connection_type, // NETNUB_TCP_LISTEN or NETNUB_TCP_OPEN
|
|
int local_port,
|
|
int remote_port,
|
|
int internet_address);
|
|
void CloseConnection(SOCKET socket_ptr); // socket address from netnub (to close)
|
|
Logical GetNextMungaPacket(
|
|
NetworkPacket *network_packet,
|
|
HostManager::RemoteHostIterator* all_iterator);
|
|
|
|
//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
// Shared Data support
|
|
//
|
|
public:
|
|
static Derivation *GetClassDerivations();
|
|
static SharedData DefaultData;
|
|
|
|
//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
// Message Support
|
|
//
|
|
public:
|
|
//
|
|
// Message IDs
|
|
//
|
|
enum
|
|
{
|
|
AcknowledgeEggFileMessageID = NetworkManager::NextMessageID,
|
|
HostConnectedMessageID,
|
|
HostDisconnectedMessageID,
|
|
NextMessageID
|
|
};
|
|
|
|
//
|
|
// Message types
|
|
//
|
|
typedef L4NetworkManager__AcknowledgeEggFileMessage
|
|
AcknowledgeEggFileMessage;
|
|
typedef L4NetworkManager__HostConnectedMessage
|
|
HostConnectedMessage;
|
|
typedef L4NetworkManager__HostDisconnectedMessage
|
|
HostDisconnectedMessage;
|
|
//
|
|
// Message table
|
|
//
|
|
static const HandlerEntry MessageHandlerEntries[];
|
|
|
|
//static MessageHandlerSet MessageHandlers;
|
|
static MessageHandlerSet& GetMessageHandlers();
|
|
|
|
// Load a local egg file and kick the mission cycle - the same path
|
|
// the single-user startup takes; used by the in-game front end /
|
|
// local console for the race-after-race loop.
|
|
void FeedLocalEgg(const char *egg_path);
|
|
|
|
void ReceiveEggFileMessageHandler(
|
|
ReceiveEggFileMessage *EggMessage);
|
|
void AcknowledgeEggFileMessageHandler(
|
|
AcknowledgeEggFileMessage *AcknowledgeEgg);
|
|
void HostConnectedMessageHandler(
|
|
HostConnectedMessage *HostConnected);
|
|
void HostDisconnectedMessageHandler(
|
|
HostDisconnectedMessage *ConsoleDisconnect);
|
|
|
|
//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
// Private methods
|
|
//
|
|
private:
|
|
typedef ChainOf<L4Host*>
|
|
DroppedMessageHostSocket;
|
|
typedef ChainIteratorOf<L4Host*>
|
|
DroppedMessageHostIterator;
|
|
|
|
Logical
|
|
SendMessageToNetnub(
|
|
Message *message,
|
|
ClientID client_ID,
|
|
HostID host_ID
|
|
);
|
|
void
|
|
SendBatchedMessageToNetnub(
|
|
Message *message,
|
|
ClientID client,
|
|
DroppedMessageHostSocket *dropped_message_host_socket
|
|
);
|
|
|
|
//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
// Private Data
|
|
//
|
|
private:
|
|
int
|
|
numberOfMungaHostsConnected,
|
|
numberOfConsoleHostsConnected,
|
|
remoteHostCount;
|
|
NetworkStartupMode networkStartupMode;
|
|
Logical eggAcknowledged;
|
|
NetworkState currentNetworkState;
|
|
L4Host *myConsoleHost;
|
|
HostID nextOpenHostID;
|
|
IteratorPosition lastHostIteratorPosition;
|
|
|
|
typedef L4NetworkManager__MessageBuffer
|
|
MessageBuffer;
|
|
MessageBuffer
|
|
messageBuffer;
|
|
|
|
//WinSock Support
|
|
WSADATA* wsaData;
|
|
//hostent* thisHost;
|
|
addrinfo* thisHost;
|
|
SOCKET gameListenerSocket;
|
|
SOCKET consoleListenerSocket;
|
|
};
|
|
|
|
//~~~~~~~~~~~~~~~~~~~~~~ L4NetworkManager inlines ~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
// L4NetworkManager__AcknowledgeEggFileMessage
|
|
// This message is sent back to the console to acknowledge receipt of the
|
|
// egg, it indicates this computer has established all it's connections and
|
|
// the console can procede with the next host
|
|
//
|
|
class L4NetworkManager__AcknowledgeEggFileMessage:
|
|
public NetworkManager::Message
|
|
{
|
|
public:
|
|
L4NetworkManager__AcknowledgeEggFileMessage():
|
|
NetworkManager::Message(
|
|
L4NetworkManager::AcknowledgeEggFileMessageID,
|
|
sizeof(L4NetworkManager__AcknowledgeEggFileMessage)
|
|
){}
|
|
};
|
|
|
|
//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
// L4NetworkManager__HostConnectedMessage
|
|
// This message is generated internally by CheckBuffers and indicates that
|
|
// a host has connected up to us. This is an INTERNAL message only and is
|
|
// not ment to be sent on the network.
|
|
class L4NetworkManager__HostConnectedMessage :
|
|
public NetworkManager::Message
|
|
{
|
|
public:
|
|
L4NetworkManager__HostConnectedMessage(HostID host_id, const SOCKADDR_IN &network_address, unsigned long stream_pointer) :
|
|
NetworkManager::Message(L4NetworkManager::HostConnectedMessageID, sizeof(L4NetworkManager__HostConnectedMessage)),
|
|
hostID(host_id),
|
|
networkAddress(network_address),
|
|
streamPointer(stream_pointer)
|
|
{
|
|
}
|
|
|
|
HostID hostID;
|
|
SOCKADDR_IN networkAddress;
|
|
unsigned long streamPointer;
|
|
};
|
|
|
|
//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
// L4NetworkManager__HostDisconnectedMessage
|
|
// The console is expected to send us this message if it disconnects from us
|
|
// for any reason. This allows the local host to setup the internal state
|
|
// so the game will auto start even if the console can't stay connected to us
|
|
// for some reason (for example, the console simulator in the game code can't
|
|
// connect to more than one host at once).
|
|
// !!! CheckBuffers should poll the connection state of all the streams and
|
|
// generate this message automatically whenever a host drops off the net.
|
|
// !!! HACK At the moment this message is always treated as if it came from
|
|
// the console host, so it should only be sent down the console stream.
|
|
//
|
|
class L4NetworkManager__HostDisconnectedMessage :
|
|
public NetworkManager::Message
|
|
{
|
|
public:
|
|
L4NetworkManager__HostDisconnectedMessage(HostID host_id, unsigned long stream_pointer) :
|
|
NetworkManager::Message(L4NetworkManager::HostDisconnectedMessageID, sizeof(L4NetworkManager__HostDisconnectedMessage)),
|
|
hostID(host_id),
|
|
streamPointer(stream_pointer)
|
|
{
|
|
}
|
|
|
|
HostID hostID;
|
|
unsigned long streamPointer;
|
|
}; |