Cyd asked for an analysis of the networking stack and what would make the simulation feel better over the internet. The analysis found something more urgent than latency: the transport has been losing data silently since the arcade, and nothing in the game could see it happen. Every send result was discarded - L4NET, the console, all of it. On the 1ms arcade LAN the socket buffer never filled, so it never mattered. Over the internet it matters twice. A peer stalled in its own 10-30 second mission load stops reading, its window closes, and our nonblocking send starts answering would-block, which threw the message away; or worse, answering a PARTIAL count, and since framing on that stream is recovered purely from each message's length prefix, the bytes that never followed sheared it for good. Both are reachable in an ordinary race, because every race has a load in it. So sends go through a bounded per-connection queue now. What the wire will not take is kept, byte-exact, and retried at three flush points - before the render (the present blocks on vsync, and this frame's state should be travelling while it does), at the top of the receive pump, and before a connect sequence. Nothing is ever dropped from the middle: these are reliable ordered messages carrying entity creation, damage and race control, so a queue that overflows its 256K declares the connection dead and lets the disconnect path run rather than quietly desyncing the stream. RP412NETSENDQ=0 restores the old behaviour and still logs what it would have lost, which is the honest way to A/B it. On Steam the same queue finally surfaces k_EResultLimitExceeded, which the old code collapsed into -1 and discarded - that was backpressure, unlogged. The receive side gained the check the release build never had. The length prefix is untrusted input; Verify() compiles away in release, so a corrupt one went to memmove as a negative, or copied 4096 bytes of assembled packet into a 1600-byte stack buffer, or named a size the pad could never complete and wedged the connection forever. It is now validated against the same bounds the sender works to, and a stream that fails them is dropped like any other lost peer. And a fry that never ends: drop zones are map entities dealt round-robin at load, ownership transfer is not implemented, so a leaver's pads stay in the DropZones group. The respawn request dispatched to one goes to a host that is gone - dropped at the send, the 'no host N in the table' path - and the two-second retry re-dispatches to the same dead owner forever. The pad scan now skips zones whose owner has left, and re-validates one assigned earlier before reusing it. The rest is measurement, because the symptoms this work exists to chase are all reported in prose and none of them are in any log. Sixteen logs from the six-player night contain zero player-facing latency lines. A race now ends with a NetLog summary: per remote pod, how many updates arrived and how evenly (median and p95 out of a log2 histogram), the widest gap, how many gaps were long enough to mean a quiet sender versus short enough to mean OUR loop stalled, how often its motion snapped instead of blending, and how far arriving updates moved it. Per peer, whether the clock alignment ever had to step mid-race - which is the input for deciding if it needs slewing, rather than guessing. The mission t0 tick goes in the log too, alongside the console's per-pod RunMission send ticks, because nothing has ever measured how far apart the machines actually start; the clockwork doors inherit that skew directly. RP412NETSTATS adds the transport's own view - per connection: messages, bytes, wire writes, partials, refusals, how much sat queued - and on Steam the first read this codebase has ever taken of GetConnectionRealTime Status. Ping, quality, pending and unacked bytes, and one route description per connection at teardown. The API was vendored and never called; there was no RTT number anywhere in the game. Finally, rpl4opt -spoolstats reads any recording offline. The data was already in every spool ever made and nothing read it that way: the recorder restamps each packet with local arrival time while the update records inside keep the sender's sim-grid stamp, so the difference is clock offset plus one-way delay, and the same running-minimum estimator the game runs live separates them. It prints delay above the per-host minimum, and decomposes each entity's gaps into sender pacing versus delivery jitter - which no live counter can do. It lives in the game exe rather than RPL4TOOL because the tool is deliberately not /Zp1 and would misread every struct in the file. Verified on the two-pod loopback harness: mesh up, egg fed, 60s raced, stopped on command, scores collected, and both summaries reading exactly what a pair of PARKED pods should read - heartbeat cadence, one snap per heartbeat, sub-quarter-metre corrections, no clock steps. The t0 ticks and the netclock offsets agree with each other to the two seconds the pods launched apart. The latency tier is deliberately NOT here. TCP_NODELAY, the Steam NoNagle flag, per-frame coalescing and the pre-sim receive drain are all scoped and all wait on this build's numbers, because the point of shipping measurement first is to find out whether the thing we would fix is the thing that hurts. Nagle is still on. Interest management is still inert. The wire format is untouched, so this build and the last one still race each other. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
474 lines
12 KiB
C++
474 lines
12 KiB
C++
#pragma once
|
|
|
|
#include "state.h"
|
|
#include "receiver.h"
|
|
#include "time.h"
|
|
#include "resource.h"
|
|
#include "hostid.h"
|
|
|
|
//##########################################################################
|
|
//########################### Net clock ##############################
|
|
//##########################################################################
|
|
//
|
|
// Aligning a peer's clock with ours, so a replicant is dead-reckoned from
|
|
// when its update was SENT rather than when it happened to arrive.
|
|
//
|
|
// Every update record carries the sender's own timestamp. The receiver
|
|
// used to throw it away and stamp lastUpdate with its own Now() - the
|
|
// original code says so: "HACK - should be based upon message->timeStamp".
|
|
// The dead reckoner then extrapolates over (lastPerformance - lastUpdate),
|
|
// so starting that clock at ARRIVAL rather than at SEND leaves every
|
|
// remote vehicle exactly one network latency behind where it should be.
|
|
// On the 1 ms arcade LAN that was invisible. Over Steam Datagram Relay it
|
|
// is a constant 50-150 ms of positional lag - a bias, not jitter.
|
|
//
|
|
// The timestamp cannot be used raw: two machines' clocks share no epoch,
|
|
// both being QueryPerformanceCounter since their own boot. So we estimate
|
|
// the offset per peer. Each arriving record gives
|
|
//
|
|
// sample = ourNow - theirStamp = trueOffset + oneWayLatency
|
|
//
|
|
// and since latency is never negative, the SMALLEST sample seen is the
|
|
// closest to the true offset. Taking a minimum over a short rolling
|
|
// window tracks crystal drift and re-adapts when the route changes,
|
|
// instead of being pinned forever by one lucky packet.
|
|
//
|
|
// RP412NETCLOCK=0 turns the whole thing off and restores the arrival-time
|
|
// behaviour, so a test machine can A/B it without a rebuild.
|
|
//
|
|
void NetClock_BeginUpdate(HostID sender); // around one message's records
|
|
void NetClock_EndUpdate();
|
|
void NetClock_Reset(); // forget every peer (new mission)
|
|
void NetClock_ReportRaceStats(); // RP412NETLOG race-end peer lines
|
|
|
|
class Simulation__SharedData;
|
|
class Simulation__IndexData;
|
|
struct Simulation__IndexEntry;
|
|
class Simulation__AttributeIndexSet;
|
|
class MemoryStream;
|
|
|
|
//##########################################################################
|
|
//################# Simulation::UpdateRecord #########################
|
|
//##########################################################################
|
|
|
|
struct Simulation__UpdateRecord
|
|
{
|
|
public:
|
|
size_t recordLength;
|
|
Word subsystemID;
|
|
Word recordID;
|
|
Time timeStamp;
|
|
Enumeration simulationState;
|
|
};
|
|
|
|
//##########################################################################
|
|
//####################### Simulation #################################
|
|
//##########################################################################
|
|
|
|
class Simulation:
|
|
public Receiver
|
|
{
|
|
//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
// Shared Data Support
|
|
//
|
|
public:
|
|
typedef Simulation__SharedData SharedData;
|
|
SharedData*
|
|
GetSharedData();
|
|
|
|
static Derivation *GetClassDerivations();
|
|
|
|
static SharedData DefaultData;
|
|
|
|
//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
// Construction and Destruction Support
|
|
//
|
|
protected:
|
|
Simulation(
|
|
ClassID class_ID,
|
|
SharedData &shared_data
|
|
);
|
|
|
|
public:
|
|
~Simulation();
|
|
|
|
//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
// Attribute Support
|
|
//
|
|
public:
|
|
typedef Enumeration AttributeID;
|
|
typedef Simulation__IndexData IndexData;
|
|
typedef Simulation__IndexEntry IndexEntry;
|
|
typedef Simulation__AttributeIndexSet AttributeIndexSet;
|
|
typedef int Simulation::*AttributePointer;
|
|
|
|
enum {
|
|
AnyAttributeID = 0,
|
|
SimulationStateAttributeID,
|
|
NextAttributeID
|
|
};
|
|
|
|
static const AttributePointer NullAttribute;
|
|
|
|
void*
|
|
GetAttributePointer(AttributeID attribute);
|
|
void*
|
|
GetAttributePointer(const char* attribute_name);
|
|
|
|
private:
|
|
static const IndexEntry AttributePointers[];
|
|
|
|
protected:
|
|
//static AttributeIndexSet AttributeIndex
|
|
static AttributeIndexSet& GetAttributeIndex();
|
|
|
|
//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
// Simulation Support
|
|
//
|
|
public:
|
|
typedef void
|
|
(Simulation::*Performance)(Scalar time_slice);
|
|
typedef void
|
|
(Simulation::*Encore)();
|
|
typedef Simulation__UpdateRecord UpdateRecord;
|
|
|
|
void
|
|
SetPerformance(Performance performance)
|
|
{Check(this); activePerformance = performance;}
|
|
void
|
|
Perform(Scalar time_slice)
|
|
{
|
|
Check(this);
|
|
(this->*activePerformance)(time_slice);
|
|
}
|
|
|
|
virtual void
|
|
PerformAndWatch(
|
|
const Time& till,
|
|
MemoryStream *update_stream
|
|
);
|
|
|
|
//
|
|
// The two halves of PerformAndWatch, so an ENTITY can interleave its
|
|
// subsystems' physics with its own, step by step, and still run the
|
|
// watchers and the update stream once per frame. PerformTo advances
|
|
// the simulation to the given time - in fixed steps when
|
|
// RP412PHYSICSHZ names a rate, in one variable slice otherwise.
|
|
//
|
|
void
|
|
PerformTo(const Time& till);
|
|
void
|
|
WatchAndWrite(MemoryStream *update_stream);
|
|
|
|
//
|
|
// Called by the entity interleave at the TOP of every fixed step,
|
|
// before any subsystem adds its forces for that step. Per-frame set-up
|
|
// work - clearing a force accumulator, deriving local velocity from
|
|
// world state - belongs here when the fixed step is on, because "once
|
|
// per frame" is a wall-clock cadence and the whole point is that wall
|
|
// clock no longer reaches the physics. Default: nothing.
|
|
//
|
|
virtual void
|
|
BeginStep();
|
|
|
|
//
|
|
// Render interpolation hooks, called by PerformTo around the fixed
|
|
// step. They live HERE rather than on the entity interleave because a
|
|
// REPLICANT never runs that interleave - it reaches PerformTo through
|
|
// Simulation::PerformAndWatch instead - and a replicant is exactly what
|
|
// every remote pod is. Hanging the snapshot off BeginStep left the
|
|
// watched car uninterpolated while the camera watching it was smooth,
|
|
// which is most of the way to nowhere.
|
|
//
|
|
// Defaults do nothing; Entity overrides them because it owns the
|
|
// origin. See Entity::GetRenderToWorld.
|
|
//
|
|
virtual void
|
|
SnapshotRenderOrigin();
|
|
virtual void
|
|
SetRenderStepFraction(Scalar fraction);
|
|
|
|
//
|
|
// The fixed step in seconds, 0 when frame-coupled. Global on purpose:
|
|
// a mixed-rate simulation would be a worse bug than either mode.
|
|
//
|
|
static Scalar
|
|
FixedStep();
|
|
|
|
void
|
|
DoNothingOnce(Scalar time_slice);
|
|
void
|
|
DoNothing(Scalar time_slice);
|
|
|
|
void
|
|
SetLastPerformance(const Time& when)
|
|
{Check(this); Check(&when); lastPerformance = when;}
|
|
const Time&
|
|
GetLastPerformance() const
|
|
{Check(this); return lastPerformance;}
|
|
void
|
|
RequestEncore(Encore encore);
|
|
|
|
virtual void
|
|
ReadUpdateRecord(UpdateRecord *message);
|
|
virtual void
|
|
WriteUpdateRecord(
|
|
UpdateRecord *message,
|
|
int update_model
|
|
);
|
|
void
|
|
WriteSimulationUpdate(MemoryStream *update_stream);
|
|
|
|
enum {
|
|
DefaultUpdateModelBit=0,
|
|
NextUpdateModelBit
|
|
};
|
|
|
|
enum {
|
|
DefaultUpdateModelFlag = 1<<DefaultUpdateModelBit
|
|
};
|
|
|
|
void
|
|
ForceUpdate(Word model=DefaultUpdateModelFlag)
|
|
{Check(this); updateModel |= model;}
|
|
|
|
protected:
|
|
Time
|
|
lastPerformance;
|
|
Time
|
|
lastUpdate;
|
|
Word
|
|
updateModel;
|
|
|
|
Performance
|
|
activePerformance;
|
|
|
|
//##########################################################################
|
|
// Flag Support
|
|
//
|
|
public:
|
|
enum {
|
|
DelayWatchersBit,
|
|
DontExecuteBit,
|
|
NextBit
|
|
};
|
|
enum {
|
|
DelayWatchersFlag = 1<<DelayWatchersBit,
|
|
DontExecuteFlag = 1<<DontExecuteBit
|
|
};
|
|
|
|
LWord simulationFlags;
|
|
|
|
void
|
|
SetWatcherDelay()
|
|
{Check(this); simulationFlags |= DelayWatchersFlag;}
|
|
void
|
|
ClearWatcherDelay()
|
|
{Check(this); simulationFlags &= ~DelayWatchersFlag;}
|
|
Logical
|
|
AreWatchersDelayed()
|
|
{Check(this); return (simulationFlags & DelayWatchersFlag) != 0;}
|
|
|
|
void
|
|
NeverExecute()
|
|
{Check(this); simulationFlags |= DontExecuteFlag;}
|
|
void
|
|
ExecuteOnUpdate()
|
|
{Check(this); simulationFlags |= DontExecuteFlag;}
|
|
void
|
|
AlwaysExecute()
|
|
{Check(this); simulationFlags &= ~DontExecuteFlag;}
|
|
Logical
|
|
IsReplicantExecutable()
|
|
{
|
|
Check(this);
|
|
return
|
|
(simulationFlags&DontExecuteFlag) == 0
|
|
|| lastUpdate >= lastPerformance;
|
|
}
|
|
Logical
|
|
IsNonReplicantExecutable()
|
|
{Check(this); return (simulationFlags&DontExecuteFlag) == 0;}
|
|
|
|
//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
// State support
|
|
//
|
|
public:
|
|
enum {
|
|
DefaultState = 0,
|
|
StateCount
|
|
};
|
|
|
|
unsigned
|
|
GetSimulationState()
|
|
{Check(this); return simulationState.GetState();}
|
|
unsigned
|
|
GetOldSimulationState()
|
|
{Check(this); return simulationState.GetOldState();}
|
|
void
|
|
SetSimulationState(unsigned new_state)
|
|
{Check(this); simulationState.SetState(new_state);}
|
|
|
|
StateIndicator
|
|
simulationState;
|
|
|
|
//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
// Watcher Support
|
|
//
|
|
public:
|
|
void
|
|
AddAudioWatcher(Component *watcher)
|
|
{Check(&audioWatcherSocket);audioWatcherSocket.Add(watcher);}
|
|
void
|
|
AddVideoWatcher(Component *watcher)
|
|
{Check(&videoWatcherSocket);videoWatcherSocket.Add(watcher);}
|
|
void
|
|
AddGaugeWatcher(Component *watcher)
|
|
{Check(&gaugeWatcherSocket);gaugeWatcherSocket.Add(watcher);}
|
|
|
|
void
|
|
AddEffectWatcher(Component *watcher)
|
|
{Check(&effectWatcherSocket); effectWatcherSocket.Add(watcher);}
|
|
|
|
void
|
|
ExecuteWatchers();
|
|
|
|
private:
|
|
SChainOf<Component*>
|
|
audioWatcherSocket;
|
|
SChainOf<Component*>
|
|
videoWatcherSocket;
|
|
SChainOf<Component*>
|
|
gaugeWatcherSocket;
|
|
SChainOf<Component*>
|
|
effectWatcherSocket;
|
|
|
|
//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
// Test Support
|
|
//
|
|
public:
|
|
Logical
|
|
TestInstance() const;
|
|
static Logical
|
|
TestClass();
|
|
};
|
|
|
|
//##########################################################################
|
|
//################### Simulation::IndexEntry #########################
|
|
//##########################################################################
|
|
|
|
struct Simulation__IndexEntry
|
|
{
|
|
Enumeration
|
|
entryID;
|
|
const char *
|
|
entryName;
|
|
Simulation::AttributePointer
|
|
entryAddress;
|
|
};
|
|
|
|
#define ATTRIBUTE_ENTRY(class,name,attribute)\
|
|
{\
|
|
class::name##AttributeID,\
|
|
#name,\
|
|
(Simulation::AttributePointer) &class::attribute\
|
|
}
|
|
|
|
//##########################################################################
|
|
//################# Simulation::AttributeIndexSet ####################
|
|
//##########################################################################
|
|
|
|
class Simulation__AttributeIndexSet:
|
|
public Receiver::InheritanceSet
|
|
{
|
|
//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
// Construction and Destruction
|
|
//
|
|
public:
|
|
Simulation__AttributeIndexSet(
|
|
Simulation::AttributeID count,
|
|
const Simulation::IndexEntry index_table[],
|
|
const Simulation::AttributeIndexSet &inheritance
|
|
)
|
|
{Build(count, index_table, &inheritance);}
|
|
|
|
Simulation__AttributeIndexSet(
|
|
Simulation::AttributeID count,
|
|
const Simulation::IndexEntry index_table[]
|
|
)
|
|
{Build(count, index_table, NULL);}
|
|
|
|
Simulation__AttributeIndexSet()
|
|
{attributeIndex = NULL; entryCount = 0;}
|
|
|
|
~Simulation__AttributeIndexSet();
|
|
|
|
protected:
|
|
void
|
|
Build(
|
|
Simulation::AttributeID count,
|
|
const Simulation::IndexEntry index_table[],
|
|
const Simulation::AttributeIndexSet *inheritance
|
|
);
|
|
|
|
//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
// AttributeIndexSet Functionality
|
|
//
|
|
protected:
|
|
Simulation::IndexEntry
|
|
*attributeIndex;
|
|
|
|
public:
|
|
Simulation::AttributePointer
|
|
Find(Simulation::AttributeID attribute) const
|
|
{
|
|
Check(this);
|
|
Verify(attribute > 0);
|
|
if (attribute<=entryCount)
|
|
return attributeIndex[attribute-1].entryAddress;
|
|
else
|
|
return Simulation::NullAttribute;
|
|
}
|
|
Simulation::AttributePointer
|
|
Find(const char* attribute_name) const;
|
|
const Simulation::IndexEntry*
|
|
FindEntry(const char* attribute_name) const;
|
|
|
|
static const Simulation::AttributeIndexSet
|
|
NullSet;
|
|
|
|
//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
// Test Support
|
|
//
|
|
public:
|
|
static Logical
|
|
TestClass();
|
|
};
|
|
|
|
//##########################################################################
|
|
//################### Simulation::SharedData #########################
|
|
//##########################################################################
|
|
|
|
class Simulation__SharedData:
|
|
public Receiver::SharedData
|
|
{
|
|
public:
|
|
Simulation__SharedData(
|
|
Derivation* derivation,
|
|
Receiver::MessageHandlerSet &message_handlers,
|
|
Simulation::AttributeIndexSet &attribute_index,
|
|
int state_count
|
|
):
|
|
Receiver::SharedData(derivation, message_handlers),
|
|
activeAttributeIndex(&attribute_index),
|
|
stateCount(state_count)
|
|
{}
|
|
|
|
Simulation::AttributeIndexSet* activeAttributeIndex;
|
|
int stateCount;
|
|
};
|
|
|
|
inline Simulation::SharedData*
|
|
Simulation::GetSharedData()
|
|
{return Cast_Object(SharedData*,sharedData);}
|