D1 phase 5+6: UDP unreliable channel (restores the 1995 reliable/unreliable split)

The ~60Hz entity update records now ride UDP in relay mode, so a lost/late datagram is dropped (dead reckoning absorbs the gap) instead of head-of-line- blocking the reliable TCP stream -- the fix for internet rubber-banding. Reliable traffic (make/damage/death/control/egg/launch) stays on TCP. This RESTORES the authentic 1995 NETNUB split, it doesn't invent one: Receiver::Message defaults messageFlags=ReliableFlag; the update path clears it (flags=0, ENTITY.cpp:590); Mode(UnreliableMode) fires at LoadingMission (APP.cpp:704 -- the 2007 port's Mode() was a no-op, now it STORES the mode). Routing gate = (mode==UnreliableMode && !(flags & ReliableFlag)) -- flag AND mode, so the map-stream creation messages (also flags=0 but flowing during the still-Reliable CreatingMission window) ride TCP for free. Client (L4NET): - Mode() stores currentNetworkMode (was ignored). - ConnectRelayUdp: UDP socket connect()ed to the relay game port; HELLO (outbound punch-through, relay learns our endpoint); udpUp on HELLO-ACK. BT_RELAY_TCP_ONLY=1 disables the channel (UDP-blocked nets degrade to the pure-TCP checkpoint automatically -- no ACK => everything stays on TCP). - RelayUdpSendFrame: {route, fromHost, seq} envelope + frame, best-effort. - RelayUdpKeepalive: ~15s NAT-hold + ~1s HELLO retry until acked. - CheckRelayUdp: HELLO-ACK sets udpUp; per-sender seq gate drops stale/ out-of-order datagrams; frames handed up envelope-stripped like TCP. - Send / ExclusiveBroadcast: route unreliable-window traffic to UDP, else TCP. - CheckBuffers polls the UDP channel first, then TCP; keepalive tick. (The relay-side UDP forwarder + endpoint learning + --udp-drop hook landed in phase 1.) Verified 2-node localhost relay: during a drive the update stream flows on UDP (relay udp tx 2->190+, TCP frozen at 53) and the peer's replicant tracks the master to sub-2u; 15% forced drop (--udp-drop 15) stays coherent (tracked to 0.1u); TCP-only fallback confirmed (udp-known drops the TCP-only pod); no crashes; mesh mode un-regressed. KB: context/multiplayer.md gains the D1 RELAY MODE section. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-18 10:00:26 -05:00
parent c029df6e6b
commit dbb9af2dfa
3 changed files with 293 additions and 5 deletions
@@ -261,6 +261,53 @@ Base fns in every subsystem vtable: slot 6 = 41bd34 (Read/apply), slot 7 = 41c50
  the same read; the twist clamp contains it (on the fixed-torso BLH it pins to ±0.01° —
  invisible). First real twist record overwrites it.

+## D1 RELAY MODE — internet play without rearchitecting (2026-07-18) [T2 live]
+Opt-in relay/dispatcher topology so pods play over the internet behind NAT with no port
+forwarding, WHILE the authentic distributed-authority simulation is UNTOUCHED (each pod still
+masters its own mech; peers stay replicants). Env-gated: **`BT_RELAY=<host>:<consolePort>`**
+(host may be a DNS name; game port = +1; UDP = same number) + **`BT_SELF=<exact [pilots]
+entry>`** (NIC matching can't identify a pod across NAT). BT_RELAY unset ⇒ every relay branch
+is dead ⇒ the classic mesh is byte-for-byte unchanged (mesh smoke-tested un-regressed each
+phase). Plan: `~/.claude/plans/partitioned-snuggling-piglet.md`.
+- **Topology**: every pod makes ONE outbound TCP connection to the relay (an extension of
+  `tools/btconsole.py --relay`) carrying both the console protocol (egg/launch — direction
+  flipped, the pod dials OUT; the relay TERMINATES the legacy protocol) and all reliable game
+  traffic, multiplexed by an 8-byte envelope `{int32 route; uint32 length}` (route ≥2 =
+  unicast hostID, −1 = broadcast-except-sender, −2 HELLO/−3 PEER_UP/−4 PEER_DOWN). Plus one UDP
+  socket for the unreliable channel.
+- **The relay is DUMB**: routes envelopes by hostID (validated against the egg `[pilots]`
+  roster), holds no game state. PEER_UP/DOWN control frames on the client synthesize the SAME
+  `HostConnected/DisconnectedMessage` the mesh accept path routes, so the "All connections
+  completed!" gate + the whole app ladder run UNMODIFIED. Peers become VIRTUAL L4Hosts
+  (INVALID_SOCKET, flipped online by PEER_UP).
+- **O(N) upload fix**: `ExclusiveBroadcast` builds ONCE and sends ONCE with the broadcast route
+  — the relay fans out server-side (kills the mesh's N−1× duplication at the seam where
+  `EntityBroadcastToReplicants` already serialized once). Verified: relay `tcp rx==tx`, one
+  copy per broadcast.
+- **UDP channel = the 1995 NETNUB reliable/unreliable split RESTORED (not invented)**:
+  `Receiver::Message` defaults `messageFlags=ReliableFlag` (RECEIVER.h:195); the ~60Hz update
+  path sets flags=0 (ENTITY.cpp:590); `Mode(UnreliableMode)` at LoadingMission (APP.cpp:704,
+  the 2007 port's `Mode()` was a no-op — now STORES the mode). The routing gate is
+  **`(mode==UnreliableMode && !(flags & ReliableFlag))`** — flag AND mode, so the map-stream
+  creation messages (also flags=0 but flowing during the still-Reliable CreatingMission window)
+  ride TCP for free. UDP envelope `{route, fromHost, seq}` + frame; relay learns each pod's
+  endpoint per-datagram (NAT-rebind tolerant), forwards verbatim, TCP-wraps to a peer whose UDP
+  endpoint is unknown (asymmetric-block fallback); per-sender seq gate drops stale/out-of-order
+  (dead reckoning absorbs the gaps — that's what it's for). `BT_RELAY_TCP_ONLY=1` = escape
+  hatch (UDP-blocked networks degrade to the pure-TCP checkpoint automatically).
+- **VERIFIED 2-node localhost relay**: full ladder to RunningMission via PEER_UP; bidirectional
+  replication (both mechs' MakeMessages cross the relay, paint ×2 each); during a drive the
+  update stream flows on UDP (relay `udp tx` climbs 2→190+, TCP frozen at 53) and the peer's
+  replicant tracks the master to sub-2u; 15% forced UDP drop (`--udp-drop 15`) stays coherent
+  (replicant tracked to 0.1u); TCP-only fallback confirmed (`udp-known` drops the TCP-only pod);
+  no crashes; mesh mode un-regressed. Recipe: relay `python tools/btconsole.py --relay 1500
+  MP_RELAY.EGG --bind 127.0.0.1`; pods `BT_RELAY=127.0.0.1:1500 BT_SELF=10.99.0.N:1502 btl4
+  -egg MP_RELAY.EGG -net 1501` (egg `[pilots]` = arbitrary unique tags). Diag: relay
+  `[relay-stats]` per-transport census; client `BT_NET_TRACE`.
+- **STILL UNTESTED (config, not code)**: a real two-physical-machine / internet run (only the
+  relay's console+game ports need to be reachable inbound — one forwarded port total).
+  Optional future: LAN `BT_RELAY=auto` UDP-broadcast discovery.
+
 ## Remaining (P6 phase 4 / Phase 7)
 Pod-LAN config: **real-IP path VALIDATED (2026-07-15)** — an egg with the machine's real LAN IP in
 `[pilots]` (not `127.0.0.1`) connected + replicated + moved end-to-end (`Connected to
@@ -226,7 +226,9 @@ L4NetworkManager::L4NetworkManager():
 	relayPadTail			= 0;
 	relayConsoleDialedOnce	= False;
 	udpUp					= False;
+	relayLocalHostID		= 0;
 	lastUdpKeepaliveTick	= 0;
+	lastUdpHelloTick		= 0;
 	udpTxSeq				= 0;
 	memset(udpRxSeq, 0, sizeof(udpRxSeq));
 	currentNetworkMode		= NetworkManager::ReliableMode;
@@ -1398,8 +1400,202 @@ void L4NetworkManager::ConnectRelayGame(HostID local_host_ID)
 	{
 		Fail("relay HELLO transmit failed\n");
 	}
+	relayLocalHostID = local_host_ID;
 	DEBUG_STREAM << "[relay] game connection up, HELLO sent (hostID "
 		<< (int)local_host_ID << ")\n" << std::flush;
+
+	// Bring up the UDP unreliable channel (a fully UDP-blocked network just
+	// leaves udpUp False and everything rides TCP -- the fallback is automatic).
+	ConnectRelayUdp(local_host_ID);
+}
+
+//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+// ConnectRelayUdp -- create the UDP socket, connect() it to the relay game
+// port (so send/recv need no address), and send the first HELLO (the relay
+// learns our public endpoint from it -- outbound NAT punch-through).  udpUp
+// flips True on the relay's HELLO-ACK (CheckRelayUdp).  BT_RELAY_TCP_ONLY=1
+// disables the whole channel.
+//
+void L4NetworkManager::ConnectRelayUdp(HostID local_host_ID)
+{
+	const char *tcp_only = getenv("BT_RELAY_TCP_ONLY");
+	if (tcp_only != NULL && tcp_only[0] == '1')
+	{
+		DEBUG_STREAM << "[relay] BT_RELAY_TCP_ONLY: UDP channel disabled\n"
+			<< std::flush;
+		return;
+	}
+	relayUdpSocket = socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP);
+	if (relayUdpSocket == INVALID_SOCKET)
+	{
+		DEBUG_STREAM << "[relay] UDP socket() failed (WSA "
+			<< WSAGetLastError() << ") -- staying on TCP\n" << std::flush;
+		return;
+	}
+	if (connect(relayUdpSocket, (const sockaddr *)&relayGameAddress,
+			sizeof(relayGameAddress)) != 0)
+	{
+		DEBUG_STREAM << "[relay] UDP connect() failed (WSA "
+			<< WSAGetLastError() << ") -- staying on TCP\n" << std::flush;
+		closesocket(relayUdpSocket);
+		relayUdpSocket = INVALID_SOCKET;
+		return;
+	}
+	unsigned long enable = 1;
+	ioctlsocket(relayUdpSocket, FIONBIO, &enable);
+	// First HELLO (retried by RelayUdpKeepalive until acked).
+	RelayUdpEnvelope hello;
+	hello.route    = RELAY_ROUTE_HELLO;
+	hello.fromHost = (int)local_host_ID;
+	hello.sequence = 0;
+	send(relayUdpSocket, (const char *)&hello, sizeof(hello), 0);
+	lastUdpHelloTick = GetTickCount();
+	DEBUG_STREAM << "[relay] UDP HELLO sent (hostID " << (int)local_host_ID
+		<< "); awaiting ACK\n" << std::flush;
+}
+
+//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+// RelayUdpSendFrame -- one datagram = one enveloped frame on the unreliable
+// channel.  Best-effort: a failed sendto is simply dropped (the point of UDP).
+//
+Logical L4NetworkManager::RelayUdpSendFrame(
+	int route,
+	const NetworkPacket *packet,
+	int packet_size)
+{
+	if (relayUdpSocket == INVALID_SOCKET || !udpUp)
+	{
+		return False;
+	}
+	char wire[sizeof(RelayUdpEnvelope) + NETWORKMANAGER_BUFFER_SIZE];
+	if (packet_size > NETWORKMANAGER_BUFFER_SIZE)
+	{
+		return False;
+	}
+	RelayUdpEnvelope *envelope = (RelayUdpEnvelope *)wire;
+	envelope->route    = route;
+	envelope->fromHost = (int)relayLocalHostID;
+	envelope->sequence = ++udpTxSeq;
+	memcpy(wire + sizeof(RelayUdpEnvelope), packet, packet_size);
+	send(relayUdpSocket, wire, (int)sizeof(RelayUdpEnvelope) + packet_size, 0);
+	return True;
+}
+
+//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+// RelayUdpKeepalive -- hold the NAT binding (~15s) and retry HELLO (~1s) until
+// acked.  Called from CheckBuffers.  During play the 60Hz stream IS the
+// keepalive; this covers idle/lobby time and the initial handshake.
+//
+void L4NetworkManager::RelayUdpKeepalive(HostID local_host_ID)
+{
+	if (relayUdpSocket == INVALID_SOCKET)
+	{
+		return;
+	}
+	unsigned long now = GetTickCount();
+	if (!udpUp)
+	{
+		if (now - lastUdpHelloTick >= 1000)
+		{
+			RelayUdpEnvelope hello;
+			hello.route    = RELAY_ROUTE_HELLO;
+			hello.fromHost = (int)local_host_ID;
+			hello.sequence = 0;
+			send(relayUdpSocket, (const char *)&hello, sizeof(hello), 0);
+			lastUdpHelloTick = now;
+		}
+		return;
+	}
+	if (now - lastUdpKeepaliveTick >= 15000)
+	{
+		RelayUdpEnvelope keepalive;
+		keepalive.route    = RELAY_ROUTE_HELLO;		// HELLO doubles as keepalive
+		keepalive.fromHost = (int)local_host_ID;
+		keepalive.sequence = 0;
+		send(relayUdpSocket, (const char *)&keepalive, sizeof(keepalive), 0);
+		lastUdpKeepaliveTick = now;
+	}
+}
+
+//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+// RelayShouldUseUdp -- the AUTHENTIC 1995 reliable/unreliable gate.  Route to
+// UDP only in the unreliable window AND for a message the engine itself
+// flagged unreliable (messageFlags&ReliableFlag==0).  This protects the
+// map-stream creation messages (which also lack ReliableFlag but flow during
+// the still-Reliable CreatingMission window) for free.
+//
+Logical L4NetworkManager::RelayShouldUseUdp(const Message *message) const
+{
+	return (udpUp
+		&& currentNetworkMode == NetworkManager::UnreliableMode
+		&& (message->messageFlags & Message::ReliableFlag) == 0) ? True : False;
+}
+
+//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+// CheckRelayUdp -- poll the UDP socket.  HELLO-ACK sets udpUp; game datagrams
+// pass the per-sender sequence gate (drop stale/out-of-order -- dead reckoning
+// tolerates gaps), then hand the frame up envelope-stripped exactly like the
+// TCP path.  Returns True with one game frame.
+//
+Logical L4NetworkManager::CheckRelayUdp(NetworkPacket *network_packet)
+{
+	if (relayUdpSocket == INVALID_SOCKET)
+	{
+		return False;
+	}
+	for (;;)
+	{
+		char datagram[sizeof(RelayUdpEnvelope) + NETWORKMANAGER_BUFFER_SIZE];
+		int received = recv(relayUdpSocket, datagram, sizeof(datagram), 0);
+		if (received == SOCKET_ERROR)
+		{
+			return False;				// WSAEWOULDBLOCK etc: no more datagrams
+		}
+		if (received < (int)sizeof(RelayUdpEnvelope))
+		{
+			continue;
+		}
+		RelayUdpEnvelope *envelope = (RelayUdpEnvelope *)datagram;
+		if (envelope->route == RELAY_ROUTE_UDP_ACK)
+		{
+			if (!udpUp)
+			{
+				udpUp = True;
+				lastUdpKeepaliveTick = GetTickCount();
+				DEBUG_STREAM << "[relay] UDP channel UP (unreliable records now "
+					"ride UDP)\n" << std::flush;
+			}
+			continue;
+		}
+		int frame_length = received - (int)sizeof(RelayUdpEnvelope);
+		if (frame_length < (int)sizeof(NetworkPacketHeader))
+		{
+			continue;
+		}
+		// Per-sender sequence gate: drop stale/duplicate/out-of-order.
+		int from_host = envelope->fromHost;
+		if (from_host >= 0 && from_host < 64)
+		{
+			if (envelope->sequence != 0
+				&& envelope->sequence <= udpRxSeq[from_host])
+			{
+				continue;				// stale -- newer state already applied
+			}
+			udpRxSeq[from_host] = envelope->sequence;
+		}
+		Mem_Copy(network_packet, datagram + sizeof(RelayUdpEnvelope),
+			frame_length, NETWORKMANAGER_BUFFER_SIZE);
+		if (getenv("BT_NET_TRACE"))
+		{
+			DEBUG_STREAM << "[net-rx] relay-udp client="
+				<< (int)network_packet->clientID
+				<< " msgID=" << (int)network_packet->messageData.messageID
+				<< " len=" << (int)network_packet->messageData.messageLength
+				<< " from host " << (int)network_packet->fromHost
+				<< " seq=" << envelope->sequence << "\n" << std::flush;
+		}
+		return True;
+	}
 }

 //~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -1664,7 +1860,14 @@ void L4NetworkManager::Send(
 		int relay_packet_size;
 		NetworkPacket *relay_packet =
 			BuildRelayPacket(message, client, &relay_packet_size);
-		RelaySendFrame((int)host_ID, relay_packet, relay_packet_size);
+		if (RelayShouldUseUdp(message))
+		{
+			RelayUdpSendFrame((int)host_ID, relay_packet, relay_packet_size);
+		}
+		else
+		{
+			RelaySendFrame((int)host_ID, relay_packet, relay_packet_size);
+		}
 		free(relay_packet);
 		CLEAR_SEND_PACKET();
 		return;
@@ -2092,7 +2295,18 @@ void
 			int relay_packet_size;
 			NetworkPacket *relay_packet =
 				BuildRelayPacket(message, client, &relay_packet_size);
-			RelaySendFrame(RELAY_ROUTE_BROADCAST, relay_packet, relay_packet_size);
+			if (RelayShouldUseUdp(message))
+			{
+				// The 60Hz update-record fan-out -- the whole point of the UDP
+				// channel (unreliable, latest-wins, no head-of-line blocking).
+				RelayUdpSendFrame(RELAY_ROUTE_BROADCAST, relay_packet,
+					relay_packet_size);
+			}
+			else
+			{
+				RelaySendFrame(RELAY_ROUTE_BROADCAST, relay_packet,
+					relay_packet_size);
+			}
 			free(relay_packet);
 		}
 		return;
@@ -2618,10 +2832,19 @@ Logical L4NetworkManager::CheckBuffers(NetworkPacket *network_packet)
 	// The classic per-host loop below still services the CONSOLE host; game
 	// hosts are virtual (skipped via the relay gate in the OnLine recv block).
 	//
-	if (relayMode && CheckRelay(network_packet))
+	if (relayMode)
 	{
-		CLEAR_CHECK_BUFFERS();
-		return True;
+		RelayUdpKeepalive(relayLocalHostID);
+		if (CheckRelayUdp(network_packet))		// unreliable channel first
+		{
+			CLEAR_CHECK_BUFFERS();
+			return True;
+		}
+		if (CheckRelay(network_packet))			// reliable + control
+		{
+			CLEAR_CHECK_BUFFERS();
+			return True;
+		}
 	}

 	//
@@ -350,6 +350,22 @@ private:
 		RelayGameDown(const char *why);
 	Logical
 		CheckRelay(NetworkPacket *network_packet);
+	//
+	// UDP unreliable channel (restores the 1995 NETNUB reliable/unreliable
+	// split): the ~60Hz update records ride UDP so a lost/late datagram is
+	// dropped (dead reckoning absorbs the gap) instead of head-of-line-
+	// blocking the reliable stream.  Reliable traffic + control stay on TCP.
+	//
+	void
+		ConnectRelayUdp(HostID local_host_ID);
+	Logical
+		RelayUdpSendFrame(int route, const NetworkPacket *packet, int packet_size);
+	void
+		RelayUdpKeepalive(HostID local_host_ID);
+	Logical
+		RelayShouldUseUdp(const Message *message) const;
+	Logical
+		CheckRelayUdp(NetworkPacket *network_packet);

 //~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 // Private Data
@@ -390,7 +406,9 @@ private:
 	int relayPadTail;
 	Logical relayConsoleDialedOnce;		// first dial gets the long timeout
 	Logical udpUp;						// UDP HELLO acked by the relay
+	HostID relayLocalHostID;			// our hostID (for UDP HELLO/keepalive/tx)
 	unsigned long lastUdpKeepaliveTick;
+	unsigned long lastUdpHelloTick;		// HELLO retry until acked
 	unsigned long udpTxSeq;
 	unsigned long udpRxSeq[64];			// per-fromHost last-seen sequence
 	NetworkMode currentNetworkMode;		// Reliable/Unreliable (Mode() stores)