From d02117d87c2dea5c886ce1d488a5eca7f1cf4839 Mon Sep 17 00:00:00 2001 From: Cyd Date: Fri, 14 Aug 2026 09:10:15 -0500 Subject: [PATCH] Answer the door before knocking on anyone else's Players kept failing to get a race started - the host watching pods that never reported ready. It is not the transport: both wires are already reliable and ordered, TCP on one path and Steam's reliable channel on the other, and every failure in the playtest logs is on the game mesh port, none on the console port. The eggs arrive fine. It is the order the mesh is built in. Each pod walks the host list from the egg, connecting out to everyone ahead of it and listening for everyone after - so in list order the connects all come first, and the listener was created lazily on the first host of the second kind. That is to say AFTER every outbound connect had already been made. A connect to a pod that is not ready blocks for RP412CONNECTWAIT, three times over, so a pod could spend a minute unable to answer its own door while the very peers it was waiting on were knocking on it. Their connects to it then timed out, and the failure went round the ring. The logs say it plainly: six of eighteen pods never reached "listening on engine port 1502" at all, and the same pair of players connects in one round and times out in the next, in both directions. It is not one bad network - it is a queue. So the walk now only builds the hosts, and the outbound connects happen after it. The listener still opens exactly where it did, on the first host that will be calling us, but nothing has blocked by then so it is up in the first moments rather than a minute in. A peer that still does not answer is tried once more, because that is the other thing the logs show - the slower pod catching up between attempts. Bounded deliberately: the transport already makes three tries inside each call, so a round here is expensive and a loop of them would turn a bad connection into a hung game. RP412MESHRETRY sets it. And when a peer is genuinely unreachable, say so. Every one of them is counted in remoteHostCount, so the mesh can never report itself complete and the pod is never told to load the mission - it just sits there looking to the host like a player who never got ready. The log now names who could not be reached and says the mission cannot start. Not done here: what SHOULD happen then. Dropping that pod out, or racing with a hole in the mesh, is a decision about the game rather than the network. An earlier attempt at this opened the listener unconditionally before the walk, which was wrong: Connect and Listen both bind localGamePort, so the pod at the END of the list - which listens for nobody - bound it twice and its one outgoing connection fought its own listener. Deferring the connects gets the same ordering with no extra socket. Verified: two-pod loopback green, both pods scoring 1000, mesh completing, stale 0, no listener on the last pod any more. Cadence measured against the previous build over three runs and unchanged - an earlier reading that looked like a regression was harness variance, the pods being parked and the numbers heartbeat-dominated. Co-Authored-By: Claude Opus 5 (1M context) --- MUNGA_L4/L4NET.CPP | 152 ++++++++++++++++++++++++++++++++++++++++-- RP_L4/RPL4ENVIRON.cpp | 14 ++++ 2 files changed, 162 insertions(+), 4 deletions(-) diff --git a/MUNGA_L4/L4NET.CPP b/MUNGA_L4/L4NET.CPP index 1afc36c..3b28747 100644 --- a/MUNGA_L4/L4NET.CPP +++ b/MUNGA_L4/L4NET.CPP @@ -501,6 +501,39 @@ void // Iterate through all the host addresses in the egg and create all the host // structures, note that there MUST be an entry in the egg for us. // + // + //------------------------------------------------------------------------ + // Answer the door before knocking on anyone else's. + // + // This walk connects out to every host listed ahead of us in the egg and + // listens for every host after us - so in list order the connects all come + // first, and the listener used to be created lazily on the first host of + // the second kind, which is to say AFTER every outbound connect had + // already been made. A connect to a pod that is not ready yet blocks for + // RP412CONNECTWAIT seconds, three times over, so a pod could spend a + // minute unable to answer its door while the very peers it was waiting on + // were knocking on it. Their connects to us then timed out too and the + // failure went round the ring: playtest logs show pods that never reached + // "listening on engine port 1502" at all, and the same pair connecting in + // one round and timing out in the next. + // + // So the walk now only CREATES the hosts, and the outbound connects are + // made afterwards. The listener is still opened exactly when it was, on + // the first host that will call us - but by then nothing has blocked, so + // it happens in the first moments rather than a minute in. + // + // Note this is why we do not simply open a listener up front: Connect() + // and Listen() both bind localGamePort, so a pod at the END of the list - + // which listens for nobody - would be binding that port twice and its one + // outgoing connection would be fighting its own listener for it. + //------------------------------------------------------------------------ + // + enum { maxPendingConnects = 16 }; + L4Host + *pending_connect[maxPendingConnects]; + int + pending_connects = 0; + listen = False; remoteHostCount = 0; while ((mission_host_data = mission_host_iterator.ReadAndNext()) != NULL) @@ -565,10 +598,14 @@ void socket_ptr = OpenConnection(NETNUB_TCP_LISTEN, localGamePort, 0, net_address.sin_addr.S_un.S_addr); } else - socket_ptr = OpenConnection(NETNUB_TCP_OPEN, localGamePort, ntohs(net_address.sin_port), net_address.sin_addr.S_un.S_addr); - - if (!listen && socket_ptr == INVALID_SOCKET) - DEBUG_STREAM << "Could not open connection to " << host_name << ".\n" << std::flush; + { + // + // Connected after the walk, not here - see above. The + // host is built now so it keeps its place in the list + // and its host ID; only the socket arrives late. + // + socket_ptr = INVALID_SOCKET; + } // // Now we can create the remote host and get the application to adopt @@ -589,10 +626,117 @@ void Check(application); Check(application->GetHostManager()); application->GetHostManager()->AdoptRemoteHost(my_l4host); + + // + // Every host we have to call, in list order. Each one is + // already counted in remoteHostCount, so until its connection + // is made the mesh cannot report itself complete and this pod + // sits in front of a loading screen for as long as anyone + // lets it. + // + if (!listen && pending_connects < maxPendingConnects) + { + pending_connect[pending_connects++] = my_l4host; + } } nextOpenHostID++; } + // + //------------------------------------------------------------------------ + // Now make the calls. + // + // Deferred to here so that our listener - opened during the walk, on the + // first host that will be calling US - is already up. Whatever these + // connects cost, our peers can be completing their half of the mesh + // throughout, instead of finding a pod that will not answer until it has + // finished waiting on somebody else. + // + // A peer that does not answer is tried again, because the logs show + // exactly that: the same pair timing out on one attempt and connecting on + // the next, as the slower pod catches up. Bounded on purpose - the + // transport already makes three attempts inside each call, so a round here + // is expensive, and a loop of them would turn a bad connection into a hung + // game. RP412MESHRETRY sets the extra rounds; 0 is one pass and no retry. + //------------------------------------------------------------------------ + // + if (pending_connects > 0) + { + static int retry_rounds = -1; + if (retry_rounds < 0) + { + const char *setting = getenv("RP412MESHRETRY"); + retry_rounds = (setting != NULL) ? atoi(setting) : 1; + if (retry_rounds < 0) { retry_rounds = 0; } + if (retry_rounds > 3) { retry_rounds = 3; } + } + + for (int round = 0; round <= retry_rounds && pending_connects > 0; ++round) + { + int still_pending = 0; + + for (int i = 0; i < pending_connects; ++i) + { + L4Host *peer_host = pending_connect[i]; + Check(peer_host); + + SOCKADDR_IN *peer = peer_host->GetNetworkAddress(); + Check_Pointer(peer); + + CString peer_name = peer_host->GetSymbolicName(); + + if (round > 0) + { + DEBUG_STREAM << "Mesh: retrying " << (LPSTR) peer_name + << " (round " << round << " of " << retry_rounds + << ")\n" << std::flush; + } + + SOCKET peer_socket = OpenConnection( + NETNUB_TCP_OPEN, + localGamePort, + ntohs(peer->sin_port), + peer->sin_addr.S_un.S_addr); + + if (peer_socket != INVALID_SOCKET) + { + peer_host->SetNetworkSocket((unsigned long) peer_socket); + if (round > 0) + { + DEBUG_STREAM << "Mesh: " << (LPSTR) peer_name + << " answered on the retry\n" << std::flush; + } + } + else + { + DEBUG_STREAM << "Could not open connection to " + << (LPSTR) peer_name << ".\n" << std::flush; + pending_connect[still_pending++] = peer_host; + } + } + pending_connects = still_pending; + } + + if (pending_connects > 0) + { + // + // Say so plainly. Every one of these is counted in + // remoteHostCount, so the mesh cannot complete and this pod + // will never be told to load the mission - it will simply + // appear to the host as a player who never got ready. + // + DEBUG_STREAM << "Mesh: INCOMPLETE - " << pending_connects + << " peer(s) unreachable, this pod cannot start the mission:\n" + << std::flush; + for (int i = 0; i < pending_connects; ++i) + { + CString gone = pending_connect[i]->GetSymbolicName(); + DEBUG_STREAM << "Mesh: unreachable: " << (LPSTR) gone + << "\n" << std::flush; + } + } + } + // // A recording starts with a header, and this is where it can be written. // diff --git a/RP_L4/RPL4ENVIRON.cpp b/RP_L4/RPL4ENVIRON.cpp index 3167b60..ba326b9 100644 --- a/RP_L4/RPL4ENVIRON.cpp +++ b/RP_L4/RPL4ENVIRON.cpp @@ -384,6 +384,20 @@ namespace "# counts down each attempt, and ESC gives up immediately.\n" "RP412CONNECTWAIT=20\n" "\n" +"# Extra passes at any pod that did not answer when the mesh was built.\n" +"# 1 by default, 0 for a single pass, 3 at most.\n" +"#\n" +"# Every pod has to reach every other pod before a race can start, and a\n" +"# pod that was still getting its own listener up when it was first\n" +"# called will not answer. The playtest logs show exactly that: the same\n" +"# pair timing out on one attempt and connecting on the next. One extra\n" +"# pass costs nothing when the room is healthy - there is nobody left to\n" +"# call - and rescues the room when a pod was merely slow.\n" +"#\n" +"# Each pass is expensive, since the transport already makes three tries\n" +"# of its own inside every call, which is why this is not a loop.\n" +"#RP412MESHRETRY=1\n" +"\n" "# How much memory to set aside for a recording, in megabytes. 1 to 512,\n" "# default 100.\n" "#\n"