Radical Heights’ Technical Director Explains Reasons Why Players Are Getting Stuck Behind Loading Screen

Lots of games are now trying to follow the trend of the Battle Royale genre, and Radical Heights is one such game that entered Early Access last week. As with all games during such an early phase, the game is faced with its share of technical problems. The most glaring issue however is one where players have found themselves stuck in an Infinite Loading Screen of Death.

Boss Key’s technical director Romain Dura has now tried explaining to the community exactly what the problem is. Initially, the problem was identified to be caused by a matchmaking error. However, it seems that this problem is quite a persistent one as it has continued plaguing the game even after the developers tried to fix the issue. You can read Dura’s full explanation of the problems below.

“Matchmaking

The matchmaking server is in charge of grouping players together and telling the game client which game server to connect to start a match. After leaving a match, players could get stuck when they queued again. The first issue discovered was that the game client was sometimes sending a message to matchmaking indicating a network disconnection instead of an event to deliberately leave a match from the Pause menu. Matchmaking was configured to preserve player slots in games, in case they crashed or got briefly disconnected and tried to rejoin. Upon finding a match again, the matchmaking server responded to clients to connect to the same game server that they were just connected to, as if they were trying to rejoin after a disconnect. The game server was then kicking players because they had already died in that match, and the game became stuck on the loading screen instead of returning to the main menu.

To avoid this behavior without having to patch the game with around 8000 concurrent live players on the second day, we attempted to disable the functionality to preserve disconnected player slots in matches, by toggling a live configuration switch. Unfortunately, this configuration hadn’t been widely tested under load, and caused all ending matches to never reset properly, leading to a starvation of the game servers, and thousands of players stuck in matchmaking queue for about 30 minutes (starting at A on the chart).

We reverted the switch change, but that did not release all stuck game servers. At this point we had to fully reset the matchmaking server and all game servers. This led to the loss of the Play button in the menu for a few minutes, and things returned to normal after that (B on the chart).

The next attempt was to force matchmaking to treat all match disconnection events as deliberate, via a code change. We deployed the matchmaking server update, which was successful in avoiding the bug where players were sent again to their previous match server.

Matchmaking, again

Unfortunately, there were still reports from players being stuck on the loading screen. Deep sigh.

Despite the previous fix, there was still an issue where players wouldn’t be removed from their match on the matchmaking server if they appeared to still be connected to their game server. This too, could make the matchmaking server force clients to reconnect to their previous server. The source of the problem was a race condition where the network message indicating that the player had disconnected from the game server came after the event to deliberately leave the match, instead of before. This happened because of a misconfiguration of a built-in delay to handle this specific case. We corrected the live delay configuration, and this problem was resolved.

Game servers

UNFORTUNATELY, later during the day we were made aware of more reports from players stuck on the loading screen. DEEP sigh.

It was then discovered that some clients were assigned a correct server, but did not appear to connected to it. This particular server was one of the new game servers that we added to meet the increasing number of concurrent players. We found that one of the new game servers had been setup with a firewall configuration rule that blocked all network traffic, instead of allowing specific game ports. This prevented players from actually connecting to the assigned game server, and loading into the match, staying permanently stuck on the loading screen. This has affected thousands of players in the afternoon of Thursday April 12th.

We corrected the firewall configuration rule at 6:10pm EST, and players could load into the game.

Steam, The Next To Last loading bug

We received more reports of players stuck on the loading screen, AND some players were unable to login into the game at all. There are no words.

We quickly discovered the source of the issue was a wide scale Steam maintenance affecting many of their services including authentication for clients and game servers. The Steam services came back online within several minutes.

MOAR issues, omg

After all this, we still receive reports of players stuck on the loading screen. Dark times.

We identified that some servers in the OCE (Sydney) and SEA (Singapore) regions, were stuck awaiting for map travel, an operation that happens at the end of a match to reset the game state. This was due to a combination of factors. Some game servers seem to be frequently disconnected from the matchmaking server while reloading the map at the end of a match, and on top of that, the matchmaking server had an incorrect configuration where it waited for a confirmation that the map reload was complete, in order to let clients join it. This happened specifically when a match ended with 8 connected players (players in games and spectators). We corrected the configuration to always force a full reset of the match state on the matchmaking server.”

We’ll have to wait and see if the issue has finally been resolved for all players or whether players will still find themselves stuck on the loading screen.