Following the recent 15-hour network outage that left many Xbox users unable to access even games they already owned, division CTO Scott Van Vliet took to social media to apologize for the downtime. In a lengthy post, Vliet spoke about the reasons behind the Xbox Live outage, how the Xbox team was able to fix it, and how the company plans to improve its infrastructure to avoid it happening again.
“Earlier today, many players began experiencing issues with sign-in, enumerating their game libraries, and playing games. The team has worked hard throughout the day to fix the issue, and we’ve restored service as of 2:30 PM PDT. Nonetheless, this is an unacceptable situation, and we must do better to support you,” he said.
“Over the coming weeks, you’ll hear more from me about our services, our platform, and the work we’re doing to make XBOX better. And that starts with some transparency about this incident.”
Xbox Live suffered an outage due to an external licensing service failing. This led to users facing errors when attempting to sign in to the service, and many also started failing entitlement checks where the console would try to make sure that a user owned the game that they are trying to play. This, in turn, became a cascading issue, with even the Full Library view being disabled, as well as the console’s general failure to launch games. Similarly, publishing and store partners also started facing problems.
The issues were caught through automated monitoring and the Xbox Live team immediately branded it a “major incident” before getting to work on fixing it. Isolating the failing infrastructure, the team started diverting network traffic to healthier parts of the system.
As for the path forward, Vliet has said that while a service outage on its own was certainly a bad thing, the fact that it was able to cascade into becoming such a major incident is the real cause for concern.
“I care less about the one-line root cause and more about the real questions: why a failure in one service was able to take down this much, why recovery took as long as it did, and what we change so a single point of failure can’t ruin your night again,” he explained. “That means hardening the dependencies underneath sign-in and game launch, improving how we detect and contain this class of failure, and being faster and clearer with you when something breaks.”
While the service outage was ultimately fixed by Xbox, the incident has ended up shining a spotlight on the follies of an all-digital library for games. The issues with the software failing when checking for ownership and entitlement only affected players trying to play their digitally-purchased game. Discs, on the other hand, were unaffected, since the physical presence of the disc itself acts as verification that the user owns the game they are trying to play.
It is worth noting that, aside from its business decisions and services like Game Pass that indicate a heavier move towards an all-digital future, Xbox hasn’t made any announcement regarding the potential end of disc production, unlike Sony.
