Update on League of Legends service in NA
Lance, aka CaptSturm here. For those of you who donít know me, Iím charged with maintaining the health of the League of Legends live service around the world. As you know, itís been a rough week here in North America.
We are painfully aware of the problems many of you have been experiencing:
- Extended outages: The North American service has been unavailable for significant stretches of time
- Long queue times: Weíve had to throttle logins and concurrent players due to these issues
- Weíve also confirmed that some players are missing content: runes, rune pages, champions, etc.
This is frustrating for you, and we feel that frustration deeply. This isnít the service youíve come to expect and it isnít what we want to deliver. Itís been code red for engineering and support Rioters, and weíve been working literally around the clock to identify and resolve these problems.
Many of you have asked for more information on whatís causing these problems, and we understand that desire Ė Iím going to try my best to give you some insight into what weíre working on, but please understand that our first priority is to put out the fire rather than exhaustively document it.
- This patch is like an iceberg Ė what you see on the surface is small in comparison to the behind-the-scenes infrastructure changes it includes.
- The problems weíre experiencing are database-related Ė specifically, itís a data throughput issue. Our infrastructure handles a massive volume of data and requests every day, and the backend changes in this patch introduced a new bottleneck.
- We put every patch through the wringer including load testing and PBE player testing, and this patch passed those tests. The problems didnít crop up until we pushed it to live. This has exposed some load testing gaps that weíre going to address (simulating millions of concurrent players is truly challenging).
- Over the last 48 hours, weíve made substantial hardware configuration changes that we believe will address the problems. Weíre actively monitoring the service, but weíre still not out of the woods.
I know the thing you want to hear most is when everythingís going to be back to normal. Iíd love to give you a timeframe, but we want to be absolutely confident in our solution before signaling the all-clear. Our number one priority is restoration of a stable game experience.
Thanks for your patience. Iím going to continue keeping you guys up-to-date as we work to fully restore League of Legends to the service youíve come to expect.