Hey Everyone! I'm posting on behalf of our Loot Tech Lead, Riot Rainbow Pi. She is still getting her PBE credentials set up at RITO HQ, so I'm passing this message along for you all.
Hi, I’m the tech lead on loot, and I wanted to apologize and talk about why loot has been down for the last week.
TLDR: We managed to find a unique and arcane bug that we couldn’t repro in our internal environments. It caused a ~50% failure rate of any transaction in loot. (Crafting, add loot via missions or store, etc). We decided it would be safest to disable access to players while we triaged the issue.
The longer version:
We started rolling out a new version of the loot service to PBE last Tue in preparation for Lunar Revel. We had done a bunch of testing on internal environments and everything looked fine. Once we did the deploy we starting running our test suite against it to verify the upgrade worked. Unfortunately our suite found issues we had never seen in internal envs. About 50% of transactions were failing to go through, which meant you wouldn’t be able to craft, there would be sporadic failures of content from missions (fixed upon retry), and a generally degraded experience.
We decided to spend some time triaging it, we had found a root cause error but we weren’t able to find a quick fix. We tried to roll it back to the previous working version so we could get loot back up and running, but for some reason the previous version was no longer behaving. In an attempt to minimize player pain, we made a decision to disable loot while we looked into this. Over this past week, we have been researching, experimenting, and trying to get loot back up. Half of our team, plus a number of people outside of the loot team were working on this. It wasn’t until today that one of our experiments proved successful.
Since this issue didn’t crop up on other envs, there was a chance that this would not impact a live deploy for lunar revel. But we didn’t want to take that chance, so we’ve been working to get this fixed. We have a fix out, and loot should be back on. If you run into any further issues please let us know.
Again, I’m sorry we had it down so long. Hopefully we have this particularly nasty bug nipped in the bud for good.
Riot Rainbow Pi