3.7.4 Deployment Downtime Incident Report

Thank you for posting this very detailed guide on what went wrong.

As somebody who slept through this I wasn't thankfully affected but appreciate the fullness and honesty with which you have shared the events leading up to the rollback.

Loving Legion League!!

Thanks, GitLab Gear Games.


ps: if it's possible, because i dont know if this information was already posted before - which programming language and migration system was used? just my own interest.

There are no other companies being this open with info on what happens when things are messed up. You guys are the best! People do mistakes. You learn from the mistakes and do better next time. That's how we learn. It might not be the sort of things you want to happen ever since it affected the players this much, but now you will have added levels of quality control for those kinds of processes too.

<3

Mistakes can and always will happen, the important thing is on how you react as a publisher/developer to it and being transparant and honest is the best way to do it.
Keep up the work!

Interesting read.

As many before me, i just want to say thank your for the great communication of what happened. Love you guys, you are the best <3

"

lupo29x wrote:

this reminds me of the time back in closed beta when someone mixed up the 32 and 64 bit .DLL files.
yeah.
i remember that. and i remember how you guys worked with the players in the middle of the night to figure out what the hell went wrong. and then went on to apologize and take the blame for what happened.
it's so wonderful to see the company hasn't changed.
you've earned my money and as soon as i can afford it i'm buying another pack.

good job GGG.

It's kinda sad to see they haven't changed. They are a multi million chinese company now, but still develop like an indie college garage band. They apparently have more than one faulty team they can't get up to standard. The only other company that does that (which isn't outright scammy) is Riot Games. This game is now 6 years old since official release, almost 1 year since Acquisition.
Though it's great to see the patch actually didn't break anything, they just missed a fucking database migration. That is the really interesting detail they could elaborate on in a follow up. Chances are high they won't.

It's too bad it happened and that the QA processes aren't robust enough, but consider that 30 minutes of rolled-back gameplay is hardly a disaster.

Also, you're human. You're supposed to make mistakes from time to time.

Thanks for the transparency. Keep it up in the future, and continue improving your QA processes, please.

As someone frequently wrangling with both Liquibase and SQL Server Data Tools, I feel your anguish.
Excellent write-up and hats off for transparency!

Very competently and correctly described everything. Erroneous actions in stressful situations are justified.
Well done boys!