2012-03-30 19:56 NZDT Unexpected downtime Incident Report

This error was caused by a simple syntax mistake in the realm configuration file. In order to rectify the mistake we had to fix the configuration file and manually transfer it to all of the machines in our realm.

All binaries and assets of a Path of Exile release are tested by our automated test system. This normally prevents this kind of basic error from occurring. However, the primary difference between the online realm and our test harness is this very configuration file. Thus we did not have a test to cover this.

We are sorry for the incident and will be taking steps to make sure it doesn't happen in the future.

In order to stop this kind of mistake we will be adding an automated test to verify that the realm configuration file is able to load correctly and has some basic correctness properties during the build process.
Path of Exile II - Game Director
Last edited by Chris on Oct 31, 2012, 5:44:07 AM
"
Jonathan wrote:
At 19:56 NZDT the website and game were down for 16 minutes during the redeploy of our servers. After the redeploy, the servers did not come back up.

This error was caused by a simple syntax mistake in the realm configuration file. In order to rectify the mistake we had to fix the configuration file and manually transfer it to all of the machines in our realm.

All binaries and assets of a Path of Exile release are tested by our automated test system. This normally prevents this kind of basic error from occurring. However, the primary difference between the online realm and our test harness is this very configuration file. Thus we did not have a test to cover this.

We are sorry for the incident and will be taking steps to make sure it doesn't happen in the future.

In order to stop this kind of mistake we will be adding an automated test to verify that the realm configuration file is able to load correctly and has some basic correctness properties during the build process.

No worries! The up time of this beta rivals games fully released!

Keep up the good work!
Even though it was short, I'm happy that you explained why for any possible inconveniences you may have caused.

You hear it a lot, but do keep up the good work. You guys rock!
Feel free to add me on Steam
http://steamcommunity.com/id/Xorjt
GGG_FanBoy_101
16 minutes? You guys are lucky. My internet went down for 2 days and life came to a screaming halt. I was thinking of packing up and movin' on out to Silicon Valley 'cause

<grizzled old prospector voice>
I heards they still got some internets out they're.
</grizzled old prospector voice>

The fact this is my first stop upon getting my connection back (never mind all the work stuff I need to do) says something about the game you guys are building. Keep up the great work.
You guys do real awesome incident reports. Real talk, not just a template response.
Available to mediate cross-league trades and mirror services. Send me a PM here or in-game.
IGN: SpudOfDoom | The Exiled - Path Of Exile's oldest clan
"
SpudOfDoom wrote:
You guys do real awesome incident reports. Real talk, not just a template response.


Real talk, you got that right. A syntax error is such a classic programming problem and they admit to it. Lets us newbs know it happens to the best of us!
NP, we still love you :)
"You guys do real awesome incident reports. Real talk, not just a template response."

This is totally true, I love your reports about downtimes etc. Not just the typical "We are having problems we will be back in X" I really love this reports and I hope other companies do the same with their users, you are awesome!
You are the best really, i love the way you always explain to us any problem or any ideas you have, really close, go go. ^^
The five minute patch downtimes that this game seems to have are amazing compared to other multiplayer games, and it really reflects well upon the technical skills of your programming team that they are so short.

Report Forum Post

Report Account:

Report Type

Additional Info