Incident Report for April 1 Catastrophe

During the week of March 23rd, one of our senior gameplay programmers was performing a routine refactor on our core skill code. In preparation for our upcoming expansion, changes were made to the functionality of Trigger Gems. The goal of the refactor was to improve the reliability of the gems that trigger spells on various conditions - Cast on Death, Cast When Damage Taken, etc. During this process, the names of the gems were entered by hand. Unfortunately, due to a combination of tiredness and the programmer's previously undiagnosed Toxoplasmosis infection, the word "Cast" was entered as "Cats" for these gems.

Unbeknownst to the programmer, the consequences of this small typo were severe.

This change was deployed to our production realm at 10:05am on Wednesday, April 1 (NZ time) as part of the 1.3.1g point release. Players reported that the Cats Trigger Gems now spawned black cats when their conditions were met. Cats When Damage Taken suddenly started creating cats in addition to endurance charges. A feline army mourned the passing of Hardcore Exiles who used the popular Cats on Death build. Needless to say, these new gems became a powerful combo with Chance to Flea.

The first report of this issue occurred at 10:09am. Our quality assurance department was able to reproduce the problem by 10:33am. At this time, an investigation into the origin of the problem began.

We are currently preparing a patch to resolve this issue and restore the Cats trigger gems to their previous functionality. We don't have an exact timeframe for the patch (our current attempted fix has issues with Leeches spawning for some reason), but we're expecting to have it resolved by the end of April 1, Pacific Time. Once this fix has been deployed, the trade value of claw weapons should begin to normalise.

In light of the fact that typos can summon animals from nowhere, we will be dramatically changing our development methodologies. Not only must programmers now copy/paste existing game terms (to avoid typographical errors), but we will be improving our code testing framework to detect the accidental introduction of animal names into the game (not only as gems but also other areas such as the Goats Reaver keystone).

On behalf of our development team, I would like to deeply apologise for this issue affecting the live realm and your play experience. This incident does not reflect the level of service you should expect from our company, and I will personally ensure that it does not happen again.
Lead Developer. Follow us on: Twitter | YouTube | Facebook | Contact Support if you need help!
Credit for this idea goes to Reddit user khai15.
Lead Developer. Follow us on: Twitter | YouTube | Facebook | Contact Support if you need help!
LOL

Report Forum Post

Report Account:

Report Type

Additional Info