Incident Report for Item Loss Bug

Over a week ago we became aware of this issue based on player reports. We have had multiple programmers working hard on fixing it since then, but it was extremely difficult to determine the root cause. This was exacerbated by the number of coincidental changes to the realm at the same time that we spent a long time looking into (for example, the deployment of the Beta realm, upgrade to GCC 4.9, release of the supporter packs, etc).

The bug was a very rare use after free issue which resulted in four bytes of arbitrary memory in the game server process being set to zero. Most of the time this caused the server to crash, resulting in no item problems. However, in very rare cases, it was possible for memory that contained tracking of what items had been deleted to become modified. In some extremely rare cases, it could modify an actual item to set properties to zero (such as number of sockets or a specific mod). In a few cases it reset characters' passive trees because that was the part of memory affected.

This issue has now been fixed.

Memory corruption can corrupt data in such a way that it looks the same as a regular user action. Unfortunately, in our logs, legitimate item deletion is indistinguishable from loss due to this bug. We are unable to restore items lost in this way.

We are extremely sorry that this problem occurred. It is unacceptable for players to lose items on the live servers and we deeply regret not being able to resolve it to everyone's satisfaction.