Information about NPC Incident 2017-04-21 & Emergency rollback on Elysium

[whitekidney 1]

Greetings adventurers.
I will try and explain as detailed as possible what happen yesterday.
Please note that this incident only affected Elysium

At approximately 18:39:58 server time a command was issued that would reload the creature_template information from the database, the intended function was to reload the values from the database and apply any potential changes to NPCs, however, what actually happened was that a significant amount of mobs in the game world became severely bugged and rewarded an unintended amount of XP, money and items to players.

The problem was identified quickly and the timeframe for the incident was isolated to have lasted approximately 184 seconds (hence why the servers were shut down so quickly).

Shortly after this incident occurred, We quickly began analyzing and parsing logs in an attempt to start a process where we would reverse any gold, xp and items rewarded to players, however we quickly realized that we would manually have to analyze nearly 30 000 lines of log entries and then reverse any items, xp or gold that was not intentionally rewarded to players. This process would have taken a very long time - and this is a process which we would have to do while the server was offline - Not ideal on a Friday evening server time.
The estimated gameplay hours rolled back is approximately 6 hours and 43 minutes (Until 12:00 server time on the 21th)

Here is a rough timeline of today's events:

18:39 - The server was put in a bugged state where NPCs were severely bugged.
18:42 - The issues were identified and the server was shut down at 18:43:02
18:55 - Senior staff members was notified.
19:40 - Meeting with lead and senior staff have concluded, the decision was made to roll back to the latest backup.
20:39 - The restore process has begun but will take a while due to the amount of data being restored.
20:42 - The world database is being restored
20:43 - World restore complete
20:43 - The log database is being restored
21:29 - The log database has completed.
21:29 - The character database is being restored.
21:58 - The character database has been restored.
22:03 - Elysium is now back online.

Timestamps are in server time

We are actively investigating why this incident occurred in the first place and we will be issuing hotfixes as soon as we have a patch ready. The backup procedures will also be slightly adjusted to make sure that backups are ran more often (I was doing maintenance on the backup solution the same day this incident occured, very unfortunate timing). We apologize for the rollback but we strongly believe that this was the only way to make sure that no illicit gold or items was introduced to the game economy.

[Hamblin 0]

22:04 - Whitekidney collapses from exhaustion

[ka.ba 0]

rawr XD

[Ragingducks 9]

You did what you had to do.

If backups will be done more often, then even better for the future of the project in case something similar happens.

[whitekidney 1]

3 minutes ago, Ragingducks said:

You did what you had to do.

If backups will be done more often, then even better for the future of the project in case something similar happens.

We're actively working on updating our infrastructure with new changes to many things, one of them being our backup solution.
Normally full backups are ran every 3 hours, but since I was doing changes to our backupsolutions (on a Friday? sigh.. Friday is no-change-Friday!) the last backup was missed. It's just bad luck.

[Aziz 1]

When are you addressing what happened to the latest azuregos kill?

[Tomyris 0]

6 minutes ago, Aziz said:

When are you addressing what happened to the latest azuregos kill?

What is to be done? Learn to kill Alliance better.

[Haitharn 2]

3 hours ago, Aziz said:

When are you addressing what happened to the latest azuregos kill?

Learn to play the game better

[Xylon666Darkstar 4]

4 hours ago, Haitharn said:

Learn to play the game better

You already know the answer for your Azuregos 'problem' then. :)

[Xylon666Darkstar 4]

4 hours ago, Haitharn said:

Learn to play the game better

You already know the answer for your Azuregos 'problem' then. :)

[Hurricane2 2]

I heard boatload of people were banned for the azuegos incident so I don't think learning to play the game better was the right answer.