Page 1 of 1

The Mana World is offline [FIXED]

Posted: 30 Dec 2018, 02:04
by gumi
The legacy game server suffered an attack and we put it offline while we work on fixing security vulnerabilities. Don't worry about data loss: we have snapshots for every hour of the last 180 days so we can do rollbacks if needed.
Please understand that we don't know when the server will be back online.

In the meantime, feel free to join us on IRC: #themanaworld on Freenode

Updates will be posted here

  • 2018-12-29 @ 02:19: attack begins
  • 2018-12-29 @ 19:25: a GM account is compromised
  • 2018-12-29 @ 19:47: the compromised account starts using GM commands
  • 2018-12-29 @ 21:56: attack reported to staff
  • 2018-12-29 @ 22:03: server taken offline
  • 2018-12-29 @ 22:03: admins are notified
  • 2018-12-29 @ 22:07: website and forums taken offline
  • 2018-12-29 @ 23:20: requesting backups (encrypted) from Amazon Glacier
  • 2018-12-29 @ 23:41: looking through logs
  • 2018-12-30 @ 01:02: forums back online, still locked
  • 2018-12-30 @ 01:20: auditing source code
  • 2018-12-30 @ 01:59: auditing source code, testing locally
  • 2018-12-30 @ 02:04: one vulnerability found, discussing about best way to tackle it
  • 2018-12-30 @ 02:16: found a second vulnerability, working on a patch
  • 2018-12-30 @ 02:44: found a third vulnerability, pondering about rewriting the auth flow entirely
  • 2018-12-30 @ 03:55: investigation still ongoing; attack successfully reproduced locally
  • 2018-12-30 @ 04:06: digging through logs
  • 2018-12-30 @ 04:33: uploading backups
  • 2018-12-30 @ 04:39: merging backups with latest data, analyzing side-effects
  • 2018-12-30 @ 04:46: data fully restored to snapshot 2018-12-29T00:07+00:00: all data beyond this point is lost
  • 2018-12-30 @ 06:10: backporting from upstream, making security patches
  • 2018-12-30 @ 09:07: reviewing patches
  • 2018-12-30 @ 15:10: making further improvements to the auth flow
  • 2018-12-30 @ 15:25: more hardening
  • 2018-12-30 @ 18:39: testing done with char server, now on to map server
  • 2018-12-30 @ 21:07: testing done with map server
  • 2018-12-30 @ 21:22: test server back online, main server still offline
  • 2018-12-30 @ 23:29: main server back online
EDIT: changed dates/times to UTC

2018-12-29 incident – post-mortem

Posted: 31 Dec 2018, 01:55
by gumi
Current status
Main server and test server are back online. 2 hours worth of data have been lost.
No personal data has been leaked.

What happened
A decade-old bug in tmwAthena, inherited from eAthena, was exploited to bypass authentication and log into GM accounts to wreck havoc on the main server (banning accounts, deleting items, etc). The bug allowed to bypass the login server althogether, making the char server believe the user was properly authenticated.

What changed
We fixed this bug and also another bug that made it possible to kick any online user simply by knowing their account id (the target would see "someone else is trying to use this account" and get disconnected). Login-, char- and map-server now all directly exchange auth info with each other so that they can validate the data that the end user sends. We also improved logging so that fail2ban can more easily spot suspicious behaviour.

Lessons learned
  • Don't use a fork that is so outdated that it is 10 years behind its upstream counterpart
  • Don't blindly trust software written by someone else without auditing it
  • woah, MadCamel is still alive? cool

Re: The Mana World is offline [FIXED]

Posted: 31 Dec 2018, 09:03
by Matt
@broadcast Platyna's revenge!

Nice write up!

Re: The Mana World is offline [FIXED]

Posted: 31 Dec 2018, 18:21
by prsm
I would like to thank everyone that came together to fix this game, so others
could play it!

It simple amazes me that someone with such thoughtful disregard feels its
okay to wreck havoc, but when push comes to shove, karma will always prevail!

Enjoy that when it hits!


Re: The Mana World is offline [FIXED]

Posted: 31 Dec 2018, 19:41
by TheManaWorld
That evening we were hunting in the crypts for many hours and thanks to this dude we lost our precious time :alt-4:

Re: The Mana World is offline [FIXED]

Posted: 01 Jan 2019, 00:33
by Tirifto
TheManaWorld wrote:
31 Dec 2018, 19:41
That evening we were hunting in the crypts for many hours and thanks to this dude we lost our precious time :alt-4:
The real treasure you have gained that evening is not experience points, but the experience of having fun playing The Mana World with nice people!

Re: The Mana World is offline [FIXED]

Posted: 02 Jan 2019, 16:28
by gumi
this thread was not intended to degenerate into a flame war..

should've locked it when I had the chance :/

EDIT: moved some posts to the flame war thread

Re: The Mana World is offline [FIXED]

Posted: 04 Jan 2019, 02:28
by gumi
Because someone is still being a jerk and flooding the server with bogus login requests we blocked a huge set of ip addresses and subnets.
Further mitigation is being undertaken and hopefully everything should be sorted out soon.

This is not a security issue, just an annoyance.
We are sorry if this affects legitimate users, we know you're paying for the actions of others, and that's not nice :/