The Mana World is offline [FIXED]

Where game and project announcements are made.
Locked
User avatar
gumi
Developer
Developer
Posts: 731
Joined: 19 May 2014, 19:18

The Mana World is offline [FIXED]

Post by gumi » 30 Dec 2018, 02:04

The legacy game server suffered an attack and we put it offline while we work on fixing security vulnerabilities. Don't worry about data loss: we have snapshots for every hour of the last 180 days so we can do rollbacks if needed.
Please understand that we don't know when the server will be back online.

In the meantime, feel free to join us on IRC: #themanaworld on Freenode

Updates will be posted here



---------------------------------------------------------------------
  • 2018-12-29 @ 02:19: attack begins
  • 2018-12-29 @ 19:25: a GM account is compromised
  • 2018-12-29 @ 19:47: the compromised account starts using GM commands
  • 2018-12-29 @ 21:56: attack reported to staff
  • 2018-12-29 @ 22:03: server taken offline
  • 2018-12-29 @ 22:03: admins are notified
  • 2018-12-29 @ 22:07: website and forums taken offline
  • 2018-12-29 @ 23:20: requesting backups (encrypted) from Amazon Glacier
  • 2018-12-29 @ 23:41: looking through logs
  • 2018-12-30 @ 01:02: forums back online, still locked
  • 2018-12-30 @ 01:20: auditing source code
  • 2018-12-30 @ 01:59: auditing source code, testing locally
  • 2018-12-30 @ 02:04: one vulnerability found, discussing about best way to tackle it
  • 2018-12-30 @ 02:16: found a second vulnerability, working on a patch
  • 2018-12-30 @ 02:44: found a third vulnerability, pondering about rewriting the auth flow entirely
  • 2018-12-30 @ 03:55: investigation still ongoing; attack successfully reproduced locally
  • 2018-12-30 @ 04:06: digging through logs
  • 2018-12-30 @ 04:33: uploading backups
  • 2018-12-30 @ 04:39: merging backups with latest data, analyzing side-effects
  • 2018-12-30 @ 04:46: data fully restored to snapshot 2018-12-29T00:07+00:00: all data beyond this point is lost
  • 2018-12-30 @ 06:10: backporting from upstream, making security patches
  • 2018-12-30 @ 09:07: reviewing patches
  • 2018-12-30 @ 15:10: making further improvements to the auth flow
  • 2018-12-30 @ 15:25: more hardening
  • 2018-12-30 @ 18:39: testing done with char server, now on to map server
  • 2018-12-30 @ 21:07: testing done with map server
  • 2018-12-30 @ 21:22: test server back online, main server still offline
  • 2018-12-30 @ 23:29: main server back online
EDIT: changed dates/times to UTC
User avatar
gumi
Developer
Developer
Posts: 731
Joined: 19 May 2014, 19:18

2018-12-29 incident – post-mortem

Post by gumi » 31 Dec 2018, 01:55

Current status
Main server and test server are back online. 2 hours worth of data have been lost.
No personal data has been leaked.

What happened
A decade-old bug in tmwAthena, inherited from eAthena, was exploited to bypass authentication and log into GM accounts to wreck havoc on the main server (banning accounts, deleting items, etc). The bug allowed to bypass the login server althogether, making the char server believe the user was properly authenticated.

What changed
We fixed this bug and also another bug that made it possible to kick any online user simply by knowing their account id (the target would see "someone else is trying to use this account" and get disconnected). Login-, char- and map-server now all directly exchange auth info with each other so that they can validate the data that the end user sends. We also improved logging so that fail2ban can more easily spot suspicious behaviour.

Lessons learned
  • Don't use a fork that is so outdated that it is 10 years behind its upstream counterpart
  • Don't blindly trust software written by someone else without auditing it
  • woah, MadCamel is still alive? cool
Matt
Knight
Knight
Posts: 1751
Joined: 07 Aug 2004, 11:47
Location: Germany->Bavaria

Re: The Mana World is offline [FIXED]

Post by Matt » 31 Dec 2018, 09:03

@broadcast Platyna's revenge!
Really?


Nice write up!
User avatar
prsm
Game Master
Game Master
Posts: 1444
Joined: 24 Mar 2009, 18:18

Re: The Mana World is offline [FIXED]

Post by prsm » 31 Dec 2018, 18:21

I would like to thank everyone that came together to fix this game, so others
could play it!

It simple amazes me that someone with such thoughtful disregard feels its
okay to wreck havoc, but when push comes to shove, karma will always prevail!

Enjoy that when it hits!

Prsm
ego is the anesthesia that deadens the pain of stupidity!
User avatar
TheManaWorld
Peon
Peon
Posts: 47
Joined: 17 Nov 2018, 13:13

Re: The Mana World is offline [FIXED]

Post by TheManaWorld » 31 Dec 2018, 19:41

That evening we were hunting in the crypts for many hours and thanks to this dude we lost our precious time :alt-4:
let me be your ⛈️
User avatar
Tirifto
Peon
Peon
Posts: 64
Joined: 19 Aug 2015, 11:38
Location: Esperantujo

Re: The Mana World is offline [FIXED]

Post by Tirifto » 01 Jan 2019, 00:33

TheManaWorld wrote:
31 Dec 2018, 19:41
That evening we were hunting in the crypts for many hours and thanks to this dude we lost our precious time :alt-4:
The real treasure you have gained that evening is not experience points, but the experience of having fun playing The Mana World with nice people!
User avatar
gumi
Developer
Developer
Posts: 731
Joined: 19 May 2014, 19:18

Re: The Mana World is offline [FIXED]

Post by gumi » 02 Jan 2019, 16:28

this thread was not intended to degenerate into a flame war..

should've locked it when I had the chance :/



EDIT: moved some posts to the flame war thread
User avatar
gumi
Developer
Developer
Posts: 731
Joined: 19 May 2014, 19:18

Re: The Mana World is offline [FIXED]

Post by gumi » 04 Jan 2019, 02:28

Because someone is still being a jerk and flooding the server with bogus login requests we blocked a huge set of ip addresses and subnets.
Further mitigation is being undertaken and hopefully everything should be sorted out soon.

This is not a security issue, just an annoyance.
We are sorry if this affects legitimate users, we know you're paying for the actions of others, and that's not nice :/
Locked