Poll: Rollback from today's update, or not?

[Frost]

The update today changed what code the main server uses.
It also changed the update hosting to use a distributing "Content Delivery Network" (kind of like Akamai for non-profits).

The new code choked on some things in the game saves, which caused some people to see their characters were lost.
To solve that, we had to hand-edit the saves and replace the things which caused indigestion.

At this point, the most obvious problem is that people must remove all clothes to trade in the game. Any fix for that will take a few days (and maybe a little longer for testing).

Let's not worry about the new update hosting until someone reports a problem.

So. We need to hear what players want. Here's the question.

Do we keep the current situation and don't trade or such for a few days?
Do we roll back the code change?
If so, do we also go back to the saved files before they were "fixed by hand" to remove certain items?

[o11c]

I am now quite confident in my ability to fix the trade bug by tomorrow. If I hadn't been forced to fix the characters/storages issue ahead of my schedule, I probably would have already fixed the trade bug.

Rollbacks should *never* be a first resort.

Edit: I'll probably fix the trade bug by the time this poll closes anyway, so what's the point?

If you don't want bugs like this getting on the main server, use the freaking test server. I tested a LOT of things there and locally, but apparently the only times I traded with myself I happened to fall into the case where it does work.

[AnonDuck]

I suppose it depends what all it was choking on and if it's likely to continue randomly choking?

If the mess has already been cleaned up, no sense re-making it. Keep the data at least. I don't see naked trading as much of an issue (I think it's funny and should be fixed rather quickly)

What are some of the other problems? I don't have enough information to make an informed opinion.

[o11c]

I suppose it depends what all it was choking on and if it's likely to continue randomly choking?

It was not random at all: the new code just rejected characters and storages that had corrupted items, and logged their deletion. I managed to remove the corrupted parts of the characters and storages and preserve the rest.

That problem has been fixed, and it is impossible for it to happen again unless someone, say, copies /dev/urandom to the savefile.

[AnonDuck]

Well that seems harmless enough.

Many saves were silently and oddly corrupted on the old server. Some saves/storages were broken all along but nobody noticed until the new server validated the save data more carefully. It's good that the data has been properly fixed to prevent future issues and bugs.

Rolling back the data would require re-fixing all these corrupted saves, so it doesn't make sense to me to do that. It'd be a pretty big step backwards.

It's like you moved a container of milk in the back of your fridge and found a HUGE mushroom behind it. That mushroom's been there for years but you never noticed it. Ewww. Do you A) Clean up the mushroom even if it's hard, or B) Put the milk back in front of it so you don't have to look at it?

In this case the mushroom has already been cleaned up and everyone's voting to put the mushroom back in the fridge and hide it behind the milk again. Wtf?

[Frost]

As I understand it, the new code choked on some data that the old code didn't.
Reason: new code is correct, and data is bad.
Solution: change the data until the new code is happy. I think o11c said he fixed "negative values" and changed them to zero.

I don't know exactly what was changed. I tried to 'diff' the before and after save files, but I couldn't find complete copies of the files to look at. The storage file blew up 'diff' with a wall of changes. We don't know exactly what was lost in the "fix."

The basic reason I can see for rolling back the save game is to get us to a known place. We trade a few hours of game time for confidence. Let's hear what people prefer.

[cinderweb]

This is rediculous I can't even trade with my wife without getting naked first and the manamarket won't work in this fashion either. What was changed in the flurry of fixes that couldn't work with the new code but could with the old? I know my wife couldn't even log in because her name this time wasn't capitalized and it never has been before so is that a feature or a bug?

[Hello=)]

Voted for revert code and data, because:
1) Bugs like broken trades are not OK for main server. And I can admit that granted server's code changes, it would take ages to test it "properly" as far as I understand list of changes. In this case "properly" seems to mean "retest everything". So in ideal world with ideal code and so on, there ideally should be team of players who actually plays on test server for a while, doing quests and other stuff which could fail, with more or less realistic/pre-existing data and without heavy use of debug cheats/GM commands (which could hide bugs). Yet I'm not sure TMW can afford it. Test servers are usually quite empty so multiplayer interactions arent really tested on test servers at all .

2) IMO data should be reverted to known good, because: as far as I know, there was data corruption, including some garbage in inventory. And since there is time pressure looks like data were not thoroughly verified, some things were fixed by hands. But it can be just top part of iceberg. Correct me if I'm wrong on this.

p.s. and I havent really got idea why it so damn necessary to screw up logins. It worked for years this way, I never heard loud complaints about this part, ever. Looks like it has been okay. Why it have to be changed? Just because o11c got to think it's somehow "better"? Now I can see "better" means a lot of grief for those who can't log in. Are there anything that justifies such a destructive change, preventing some players from logging in? It does not looks like if TMW haves so much players there is strong need to get rid of some of them, no?

[o11c]

I don't know exactly what was changed. I tried to 'diff' the before and after save files, but I couldn't find complete copies of the files to look at. The storage file blew up 'diff' with a wall of changes. We don't know exactly what was lost in the "fix."

I explained *exactly* what I did, and how. It is not my fault you are incapable of reading when I communicate with you.

Please, STOP trying to give me a bad name. I have done everything possible to make this go as smoothly as possible.

The basic reason I can see for rolling back the save game is to get us to a known place. We trade a few hours of game time for confidence. Let's hear what people prefer.

So it's better to have a known bad place than a guaranteed good place?

Voted for revert code and data, because:
1) Bugs like broken trades are not OK for main server. And I can admit that granted server's code changes, it would take ages to test it "properly" as far as I understand list of changes. In this case "properly" seems to mean "retest everything". So in ideal world with ideal code and so on, there ideally should be team of players who actually plays on test server for a while, doing quests and other stuff which could fail, with more or less realistic/pre-existing data and without heavy use of debug cheats/GM commands (which could hide bugs). Yet I'm not sure TMW can afford it. Test servers are usually quite empty so multiplayer interactions arent really tested on test servers at all .

True. And if things were done *my* way, I would have fixed the trade bug today and left the storage/chars corruption until tomorrow, because it wouldn't get worse by waiting, whereas trades affected people constantly.

I tried time and time again to get people to go on the test server. What more could I have done?
(It *really* hurt this cause when Frost stopped running the code to test, but eventually we found a new test server)

2) IMO data should be reverted to known good, because: as far as I know, there was data corruption, including some garbage in inventory. And since there is time pressure looks like data were not thoroughly verified, some things were fixed by hands. But it can be just top part of iceberg.

There was no "known good" data before. Do you really suggest running the version of the server that *generated* the corrupt file as opposed to one that, by symmetry, can never write a savefile that it can't read?

p.s. and I havent really got idea why it so damn necessary to screw up logins. It worked for years this way, I never heard loud complaints about this part, ever. Looks like it has been okay. Why it have to be changed? Just because o11c got to think it's somehow "better"? Now I can see "better" means a lot of grief for those who can't log in. Are there anything that justifies such a destructive change, preventing some players from logging in? It does not looks like if TMW haves so much players there is strong need to get rid of some of them, no?

There were, I don't remember, something like a hundred accounts that differed in case already. Do you suggest I should have deleted them all instead?

This is NOT an arbitrary problem. In order to have any performance at all, you MUST choose whether your names will be case-sensitive or case-insensitive, consistently.

[Frost]

I don't know exactly what was changed. I tried to 'diff' the before and after save files, but I couldn't find complete copies of the files to look at. The storage file blew up 'diff' with a wall of changes. We don't know exactly what was lost in the "fix."

I explained *exactly* what I did, and how. It is not my fault you are incapable of reading when I communicate with you.

Okay, I'm stupid. Then tell me again *exactly* what you did, and where I can get a list of what you changed. What I saw were incomplete files.

The basic reason I can see for rolling back the save game is to get us to a known place. We trade a few hours of game time for confidence. Let's hear what people prefer.

So it's better to have a known bad place than a guaranteed good place?

If by "known bad" you mean that nobody had their characters or items deleted, and by "guaranteed good" you mean after you deleted unspecified items from the save file, I guess so.

And if things were done *my* way, I would have fixed the trade bug today and left the storage/chars corruption until tomorrow, because it wouldn't get worse by waiting, whereas trades affected people constantly.

We asked you to please not make this update, because it wasn't ready. You did it your way, and we learn the dangers of running untested code. If the code wasn't tested properly, just go back and test it until it's ready.

I tried time and time again to get people to go on the test server. What more could I have done?
(It *really* hurt this cause when Frost stopped running the code to test, but eventually we found a new test server)

That server runs CentOS. You wrote code that won't compile on CentOS/RHEL. Okay.

2) IMO data should be reverted to known good, because: as far as I know, there was data corruption, including some garbage in inventory. And since there is time pressure looks like data were not thoroughly verified, some things were fixed by hands. But it can be just top part of iceberg.

There was no "known good" data before. Do you really suggest running the version of the server that *generated* the corrupt file as opposed to one that, by symmetry, can never write a savefile that it can't read?

Players don't like to lose items. This was another thing we missed in testing. In fact, once we saw the problem during the update today, that was a red flag to step back and reassess the update procedure.

p.s. and I havent really got idea why it so damn necessary to screw up logins. It worked for years this way, I never heard loud complaints about this part, ever. Looks like it has been okay. Why it have to be changed? Just because o11c got to think it's somehow "better"? Now I can see "better" means a lot of grief for those who can't log in. Are there anything that justifies such a destructive change, preventing some players from logging in? It does not looks like if TMW haves so much players there is strong need to get rid of some of them, no?

There were, I don't remember, something like a hundred accounts that differed in case already. Do you suggest I should have deleted them all instead?

This is NOT an arbitrary problem. In order to have any performance at all, you MUST choose whether your names will be case-sensitive or case-insensitive, consistently.

Again, a user-facing change that was not communicated clearly. It's up to you to justify it, not for players to justify being able to login the same way they always have.

Clearly, these code changes are controversial. Clearly, they are untested. We can and must do better.

[o11c]

As expected, the relevant bug was fixed long before this poll concluded.

[Ginaria]

Hello all :)

can someone give me a summary, what happened after i logout for update yesterday? I believe some more players would be interested in a summary and what the actually status is.
I thought me, i look at forum before i login today, if update was successfully and i am getting little nervous, what i have read in so many threads... for a neutral Pinky its look like little chaos and communication problem.

I still have not login at TMW and at the moment i can only say:
The information politics is not the best. If the update is so big and can make trouble with some items, then please inform the community earlier. I personally would prefer more details about a big update, probably some days sooner.
The most important for community is not to loose their accounts, stuff, money,... after an update. So what you expect, if players are not well informed and seeing an empty storage :o. Big updates can make always problems, its not easy to test that, but it should not influence players stuff.
Bugs, like trading is only possible if you are naked, is not a big problem, that can repaired and after some time we laugh about that.

(To be honest i dont have an good overview at the moment, i hope i have not told something wrong, else please correct me)

[Ginaria]

Some seconds ago, i got this information, probably Void can say more something about it.
"V0id lost storage in 4 accounts"

Yes 4, not 3, not 2, not 1, ITS 4, what the hell is that for an update???
Sorry o11c, normally really like that DEV spend their free time to improve project, but if this message is true, then your update is not ready for main server. Probably Voids account can be repaired manually, but he is surely not the only one. What is with players, who are not playing at the moment and return next month to TMW. How should this be handled. And i dont believe that player did screenshots of their items. How accounts can repaired manually? Who is saying what was inside storage? And thats only few questions...

I really hoped until now, that the situation is not so worse and the bugs can be repaired soon, but this info message is really a big disappointment now. How to repair and to detect all accounts + storages, which are corrupted, especially for offline players?
Now i only wish me a rollback to the status before the last update...
Please convince me that i am wrong...

[Hello=)]

I explained *exactly* what I did, and how. It is not my fault you are incapable of reading when I communicate with you.

Can you tell me one thing I fail to understand: why its impossible to do some test run on "real" dataset to see if new code chokes without affecting players? I.e. like this: copy data to safe place (other dir, separate VM, etc), launch new code with these data and see how it performs, etc. Then you'll have a plenty of time to understand breakage and deal with it, investigating which items corrupted, etc without time pressure and angry players eating your brain. Sure, I can imagine some privacy issues, but your code works with this dataset anyway, so I dont see how some extra test run screws up players privacy on it's own. Am I missing some point here?

I tried time and time again to get people to go on the test server. What more could I have done?
(It *really* hurt this cause when Frost stopped running the code to test, but eventually we found a new test server)

And what would motivate people to do such testing? Speaking for myself, it's not like if guild code removal, ditching GM impersonation and other changes similar in spirit could motivate me too much. Yes, good code is great and you did a lot in this regard, I cant deny it. But there is one catch: it worth nothing (from player or admin point of view) if it lacks features people would expect from decent RPG servers. Even ancient bugged eAthena had it somehow. And at the end of day, RPG server is not a code museum. You see, I have somewhat different idea on what is "good thing" (tm).

There was no "known good" data before. Do you really suggest running the version of the server that *generated* the corrupt file as opposed to one that, by symmetry, can never write a savefile that it can't read?

As far as I understand, it only makes broken saves on shutdown but not on timed DB saves, right? So I guess it's incorrect shutdown sequence? While its bug for sure, and really unpleasant one, its not fatal when its known. Correct me if I'm wrong. Sure, TMW team can have another idea about it. And sure, fixing it is good.

There were, I don't remember, something like a hundred accounts that differed in case already. Do you suggest I should have deleted them all instead?

Of course I don't. My suggestion could be run new code on actual data in safe location and correct all issues without time pressure, etc. That's from testing standpoint in ideal world.

This is NOT an arbitrary problem. In order to have any performance at all, you MUST choose whether your names will be case-sensitive or case-insensitive, consistently.

As far as I can remember eA protocol and overall logic, login we're referring to only (?) used by login server at begin of session. Then most of stuff refers player by ID. So is this change brings any real gain? I'm really doubt someone would ever notice if login would take 0.005 seconds instead of 0.01 seconds. But I bet they would notice inability to login to their account for sure . Are there any strong performance win to justify possible loss of players from server?

p.s. and I really fail to understand how it has come to stuation where only 1 person takes some key decisions like telling everyone else which features they should (and should not) use, f..king up logins for players etc. I'm really curious if TMWC supports all this, etc. Or they're just some insignificant jerks which are safe to ignore and all should bow down to Platyna o11c's decisions exclusively? Can someone explain me WTF? Sure, it can look like a bitching. But when some bugged and feature crippled thing lands on my head on server where I play I guess I have some reasons to be unhappy, no?

[Cassy]

I know it's an easy situation at the moment, but can someone please tell the players how high the chance of a rollback would be?
Just an indication.

At the moment many people can't be bothered playing because no one wants to waste time on doing things that could be revoked.


Also please don't be too hard with o11c... he made mistakes but remember he is the only real code dev, has a hard job, spends his free time (like all devs) and code changes are nothing that makes players say "thank you"... think about it