TMW Admin Role and Responsibilities

[Frost]

I think that TMW leadership needs to have clarity and agreement about what responsibilities and authority go with each role.
I'm posting this in public because I think that players should be part of the conversation.


Because Freeyorp has been proposed to return as head TMW admin, I'll start with that role. I think it's only fair to have a common understanding of what it means to be a "TMW admin."

This is my idea of what a TMW server admin is responsible for:

• Keep the game running and stable.
• Plan and prepare for problems (like how to recover from a crash with minimal problems.)
• Maintain backups. Keep backups of those backups, in case one set is damaged.
• Stay informed about problems on the host and fix them when possible. (For example, make sure an OS vulnerability is patched appropriately.)
• Listen to and talk with key groups in the game: Players, GMs, and Devs.
• Provide information to help GMs manage problems. For example, if someone is warned for insulting another player, and then a new character appears with more insults, look in the server data to see if that's the same person. Another example is if someone claims their account was stolen, GMs need more information than they can access directly.
• When appropriate, provide information to help players. For example, if someone needs a password reset, confirm whether they're the real "owner" and then give the password.
• Communicate clearly. When work is planned, tell GMs and announce it in the game ahead of time. When the work is planned for a certain time, communicate that time. If the work changes or is rescheduled, communicate that. If the work involves Developers, communicate and coordinate with them as appropriate. For example, a seasonal quest might need a content update at a certain time. When work is completed, communicate that too.
• Understand the work. Understand how to use relevant tools. When you make changes on the server, understand what you changed, and try to understand the effects of those changes.
• Prepare for failures and manage them. For example, if a changes goes wrong, be prepared to undo the changes.

In short: keep the server reliable. When there's a problem, manage it. If the server crashes once, blame the programmer, but if it crashes twice, blame the admin.


This is my idea of what a TMW server admin has authority for:

• Be informed about all relevant changes on the server, including configuration changes and software patches.
• Understand and approve all changes to the TMW server itself, including code, content, tools, and data files.

In short: have enough authority to do the work.


Platyna created a strict hierarchy with her at the top. Rather than work with a team, she set people against each other and directly "managed" only the admins. As an admin, I was threatened if I didn't enforce her policies. When someone had too many conflicts with her, she got that person removed. I think we left that dictator model behind when we left her hosting service. Therefore, I also have a few ideas of what the TMW admin position does not include.

• Admins are not "in charge of TMW" or of GMs. They are peers with GMs and Devs, just in different roles. GMs manage player situations, Devs create software and content, Admins keep the server running.
• Admins do not set TMW game policies. They might need to set policies related to the server (if someone tries to brute-force logging in to game accounts, the admin could respond) but not to game play (don't make a new rule and tell GMs to enforce it.)

These are just my ideas, and I'm interested in what others have to contribute.

[Ablu]

Meh. This is silly. If you need such strict silly rules it is not going to work anyway. You need to be behind of the plans of the project anyway to work for it. Anything else is just nonsense.

Regards,
Ablu

[tux9th]

Admins should do all of what you said but shouldn't be forced or expected to do all of that in perfection.

I would expect an Admin to be a supportive member which helps at points where no one else has the access to do so.

- Password resets
- locating stolen items, stolen accounts
- managing the Server (OS updates, Machine)
- Backing up the Server (including gamedata etc.)

Over regulating this position will lead to deadlock because everyone will try to find fault in another ones behavior. The rough rules should be defined (probably in a manner I suggested) the rest should be common sense. People we trust with all the data should be sane enough to handle all of the requests rationally and in a matter which is acceptable to everyone.
If something is not acceptable there are ways to assess this specific behavior/situation to avoid it in the future.
Currently we have a lot of protocols strayed throughout the forums on how to handle what situation. Those guidelines could be unified and saved in a place where everyone can find them for transparency.

In my opinion Freeyorp is capable of all the things you listed above, but not all of those things should be required to become Admin. He doesn't have all the time to communicate as much with players as Frost does but he is perfectly suited for the techincal and supportive aspects of this position. In this word once again I voice my approval for Freeyorp as a new addition to the Server Admin - Team.

with best regards
tux9th

[o11c]

I'm not sure why we need to go into further detail than what is already in the charter

2. B. Server Admins. Admins are those that have ssh access to the server and are responsible for server restarts, maintenance, and the general health of the server overall.

In particular, it is not their responsibility to:

• make any decisions related to development
• take any part in GM events, except if approved by the TMWC
• communicate with players on any issues apart from technical troubleshooting and password resets

Other than that, this is what I'm working by:

• a server admin must be available much of the time
• a server admin does *not* have the right to refuse certain duties while doing others (however, certain duties may *typically* be done by one)
• a server admin must communicate their actions in a persistent location such that other server admins do not need to be confused by what was done
• a server admin must communicate with all other server admins *before* taking any action that is neither urgent nor commonplace

[Frost]

Meh. This is silly. If you need such strict silly rules it is not going to work anyway. You need to be behind of the plans of the project anyway to work for it. Anything else is just nonsense.

It's a job description. I guess those exist even in Germany.
It says what the person is responsible for and what authority they have.
I gave a long list with examples and a short summary. Read either part.
I think it's important that everyone understands what's expected.

I'm not sure why we need to go into further detail than what is already in the charter

2. B. Server Admins. Admins are those that have ssh access to the server and are responsible for server restarts, maintenance, and the general health of the server overall.

In particular, it is not their responsibility to:

• make any decisions related to development
• take any part in GM events, except if approved by the TMWC
• communicate with players on any issues apart from technical troubleshooting and password resets

When I started playing the game, the admins did many things well, but we also had:

• game crashes that were blamed on things beyond the control of admins, and never solved
• rollbacks of game data where sometimes we lost days of play, because...
• backups were unreliable, and admins did not take responsibility to fix the situation
• decisions about restarts or game changes that were not clearly communicated either before or after
• admin who actually told players that the game is "pre-alpha" and therefore they had no basis to complain about anything, but just be grateful for whatever they get
• admins who did not play the game or communicate with players
• admins who were not available to assist GMs (for example, by identifying alts after a ban)
• official admin policy was that if you don't send email from the address registered to that account, you won't get help
• admins acted as "GM boss" rather than leaving that role and decisions to GMs
• admins did not explain problems to players, but retreated into excuses like "you wouldn't understand...."
• rather than openly explain decisions, admins tended to argue that they didn't have to justify anything

When I started as admin, I:

• Set up hourly backups to a remote host, because we knew that local backups were unreliable.
• Continued to play the game and relate to players as an equal.
• Made a point to explain my actions and decisions to others when asked, and to answer questions honestly and directly. I have confidence in my work, and I believe that others will have confidence when they see what I do.
• As part of that confidence, I try to be open and honest when I make mistakes, and accept criticism. I'm not perfect, but I'm also not brittle. For example, I failed to set up automated backups after we moved the server, and this directly caused a game data rollback.
• Asked Players and GMs and Devs how I could help make the game better. Within the limits of my role, I tried to do what they asked.
• Made a point to begin every troubleshooting situation with the firm belief that it was something I could fix, and my responsibility to do so. It's not "somebody else's problem" until I have evidence that identifies the problem, I have done everything I can to correct or mitigate the problem, I provided evidence of the problem to the responsible party, I suggested (as well as possible) actions to address the problem, and I communicated both the problem and my outcomes to any affected and interested people in TMW.

If this is the same charter we had before, I think we do need more clarity.
In particular, I think the admin is responsible to communicate with players. It's a dirty job, but someone has to do it.

Other than that, this is what I'm working by:

• a server admin must be available much of the time
• a server admin does *not* have the right to refuse certain duties while doing others (however, certain duties may *typically* be done by one)
• a server admin must communicate their actions in a persistent location such that other server admins do not need to be confused by what was done
• a server admin must communicate with all other server admins *before* taking any action that is neither urgent nor commonplace

I think you put your finger on exactly where you and I have had conflicts, and I agree.

The reason I'm stepping down is because as long as I'm admin, I hold myself responsible for keeping the server reliable and for fixing problems when they happen.
I advised that the code push in September was risky, and requested certain things (documentation of the changes, agreement about certain changes to GM commands, more testing, upgrade procedure, rollback procedure) before we pushed. You pushed the changes without doing those things, and since then most of my concerns became true. I was left trying to support untested code on a server that crashed using tools like tmwa-admin with undocumented changes, following the documented restart procedure that I didn't know was wrong, helping players figure out what case their login was (now that the server required strict case sensitivity), etc.

I've worked in jobs where I had responsibility to solve problems, but not the authority to do so. I know exactly what that's like, and it's not something I choose to do again. That's why I demand that I have the authority to perform my job properly -- or that the person who takes that authority also takes an active role in meeting the responsibilities.
And that's why I'm trying to get clarity about the responsibilities and the authority of the admin role. I want to see the next person succeed.


I think it's important to hear from people who are currently active in TMW. This includes GMs and Players.

[Chicka-Maria]

If this is the same charter we had before, I think we do need more clarity.
In particular, I think the admin is responsible to communicate with players. It's a dirty job, but someone has to do it.

IMO this Isn't *just* the admins responsibility at all. This should be for anybody working on the game including developers. Which Is why I like the fact we had a recent Dev meeting in game. It allows developers to gain knowledge of what the players want and need in game. GMs are also responsible for communicating with players as well to define rules and explain such to them or any other problems that occur in GMs hands. Everybody in TMWC has a role to communicate with players, whether they do it or not is another story.

I'm glad this is a public post, now everybody can see that there *is* transparency going on.

I will read over again what was said above and most likely have more to say later.

regards,
Chicka

[Ablu]

Meh. This is silly. If you need such strict silly rules it is not going to work anyway. You need to be behind of the plans of the project anyway to work for it. Anything else is just nonsense.

It's a job description. I guess those exist even in Germany.
It says what the person is responsible for and what authority they have.
I gave a long list with examples and a short summary. Read either part.
I think it's important that everyone understands what's expected.

First this is more a hobby project, right?

Second: Even for fulltime jobs: If you need a list of rules to be able to justify yourself it is becoming ridiculous.

[veryape]

Meh. This is silly. If you need such strict silly rules it is not going to work anyway. You need to be behind of the plans of the project anyway to work for it. Anything else is just nonsense.

First this is more a hobby project, right?

Second: Even for fulltime jobs: If you need a list of rules to be able to justify yourself it is becoming ridiculous.

I think this is a brilliant idea to be honest. Not just because of TMWC knowing what roles the different titles are supposed to take care of and how responsibility is shared - it is also great for the players of the game to know the difference between an Admin/Dev/Gm. This is useful since many people do not know the difference or what they can and cannot do.

Secondly, I think that raising the question of what each role should and should not do might raise the question about division of responsibilities in gray areas, and hence stress those theoretical problems before they become real problems.

It might be super silly, but hopefully this somehow leads to any sort of agreement in like if X does Y tell Z about it or, X can do Y only if he has spoken to Z first. This can (and will) save a lot of trouble down the road and might spare us shitloads of conflicts, drama and confusion.

Great initiative, i think that this is what TMWC is really about, division of power/responsibilities and transparency towards the players.

[blackrazor]

I agree with Ablu; it's silly the specific way this information is being put forth. So, Frost posted a wall-of-text a.k.a. job description, while ignoring all the other moving parts in the equation. It amounts to nothing, without a more complete, all-encompassing, underlying context. Case in point (since Frost brought up the code push in this thread already), here is the operative question you should ask:

If this admin description had existed prior, would it have stopped or consequenced the code push, in the way that it was done? If the answer is, as I believe, that it would have made zero difference, then that is the exact value of this thread, so far.

Personally, I'm relieved Frost is stepping down as admin, and I wish the best of luck to Freeyorp in his place. I think with o11c, the best approach is to harness that awesome energy and creativity which he possesses, while helping him to temper it with the realization that as a live project, the stability of the main server, and the retention of the playerbase, developers, and GMs (which makes it alive), is the absolute priority.

[Hello=)]

1) IMO Frost's description of admin role is 100% correct. Can't say it better. Admin is supposed to keep things running.

2) I have to admit o11c seems to make huge discounts to self. Let's remind he recently f...d up gameplay on main server with recent update due to inability to communicate matters and encourage people to test changes and then utterly failing to understand main server is not his personal testbed. Also, changing case sensitivity of accounts and messing up GM logs to a degree it screwed up GM duties due to "smart" speed optimization (sure, learning why server software flushes logs line by line is not for o11c, he is smarter than rest of world who are doing logging this way). Yet, for some reason o11c thinks discounts only apply to him and don't apply to others. Everyone else does not deserves discounts. So let's grill Frost's butt. That's what I call double standards.

3) I fail to understand why o11c behaves like if he haves (or should have) exclusive authority to define project future. To my taste, o11c haves BAD project management skills. He maybe good coder, but bad in project management and lacks understanding of very basic things (like what is "production server", "test server" and how they differ in management, how certain actions would impact project as whole, etc). And also bad in communicating matters. Btw, crappy project management can negate all good code and seriously harm project. Something to consider, isn't it?

4) When it comes to dealing with players, Frost is pleasant to deal with, helpful and if someone haves some personal tensions with him for more than 30 minutes - it's a really noteworthy "achievement". Sure, I managed to clash with Frost couple of times, but it always has been resolved in ways pleasant for both sides. I can see o11c haves some clashes with quite many people, not just some disagreements with Frost.

Then, o11c complains he have to do nearly all work himself. Sure, if you stick to such approaches, you'll end doing all work. Because such approach could be discouraging others. And to be honest I doubt one human can fully handle RPG creation at all. This is too large task. Server is just one part of it. On it's own, server code (almost) lacks value for anyone except author(s). Example: ManaServ. It has come to the fact ManaPlus no longer supports manaserv protocol at all. Due to lack of demand and cooperation between projects I guess. But it has been created "for good" (and maybe even haves better thought design, etc).

As for responsibility: I've seen really interesting example - on my own butt: it took me about 1.5 weeks and extra "pinging" to get onto "testers" forum group (to be able to read announces if there is something to chew on). While I can't blame admin(s) in such case, I can admit devs who complain on lack of testing and/or suggest to join this group could do a bit better in this regard. So it appears its overall state of things. Blaming just Frost for things like this would be unfair and double standards.

[blackrazor]

Tester, nobody is making double standards between Frost and o11c, but there are complex social dynamics afoot.

Frost, for his part, played THE pivotal role in bringing the correct people together, to change TMW history, which directly elevated o11c from the role of a talented coder firmly under the administration's thumb, to his current role as coder / admin, with a lot of power to push the changes he feels are needed or justified.

No one was ever under any illusions about o11c's ability / willingness to communicate and his judgement regarding social relations matters, so no one is judging him on that basis. o11c is the conjurer, making magical and potent (and sometimes disastrous) server incantations, deep from within his lab.

But if o11c is the conjurer, then Frost was the savvy politician, with the communication skills and charisma to unite many towards an uncertain but promising goal. But that power to convince others comes with a lot of feedback, if things don't pan out entirely as planned.

In any case, I don't think people are trying to heap additional consequences onto Frost. He might be, in fact, his own harshest critic. But regardless, the usefulness (or lack thereof) of a detailed administrative job description, in the absence of all the other required documentations of all the other relevant moving parts in the project, to be honest sounds more like Frost's Swan Song and his message for o11c, rather than a serious effort to fix some of the administrative issues via regulation and the written word.

[Platyna]

Platyna created a strict hierarchy with her at the top. Rather than work with a team, she set people against each other and directly "managed" only the admins. As an admin, I was threatened if I didn't enforce her policies. When someone had too many conflicts with her, she got that person removed. I think we left that dictator model behind when we left her hosting service.

What a blatant lies. You left behind not only my hosting services but also decency and integrity. Name one person I removed because they were disagreeing with me.

And I threatened you, indeed - when you along with GMs were trying to change the rules behind my back and you were lying on an IRC channel that I DDoSed/hacked you.

Such person isn't worth a spit. That's why it is so sad to see TMW in hands of someone like you.

Regards.

[o11c]

When I started playing the game, the admins did many things well, but we also had:

• game crashes that were blamed on things beyond the control of admins, and never solved
• admin who actually told players that the game is "pre-alpha" and therefore they had no basis to complain about anything, but just be grateful for whatever they get

Only with the big code update could I finally call TMWA "alpha". The release quality of a piece of software is not driven by deadlines, popularity, or what you'd like it to be.
Fixing those crashes and related things was the primary purpose of the big code update.

• rollbacks of game data where sometimes we lost days of play, because...
• backups were unreliable, and admins did not take responsibility to fix the situation

Do I need to remind you of the time this happened while you were admin, after you told me to take down my backup system?

• official admin policy was that if you don't send email from the address registered to that account, you won't get help

Which is quite a reasonable policy. If password resets were fully automated, that would be enforced by code.

I advised that the code push in September was risky, and requested certain things (documentation of the changes, agreement about certain changes to GM commands, more testing, upgrade procedure, rollback procedure) before we pushed. You pushed the changes without doing those things, and since then most of my concerns became true.

I delayed the update by about a month trying to satisfy your concerns, which put it at a time when I had less free time to support the update. Eventually, it became clear that your concern was not "how to make this update happen safely" but "how to prevent there from every being any more code updates".

I provided documentation, participated in the conversation about GM commands (and provided some solutions to the problems being raised), I offered to list the entire upgrade and rollback procedures, and you refused to hear them.

And let's not forget that I did ask for your agreement to go ahead, and you gave it.

I was left trying to support untested code on a server that crashed using tools like tmwa-admin with undocumented changes

Poor wording ... tmwa-admin never crashed, and I documented that tmwa-admin's documentation was wrong (and always has been wrong).

following the documented restart procedure that I didn't know was wrong

What documented procedure? I don't remember any documentation at all for running the main server, we always communicated privately (and the procedure still used on the main server is exactly the same as we'd been using for months before the big code update, with the possible exception of the binary name change, which is entirely optional).

helping players figure out what case their login was (now that the server required strict case sensitivity)

Which was well publicized before the restart.

In particular, I think the admin is responsible to communicate with players. It's a dirty job, but someone has to do it.

IMO this Isn't *just* the admins responsibility at all. This should be for anybody working on the game including developers. ... GMs are also responsible for communicating with players as well to define rules and explain such to them or any other problems that occur in GMs hands. Everybody in TMWC has a role to communicate with players, whether they do it or not is another story.

I go one step further. With the exception of password resets and technical troubleshooting, I think server admins do *not* have any reason to communicate with players - the devs and GMs do have reason, and do successfully receive communication with players.

Can someone give me an example of what a player would need to communicate with an admin for? There would be plenty of reasons if the admins were in charge of the GMs and devs, but that is not the case.

First this is more a hobby project, right?

I think that attitude is harmful - it has caused many problems in the past.

Remember, I do blame the older generation of TMW devs for the manasource split and for letting Platyna rise to power. ... but both of those things are in the past now.

Secondly, I think that raising the question of what each role should and should not do might raise the question about division of responsibilities in gray areas, and hence stress those theoretical problems before they become real problems.

I just follow two rules in those cases:

• Don't be the one start a problem. No one is perfect, and we're still recovering from being under Platyna, so if we spent all of our time criticizing each other we'd have time for nothing else (though we're not far from that now - I was forced to make a formal complaint after I was unable to execute my duties due to interference).
• If in doubt over whether something falls under my responsibility, ask. If the TMWC agrees on something, it is okay to do.
• If I have no doubt that it is within my area of responsibility, always post evidence, so that the TMWC can choose to override.

2) I have to admit o11c seems to make huge discounts to self. Let's remind he recently f...d up gameplay on main server with recent update due to inability to communicate matters and encourage people to test changes and then utterly failing to understand main server is not his personal testbed.

I fully admit that I did not do a good enough job with the official testing group. I *did* however, post in the dev forum whenever I did the change, which in times past always gave me bug reports, and also performed a *huge* amount of testing myself.

At no point did I intend to use the main server for testing. However, once the update happened, I made an informed decision to not do a rollback because all the bugs that appeared in the update were simpler (albeit often more obvious) than the bugs that were fixed by the update and because (due to the nature of the update) a rollback would be a relatively expensive operation.

Also, changing case sensitivity of accounts

I hate to repeat myself, but ... that is the *one* decision I've made since we broke up with Platyna on which I have not been willing to negotiate at all.
In the past, people have lost access to their accounts because of the old behavior.

and messing up GM logs to a degree it screwed up GM duties due to "smart" speed optimization (sure, learning why server software flushes logs line by line is not for o11c, he is smarter than rest of world who are doing logging this way).

Some additional information about that. I did not *remove* a flush from the GM logs. I certainly hope the rest of the world is not doing what TMWA used to do, which was open and close the log every single time a line needed to be written.

Yet, for some reason o11c thinks discounts only apply to him and don't apply to others. Everyone else does not deserves discounts. So let's grill Frost's butt. That's what I call double standards.

(the word "discount" is strange here ... is this a language/region issue?)

The reasons I am grilling Frost recently instead of following my policy to not start a conflict are:

• Frost asked me to do certain things.
• I did them.
• Frost criticized me for not doing those things, even though I actually was doing them.
• Frost did not himself do the same things that he required me to do, even when I explicitly asked him to do them.

3) I fail to understand why o11c behaves like if he haves (or should have) exclusive authority to define project future. To my taste, o11c haves BAD project management skills. He maybe good coder, but bad in project management. And also bad in communicating matters. Btw, crappy project management can negate all good code and seriously harm project. Something to consider, isn't it?

I have never claimed or asked for exclusive authority for this project's future. I have asked on many occasions for others to take a more active role in that. A lot of times, I end up doing something just because no one else bothers to do it.

I fully admit that I am not good at communication. I really wish, however, that there was someone to cooperate with. I miss Jenalya for that.

As for responsibility: I've seen really interesting example - on my own butt: it took me about 1.5 weeks and extra "pinging" to get onto "testers" forum group (to be able to read announces if there is something to chew on). While I can't blame admin(s) in such case,

You are correct that, in general, this is not an admin responsibility.

It is possible that the problem affecting forum emails is related to this, but I kind of doubt it.

I would appreciate feedback from:

• one person who is not currently a member of the tester group, nor any group with special permissions (i.e. most forum members)
• one person who is a member (but not a leader) of the tester group, but not any group with special permissions (i.e. a tester who is not a developer)

regarding whether they can see

• who are the leaders of the testing group
• whether anyone is in the request queue for the testing group

Similar to password resets, we don't always remember to check these queues, and (unless there is an email problem) there is no way to get notifications, so the best way to get dequeued quickly is to message someone who can check the queue.


... but this has gotten fairly offtopic.

[Hello=)]

What a blatant lies.

Nope, these are not lies. IIRC, Doorsman left project after asking how to challenge your decisions. Then you've told your decisions can't be challenged. That's exactly how tyrant looks like. Furthermore, fact of appearance messages like this indicates quite many people considered you tyrant, up to degree you can't just stay silent about situation.

Tester, nobody is making double standards between Frost and o11c, but there are complex social dynamics afoot.

Yes. And I can admit o11c haves troubles to interact with quite many different people due to sticking to some strange or biased approaches. Half of issues occurred due to his own attitude. Failing to evaluate how your actions would be seen by others could be bad thing as it causes unnecessary aggravation.

to be honest sounds more like Frost's Swan Song and his message for o11c, rather than a serious effort to fix some of the administrative issues via regulation and the written word.

Everyone haves tolerance limits. Looks like Frost hit them. And I can admit it's not easiest thing to achieve, so there is something wrong goes on. And to tell things directly: I've seen some signs Frost is skilled system administrator. He appears to be ready to be in charge of server stability, security, etc (within his powers). I have some doubts someone else can readily take this over and continue the same or better. And o11c or Freeyorp are NOT system administrators at all. They are devs. They probably lack understanding of key things which are required to keep server running smoothly.

Only with the big code update could I finally call TMWA "alpha". The release quality of a piece of software is not driven by deadlines, popularity, or what you'd like it to be.

In no way successful RPG project is all about perfect server code. Server without features, content and tooling to conduct interesting events could be boring. Server without client is unusable. Server without good admin(s) is PITA. Server without GMs is chaos. Server without players is wasteland. And if you want to tell about "software quality", come on. But as QA I can add something to this topic and I can't promise you'll like it . Right now I can admit you fail to get idea software quality is a complex, multi-fold thing. Especially in large projects (RPGs could be good example here). Good code is nice to have, sure. But there is no universally accepted definition of "good code" (except counting bugs per kloc, which is not what you mean). And it's not only good code what makes program quality good overall. Other things matter as well. Great example is djbdns. It got high-quality code, at least in terms of security (means "quite few bugs overall"). Very nice PoC how to write secure program, especially noteworthy for using plain C. Unfortunately, it's worth of nothing as it lacks features people consider mandatory for DNS servers these days. Can I tell it's a good quality program? Nope, its good quality artefact to expose in museum and wonderful PoC. But quite useless thing when it comes to actual DNS servers. So it mostly unused, even if it haves some valuable properties.

Fixing those crashes and related things was the primary purpose of the big code update.

Map server still crashes sometimes (probably also losing some logs in process ). Maybe even more than before (it's possible to introduce new bugs when fixing old ones, etc). I counted about 4-5 map server crashes after update and I do not remember so much crashes before. Though without crash dumps I can't tell what code crashed, sure. Also, 4144 told there are still easy ways to crash server. That's "player visible" part. I think you should be more careful about statements when grilling someone for bad job.

I delayed the update by about a month trying to satisfy your concerns

Yet it still turned "production" server to "testbed". Time to admit Frost had a point, isn’t he? Skilled admins have idea about basics of QA (just to make own life easier).

Can someone give me an example of what a player would need to communicate with an admin for?

1) Reporting issues (like insane CPU loads, packet losses, etc).
2) Advanced manipulations with accounts where GMs lack enough rights. Say, renaming troublesome nicks, adding people to manamarket, etc.
3) Handling some misconfiguration/malfunction which does not lies in the code but rather system configuration related.
4) Advanced investigations where GMs lack access to required data.
5) Dealing with high-profile abusers or attacks which are too powerful for usual GMs to handle. Like advanced bans of offenders by IP range, blocking some exceptionally nasty or troublesome things on firewall way before it can reach any program and cause issues, etc.

There would be plenty of reasons if the admins were in charge of the GMs and devs, but that is not the case.

At least sometimes I've seen Frost doing some things for GMs. For example, Frost gone through logs for GMs, etc.

I fully admit that I did not do a good enough job with the official testing group.

IMO testers should be somewhat ashamed of such releases as well. That's how it feels for me.

I *did* however, post in the dev forum whenever I did the change, which in times past always gave me bug reports, and also performed a *huge* amount of testing myself.

As I told, for me it appears you don't have real idea how much testing your changes would require. Generally, there is need to retest EVERYTHING. It have to be team of several players playing for a while, doing most quests, playing different classes, etc. Sure, it looks well above of what TMW could afford right now. But if you can be in perfectionist mode for code, let's tester go perfectionist mode as well. Then, if it's okay to postpone some removed features reimplementation to "later", I don't see why it's not to okay to postpone code release to "later" as well, until it actually tested (and ideally on par in terms of features). If you insist there is no time pressure and start talks about quality, that's what you get from tester's side .

a rollback would be a relatively expensive operation.

Firms my view about bad project management skills. Anyone who cares how project performs overall, should consider such outcome is possible and prepare fallback route if things go badly. System administrators usually could greatly help with it... if communicated properly and maybe provided some assistance. On it's own, reverting troublesome updates from production is usual practice. And it's not admins to blame for versions incompatibility and one-way tickets, obviously. It's lack of responsibility from devs to roll out incompatible things, which make it harder to do rollback. Btw this is what exactly forces "stable" linux distros to lock software versions between major releases, only backporting most daunting fixes. Software authors could be really irresponsible, so it proven to be bad idea to use software from authors directly on alive servers. Then there was strong demand for someone to take over this process. Maintainers appeared as some groups which would solve this issue and help users and administrators to run things smoothly for a while. The obvious downside is that downstreaming new code takes a while.

I hate to repeat myself, but ... that is the *one* decision I've made since we broke up with Platyna on which I have not been willing to negotiate at all.

Does not removes the fact it possibly was not good for project overall. You gained little (from player- or admin-visible sides) but lost some players.

In the past, people have lost access to their accounts because of the old behavior.

I agree case insensitivity is weird, but after fixing it, many players faced login problems. I don't see gains which can outweigh such damage. Though I can agree it could be "necessary evil" in long run. IMO you should try to avoid touching anything login-related, its delicate thing which can easily make a lot of aggravation/misunderstanding/damage to project.

Some additional information about that. I did not *remove* a flush from the GM logs. I certainly hope the rest of the world is not doing what TMWA used to do, which was open and close the log every single time a line needed to be written.

Now go and see how they do it and understand why they stick to "suboptimal" ways. You'll usually see something like this in serious server software around. There could be something like fsync() instead of close(), which is also bad in terms of performance. Maybe you failed to get idea, but I can repeat: this "strange" behaviour is intended to get logs written to disk immediately. Up to degree where physical sectors on disk should contain actual file data (remember, OS buffers file writes). With such approaches logs would contain up-to-date information and if there is crash, this approach minimizes sensitive data (logs) loss. At most 1 line could be lost (where program or system actually crashed) if logging implementation is correct. In logs data integrity counts over performance (also, synchronous flushed write of single line supposed to take sane amount of time and unless server is badly maintained or writes enormous amount of logs it should not cause major issues). When there is not enough I/O speed on heavily loaded servers, common practice is to log via network to dedicated server which can afford synchronous writes. However it only required on really heavily loaded servers which generate huge amounts of logs and have I/O busy by other tasks. Btw this is really basic knowledge of file system operations, somewhere between POSIX filesystem access api and system administration. It's about getting idea about buffered writes, flushes and amounts of data loss upon crash. The problem here is that buffering haves potential for extremely high data losses, which is not acceptable for things like logs.

(the word "discount" is strange here ... is this a language/region issue?)

Okay, if you insist, in our region we have far stronger term to describe this effect: "own sh*t does not smells". Unfortunately, this syndrome is common for programmers, so they often fail to get idea why people do not share their views. It's hard to blame self. This is well known fact.

I have never claimed or asked for exclusive authority for this project's future.

But you often behave like if it was the case.

A lot of times, I end up doing something just because no one else bothers to do it.

And quite many times it's because nobody else motivated to do it. And if you'll handle things like it has been with rules and Nard, server administration and Frost, etc - sure, you can end up doing even more of work yourself. Not to mention that it's impossible to be good in everything.

I fully admit that I am not good at communication. I really wish, however, that there was someone to cooperate with. I miss Jenalya for that.

I can understand you in this, cooperation is fun. And to be honest, I really prefer to tell you something like "thanks for your hard work" rather than what I have to type here. And btw, I really doubt someone haves sole purpose to clash with you or make you unhappy. There could be some misunderstanding, but it usually happens when at least one side is uncooperative and not in mood for consensus.

[Frost]

t3st3r summarized my thoughts very well.
I have years of experience managing servers. This basically means evaluating risk vs. benefits for each change, communicating about changes, and coordinating what happens on the server. That includes platform changes, application software changes, content changes, usage patterns, and trying to keep expectations aligned with what is delivered. At this point in TMW, I lack the authority to control changes and the information to prepare for changes.

I could dissect o11c's arguments point by point, but I think this is central to how my vision of the game and of admin responsibilities differ from his:

Only with the big code update could I finally call TMWA "alpha". The release quality of a piece of software is not driven by deadlines, popularity, or what you'd like it to be.

Alpha is very early in a product lifecycle. It's when you finally have enough stuff put together to begin testing. It's understood that there are serious problems in the code, which is why you don't trust it yet.
http://en.wikipedia.org/wiki/Software_r ... life_cycle
http://www.techterms.com/definition/alpha_software

I think that alpha-quality code should be tested on private servers until it's stable enough to be useable and progress to beta. Beta code would go on the public testing server until there's enough confidence that it won't destroy the live game server, and the changes are understood enough to put on the live server.

Many people have invested many many hours in The Mana World. I'm sad and frustrated to see the new dev/admin goal is to push unstable and untested code on the live server, knowing that years of game data could be lost.