Forming a beta testing team

[Derpella]

I think we should sort everything out. Chaos isn't a friend of any project...

My idea:


1. One beta-testing team should consist of trusted people with minimal knowledge of coding. Not to code, to understand.
I believe Nard, Taree, Okur and me posses those qualities.

a.) Beta team should have knowledge of EVERY smallest snippet of code changed. Even economical changes closed to others.
It would prevent problems like Orange Towel issue or Summer Shirt Frenzy.
b.) No smallest change should be implemented untested.
c.) Server for beta-testers would be closed for others.



2. There should be another testing server, because small group cannot simulate economical and social conditions well. But it should be attractive, playing-focused, and dynamic. My vision:

a.)Double or tripled experience and drop rates
b.)PvP enabled besides Hurnscald and Tulimshar town areas.
c.)Killing players would give experience and drops from last victim's pick-up.
On server like this many players would happily participate, and by creating social and economical structures developers would be able to test economical and battle balance without endangering main server's players with game-breaking changes.

[Nard]

Bug tracking:

If I understood well there are actually 4 ways to report bugs. Order is not significant.

• IRC channel: irc.themanaworld.org#tmw-dev
• Forum: Community>Player talk:http://forums.themanaworld.org/viewforum.php?f=12
  tmwAthena server development>Content development: http://forums.themanaworld.org/viewforum.php?f=13
• Mantis: http://mantis.themanaworld.org/
• Manasource: http://bugs.manasource.org/ (Official Client)

I think this may be good for members of development team but completely unreadable for a an ordinary player.
More there might be considerable developers' time loss to locate the bugs.
I suggest one forum for bug reporting (and testing?), built on "GM Community Suggestions" model, with indications for bug report.
When a contribution appears to feature a bug, forum moderators could move it there.
Bug tracker is a powerful tool but needs registration and users need to be a bit more aware of the way game works. mantis should be cleaned up to allow easier reporting. (I summarize what I read in other threads)

[Crush]

@Derpella:

1. Why do you want a closed beta testing server? I fail to see any advantage over a public one, but a lot of disadvantages (Linus' Law). The only situation where I see the need for a private beta server is for testing holiday events to avoid spoilers.

2. The way you describe the public beta testing server (more exp, faster updates and pvp) it seems to me like it would be more attractive than the normal server to many people. The result would very likely be that a lot of players will only play on the testing server. This means essentially an unnecessary fragmentation of the player community and double administration and moderation work. Also, with the PVP rules you describe, the socio-economical structures would be quite different, nullifying the reason why you want such a server.

@Nard:
I could be mistaken, but I think you can enable guest posting on Mantis with a configuration option.

[Nard]

Second create a forum for testing, so the testing activities will not inflate the developers discussions too much and make them harder to read. One thread = One code or script portion= one testing task

I don't understand why you want a new forum for this. We use to create topics in the content development section which contain "TEST" in their topic, and describe what needs to be tested. What do you think can be improved on that?

I suggest that because I think testers should not have to look into developers discussions to find the job they have to do. Another reason is testers objectivity: they must have as least contact with the code writer as possible. This point seems to me very important when testing a new quest.

Third define a testing procedure: I suggest testers use char(s) with status similar to official server and char(s) with variable stats to be able to fit various situations.
a tester is assigned one task (or a sequence of tasks) by the content leader (or by a testers' leader if needed). If developers join the team they are assigned different tasks than those they worked on. No playing with unreleased items and no unrealistic status while testing is allowed or the test is biased. Initial conditions must be specified if needed.

The purpose of this procedure (these procedures) is again tester objectivity and realism of tests.
!: It is (they are) supposed to be applied to a future development team and I have no right to say anything about any developer choices to test his own work.

[Jenalya]

a.) Beta team should have knowledge of EVERY smallest snippet of code changed. Even economical changes closed to others.

There are no closed changes (beside of events sometimes), changes can be seen in the repositories.

c.) Server for beta-testers would be closed for others.

As Crush said, I don't see any sense in that. The sole purpose of setting up a public testserver was to make testing possible for everyone, especially people who don't have the technical knowlegde to set up a local testing server. If we want testing exclusively by certain people we wouldn't need a public testserver.

2. There should be another testing server, because small group cannot simulate economical and social conditions well. But it should be attractive, playing-focused, and dynamic. My vision:

a.)Double or tripled experience and drop rates
b.)PvP enabled besides Hurnscald and Tulimshar town areas.
c.)Killing players would give experience and drops from last victim's pick-up.
On server like this many players would happily participate, and by creating social and economical structures developers would be able to test economical and battle balance without endangering main server's players with game-breaking changes.

I'm unsure about this - wouldn't it just create a different "main" server?

I suggest that because I think testers should not have to look into developers discussions to find the job they have to do.

But having dedicated topics for the testing would serve this purpose, wouldn't it?

[Wombat]

This beta testing team isn't getting any special privileges. Just test and report bugs. It isn't hard and there is no need to add to the infrastructure to make this happen. Communicate with Jenalya and Frost about beta testing on #tmw-dev.

[Derpella]

This beta testing team isn't getting any special privileges. Just test and report bugs. It isn't hard and there is no need to add to the infrastructure to make this happen.

Wombat, those things I wrote down show why I want testers team to be "special". Those aren't privileges: we wouldn't get any rights or prospects in main game. And solid planning would prevent things from getting messy and chaotic, information from getting split and incomplete, and in the end the whole commotion from quickly becoming useless. It's not about getting pink or Venetian Beige nicks on forum. We want to help, but I don't want the whole idea getting ridiculed in chaos and bad planning.
And right now, as Nard's post shows, the "no infrastructure" system is not working.

There are no closed changes (beside of events sometimes), changes can be seen in the repositories.

I already heard of few changes that happened from nowhere and were made by nooneknowswho.

As Crush said, I don't see any sense in that. The sole purpose of setting up a public testserver was to make testing possible for everyone, especially people who don't have the technical knowlegde to set up a local testing server.

"Economical" spoilers. If things that would change the economy will go untested because it would be dumb to give such info to anyone who wants it, that will lead to another catastrophes. Just knowing about the change isn't enough- it would be giving opinions, not testing.

If we want testing exclusively by certain people we wouldn't need a public testserver.

I don't mean to close the current testing server, just to make one being exactly the main one two weeks in the future, with changed prices included.

The way you describe the public beta testing server (more exp, faster updates and pvp) it seems to me like it would be more attractive than the normal server to many people. The result would very likely be that a lot of players will only play on the testing server.

This is the most harsh critique of TMW I have ever read. If you think a very big part of players would abandon their chars they leveled by thousands hours of pressing buttons, and items they collected for years, to move to a server on which they would have to start again, it's like admitting some kind of failure.
And yes, that was my point in some parts: 10 testers won't ever create a community that would simulate the main game well. And the changes? I want it to be fast and intense, just like scientist don't test some things on chimpanzees, but on fast-reproducing fruit flies. Otherwise we would have nothing to observe. And among PvPers you would better see advantages and disadvantages of changes, with harsh "natural selection" they will easily change their chars to fit conditions after every rebalancing.

[Crush]

I got something for you to read, Derpella. It's the famous essay "The Cathedral and the Bazaar" by Eric S. Raymond. It sums up the development philosophy of the TMW team pretty well.
http://www.catb.org/~esr/writings/cathe ... al-bazaar/

When you want to work in a cathedral, and not on a bazaar, you might not fit well into the TMW team.

[Derpella]

Hmm... I see many things about a need to be transparent and to respect other's opinions. I can't see "despise your users and tell them to gtfo when they're reporting bugs", or "if you do things for free, any critique is a terrible ungratefulnesses".
And I don't really get what's so evil and microsofty in my ideas. Closed beta servers for testing economical things? Or making sure I won't be uninformed, and then promptly told to go away? Or maybe making everything neat and easy to do and understand = EVILNESS?

And some things are very, very chaotic. I remember once I spent few hours to install official client with manaserv support, googled for a long, long time without succeeding, then finally someone on IRC told me I should install it from git main branch, and in the end I discovered it wasn't that version. Very cute and bazaary.

[Wombat]

You just need to check in with Frost and/or Jenalya about what to test and test it. Communicate on irc #tmw-dev about this. Really, we use irc a lot and it helps to stay logged in to irc for extended periods. I typically keep my computer running for several days logged in to irc and restart only when it is needed.

[Nard]

This beta testing team isn't getting any special privileges. Just test and report bugs. It isn't hard and there is no need to add to the infrastructure to make this happen. Communicate with Jenalya and Frost about beta testing on #tmw-dev.

• Nobody asked for any privilege for testing, or I missed something.
• In my opinion testing is first directed to bug detection, but it is also your first feedback from external players with no kind of preexistent opinion. It can lead to discussion interesting developers which are not development. Testers are players, not developers; they should not be obliged to read all development history (of a questfor example) to know when they have to begin testing.
• To my experience IRC can fit bug reporting only if the person you try to reach is on line. There is information loss on it (persistence of bad luck?)
• a specific forum allows information sorting, and everything concerning a single subject can belong to a single thread.
• a specific forum allows heavier moderation rules than player talk and than no moderation irc.
• the extra infrastructure needed is not that heavy
• I cannot keep the computer i actually use for irc online 24/24: it is a multi user one, more I use several computers and several locations to connect here, on irc and in game. news groups would fit better to asynchronous communication. Irc is intended to synchronous ones.

[Wombat]

Nobody asked for any privilege for testing, or I missed something.

You missed something.

[*]In my opinion testing is first directed to bug detection, but it is also your first feedback from external players with no kind of preexistent opinion. It can lead to discussion interesting developers which are not development. Testers are players, not developers; they should not be obliged to read all development history (of a questfor example) to know when they have to begin testing.

It is up to you how you approach testing.

[*]To my experience IRC can fit bug reporting only if the person you try to reach is on line. There is information loss on it (persistence of bad luck?)

Create a log of the discussions that go online.

[*]a specific forum allows information sorting, and everything concerning a single subject can belong to a single thread.

I strongly disagree. Several threads will be needed to properly track testing, based on project and concern. We also don't need any new forums.

[*]a specific forum allows heavier moderation rules than player talk and than no moderation irc.

IRC and forum moderation can occur with heaviness or lightness. There is no difference other than medium and method.

[*]the extra infrastructure needed is not that heavy

We don't need extra infrastructure.

[Nard]

Nobody asked for any privilege for testing, or I missed something.

You missed something.

can you tell me which privilege was asked for please?
(with the exception of a closed testing server from Derpella, which is not really a privilege, I don't support this idea)

[Jenalya]

To my experience IRC can fit bug reporting only if the person you try to reach is on line. There is information loss on it (persistence of bad luck?)

It happens often that someone reports a bug on IRC e.g. while I am asleep. When I check the backlog in the morning, I see it and react. It works really good.

a specific forum allows information sorting, and everything concerning a single subject can belong to a single thread.
a specific forum allows heavier moderation rules than player talk and than no moderation irc.

I'd suggest the feedback forum for bug reports rather than playertalk. Topics about testing something specific fit to content development.

I cannot keep the computer i actually use for irc online 24/24: it is a multi user one, more I use several computers and several locations to connect here, on irc and in game. news groups would fit better to asynchronous communication. Irc is intended to synchronous ones.

In that case you might like quassel. There you can have a quasselcore which keeps the connection to the irc-server all the time and you can connect to it with a quasselclient.

[Nard]

log of the discussions that go online.

impossible for me, some players cannot even play while their browser is open.

Several threads will be needed to properly track testing, based on project and concern.

I have nothing against several threads for one testing operation if required

IRC and forum moderation can occur with heaviness or lightness. There is no difference other than medium and method.

I will not go further on this topic. I already said what I had to say.

We also don't need any new forums.... We don't need extra infrastructure.

I don't agree with you, and go on thinking, that a specific forum would be more efficient, if not to you (you would have one more to read) at least to testers and bug reporters.