Discussion: Testserver setup

[Jenalya]

I just read the discussion in IRC about the testserver and have a lot to say in response, so I decided to post in forums instead. This might also help to make the discussion more clear and structured.

It seems to me that we have two different ideas of the purpose of the testserver:

o11c, you said you think the testserver should exactly track mainline/master. Did you say why? (If so, I oversaw.)

My idea of the testserver is that it can be used to test unfinished content before it is added to mainline.
tux9th's new map are a great example of why I think this useful/needed, but the idea can be applied to any content: he created his maps, but there might be a number of mapping errors which are very cumbersome to find alone. So putting them on the testserver is a way to make the new content comfortably available to testers, find remaining issues and fix them before pushing the content to mainline repo.
If you say commits need to be ready for mainline before being added to the testserver, you have a deadlock:
content needs testers for getting ready, but for getting tested it needs to be ready.

I don't think that having several instances of the testserver each with commits from different forks is helpful, because it adds complication for the tester. The original idea when setting up the testserver together with Frost is to have a comfortable and uncomplicated way to access content to be tested. This way we can reach the broadest testing audience. Any complications as having several instances of the testserver, the users needing to start the client with local client-data, or anything like this, is in contrast with the original idea behind the testserver.
If meeting this requirement of making it easy for testers results in additional complications on the side of developers or hosting, this should in my opinion be dealt with in the backend, and should not affect the testers experience.

The problem that caused this discussion is that when adding content to test from different forks, there might be merge conflicts, but the automated script to restart the testserver brought the servers up even if there are unmerged files.
I propose to adapt the script to abort with a meaningful error message in case of a conflict.
The person doing the restart would then have to manually resolve the conflict and then restart properly. That's how I worked so far when adding content from forks (though I usually rebased the changes from the forks onto mainline/master).
If someone's restarting the testserver who doesn't know how to do that, this is a great opportunity to learn how to do that.

[Crush]

maybe there should only be a commit on the test server data files when a test was successful. Testing a feature on the test server could then work like that:

1. test server pulls from mainline
2. developers upload changes to test server, but don't commit them
3. testers go testing
4. when errors are found go back to step 2
5. when the test was successful, a commit is made on the test server
6. mainline pulls from test server

[Kandiman]

Although I haven't been very active myself lately, I have been watching the development of this topic with some interest. After some offline discussion with Frost, I figured I would throw my 2 cents into the pile.

I do see some merit to having perhaps 2 testing server instances: an alpha and a beta.

The alpha server would be used for testing things which require a group of developers working on the same project, but is not ready for public testing. A separate repo could be tracked for this. If client updates are needed, a secondary update server could be hosted.

The beta server would be used for public testing to determine if content is ready for deployment and could track a testing repo. Once approved, the changes committed to the testing repo could be pulled to mainline.

In both cases, the servers should not have any interdependence on each other. Changes to one, or a restart, should not affect any content or availability of the other.

On a somewhat related note, I have privately offered to Frost, and now publicly to the content group, to host updates again. If multiple instances are required, we need only specify unique DDNS names for each (such as tmw-alpha.no-ip.org and tmw-beta.no-ip.org) and configure the test server to point to the appropriate update server.

[o11c]

As I see it, there are 2 contradictory purposes for the test server:

• Widespread testing of changes that are believed to be complete and ready for mainline, to see if anything was missed (master branch). Maybe this could be called "QA", and have a fixed time or number of testers required before applying to the main server.
• Narrow (but multiperson) testing of things that are being prepared for mainline. This somewhat overlaps with what a testserver on your local machine *can* be used for, but is more convenient.

It would be easy to specify in each '[TEST] topic which of these servers contains the content. This would often start with the experimental server and then change to the QA server.


The o11c-restart.sh script will abort if any subcommand fails - including the automatic merge. I "fixed" the impossible condition that was happening where half the time the map-server didn't start. (the old-restart.sh script was also vulnerable but less likely to exhibit the bug)

Currently updates are only pulled for the main server (which in my plan tracks the repository directly); this could easily be extended to the alternate but I'm not convinced that it's a good idea.

Currently, restarting kills all the servers on the same user at the same time. This could be changed by , or when I write a proper monitor (you would say things like "monitor restart main" "monitor restart all", "monitor stop jenalya")

The only downside that I see is that currently updates can only be specified per login-server, not per-char-server. I'm looking into whether the client can be tricked into downloading updates in 2 passes.
In the mean time, client updates can be set to pull from the experimental branches, rather than the master, since client data is usually just added. If more complex tasks are needed there's always -u -d

Note: the update-generating script may need to be modified to pull from a branch that can be rebased. This would be done as follows:
1. Generate updates from the main client-data as currently.
2. Generate a single .zip containing whatever is not in master but is in the experimental branch.

[Frost]

As I see it, there are 2 contradictory purposes for the test server:

• Widespread testing of changes that are believed to be complete and ready for mainline, to see if anything was missed (master branch). Maybe this could be called "QA", and have a fixed time or number of testers required before applying to the main server.
• Narrow (but multiperson) testing of things that are being prepared for mainline. This somewhat overlaps with what a testserver on your local machine *can* be used for, but is more convenient.

I see a subtle difference between those two purposes, and each can be hosted independently of the other. What is the contradiction?
The "QA" server is an interesting proposal. I don't know if we have enough testers to make it useful.

It would be easy to specify in each '[TEST] topic which of these servers contains the content. This would often start with the experimental server and then change to the QA server.

This adds complexity and confusion. Let's hear how much value the testers and content devs would see in creating an additional server and additional process for "QA."

The o11c-restart.sh script will abort if any subcommand fails - including the automatic merge. I "fixed" the impossible condition that was happening where half the time the map-server didn't start. (the old-restart.sh script was also vulnerable but less likely to exhibit the bug)

The script is undocumented. It fails without explanation if specific conditions are not met. The existing restart script has worked acceptably and reliably for the last year. The new script should be finished and approved by the team before we make it part of the team workflow.

Currently updates are only pulled for the main server (which in my plan tracks the repository directly); this could easily be extended to the alternate but I'm not convinced that it's a good idea.

I don't know what you mean, but the testing server pulls updates independently of the main server.

Currently, restarting kills all the servers on the same user at the same time. This could be changed by , or when I write a proper monitor (you would say things like "monitor restart main" "monitor restart all", "monitor stop jenalya")

Needless complexity to solve a nonexistent problem. At present, different servers run as different users.

The only downside that I see is that currently updates can only be specified per login-server, not per-char-server. I'm looking into whether the client can be tricked into downloading updates in 2 passes.
In the mean time, client updates can be set to pull from the experimental branches, rather than the master, since client data is usually just added. If more complex tasks are needed there's always -u -d

The existing setup has independent testing servers running as independent users with independent configurations and able to use independent sets of updates. The different servers are for separate audiences.

Your suggestion adds significant complication for both devs and testers, and introduces risks and dependencies.
Where is the demand for this coming from?

Note: the update-generating script may need to be modified to pull from a branch that can be rebased. This would be done as follows:
1. Generate updates from the main client-data as currently.
2. Generate a single .zip containing whatever is not in master but is in the experimental branch.

That sounds like a lot of work for little benefit. It's true that the first time an "experimental tester" (whatever that means) pulls updates, it would be simpler to download just one file. However, that monolithic file will change every time there is any change to client-data. This will increase bandwidth and disk space requirements to simplify a process that is already transparent to the user.

[Jenalya]

I still don't see the point in having multiple testservers.
From a content developers point of view, I just want to have a server where I can put in content to test, sometimes before I put it into the mainline repo.

We could try a setup with two different independent testservers, one which we use as before, and one additional tracking only mainline/master. But I seriously doubt that this one would find much use.

[v0id]

I still don't see the point in having multiple testservers.
From a content developers point of view, I just want to have a server where I can put in content to test, sometimes before I put it into the mainline repo.

We could try a setup with two different independent testservers, one which we use as before, and one additional tracking only mainline/master. But I seriously doubt that this one would find much use.

In my opinion, the case in bold should not happen right before an official release, as the tested content is being validated against mainline. Hence no additional changes (even if they aren't supposed to interfere with the tested content) should be applied.

So it's only in this case that a second test server would be useful. However, the timeframe where a tested content is in its final validation is usually short, so, as Jenalya pointed out, this second server would find little use.

A simple rule like "No changes out-of-sync with mainline can be applied on the test server <insert duration here> before a planned release" would do it.

[Nard]

So it's only in this case that a second test server would be useful. However, the timeframe where a tested content is in its final validation is usually short, so, as Jenalya pointed out, this second server would find little use.

There was this kind of server when Frost made it available to us. The conditions of testiing made it pretty useless.

A simple rule like "No changes out-of-sync with mainline can be applied on the test server <insert duration here> before a planned release" would do it.

my orange towel! Did Big Crunch and his fellow ate it?
Seriously I find the testing server nice as is at the moment, and useful too: the number and importance of bugs after release have been considerably reduced. For players it is a success, and a pretty good one.

I noticed though Kandiman's proposal:

The alpha server would be used for testing things which require a group of developers working on the same project, but is not ready for public testing. A separate repo could be tracked for this. If client updates are needed, a secondary update server could be hosted.

I would call it a development server rather than alpha. Developers could find it useful to discuss about unfinished ideas invite some players sometimes... The question is: Is there enough to feed it, do you have time enough, resources enough to manage a 3rd server?

Nard

[o11c]

The o11c-restart.sh script will abort if any subcommand fails - including the automatic merge. I "fixed" the impossible condition that was happening where half the time the map-server didn't start. (the old-restart.sh script was also vulnerable but less likely to exhibit the bug)

The script is undocumented. It fails without explanation if specific conditions are not met. The existing restart script has worked acceptably and reliably for the last year. The new script should be finished and approved by the team before we make it part of the team workflow.

The o11c-restart.sh has exactly the same documentation as the previous restart script, because it does exactly the same thing.

It fails if any subcommand fails, and all subcommands are believed to print error messages in that case.

The existing restart script has not worked perfectly reliably and at times resulted in broken server data.

Currently updates are only pulled for the main server (which in my plan tracks the repository directly); this could easily be extended to the alternate but I'm not convinced that it's a good idea.

I don't know what you mean, but the testing server pulls updates independently of the main server.

I meant, in the context of o11c-restart.sh, the main testing server ("QA") as opposed to the experimental one(s).

The only downside that I see is that currently updates can only be specified per login-server, not per-char-server. I'm looking into whether the client can be tricked into downloading updates in 2 passes.
In the mean time, client updates can be set to pull from the experimental branches, rather than the master, since client data is usually just added. If more complex tasks are needed there's always -u -d

The existing setup has independent testing servers running as independent users with independent configurations and able to use independent sets of updates. The different servers are for separate audiences.

What separate audiences are those?

Your suggestion adds significant complication for both devs and testers, and introduces risks and dependencies.
Where is the demand for this coming from?

A significant mantra of software development is "don't break the repository". Any decent-sized project has an automated test suite that must be passed on every single commit. Since we're kind of multiplayer we are restricted to manual testing (even if we developed automated methods, they would too easily be used for evil). Then, there is a second stage of testing before a release.

Using a single server for pre-commit and QA testing is problematic (this seems to be the main point that people disagree on).
Using a separate login-server means that (although it does mean we can specify an alternate test server without technological changes) no one will use the one that's not on :6901.
Using a single login-server with two (or more) versions of the char-server/map-server solves this problem.

I believe, and seek concurrence with, the statement that it is not profitable to us, if people randomly go on the test server without reading the [TEST] topics.

Note: the update-generating script may need to be modified to pull from a branch that can be rebased. This would be done as follows:
1. Generate updates from the main client-data as currently.
2. Generate a single .zip containing whatever is not in master but is in the experimental branch.

That sounds like a lot of work for little benefit. It's true that the first time an "experimental tester" (whatever that means) pulls updates, it would be simpler to download just one file. However, that monolithic file will change every time there is any change to client-data. This will increase bandwidth and disk space requirements to simplify a process that is already transparent to the user.

copying and editing a single line to an update script is not terribly complex ... You certainly misunderstand what I mean by "one zip". I mean one zip containing at most a couple of maps and a couple of graphics, because everything else is already in the master branch and caught by the main part of the update script.
(I'd like to see a proper client-update-packaging script sometime, but that's low on the priority list, and from the sounds of it people would shout "there's nothing wrong with the one we have now" despite the obvious and subtle problems with the current one)

Currently, restarting kills all the servers on the same user at the same time. This could be changed by , or when I write a proper monitor (you would say things like "monitor restart main" "monitor restart all", "monitor stop jenalya")

Needless complexity to solve a nonexistent problem. At present, different servers run as different users.

Regardless of anything else, I *will* create a proper monitor with start/stop/status commands that can be used on groups. The current method using killall is horribly unsound; among other problems, it can (and does) result both in meaningless error messages and in errors that cannot be reported.

The missing part of my original line was "This could be changed by running the char-server/map-server on another user and connecting to the login-server on the main test user." It's apparent that there are more major concerns than the details of how the servers are restarted.

A simple rule like "No changes out-of-sync with mainline can be applied on the test server <insert duration here> before a planned release" would do it.

The problem is that we don't have that kind of plans. Content basically gets released whenever no bugs are noticed in it.

[Frost]

[edit: repeated argument removed.]
o11c, you can do as you please if you work alone. Before you make functional changes to a team environment, you must work with -- and listen to -- other members of the team.

[Jenalya]

During a conversation with tux9th, we had the idea that it'd be reasonable to alter the restart script not to pull from any repo at all. The person restarting the testserver will update the repository manually, which is, in case there is no content from forks, only a pull from mainline/master as it was done in the script previously. If there's content in forks that should go onto the testserver, this needs an additional merge.

[o11c]

[edit: repeated argument removed.]
o11c, you can do as you please if you work alone. Before you make functional changes to a team environment, you must work with -- and listen to -- other members of the team.

the o11c-restart.sh script does not contain function changes, only additions.

The split testserver setup (QA/experimental) is admittedly a change. If the previous discussion was not enough for you, this thread contains one. So can you please actually address the points of the argument rather than attacking me in a meta-argument?

During a conversation with tux9th, we had the idea that it'd be reasonable to alter the restart script not to pull from any repo at all. The person restarting the testserver will update the repository manually, which is, in case there is no content from forks, only a pull from mainline/master as it was done in the script previously. If there's content in forks that should go onto the testserver, this needs an additional merge.

That could easily be done by adding a --auto or --manual flag to my script, depending on what the default should be.

The problem with that is: on the occasions when I restarted the test server, I had full ability to resolve the merge conflicts I noticed. But I had no clue in what way they *should* be resolved. So I was left with 2 options: don't test my changes at all, or test on a broken server.

[Jenalya]

The split testserver setup (QA/experimental) is admittedly a change. If the previous discussion was not enough for you, this thread contains one. So can you please actually address the points of the argument rather than attacking me in a meta-argument?

The problem with that is: on the occasions when I restarted the test server, I had full ability to resolve the merge conflicts I noticed. But I had no clue in what way they *should* be resolved. So I was left with 2 options: don't test my changes at all, or test on a broken server.

As I already said in my first post, which arguments you chose to completely ignore, I think that keeping it simple for the tester/user should be priority, to make it accessible for as much testers as possible.
Having several instances adds in my opinion complication both for the testers as well as for the developers, with little benefit.

Regarding the situation of merge conflicts of which you don't know how they're supposed to be resolved, I dare to say that this situation occurs rarely, and if it occurs it rather shows flaws in the team's communication than a problem with the testserver setup.

[o11c]

So can you please actually address the points of the argument rather than attacking me in a meta-argument?

As I already said in my first post, which arguments you chose to completely ignore, I think that keeping it simple for the tester/user should be priority, to make it accessible for as much testers as possible.

That part was addressed to Frost.

I have addressed your concerns by just saying to post in the topic on which server things are. It's not a horribly complicated change on the tester's part to say "feature xxx is in experimental on map yyy" instead of "feature xxx is on map yyy".

Will you give agreement to my statement that people on the test server who haven't read the topics (or perhaps received some other method of communication from the developers) are not helping?

Regarding the situation of merge conflicts of which you don't know how they're supposed to be resolved, I dare to say that this situation occurs rarely, and if it occurs it rather shows flaws in the team's communication than a problem with the testserver setup.

Admittedly it's a communication problem, but use of a cleaner testserver setup avoids the problem entirely.

I informed you of the problem the first time I encountered it, but it was still present the next time I needed to restart the test server.

My setup is a small amount more work for the developers when comparing the best cases, but is less work when comparing the worst cases.

[Jenalya]

After discussion in IRC we agreed on trying a setup with two char-/map-servers running with the same login-server.
One of them will contain new things to test, which are announced in test topics, the other one will track mainline/master and can be used for release testing.

We're going to use this setup for a while and see if it's easy to use and if both servers are needed.