tileset order canonicalization

[o11c]

Background information:
Now that we're using CSV, compression of map data can happen at a higher level.

git is likely to use deltas between versions of a file, and *may* use deltas between two different files if they are similar enough.

the .zip is format is not capable of cross-file compression, but .7z is (calling it "solid compression"), and is supported by physfs since 1.1, which was released in 2005.

Unfortunately, cross-file compression is impossible if identical high-level data (which tileset is this from?) differs at a low level due to the order tilesets are imported.

---
What should happen:
Reorder the tilesets in every map file to a canonical ordering.

The collision tileset should come first so it gets tile IDs 1 and 2; any instances of 1 (the transparent tile in the collision tileset) should then be replaced by tile 0 (the global empty tile)

There should be a canonical ordering for every clime: desert, woodland, snow, cave, and whatever indoors versions we need. Every map in those climes should import every tileset of that clime, even if it does not use it. The exception is if the unused tileset is at the end of the tileset; in that case it can be dropped.

Tilesets that are map-specific, or nearly so, should be added at the end - such as the graveyard tileset.

Finally, all tilesets should have there name changed to something canonical. I suggest the exact name of the tileset (no capitalization or replacing underscores with spaces), minus the extension.

---
Suggested attack plan:

• fix tileset names
• Edit all maps to remove tileset they aren't using. To alleviate boredom, also canonicalize the exact name and visibility of map layers. I suggest "Collision" and "Objects" be invisible.
• Do stats to find out the most used tilesets, and put them first in their noncolliding sets.

  Code: Select all

  grep '<tileset' maps/*.tmx | sed 's/:.*name="/\t/;s/".*//' | sort -k 2

  Code: Select all

  grep '<tileset' maps/*.tmx | sed 's/.*name="/\t/;s/".*//' | sort | uniq -c

• Edit all maps to add and order the canonical tilesets

---
Future possibilities, not considered at this time:
switch all images to an uncompressed format (tga?)
split images into smaller files, so partial changes don't force large data updates.

[Crush]

Note that when a map references a tileset without using it, the client will still load it from disk and keep it in memory. This means longer load times and higher memory footprint for the client.

[o11c]

Note that when a map references a tileset without using it, the client will still load it from disk and keep it in memory. This means longer load times and higher memory footprint for the client.

Doesn't it already have it in memory because the tileset is used by other maps, though? Please don't tell me the client drops and reloads the tileset every time it's used by a different map ...

If the surfaces are set up correctly the tilesets will stay in the X server, not in client memory at all. This is much more efficient than decompressing it (twice!) and sending it to X every time a map using it is loaded.

The total size of all tilesets is 38 171 648 bytes. 38 MB is not enough to be pushed into swap unless you're screwed anyway, and the L2 cache is better served by letting it sit in one place in RAM until needed, than by doing all the work of uncompressing it, which involves touching at least 3 areas of memory.

[Crush]

I was just wondering. You are investing a ton of time and energy into tweaking really miniscule details, so I thought you might be equally interested in saving a few bytes and milliseconds on the clients.

Doesn't it already have it in memory because the tileset is used by other maps, though? Please don't tell me the client drops and reloads the tileset every time it's used by a different map ...

IIRC the resource manager drops cached resources which weren't used for a few minutes.

Edit: it's 30 seconds. See method ResourceManager::cleanOrphans().

[o11c]

I have this almost done, but am hung up on tall tilesets (_x2, etc) misimported as 32x32. Also, at least one animated tileset is imported once as nonanimated.

Using the external tileset feature will make these problems unlikely to happen. The client is supposed to support this; I will test it with 0.5.0; even if not it is a valuable tool in detecting this problem.

Currently I am using a new top-level directory, tilesets/. Feedback?

There are currently two tilesets that are (always) imported as extra-wide - Snow_tree_x3 and barbarians-yurt_x4. As I recall this caused problems with older clients - do we remember which? We still support mana 0.5

Due to the difficulty of rebasing past this stage, I request a lock on existing maps/ in client-data. (rebasing in the earlier stage was nearly trivial). I see that Jenalya has changed a couple tiles, but otherwise just added a map. (Adding a map will require a little additional work if I use external tilesets, but does not cause conflicts)

Additional problems beyond this scope:

• some tilesets mix a number of x2 or x3 features on an x1 tileset
• some tilesets appear to contain duplicate tiles.
• some tilesets contain tiles that should be implemented as transparency + the layer below.
• some tilesets have many unused tiles.
• no tileset should contain a fully transparent tile.

[o11c]

Tested and pushed. I have not fixed the bad tileset imports.
Offending maps/tilesets are:

Code: Select all

027-3 Crypt-Fringe_x3
027-4 Crypt-Fringe_x3
009-2 Pots_x2
027-2 Pots_x2
009-4 Woodland_x2
033-1 barbarians-outdoor
011-6 cave_ani_wetwall
009-4 cave_x2
011-6 cave_x2
028-3 cave_x2
011-6 woodland_indoor_x2
027-2 woodland_indoor_x2
009-4 woodland_indoor_x3
027-2 woodland_indoor_x3

[Jenalya]

Currently I am using a new top-level directory, tilesets/. Feedback?

I have no objections, but please update the wiki article with the mapping tutorial to reflect the currect status.
And on a sidenote... why do you ask here for feedback if you push your changes before giving people the time to reply?

Due to the difficulty of rebasing past this stage, I request a lock on existing maps/ in client-data. (rebasing in the earlier stage was nearly trivial). I see that Jenalya has changed a couple tiles, but otherwise just added a map. (Adding a map will require a little additional work if I use external tilesets, but does not cause conflicts)

These changes are from tuX9th, not me. His maps are almost done, so if you explain to him what to change to comply with the new system, those could get in, and then you can have your lock.

[o11c]

please update the wiki article with the mapping tutorial to reflect the currect status.

Done.

And on a sidenote... why do you ask here for feedback if you push your changes before giving people the time to reply?

Oops. I was too distracted by the euphoria of getting it done. That said, if I'd thought about it, the difficulty of rebasing means "get it in now" would have taken priority over "see if this is consensual" if I'd thought about it.

Due to the difficulty of rebasing past this stage, I request a lock on existing maps/ in client-data. (rebasing in the earlier stage was nearly trivial). I see that Jenalya has changed a couple tiles, but otherwise just added a map. (Adding a map will require a little additional work if I use external tilesets, but does not cause conflicts)

These changes are from tuX9th, not me. His maps are almost done, so if you explain to him what to change to comply with the new system, those could get in, and then you can have your lock.

For the map on which only one thing is changed, it's probably simpler to just redo the changes on my version of the map. However, the following instructions should still mostly apply, with the additional constraint that you should check that 'git diff' shows only a couple of tiles changed.

For a new map:
First, change the order of the tileset to match one of the orders of an existing map, by dragging the tabs around. You can use tool/list-tileset-orders for this.
Then, for each tileset used by the map, click the "export" button in the tileset pane. Navigate to ../tilesets/ and overwrite the existing tsx file. The -fixme versions are for _x2, _x3, etc. tilesets that are supposed to be imported as 64, 96, etc. pixels high, but were erroneously imported as 32x32. We will reject any new maps that use a -fixme tileset.
Finally, check to see that the tsx files haven't changed in git. If you somehow manage to hit a -fixme tileset, the name will likely change; keep the version from the repository (git status will tell you how to do this).

If you need to add a new tileset, obviously there won't be one to overwrite, and you'll need to 'git add' the tsx file afterward.

If you need further instruction, I should be on IRC tomorrow. Remember, 'git status', 'git diff', and 'git diff --cached' are your best friends. (git add --patch is pretty cool too)

[o11c]

Based on some experimentation, there isn't a problem with tiles wider than 64 pixels on mana 0.5.1. So, that's another possible future fix.