What are the reasons for lag in-game

Ask for help regarding any technical issue or report any bug or OS independent issues.
Post Reply
Speiros02
Novice
Novice
Posts: 111
Joined: 08 Nov 2012, 09:38
Location: Australia
Contact:

What are the reasons for lag in-game

Post by Speiros02 »

I'm aware that this is an ongoing problem, and might not be specific to my system, but I would like to get some ideas...as many as possible actually...as to what is causing the lag in-game. This would be great to see an exhaustive list as such, and then maybe AFTER that, a new post on how each one can be addressed...

So, what do YOU know of that causes lag in-game?

A name is to be chosen rather than abundant riches...Solomon's words (Pr.22:1)

[url=http://speirosmusic.tumblr.com]Speiros' music page[/url]

A.K.A "Speiros"...(not being smart...this name is speiros)

User avatar
o11c
Grand Knight
Grand Knight
Posts: 2262
Joined: 20 Feb 2011, 21:09
Location: ^ ^

Re: What are the reasons for lag in-game

Post by o11c »

There are a number of reasons, but it usually comes down to "your internet is crap":
  • You're on wifi.
  • Your ISP is dropping packets. This is especially a problem in parts of Europe, which due to business politics, has terrible backbone routing even though end-user internet connections have higher bandwidth than the US. This can sometimes be fixed by making your own tunnel, but I don't know where to put it.
  • The DDoS-resistant datacenter we're in might be fighting off an attack against a bigger target in the same datacenter.
  • Occasionally there may be spikes due to server load. I *think* I have fixed all of these.
Former programmer for the TMWA server.
Speiros02
Novice
Novice
Posts: 111
Joined: 08 Nov 2012, 09:38
Location: Australia
Contact:

Re: What are the reasons for lag in-game

Post by Speiros02 »

o11c wrote:There are a number of reasons, but it usually comes down to "your internet is crap":
  • You're on wifi.
  • Your ISP is dropping packets. This is especially a problem in parts of Europe, which due to business politics, has terrible backbone routing even though end-user internet connections have higher bandwidth than the US. This can sometimes be fixed by making your own tunnel, but I don't know where to put it.
  • The DDoS-resistant datacenter we're in might be fighting off an attack against a bigger target in the same datacenter.
  • Occasionally there may be spikes due to server load. I *think* I have fixed all of these.
lol, well I can't argue about my internet being crap...Australia is certainly behind the rest of the world in internet, due to the distance from other continents, and the small population. It's also hell expensive. Thanks for the points.

A name is to be chosen rather than abundant riches...Solomon's words (Pr.22:1)

[url=http://speirosmusic.tumblr.com]Speiros' music page[/url]

A.K.A "Speiros"...(not being smart...this name is speiros)

User avatar
Hello=)
TMW Classic
TMW Classic
Posts: 650
Joined: 11 Jun 2009, 12:46

Re: What are the reasons for lag in-game

Post by Hello=) »

Ahh, people. You can use traceroute, mtr or similar tools to get idea where packet loss/delay happens and measure it in technical ways, which would allow to distinguish "wireless crap" and other cases. Internet is complicated thing where many things can fall apart. Failure of any part of quite lengthy chain would make you unhappy. Use proper tools to get idea what wrong and troubleshoot it (if you can). If packets are lost or delayed on your area - your fault. If on ISP area - their fault. Some intermediate routers? Good luck to explain their owners it suxx, when you're not their customer, etc 8).
User avatar
Nard
Knight
Knight
Posts: 1113
Joined: 27 Jun 2010, 12:45
Location: France, near Paris

Re: What are the reasons for lag in-game

Post by Nard »

I also still have lags while I have a somewhat good average ping (160-210ms). They occur mostly when number of players gets over (about) 45. They can be repeated small lags (( less or about a second, with period of a few seconds), or spikes of several seconds ("rare").
"The language of everyday life is clogged with sentiment, and the science of human nature has not advanced so far that we can describe individual sentiment in a clear way." Lancelot Hogben, Mathematics for the Million.
“There are two motives for reading a book; one, that you enjoy it; the other, that you can boast about it.” Bertrand Russell, Conquest of Happiness.
"If you optimize everything, you will always be unhappy." Donald Knuth.
User avatar
veryape
TMW Adviser
TMW Adviser
Posts: 558
Joined: 06 Dec 2012, 12:08
Contact:

Re: What are the reasons for lag in-game

Post by veryape »

I havn't actually noticed network lags, but sometimes when i run to the GY alone it feels like it takes some time for the mobs to spawn. So i run there, see that there are no mobs in the initial part of the gy. I continue on in searching for the people i think have cleared them and end up in the middle of all the spawns and die.

Other than that I havn't had any kind of lag problems have happened often enough for me to think that it has to do with the server/location of the server.
Characters: veryape / Captain Dunce / Elvara / veryapeGM
Chay
Novice
Novice
Posts: 78
Joined: 02 Dec 2013, 07:56

Re: What are the reasons for lag in-game

Post by Chay »

veryape wrote:but sometimes when i run to the GY alone it feels like it takes some time for the mobs to spawn. So i run there, see that there are no mobs in the initial part of the gy. I continue on in searching for the people i think have cleared them and end up in the middle of all the spawns and die.
that's not lag, the mobs are killed only every xy seconds via a fixed timer, the spawn you see is just the usual (fixed timed) respawn after mob exitus, not triggered by anyone entering the map . (so in fact if you stay dead on the gy you can also see skels rise from the death and soon crumble to dust again
User avatar
Nard
Knight
Knight
Posts: 1113
Joined: 27 Jun 2010, 12:45
Location: France, near Paris

Re: What are the reasons for lag in-game

Post by Nard »

veryape wrote:I havn't actually noticed network lags,
Indeed, The lags I am speaking about have nothing to do with network. (nor my vido settings or any kind of computer overload from my side. I checked with only manaplus lanched, and also with different computers an OSes)
"The language of everyday life is clogged with sentiment, and the science of human nature has not advanced so far that we can describe individual sentiment in a clear way." Lancelot Hogben, Mathematics for the Million.
“There are two motives for reading a book; one, that you enjoy it; the other, that you can boast about it.” Bertrand Russell, Conquest of Happiness.
"If you optimize everything, you will always be unhappy." Donald Knuth.
User avatar
AnonDuck
TMW Adviser
TMW Adviser
Posts: 645
Joined: 02 Jan 2009, 04:19
Location: Catland

Re: What are the reasons for lag in-game

Post by AnonDuck »

Nard wrote:I also still have lags while I have a somewhat good average ping (160-210ms). They occur mostly when number of players gets over (about) 45. They can be repeated small lags (( less or about a second, with period of a few seconds), or spikes of several seconds ("rare").
The occasional lost TCP packet will do that. When TCP loses a packet the stream has to recover. TMW uses fast-recovery TCP options to make this happen as quickly as possible but it can still take some time to recover and lead to instability of the stream for a small time afterwards. "Peak internet usage" hours could easily explain the relation to online players and packet loss. The output of netstat -s corroborates this hypothesis There are many TCP recoveries happening and Quick-Ack mode is activated quite often:
Spoiler:
TcpExt:
61079 invalid SYN cookies received
1294 resets received for embryonic SYN_RECV sockets
73 packets pruned from receive queue because of socket buffer overrun
232 ICMP packets dropped because they were out-of-window
1 ICMP packets dropped because socket was locked
987714 TCP sockets finished time wait in fast timer
112558 packets rejects in established connections because of timestamp
6951080 delayed acks sent
9855 delayed acks further delayed because of locked socket
Quick ack mode was activated 359900 times
623224 packets directly queued to recvmsg prequeue.
116657 bytes directly in process context from backlog
28339978 bytes directly received in process context from prequeue
69796737 packet headers predicted
608904 packets header predicted and directly queued to user
71763295 acknowledgments not containing data payload received
161088344 predicted acknowledgments
5734 times recovered from packet loss due to fast retransmit
759392 times recovered from packet loss by selective acknowledgements
464 bad SACK blocks received
Detected reordering 751 times using FACK
Detected reordering 3574 times using SACK
Detected reordering 12 times using reno fast retransmit
Detected reordering 4581 times using time stamp
18699 congestion windows fully recovered without slow start
56364 congestion windows partially recovered using Hoe heuristic
59069 congestion windows recovered without slow start by DSACK
158575 congestion windows recovered without slow start after partial ack
575569 TCP data loss events
TCPLostRetransmit: 36484
5351 timeouts after reno fast retransmit
297671 timeouts after SACK recovery
14245 timeouts in loss state
1168700 fast retransmits
74423 forward retransmits
279472 retransmits in slow start
596494 other TCP timeouts
218 classic Reno fast retransmits failed
19929 SACK retransmits failed
3779 packets collapsed in receive queue due to low socket buffer
394485 DSACKs sent for old packets
3215 DSACKs sent for out of order packets
378572 DSACKs received
2058 DSACKs for out of order packets received
21143 connections reset due to unexpected data
887 connections reset due to early user close
10117 connections aborted due to timeout
TCPSACKDiscard: 275
TCPDSACKIgnoredOld: 2348
TCPDSACKIgnoredNoUndo: 144156
TCPSpuriousRTOs: 30125
TCPSackShifted: 1109285
TCPSackMerged: 2729677
TCPSackShiftFallback: 4091251
TCPBacklogDrop: 13
TCPChallengeACK: 1193
TCPSYNChallenge: 94
Another possibility is that the server is overloaded or stalling. Judging by the CPU load on the server it doesn't look like there is any sort of bottleneck in processing power. The map-server may be stalling somehow without using much CPU. There is no insturmentation in place to determine if this is happening. Knowing the code-base and o11c's attention to detail (and simply watching the machine carefully with an experienced eye) this possibility seems more unlikely to me than the first.
Head of the TMW Illuminati
User avatar
o11c
Grand Knight
Grand Knight
Posts: 2262
Joined: 20 Feb 2011, 21:09
Location: ^ ^

Re: What are the reasons for lag in-game

Post by o11c »

There *used* to be a bit of server code that stuck the process in D state every 1024 seconds, I changed it to fork so it could execute asynchronously.

There might still be a tiny delay, but I don't expect it to be visible. That said, I do still have plans to fix it by switching to syslog.
Former programmer for the TMWA server.
User avatar
Nard
Knight
Knight
Posts: 1113
Joined: 27 Jun 2010, 12:45
Location: France, near Paris

Re: What are the reasons for lag in-game

Post by Nard »

MadCamel wrote:The occasional lost TCP packet will do that. When TCP loses a packet the stream has to recover.[...]
Another possibility is that the server is overloaded or stalling. Judging by the CPU load on the server it doesn't look like there is any sort of bottleneck in processing power. The map-server may be stalling somehow without using much CPU. There is no insturmentation in place to determine if this is happening. Knowing the code-base and o11c's attention to detail (and simply watching the machine carefully with an experienced eye) this possibility seems more unlikely to me than the first.
On several occasions, I tried to find correlations betwen the "lags" I observed, manaplus ping and netstat logs (various time intervals). Unfortunately the "measures" were not precise and synced enough to show anything more than the most obvious ones, which were network related.
Packet loss probably occur in the following issue that I should have reported again along with lags: After disconnecting an account, it happens that the server does not receive the message and the char stays online. It can stay online several hours, the maximum i could see at the moment is about 12h, until I reconnect. I should perhaps call that client-server misunderstanding instead of packet loss.
In the same order of ideas, The mobs ghosting have "recently" been significantly reduced (I sincerely can't say when it occured).

About bottlenecks: It may be the time to test again playing configurations where it was known that they caused lags (Illia for example, before V0id made the "exploit" mod). If they still occur, It might be interesting to develop instruments on both client and server sides, but it would require to define properly what we call a lag and which points to look at.

@o11c: I never noticed (nor heard about from players) anything like about 20mn periodic. The time period is long enough though so we can have missed it. Glad to know that this issue is fixed anyway. Thank you.
"The language of everyday life is clogged with sentiment, and the science of human nature has not advanced so far that we can describe individual sentiment in a clear way." Lancelot Hogben, Mathematics for the Million.
“There are two motives for reading a book; one, that you enjoy it; the other, that you can boast about it.” Bertrand Russell, Conquest of Happiness.
"If you optimize everything, you will always be unhappy." Donald Knuth.
User avatar
Deimheucadh
Novice
Novice
Posts: 57
Joined: 27 Jan 2011, 11:13
Location: Yekaterinburg, Russia

Re: What are the reasons for lag in-game

Post by Deimheucadh »

here is my tracert output:

Yekaterinburg - Moscow - Poland - Stockholm - Frankfurt - Duesseldorf - Paris - Washington - Houston
200 ms each hip. what do you think, do i feel lags? :D
Attachments
tracert.png
tracert.png (7.76 KiB) Viewed 3938 times
User avatar
o11c
Grand Knight
Grand Knight
Posts: 2262
Joined: 20 Feb 2011, 21:09
Location: ^ ^

Re: What are the reasons for lag in-game

Post by o11c »

Deimheucadh wrote:here is my tracert output:

Yekaterinburg - Moscow - Poland - Stockholm - Frankfurt - Duesseldorf - Paris - Washington - Houston
200 ms each hip. what do you think, do i feel lags? :D
The 200 ms is a total, and 200 ms go get to the other side of the world is as good as you can get.

The troubling thing is your hop 2, which shows an average of 260 ms and a max of 411 ms, so you're losing over 50% of your packets there.
Former programmer for the TMWA server.
Post Reply