Thorough Traceroute set - March 21

Last night around 8:30PM, I logged on and was seeing the sorts of latency spikes that have been occurring for the past couple months. I was looking earlier around 6PM, but they were not present at that time, so I cannot say exactly when it began. I ran a set of traceroutes to the four servers closest to me (I live in southern Ontario, Canada.) I had a previous thread up, but I posted a couple times and got no replies beyond the first, probably because the TR's were not run for very long. For sake of clarity and readability, I decided to make a new thread.

The four servers I tested were:
Canada East (Toronto): https://pastebin.com/3caXYmtH
Washington, D.C.: https://pastebin.com/7hxZ8Un8
Texas: https://pastebin.com/8XqhTeJz
California: https://pastebin.com/8krsFBie

As I would be playing on Canada East, that is the server I initially logged into, and the one on which I noticed the spikes. Although I have seen them on Washington and Texas in the past(never tried California previously,) this time I did not see significant latency spikes on Washington, Texas, or California, which seemed unusual given they are farther away from me. I saw occasional spikes, but it seemed to largely be when in my hideout, and not with the complete regularity that has defined this issue(while it is present, which seems to be evenings) since I started trying to play again a couple months ago.

However, while running traceroutes to these three US servers, I did often see significant packet loss roughly halfway to the server, sometimes as much as 40 or 50%! Given that I was seeing a relatively stable connection in-game, I found this surprising as well. In comparison, the traceroutes to the Toronto server (where I was seeing the rampant lag spikes in-game) showed 15-20% packet loss when I first tested, and only on the three final hops (including the destination server.)

As I was testing the three US servers (Washington, Texas, then California, in that order) I periodically logged back in to the Toronto server to see if I was still getting lag spikes there, as I found the lack of lag spikes on the US servers strange. While the lag spikes to Toronto did remain throughout the evening, they became less consistent as time wore on (I tested for roughly a 2.5 or 3 hour period, from 8:30 to just before 11:30PM.) I first tested Toronto around 8:30PM, and saw an average of 16% packet loss on the 3 final hops. Halfway through testing, I ran another TR to Toronto from Oriath, and this time saw only 4-5% on the final 3 hops. This coincided with the reduced frequency of spikes as the evening wore on.

While the spikes continued to occur to the same levels, and in the same rough proportion to each other, there were greater and greater periods *between* the spikes. When I began testing, the spikes were incessant, with essentially no time between them. When I ran my second TR to Toronto, around 10:20PM, there were beginning to be small breaks in-between the spikes, and when I finished testing California, around 11:30PM, the spikes were occurring seconds apart, though still fairly consistent.

There are three separate "levels" of latency that the spikes will go to. Mostly, it spikes from normal latency (30ms on Toronto or Washington, 40 on Texas, and 60-70 on California) to 100+. Semi-regularly, there will be higher spikes of 300ms or so, and perhaps every minute or two, there will be spikes of over 1000ms. As I mentioned, as the evening wore on, there began to be greater and greater periods between these spikes, though I still saw all three "levels" of them throughout the night. In comparison, I normally never see my ping spike more than 10ms during daytime when this issue is not present.

I hope this all makes sense, as it's a lot of information to try and organize and I've been pretty busy the past few months so my brain is pretty fried as-is. I'm gonna take a break to eat and then make a second post to try and explain some strange behaviour I saw at the end of testing, in the event it may be relevant. Please let me know when you're able to if this information is of any help, or if there are any further tests I can run for you, as I really want to do whatever I can to assist in getting this solved. As I finish writing this, I am just beginning to see occasional spikes of 60-100ms on the Toronto server, so it would seem that this is roughly the time that it begins to occur.

It seems that the issue is now more-or-less in full swing. I just finished running a TR to a Toronto IP that was hosting Oriath, and was seeing the same final 3 hops showing loss, of 2-3%. I'll run a few more over the next couple hours to Toronto and Washington for comparison, and post a complete set when I'm done.

For reference, this was my in-game latency bar at 7:10PM tonight:

at 7:45PM:

and at 8:00PM:

Alrighty, my apologies for not uploading this earlier, I haven't been sleeping well. I ran some more tests on the evening of the 22nd, both to Canada East and Washington. I also took some screenshots of the latency bar at various times, as it's kind of hard to describe through text the nature of the lag spikes but it seems like it might help in pinpointing the packet-dropping behaviour.

What's strange is that the issues with the connection to Canada East wind down usually around 11PM, but the issues halfway along the path to Washington remain in significant numbers (up to 33% at last test), even though the connection to Washington shows as stable while in-game.

Canada East: https://pastebin.com/GZ52AVbz
Washington: https://pastebin.com/jMnStgQu

To demonstrate the point I was trying to get at, here's what the in-game latency reading shows when the lag is worst:
8:30PM

9:00PM

Those peaks are massive spikes of like a thousand or more, but as you can see between them, the connection is never stable.
A few hours later, it becomes more intermittent:
10:50PM


The peaks in the first picture there are less random, and you can see that in between them are much more regular jumps up-and-down. In the second, taken minutes later, the peaks are much less severe as the baseline becomes reliable again. While there will still be occasional spikes of 1000+ around this time, it's not really in the sort of quick succession seen earlier in the evening. A short time later, just after 11:00PM:

It's more ore less back to normal, although still very jittery, only getting the occasional singular peak.

For comparison, here is the connection to Washington around 9:00PM:

and around 11:00PM:

That is despite the packet loss seen roughly halfway along the path to D.C. When switching from an instance hosted on Toronto to an instance hosted on Washington while in-game, the difference is stark(9:30PM or so):


On that note, I observed some unusual behaviour and I wasn't sure if it was normal, or possibly related somehow. When I entered my hideout on Canada East and opened up a set of portals to a map for testing, and then logged out and logged back in on Washington, entering my hideout or the portals opened on the Toronto server continues to connect me to Toronto IPs. Due to this, I was actually only able to test the connection to Washington from Oriath, as going anywhere else would return me to a Toronto IP (and returning to Oriath would connect me back to a Washington IP.) I'm not sure if this persistence behaviour is normal or not, as when I tested on the evening of the 21st(as shown in my original post in this thread,) logging out and back in from a new gateway would bring me to a hideout in the proper region, with the portals I had opened on the old gateway no longer there. I was getting a separate hideout each time I changed gateways, UNTIL I finished my fourth set of tests on California. After finishing up my final tests and returning to the Canada East gateway, I was connecting to a Toronto IP in Oriath, but still connecting to a California IP in my hideout or map portals. It was very confusing, as the behaviour only started after finishing up my final set of tests. When testing on the 22nd, however, this behaviour was the standard, and I observed it from the first time I hopped over to Washington. I feel like it may be due to the amount of hopping around I have been doing, possibly setting some sort of flag on my account, but I've never previously changed my gateways around like this so I have no prior experience to compare it to.

Again, I hope I'm being clear enough here, as I was hoping to post this earlier but was never able to get around to it. I can continue to run tests if it would help, I'd just need to know specifically what sort of parameters you're looking for, such as how many tests to run and what time intervals (every half an hour, hourly) as well as how many gateways to run tests to when I do them. It's kind of time consuming, but if it helps you get Cloudflare or any other upstream provider to listen, it's worth it.

Finally, if you're looking for other users in the area to post their own results, I would recommend making a sticky thread so they can see it before they each post their own personal thread seeking help. I'd be more than happy to re-post my pastes with a much more concise and simplified writeup as a sort of template for others to follow. If there's anything I can do, feel free to let me know!

I was waiting for a reply to see what would be most helpful, but as one hasn't been posted yet I'll just start dumping pastes of the results of traceroutes to IPs from Washington and Toronto, and see what we can figure out.

https://pastebin.com/ie7E05ch

"

Sunyata_ wrote:

I was waiting for a reply to see what would be most helpful, but as one hasn't been posted yet I'll just start dumping pastes of the results of traceroutes to IPs from Washington and Toronto, and see what we can figure out.

https://pastebin.com/ie7E05ch

Lastest WinMTR report doesn't show the problem anymore.....

Actually, there is still packet loss at the same spots, and to the same degree that it's been reduced, the problem is reduced but still present in-game. Path of Exile isn't really playable with random latency jumps, and so the problem is in fact still a problem.

I don't generally see you providing much in the way of support, Han. You mostly just seem to dismiss people's requests for it. I'm not sure what you're trying to accomplish.

"

Sunyata_ wrote:

Actually, there is still packet loss at the same spots, and to the same degree that it's been reduced, the problem is reduced but still present in-game. Path of Exile isn't really playable with random latency jumps, and so the problem is in fact still a problem.

I don't generally see you providing much in the way of support, Han. You mostly just seem to dismiss people's requests for it. I'm not sure what you're trying to accomplish.

Oh shit sorry, I only looked at the first one --> Canada East 7:05PM
where there is no packet loss.

But your WinMTR clearly shows that the problem can reappear with Canada servers, but not with the Washington servers. That could very well look like a performens problem.....

Hey no worries man I've just been dealing with a lot lately and reading my post again I feel like that was kind of rude of me.

The Washington TRs show loss halfway along the route, Ian said it was a problem with Cloudflare in Toronto, which makes sense as any internet connection in this area would end up going through Toronto on its' way to the US. I'm just trying to give as much specific data as possible to GGG so they can actually raise a stink with Cloudflare, or whoever else is dropping PoE packets. I imagine they're very overwhelmed with all of the recent issues plus the new patch, so if there's no reason to ask me to do anything else, I'm just trying to offer whatever I can.

he he no worries, I can take it :)

I have been "pissed at" / "looked down at", on work for the last 4 years because I want things done correctly. I just resigned and my boss got so upset and told me whatever I wanted, I could get if just I would stay. I declined.

So in about 3 weeks they have noone left that knows anything about their IT infrastructure, because my collegue (we were the 2 who made it all from the buttom on up) left 1 week ago......