ScrotieMcB vs Desync Dev Manifesto (Massive Wall Map of Text)

Before I start in on this, I want to say I love Path of Exile, I truly respect GGG as indie developers, and I am amazed by Chris's constant communication with the community here. I don't really believe in petty name-calling, so don't do it.

"

Chris wrote:

Spoiler

"Desync" is a very hot topic at the moment. At best it's a minor annoyance when it occurs and at worst it can cause characters to get killed in situations where they thought there were no monsters around. We have many changes coming that will substantially improve the situation, but would like to also explain how our synchronisation systems work in case you're interested, and to make it clear that game state synchronisation is a problem that all online games need to deal with.

In this article I'm going to try to clearly explain:

• How different types of online games handle latency
• How our system of action prediction works
• Why sync problems occur with this system and how they manifest
• Why desync has to exist and why rubber-banding is good
• Why some other games don't appear to have similar problems
• What we're planning to do to improve synchronisation

How different types of online games handle latency
Any game has calculations that occur to determine the result of actions. In RPGs, these can range from combat calculations (who did what damage) to important economic transactions involving game items. To prevent players cheating, it's important that these calculations are not done on the gamer's computer, because they can easily modify the result of such calculations.

Because of this, all calculations that affect someone's progress must be done on servers that we control. These servers exist all over the world (Texas, Amsterdam and Singapore, currently), but due to the speed of light and other physical limitations, it's not instant to send or receive data from them. We typically see response times between our players and the servers of around 50-250ms.

All online games have this situation. The server has to dictate whether things happen or not, but there's a 50-250ms delay before data gets to the server and back. There are three ways that games can solve this:

• Trust the client. This means people can cheat, but the results are instant. We will not do this.

Let's all take a moment and agree that trusting the client leads to all kinds of hacks, which would be problematic enough in itself but made even worse by bots, who can use the hacks themselves to augment their power... and thus this method is insanely stupid.

All on the same page? Good.

Except GGG does do this, in a limited capacity: map generation, or to be more specific map concealment. The entire map is currently sent by the server to the client when an instance first loads (loading screen), and then the client is trusted to reveal the map slowly as the player explores it.

This has, of course, been discovered and exploited by map hacks, which in turn allows bots to determine optimal pathing through an instance without the need to actually explore it.

I pointed this out nearly 2 months ago (thread). On the one hand, Chris himself responded to the thread amazingly fast; in less than two hours he managed a reply despite being at a conference at the time. On the other hand, it's been two months, and it's still a part of the game. Really, outside of replying in my little thread, GGG hasn't done anything.

They haven't even edited the little part where they lie to misinform the readers of the Development Manifesto, saying that they won't do something which they do. Considering what a hot issue desync is, you'd think they'd want to correct an error like that, so that the knowledgeable reader isn't thinking "bullshit" before they get too far into the article.

"

Chris wrote:

• Wait until data arrives back from the server before doing anything. This is a very common strategy in RTS and MOBA games. If you click to move, the unit will only start moving once the server says so, which is 50-250ms later. If you are close to the server, you'll quickly get used to the lag and everything feels pretty good. If you're far away (New Zealand, for example), it feels like you're playing drunk. Every time you issue an order, nothing happens for quarter of a second. This does not work for Action RPGs.
• Start predicting the result of the action as though the server said yes, immediately. When the server later gets back to you with a result, factor it in. This is what Action RPGs including Path of Exile do. It means that when you click to move, or click to attack, it occurs instantly and feels great. The problem is what happens when the server decides that the action can't have occurred - that's when the game gets badly out of sync.

Which brings us to Lie Inaccuracy #2. Yes, this early.

Maybe your genre assignment differs from mine, but I consider Diablo 3 to be an Action RPG. Not a very stellar one: lack of build commitment, stale itemization, auction houses, et cetera, et cetera. I know we don't like Diablo 3 around here, but the one thing I haven't heard a lot of complaints about is the fluidity of its play and how it handles latency issues. And, if you couldn't guess already, Diablo 3 waits until data arrives back from the server before doing anything. Which apparently can work.

Now I'm not a proponent of the "wait for the server" method; I have a rather strong preference for the "start predicting the result" method. However, I think it's important for people to really understand the decision-making tree here, to understand why the Diablo 3 method of dealing with latency can work, and why saying that it can't is particularly troubling.

The first decision is to not trust the client for determining the game state, because (just in case you forgot) that's insanely stupid; it wasn't a very hard decision, but a decision nevertheless. Not trusting the client means that the server version of the game state is the only version that actually matters. This in turn means that the client will never have a perfectly accurate view of the game state, due to the 50-250ms latency.

Which brings us to our second decision: while we are waiting on accurate game-state data from the server, which kind of lie do we want to feed the player? This is a necessary evil, unfortunately. At the time the client is required to report the game-state to the player's eyeballs, the client doesn't have it; perhaps it has the gamestate from 50-250ms ago, but that's not quite the same thing. Our choices are essentially the "wait on the server" method, where instead of seeing the game state as it truly is now you're continuously playing 50-250ms in the past, and the "predicting to fill in the gaps" method, where the game uses the game-state data from 50-250ms in the past to predict, to the best of its ability, what the true game-state is now.

The thing to remember here, though, is that both methods deliver fiction. This isn't a choice between truth and truth here, and thus the decision is largely an aesthetic one. At the end of the day, what is really important is the reliable delivery of game-state data that is as fresh as possible; the "wait on the server" method will wait less, and the "fill in the gaps" method will have to rely less on predictions because it will have more facts.

Thus, it's not just that "wait on the server" can work for a game like Diablo 3; if "wait on the server" would not work with PoE, then "fill in the gaps" would not work either. The choice made here is able to "cover for" poor latency issues. Which makes this a particularly disturbing lie; if GGG isn't able to implement "wait on the server," then what is going on with their current "fill in the gaps?"

edit: There seems to be some confusion along the lines of "if the wait-on-the-server method is suboptimal, and the fill-in-the-gaps method isn't working for us, then we should just go back to the trust-the-client method." Which is still, of course, insanely stupid. The fill-in-the-gaps method can work, and does work on a regular basis for GGPO, a netcode add-on that allows old-school fighting games (SF2, KoF, et cetera, which arguably have even tighter response issues than ARPGs) to be played PvP across continents in an environment which "provides the illusion of lag-free gameplay" (it's even honest with its customers, calling an illusion an illusion). Philosophically, GGPO is the same system as PoE's; it calls them "rollbacks" while PoE calls them "rubberbands" but they're essentially the same thing. Because of GGPO, we know that the fill-in-the-gaps method can work; the question for PoE becomes "why don't they?" (excluding the fact that PoE has more data than an arcade emulator)

"

Chris wrote:

Action RPGs have to use the third system (action prediction) to feel responsive. The problem is, the second you start moving, you're implicitly out-of-sync by definition. Your client has drawn the first few frames of movement (to be nice and responsive), but the server has no idea you clicked a button yet until the data arrives. Action prediction is mandatory for this type of game but results in you being slightly out-of-sync almost all of the time. This is generally no problem, but once too many predictions get made based on incorrect data, very bad things happen. The challenge is detecting and correcting the situation before this occurs.

This issue isn't exclusive to the "fill in the gaps" method; it applies to "wait on the server" too. The difference is that "fill in the gaps" lies by commission; when it desyncs it tells you things are in places they never were. "Wait on the server," on the other hand, lies by omission, continuing to report that things are where they were without updating the user on where those things are now.

Perfect sync is only possible in 0ms latency situations, but we don't need perfection. The only way to improve sync is to reduce latency, allowing the client to receive the full game state more often.

"

Chris wrote:

How our system of action prediction works
Let's say you're playing with 200ms round-trip latency and you click a monster that is 2 seconds of travel distance away from you. Assume your attack animation has its contact point after 300ms (which is where damage is dealt).

0ms: You click the monster. Your character starts running towards it on the client.
100ms: Your click arrives at the server. The character there starts running towards the monster also. At this stage your local character is already 5% of the way there.
2000ms: Your character arrives at the monster on the client. It's not there yet on the server. You don't even know if it'll ever arrive for sure (it might get interrupted by an attack still). Your client starts to animate the sword swing:
2100ms: Your character arrives at the monster on the server. The server immediately performs the combat calculation in advance of the contact point and sends the tentative result back to the client.
2200ms: You receive the notification from the server about what type of damage you will deal and roughly how much. Thankfully it arrived before the contact point of the animation! This is not always the case.
2300ms: You hit the contact point on the client. Because you have the damage information in advance, you can draw a pleasing blood splatter, fire effect and so on. This hit has not even occurred yet on the server.
2400ms: You hit the contact point on the server. The damage is locked in and actually applied to the monster. It dies. Experience and item drops are calculated and sent to the client.
2500ms: Your client receives an experience update and the information of what items to show falling to the ground.

I assume most of you have noticed that, even with less than 250ms latency, you can be 2500ms (or more) out of sync. This quote gives a window as to how: "You receive the notification from the server about what type of damage you will deal and roughly how much," "Your client receives an experience update and the information of what items to show falling to the ground." This isn't communicating the full game-state; it's a partial game-state, only the things that changed recently.

What this shows is that GGG is trying to use the "fill in the gaps" method not only to fill the gaps, but as an actual replacement for network communication. The server is only seeing fit to give the client the parts of the picture which it considers relevant, which leads to things like the position of a particular monster (or even the player) being ignored for seconds at a time.

Folks, if you really want to know why PoE desyncs so much, it's not actual network latency, and it's not the "fill in the gaps" method per se. It's actually a policy of minimalist communication between the client and the server.

The most famous example: you hover of a monster on the other side of a door, then use a skill on it by clicking or pressing a key. A few seconds later you've magically teleported into the room, surrounded by scary threats, and either die or come perilously close? What happened?

Well, when you used the attack skill on the monster, it turns out it wasn't where you said it was; it was actually in a corner of that other room. So your character wanders to the doorway (if ranged, for line of sight) or all the way over to that corner (if melee), then attacks it.

Most people just shrug here, but they should be asking harder questions. First off, why didn't my client tell the server what position I was attacking, in addition to which monster? If it had the attack position in addition to the attack target, couldn't the server tell I was way out of sync and interpret my command differently? Second, did I really go that whole two seconds spamming the skill and not receive a single update on the position of that monster? At the very least, you would think that attacking a specific monster would clue the server in on updated the position data on that monster, if not all monsters in general. So why did it take so long to get the correct position data?

And for that matter, why isn't the position data — I'm not talking monster frames or types of complicated things, I'm just talking basic x,y position coordinates on all players and monsters — updated with pretty much every single communication from server to client? Let's break it down: up to 256 identifiers for players/monsters, communicated previously, 1 byte; x coordinate and y coordinate both ranging from 1 to 65536 (if 5 is default one-handed melee range, about 500 screens each direction), 2 bytes each, total of 5 bytes per monster, which means a whopping 1 kilobyte with 100 monsters moving around. Is that too much bandwidth to inform the client about what's going on every 250ms or so? Would waste a whopping 0.25% of my bandwidth.

The way that the Path of Exile servers send information to the client is treated on a "need to know only" system of some sort. For certain larger pieces of data, this makes sense, but for the real basic stuff like positions — stuff that's extremely compact from a data perspective — it's completely nonsensical. It's a system with nearly zero redundancy even with the lightweight systems.

Therefore, massive desync.

"

Chris wrote:

Why sync problems occur with this system and how they manifest
This above example assumes that everything went smoothly. It's entirely possible for the 2 second travel time to be completely different on both ends, or for a lag spike to occur causing the timing to get completely out of sync. If the attack is interrupted on the server before it starts (during movement) but not on the client, then you have a long animation playing that can't be cancelled because the communication time is a decent length of the animation.

Even if no strange lag occurs, the monsters that are nearby are pathfinding on the client to where they think you are - which by definition is different than on the server because of latency. These entities have to find paths that go around the other monsters, which of course are in subtlety different positions on both ends. The differing paths further contribute to the monsters being in the wrong place.

It's worth stressing that in 99% of combat events, everything feels fine. Although the simulation is out-of-sync due to the speed of data transmission, the timing generally works out and monsters who are following weird paths get to you at roughly the right times and in roughly the right places. It's hard to really know that anything's wrong... except when it's horribly wrong.

Unfortunately, when things are very out of sync, players have a pretty bad time. They take damage out of nowhere or find that they're actually trapped between monsters that didn't appear in the right places on their client. We have code to detect these situations and hopefully resync (rubber-band) the entities back into place quickly, but it's often not good enough.

Once again, this is evidence of an on-demand system, rather than a routinely-delivering-required-information system. Properly identifying what the client needs to know and when would mean that a backup system that makes specific requests would be a failsafe, not a required crutch.

"

Chris wrote:

Why desync has to exist and why rubber-banding is good
The key thing to understand is that Action RPGs have to use an action prediction system like this. If they wait for confirmation of every action from the server then it feels terrible to control.

Even if our resyncing code was perfect, there would be situations where the game gets out of sync just because of tiny timing differences. Imagine you're running near a large rock, and you arbitrarily click on the other side of it. Both the client and the server attempt to find the shortest path around the rock. Because your client is ahead of the server by definition (as the movement was processed there approximately 50-250ms earlier, so that it was responsive), there are cases where the client may choose to go a different way around the rock than the server. If you were hit by a monster en-route, then your movement will be interrupted in a different place on both simulations. You are now out of sync. Intelligent resync code would detect this and rubber-band you across the rock back to where you're meant to be.

If the resyncing code was perfect, small data like position would be delivered so frequently that it would have to be a rather small rock for it to not resync much earlier. Technically, that would still be a rubberband, but no "intelligent resync code" would be necessary; instead, very dumb, repetitive resync code would make the issue moot.

"

Chris wrote:

The key observation here is that improved resync code involves more rubberbanding than we have at the moment. If we do it properly, monsters and players will be corrected to better positions more frequently, to prevent anything getting drastically out of place.

This is the single truest part of the entire article. No nonsense about "intelligent" code, implying it makes decisions; instead, "improved" and "more frequently." This is what needs to happen.

"

Chris wrote:

Many players interpret the rubber-banding itself as "desync", when in reality it's what is fixing the problem as it is detected. It's not going to be easy explaining that the increased rate of rubber-banding is not only good, but also the ideal solution.

A smart player interprets multiple small rubber-bands as a good thing, and sporadic, long-distance rubberbands — that which PoE feeds us regularly — as a symptom of a disease.

"

Chris wrote:

Why some other games appear to not have similar problems
Games using the "wait until server responds" method (RTS and MOBA games) have much higher input latency but don't have the same sync issues that we do. They have their own class of game state synchronisation problems that we thankfully don't have to deal with.

Games using client action prediction like ours run into exactly the same sync issues that we do unless they cheat on certain aspects of the simulation. For example, it's common for Action RPGs to do some combination of the following:

• Entities can hit each other from a long distance away
• There's no chance to hit - all hits occur for sure
• Various speed/collision concessions that make it easy to speedhack and/or walk through monsters with modified clients
• Attack animations cannot be interrupted (i.e. what we treat as Stun).
  

Unfortunately, we don't want to do any of those things! They each individually ruin part of the hardcore experience: by allowing combat/movement cheats, preventing accuracy from existing as a mechanic, prevent stunlock, preventing people getting blocked in, etc.

Due to the fact that we want to have hardcore game mechanics (i.e. ones where position matters and it's difficult to cheat in PvP), the only option for us has been to put a lot of work into improving our combat simulation and resync code.

And here I agree with the manifesto whole-heartedly. There are plenty of ways which GGG could improve the desync issue without conceding interesting mechanics like there; for example, fix any of the numerous issues listed earlier in this post.[quote="Chris"]What we're planning to do to improve synchronisation

There are a lot of changes that we're experimenting with that may individually improve the synchonisation of the combat simulation (along with their potential drawbacks):

• Have monsters on the client attack your server location rather than client location to reduce entropy. Maybe compromise on them attacking a mid-way point between the two. The drawback here is that it means they'll appear they are swinging at the air, but they're technically more in sync.

[/quote]Is it really that hard to just communicate simple position data to the server? You'd rather have monsters fight like they're drunk?
[quote="Chris"]

• Display blood and elemental effects at the contact point on the client rather than as damage confirmation. This will mean that combat feels more impactful, but we lose the communicated visual information about whether damage was actually dealt. It could be that this is easier to apply to effects from spells because they generally don't have a hit/miss calculation.

[/quote]Until you manage to get desync truly handled, it would be best to keep this unchanged so that savvy players can use the gore as sync confirmation.
[quote="Chris"]

• Resync entities that successfully hit you when nothing is on the client near you. This may actually pull the entity even more out-of-sync if you're in the wrong place yourself.

[/quote]When nothing is on the client near you? Assuming I accept the ridiculous premise that positional resyncs are not possible, why wouldn't the server at least update positional data for Monster X when Monster X hits you, regardless of what else is near you?
[quote="Chris"]

• Resync everything in an area around a desynced entity. This reduces overall entropy massively but would be pretty jarring.

[/quote]I'm confused. So when your "intelligent" code detects that a player is out-of-sync, it doesn't resync everything, it just resyncs the lone elements which it detects are out-of-sync? Wow.
[quote="Chris"]

• Delay actions if the client was ahead on its path. This will solve the case where monsters die before you get to them (if you were out of sync) but technically results in lower combat efficiency for players in these cases.

[/quote]Desync lowers our efficiency enough.[quote="Chris"]

• Improve the distance-based resyncing that occurs for things that are far away from where they should be. It doesn't currently take movement speed into account properly. This is why Rhoas feel quite out of sync when charging.
• Measure overall entropy around the player and force a resync if it exceeds some threshold. The problem is that by the time the resynced information gets to the client, more actions could have occured.
• Fix bugs with specific skills that cause them to act differently on the client and server (Whirling Blades, for example, sometimes fails to trigger based on distance on one end).

[/quote]Both of these are obviously good... and the problem isn't really a problem if the client knows which messages the server sent first thanks to some kind of timestamp. You guys do use some kind of timestamp, right?[quote="Chris"]At this stage it looks like the biggest gains will come from improving the resync code so that it rapidly and reliably resyncs the combat situation if things get too desynchronised. This will mean more rubber-banding (as explained earlier), but will massively reduce deaths that occur from the player not being able to see the true locations of entities.

I explained the above changes with their drawbacks because I want to make it clear that this problem is intrinsically difficult to solve. We're fighting against both the laws of physics (travel speed of data) and the desire to not compromise gameplay mechanics. I have full confidence that we will incrementally deploy changes during Open Beta that substantially improve this situation.

I'm sorry about the wall of text. I hope I explained it clearly enough. I am also sorry that it has taken us this long to prepare changes for this issue. We are very careful to not introduce additional problems to the combat simulation and want to make very sure that the changes we're deploying are big improvements. I will let you know as soon as we have a specific patch in mind that we'll start introducing changes in.[/quote]Seems like an appropriate place to put the...

TL;DR: To be honest, at this point I don't think what the PoE netcode needs is "changes." I think it needs to be thrown out completely and rebuilt from the ground up. It's clearly based on a number of logical assumptions, incomplete information, and probably a lot of inefficient communication (low information entropy). Perhaps you could even implement UDP instead of TCP for those continual position updates; it's not like resending old data matters in that case. Recently GGG put the time into an art datasize overhaul, and the results on performance were impressive. Netcode is in dire need of a similar treatment.

There was a typo in there (Wrong word used, nothing serious) but I've lost where it was now o_O

All in all an excellent read and I would like to know the same things too.

The reason I left Desync out of my feedback post is because it has been mentioned by the community so many times there should be absolutely no doubt in GGG's mind that it is their top priority heading to launch. But it should have been a top priority for a while now, but almost nothing has been done. We get told the way it works & what they would like to achieve with it but little is ever actually done. It has lead me to believe that little will ever be done, if GGG knew a way to 'fix' it I think they would have.

I have resigned myself to this game having desync, yeah sure it lowers the quality of the game (quite a bit) but fortunately for me my playstyle results in minimal desync.

The reason I made my post was because we are fast approaching launch, GGG seem set to drop the beta tags in a couple of months but to me it still very much feels like the same beta we had a year ago. The patches for OB have mostly been content, very little has been done to balance the game & address the issues it faced back in CB. The added content has brought it's own issues with it as well.

I don't see GGG completely revamping their netcode, I don't see desync being a issue that get's put to rest, I think it is just going to be a bug bear of the game forever. Unfortunately the desync issue cripples a game like this, there is no doubt many have left due to it & many more are only hanging on at the promise it will be better. If that day never comes then there is only so long ppl will wait.

"

lethal_papercut wrote:

I don't see GGG completely revamping their netcode, I don't see desync being a issue that get's put to rest, I think it is just going to be a bug bear of the game forever. Unfortunately the desync issue cripples a game like this, there is no doubt many have left due to it & many more are only hanging on at the promise it will be better. If that day never comes then there is only so long ppl will wait.

There's still time, and pessimistic determinism solves nothing.

"

ScrotieMcB wrote:

There's still time, and pessimistic determinism solves nothing.

My advice is to get real. Wake up & smell the coffee. The game hasn't changed in this regard in well over a year, the next 2 months will not make a jot of difference. The community as a whole has been more than vocal on the subject yet GGG have only chosen to fob us off with excuses as to why it is the way it is. Now you expect them to completely overhaul the whole of the netcode before launch? It's totally unrealistic to propose.

There's no substitute for good netcode.

If it needs a rewrite, there should be a rewrite, cost be damned.

Small indie games succeed due to quality more than anything.

I think an excellent example would be Heroes of Newerth. From the perspective of an indie game it is ridiculously successful. S2Games is a small studio. A small studio that made an engine with incredibly tight gameplay and extreme responsiveness. They still have well over 100,000 concurrent users during peak time.

People are willing to forgive so many things, as long as the core gameplay is tight and responsive. There are tons of problems people have with HoN, but it would have died pretty much instantly without the gameplay that it has.

People are far LESS willing to forgive poor responsiveness, no matter how polished the rest of the game is. In the end, responsiveness and tight control are pretty much the most important possible things.

Fuck your content
Fuck your balance changes
Fuck your micro-transactions
Fuck your new skills
And most of all, fuck your uniques.

Fix your netcode. If you don't have the man power to fix it, then buy some more manpower and get it done.

Look outside of your tiny country, and ship in real experts if you have to.

"

tikitaki wrote:

Fuck your content
Fuck your balance changes
Fuck your micro-transactions
Fuck your new skills
And most of all, fuck your uniques.

Fix your netcode. If you don't have the man power to fix it, then buy some more manpower and get it done.

I forgot when writing the OP to include at least one fuck on every line, it makes reading far more enjoyable. I fucking approve.

+1 !

I know I am asking for a lot but I would love to see answers from GGG.

You're so right that Diablo3 with its system, although it's a shitty ARPG, felt like it had really fluent combat with absolutely no desync. I have no idea why it couldn't work for PoE. Excellent post, I agree with all of it. If they don't want to use the diablo3 system, I'm all for making it client based. Maphack is already possible, so it can't hurt all that much to make it like so. Bots will eventually find a way anyways.

I'd still prefer the D3/RTS system, I honestly don't get how can someone make a game that is basically based on prediction and not want to eventually change it, especially when there is proof that the combat can be fine with a different system.

If they don't want to change the system, the code is obviously flawed as it is, and shouldn't be improved at all, it should be reworked completely. I don't experience that bad desync, but it still sucks dick for me. I've played Torchlight 1 and 2, I assume that's the same shit, but it's obviously done a lot better, I had no desync issues there.

I'm not good at this, but what would happen if the game would resync everything every ~0,2 sec? Because when it identifies I am desynced, I'm already dead/in the middle of a pack of monsters.