Technical solution to eliminate desync in single-player sessions

Deterministic game state checking by the server eh. Sounds promising.

It seems the only major setback to this idea is the 1-4 second delay when logging off/tping. EXCITING!

Anyway. Clearly the only route this thread can go now is to have the big man himself post, the HEAD honcho... the boss man... JOHNATHAN.

Common John (can I call you that? Imma just go and risk it), I summon you!
"
It seems the only major setback to this idea is the 1-4 second delay when logging off/tping.


Without this, players could very easily escape death and unfortunate situations. I don't know of any other solution unfortunately. However, I do think that without lag/desync players would not need to ALT-F4 or insta-TP nearly as frequently. =)
I'd just like to say that this is the type of thread we need to see more of.
"
qwave wrote:
MeltingPoint:

"
basically server and client both have RNGs and each are 'seeded' with the same value: client pulls a value from its generator->attaches it to a snapshot->server receives snapshot->pulls number from generator->numbers should match.


Yes, but the most important thing is that both the client and server use these values to simulate the game state and entropy, including mob pathing.



"
I can't see how that validates whether the user clicked on the ground or the mob right beside him.


The mob is guaranteed to be in sync 100% of the time if both the client and server are running the exact same deterministic simulation. So if the client clicks the mob, then he clicks the mob.

Your use of absolutes detracts from your credibility. There is no 100% guarantee of anything, and the only way for the server to obtain information of user interaction is susceptible to automated manipulations.

"
qwave wrote:

"
At some point, mob position has to be calculated on the server and since mob position is highly dependent on player position would you not have to go back through all the snapshots and calculate mob position?


The mobs will be moving during each snapshot based on the entropy produced from the seed. Therefore, both the client and server will know the locations of mobs because they are sharing the same seed and thus the mobs are traveling in the same spots.

This would only be correct if the client side program would base all calculations based on the last assertion which is up to the delay between your assertions. And once again i must emphasize that such client side calculations very much ARE within the range of exploits despite what you say. Difficult hack; yes, impossible; absolutely not.

"
qwave wrote:

"
What happens for the 999 milliseconds between snapshots?


Your game client is creating the snapshots 100% of the time and streaming them to the server. This is the same way the game is currently doing it, that doesn't change at all. The difference is that the client is able to accurately run the 'simulation' without having to receive responses from the server.

If this was the case, why stream at all? In either case the server does nothing; the servers receives the packet and concludes all is ok, or it receives the packet and determines you are not where you should be(and thus assume a hack). But if your snapshot system is as fool proof as you claim it would be detected then.

"
qwave wrote:

"
Does the client have the authority to say the mob hit me? If it does, could the client not be forged to say it didn't hit me?


The client must adhere to the deterministic seed. The seed runs against the game engine, so if the mob hits you, that has to be part of the snapshot. If you send a snapshot without the mob hitting you, the server will compare the snapshot to it's own simulation and validate that you hacked the system. The client cannot forge a snapshot because the server is running the same simulation.


"
How would a seed value give you that answer? Since no validation is performed until the end of a session, how would you resolve this?


The validation can occur anytime that GGG wants. I only recommended the end of the session because it's the most efficient way to do it. The server does not 'persist' (save your new state to the database) until it has validated your snapshot. However, it CAN save the snapshot so that it can be restored/validated in the event of crashes/disconnects/etc.

Obvious hack here would be; use exalt, if you dont get what ya want force a restore.
==edit==
There are numerous exceptions you have alluded to to prevent cheats, each one has to be individually code. And every single exception needs to be anticipated before going live. How many thousands of manhours do you think it would take to design/code/and test your system? And is anyone truly in a position to say for certain he can think of all possible exceptions ahead of time? If anyone would be so arrogant im sure there are hundred of geniuses currently playing PoE who would love the challenge.
========
Overall some additional client side processing could help eliminate some lag at the cost of vulnerabilities to system security. Exactly as the devs have previously stated. And it is their call. Expend a large sum of resources to open up potential vulnerabilities... or not.

And then there is the whole issue of false positives for hack detection. What if someones internal fan kicks on slightly varying the processor speed and thus the 'determinism' of the game. Or of a disk write error which flips a bit. Or numerous other entropic processes.
For years i searched for deep truths. A thousand revelations. At the very edge...the ability to think itself dissolves away.Thinking in human language is the problem. Any separation from 'the whole truth' is incomplete.My incomplete concepts may add to your 'whole truth', accept it or think about it
Last edited by SkyCore#2413 on Nov 18, 2013, 9:25:47 PM
I for one know very little about net coding since I'm a SA/DT tech but I have to agree with Qwave's ideas. It all might not be flawless and yes it might allow for 18/1000000 to stay alive with some sort of packet injector. But on the other hand how is that different from alt+f4 or exiting the game to avoid death? Answer: it's not.

All in all, we need to know more exact details about whats being done to desync to fix the issue. It has been around for a long time with very minor changes that I have noticed. Rhoa fix being one of them. But there is still massive desync causing skills such as Incinerate while casting with GMP or LMP and turning a direction, or Cyclone, Leap slam, Lighting warp, Whirling blades.

Besides the skills, you can still desync through walls ( I have done this twice in the last week) and desync from collision.

Also all the people talking crafting, if you read the full first post by OP that would still be handled server side.

BTW best thread to grace GD in a very very long time.
Twitch.tv/Nithryok
Last edited by Nithryok#2577 on Nov 18, 2013, 9:18:31 PM
"
Your use of absolutes detracts from your credibility. There is no 100% guarantee of anything, and the only way for the server to obtain information of user interaction is susceptible to automated manipulations.


A mob cannot be out of sync because the server and client are using the same key. It is a 100% guarantee because there is no 'sync'. I can use absolutes because it IS absolute. There is nothing more absolute than a shared key.


"
This would only be correct if the client side program would base all calculations based on the last assertion which is up to the delay between your assertions. And once again i must emphasize that such client side calculations very much ARE within the range of exploits despite what you say.


The client uses the seed to continually simulate the game in the exact same way the server does. The client-side calculations MUST match the seed, any changes will result in the server rejecting the snapshot. Basically, the client BECOMES the server, while the actual game server maintains the authority to make the final judgement.


"
Obvious hack here would be; use exalt, if you dont get what ya want force a restore.


Like ive stated in at least 20 different posts, crafting and other activities that are unrelated to pathing/combat would be 100% controlled by the server. My proposal is specifically designed to eliminate desync during combat/movement. Crafting does not desync you.


"
What if someones internal fan kicks on slightly varying the processor speed and thus the 'determinism' of the game.


Are you being serious? I don't understand why you are asking me these sorts of questions if you aren't bothering to research even the most basic fundamentals of this.
Last edited by qwave#5074 on Nov 18, 2013, 9:21:57 PM
"
qwave wrote:
"
Your use of absolutes detracts from your credibility. There is no 100% guarantee of anything, and the only way for the server to obtain information of user interaction is susceptible to automated manipulations.


A mob cannot be out of sync because the server and client are using the same key. It is a 100% guarantee because there is no 'sync'. I can use absolutes because it IS absolute. There is nothing more absolute than a shared key.

Wrong wrong wrong. User interaction CANNOT be anticipated by the server. <--period Thus there is no guarantee.
Your lack of knowledge about this most fundamental aspect leads me to believe you are not near as competent as you claim to be.

"
qwave wrote:

"
This would only be correct if the client side program would base all calculations based on the last assertion which is up to the delay between your assertions. And once again i must emphasize that such client side calculations very much ARE within the range of exploits despite what you say.


The client uses the seed to continually simulate the game in the exact same way the server does. The client-side calculations MUST match the seed, any changes will result in the server rejecting the snapshot. Basically, the client BECOMES the server, while the actual game server maintains the authority to make the final judgement.


"
Obvious hack here would be; use exalt, if you dont get what ya want force a restore.


Like ive stated in at least 20 different posts, crafting and other activities that are unrelated to pathing/combat would be 100% controlled by the server. My proposal is specifically designed to eliminate desync during combat/movement. Crafting does not desync you.


"
What if someones internal fan kicks on slightly varying the processor speed and thus the 'determinism' of the game.


Are you being serious? I don't understand why you are asking me these sorts of questions if you aren't bothering to research even the most basic fundamentals of this.

Huh. So you assume everything in the real world is perfect. I have pointed out vast gaping holes here. You have attempted to validate yourself rather than actually accept that i am right. And now when chaos comes into the mix you simply insult me?
This was a well informed and serious question. Chaos occurs. Imagine if the time elapsed on my computer doesnt match the time elapsed on the server due to a power fluctuation cause by said fan kicking on. Either way the server handles such a situation, there is an exploit available. If the server responds with 'hack detected' you get a false positive. If it ignores it, a world of possibilities await the enterprising hacker.
If there is is an internal error in memory(including the very unreliable virtual memory) it may lead to snapshots being corrupt. Also leading to false positives.
What about radiation. It flips bits. Very regularly. In fact you can determine a hosts altitude by it. Again we have false positives.
What about packet loss? What if your dog rubs against your modem cause several seconds effective disconnect? What about discon altegether? How many contiguous packets are acceptable. What about receiving packets out of order(if using udp). What about when you receive 3 seconds of packets all at once? 3 minutes of packets? There is a LOT of fucking chaos out there. And your inability to comment on it yet again refutes your all knowing claims of perfection.
For years i searched for deep truths. A thousand revelations. At the very edge...the ability to think itself dissolves away.Thinking in human language is the problem. Any separation from 'the whole truth' is incomplete.My incomplete concepts may add to your 'whole truth', accept it or think about it
Last edited by SkyCore#2413 on Nov 18, 2013, 9:52:07 PM
I am not trying to insult you, but I don't understand why you would ask these sorts of questions if you have done the research on how pseudo random numbers are generated. Any wikipedia article or white paper will clarify all of these questions you have in much greater detail.


"
Wrong wrong wrong. User interaction CANNOT be anticipated by the server.


The server DOES NOT need to anticipate user interaction! The client notifies the server of the interaction, and the server validates it. This is the exact same way that the game is currently run. The only difference is that the client can simulate the game.


"
Imagine if the time elapsed on my computer doesnt match the time elapsed on the server due to a power fluctuation cause by said fan kicking on.


The client is using a delta from the last timestamp. Any sort of entropic 'clock drift' is not going to impact the data. Moreover, the server can easily correct the client to ensure synchronization.


"
If there is is an internal error in memory(including the very unreliable virtual memory) it may lead to snapshots being corrupt.


If there is a memory corruption in this manner, your game is going to crash and the server will continue the simulation without the client.
Last edited by qwave#5074 on Nov 18, 2013, 9:58:16 PM
As the programmer responsible for maintaining the client/server synchronization of the random level generation, I can tell you that this sounds like a nightmare, for a few reasons.

1) The server must validate the snapshots in real-time, not once the client leaves the instance. This is because the client can potentially muck around in one instance for huge lengths of time. Hours, days, weeks even, if the realm stays up. Processing so much all at once would lag the instance to hell and back, and changing areas has to wait on this task. Also, you will still need the ability to resync players to a recent state, when desync inevitably occurs (see next point). Also also, you need to be able to save the player's data at a moment's notice to prevent item dupes with trading.

2) Floating-point calculations will still cause desync. Suppose the client needs to perform a mathematical calculation, such as 37 / 13. The client determines the result to be 2.84615384. But when the server performs the same calculation, with the same values, the exact same bit-patterns inputs, it might instead return 2.84615385. The numbers are slightly different. Why would it do this? There are several reasons: Windows client vs. Linux servers. Different CPU architectures. Different optimizations compiling the servers vs. client. Different system drivers. Whatever the cause, you now have a divergence point, where the simulation may or may not diverge, depending on how those values are used and rounded. In that example, rounding to an integer afterwards isn't a problem, but what if the two values were 7.499999 and 7.500001? One will round to 7 and the other will round to 8. Suddenly the tiny difference isn't so tiny, and so even legit players can end up desynced.

3) The client would effectively have the ability to see into the future. A hacked client could predict when hits, crits, evades, and dodges will occur, so a player could alternately attack minions/bosses, or even just pause a certain number of milliseconds, to ensure they are almost never hit by strong attacks and almost always crit against the boss. In both of these cases, the final simulation is perfectly legitimate, so the server cannot know the client is cheating, unless you want some statistical system that punishes players for being "too lucky", which will harm legit players who get an actual lucky streak.

If item creation and currency use is done from the server, then you have monsters only dropping items 2-3 seconds after they die, and waiting 2-3 seconds for every currency item to take effect, since you have to wait for the next snapshot to be verified (and on a turbulent connection, that could be a while). Using (say) 1000 Fusings to try for a 6-link would take forever (you couldn't spam shift+right-click).

4) How do you detect death? You can't rely on the client sending the "Oops I died" packet. Otherwise hacked clients would never die. The server must be able to continue the simulation without the client, and determine when a player dies. This means that the server must keep simulating "the future", and client commands occur in "the past", which then changes the simulated future (like what Valve's Source Engine does). Now, you suddenly need the ability to effectively "rewind time" as far as the simulation is concerned. This is not a trivial task, at all.

5) This entire system breaks down completely as soon as you have more than one player. Anytime one player moves or hurts a monster, everyone else is automatically desynced and must be jolted back to the last verified snapshot. This is because you cannot recover from small amounts of desync, since even tiny differences completely change the RNG and state hash.

The "rewinding time" mechanic would somewhat solve this, but you would STILL get desync (just like in Team Fortress 2 and other Source Engine games). The classic example is being headshotted through a wall, because even though you successfully took cover on your screen, you didn't on the sniper's.

Also, what happens when the level geometry is different for different players? For example, if one player has destroyed the Undying Blockage in the Sewers, then the corridor is clear for them, but another player who hasn't done that will have the corridor blocked. The simulations for projectiles and movement will be VERY different on each client, so which one does the server think is correct?

6) Maphack would be even easier, and better, because it could show all the monsters and chests and items on the map, too.

Just my 2c.
Code warrior
"
SkyCore wrote:
"
qwave wrote:
"
Your use of absolutes detracts from your credibility. There is no 100% guarantee of anything, and the only way for the server to obtain information of user interaction is susceptible to automated manipulations.


A mob cannot be out of sync because the server and client are using the same key. It is a 100% guarantee because there is no 'sync'. I can use absolutes because it IS absolute. There is nothing more absolute than a shared key.

Wrong wrong wrong. User interaction CANNOT be anticipated by the server. <--period Thus there is no guarantee.
Your lack of knowledge about this most fundamental aspect leads me to believe you are not near as competent as you claim to be.

"
qwave wrote:

"
This would only be correct if the client side program would base all calculations based on the last assertion which is up to the delay between your assertions. And once again i must emphasize that such client side calculations very much ARE within the range of exploits despite what you say.


The client uses the seed to continually simulate the game in the exact same way the server does. The client-side calculations MUST match the seed, any changes will result in the server rejecting the snapshot. Basically, the client BECOMES the server, while the actual game server maintains the authority to make the final judgement.


"
Obvious hack here would be; use exalt, if you dont get what ya want force a restore.


Like ive stated in at least 20 different posts, crafting and other activities that are unrelated to pathing/combat would be 100% controlled by the server. My proposal is specifically designed to eliminate desync during combat/movement. Crafting does not desync you.


"
What if someones internal fan kicks on slightly varying the processor speed and thus the 'determinism' of the game.


Are you being serious? I don't understand why you are asking me these sorts of questions if you aren't bothering to research even the most basic fundamentals of this.

Huh. So you assume everything in the real world is perfect. I have pointed out vast gaping holes here. You have attempted to validate yourself rather than actually accept that i am right. And now when chaos comes into the mix you simply insult me?
This was a well informed and serious question. Chaos occurs. Imagine if the time elapsed on my computer doesnt match the time elapsed on the server due to a power fluctuation cause by said fan kicking on. Either way the server handles such a situation, there is an exploit available. If the server responds with 'hack detected' you get a false positive. If it ignores it, a world of possibilities await the enterprising hacker.
If there is is an internal error in memory(including the very unreliable virtual memory) it may lead to snapshots being corrupt. Also leading to false positives.
What about radiation. It flips bits. Very regularly. In fact you can determine a hosts altitude by it. Again we have false positives.
What about packet loss? What if your dog rubs against your modem cause several seconds effective disconnect? What about discon altegether? How many contiguous packets are acceptable. What about receiving packets out of order(if using udp). What about when you receive 3 seconds of packets all at once? 3 minutes of packets? There is a LOT of fucking chaos out there. And your inability to comment on it yet again refutes your all knowing claims of perfection.


Dude, read qwave's posts man! He's answered this crap like fifty times...
In case of real lag, desync WILL still happen. But only on a laggy connection or on rare occasions, not like it's happening today.


Why do you guys think desync doesnt exist in other major games? They use the techs describe in this post. IT'S A BRANCH STANDARD THAT IS PROVEN TO WORK.
Ask your logal net guru, I did :)



And why are some ppl so affraid that GGG can have missed an obvious solution to desync? They have done an increadable job with this game being such a small dev team and all. But they cant do everything. So accept help when it's presented.


Desync is real, but only in PoE. Go figure.

Report Forum Post

Report Account:

Report Type

Additional Info