"
_Interrupt_ wrote:
"
PGLoLo wrote:
So it will be 3Gb of data at one update.
That is exactly the problem.
Do you have an internet connection that can download 3GB of data every minute?
Does anyone have such internet connection?
Not to mention POE servers will have to send this amount of data evey minute to hundreads of people using public stash api.
How is this a problem? With said compression we're down to 500MB. From GGG's point of view, they have the API behind load balancing, assumably with international data-duplication/caching so they will not have any problem serving some hundreds of megabytes per second to some thousands of users from their end. 100 different indexing services can be querying a dozen different servers on the closest continent to the indexer.
Each of them is not behind a consumer internet connection but designed to serve gigabytes per seconds.
On the other side is the indexers. I do not know where the poe.trade database is hosted, but I assume most services that make money on this has set up shop in a data center or distributed over several data centers which also have capacity that outstrips a consumer internet line...
If you want to populate your own elasticsearch instance on your home computer and need to stream 500 MB per 1 minute, you might not be able to if you cannot download 10 MB/sec (~100MBit). Where I live most people have 100 MBit internet connections, so even that is not always an issue.
|
Posted byFatalityDance#6634on Mar 4, 2016, 10:43:59 AM
|
"
FatalityDance wrote:
How is this a problem? With said compression we're down to 500MB. From GGG's point of view, they have the API behind load balancing, assumably with international data-duplication/caching so they will not have any problem serving some hundreds of megabytes per second to some thousands of users from their end. 100 different indexing services can be querying a dozen different servers on the closest continent to the indexer.
Each of them is not behind a consumer internet connection but designed to serve gigabytes per seconds.
On the other side is the indexers. I do not know where the poe.trade database is hosted, but I assume most services that make money on this has set up shop in a data center or distributed over several data centers which also have capacity that outstrips a consumer internet line...
If you want to populate your own elasticsearch instance on your home computer and need to stream 500 MB per 1 minute, you might not be able to if you cannot download 10 MB/sec (~100MBit). Where I live most people have 100 MBit internet connections, so even that is not always an issue.
What compression are you talking about exactly?
I have looked everywhere and i haven't been able to find any references to aforementioned 'compression'.
I'm not sure what you mean by "that is not always an issue".
You mean everyone should continue using poorly optimized code just because people in your region have good internet connection?
|
Posted by_Interrupt_#7402on Mar 4, 2016, 12:55:01 PM
|
"
_Interrupt_ wrote:
"
FatalityDance wrote:
How is this a problem? With said compression we're down to 500MB. From GGG's point of view, they have the API behind load balancing, assumably with international data-duplication/caching so they will not have any problem serving some hundreds of megabytes per second to some thousands of users from their end. 100 different indexing services can be querying a dozen different servers on the closest continent to the indexer.
Each of them is not behind a consumer internet connection but designed to serve gigabytes per seconds.
On the other side is the indexers. I do not know where the poe.trade database is hosted, but I assume most services that make money on this has set up shop in a data center or distributed over several data centers which also have capacity that outstrips a consumer internet line...
If you want to populate your own elasticsearch instance on your home computer and need to stream 500 MB per 1 minute, you might not be able to if you cannot download 10 MB/sec (~100MBit). Where I live most people have 100 MBit internet connections, so even that is not always an issue.
What compression are you talking about exactly?
I have looked everywhere and i haven't been able to find any references to aforementioned 'compression'.
I'm not sure what you mean by "that is not always an issue".
You mean everyone should continue using poorly optimized code just because people in your region have good internet connection?
Everything on the internet is compressed. If it isn't then the technical people are not doing their job. GGG seems to be doing their job, if toolmakers aren't competent to use it, thats on them:
https://www.reddit.com/r/pathofexiledev/comments/48tf4q/psa_make_sure_you_are_using_http_compression_with/
"
Since the Stash Tab API went on-line yesterday, ExileTools has pulled 574.69MB of compressed data from it - however, uncompressed, the size of that is 3,496.32MB - 3.5GB!
You do NOT want to create any tools/etc. that are taking that stream down uncompressed.
https://en.wikipedia.org/wiki/HTTP_compression
|
|
|
If one compressed stash tab contains data equal to 24KB
Multiply it by 10 000 asynchrynous usage by poe players ~ 240 MB
Without cashing implemented on indexer side ( yea i know its dumb) thats 240MB to be served per request (or processed per request)
And thats not counting non changed state tabs.
I think its a fuken lot. Poe.trade scans similar amount of data but with worker set to ~ 10min crone? I think they also parse fetched data to something easier to work with than raw json data and store it to db.
If stash api works on pattern "update on request" i think its pretty fuken stupid and can lead to serious overload.
EDIT.
Better implementation would be to store information about required stash tabs to be updated on request and launch worker that will do this job every X minutes in bulks. Saving such a data will be instantious. Small delay on refresh is also not a problem, xyz works this way from the beggining...
IGN: PojzonAbyss
[3.2] Immortal Indigon Poet's Pen Hirophant - https://www.pathofexile.com/forum/view-thread/2129766
[3.1] Yamata No Orochi - Oni-Goroshi Jugg http://poeurl.com/bHB9
[2.4] The True Queen Of The Forest - Poison LA/BR Pathfinder http://poeurl.com/M6u [Retired] Last edited by eragon1111#0889 on Mar 4, 2016, 8:59:35 PM
|
Posted byeragon1111#0889on Mar 4, 2016, 8:54:13 PM
|
