1.5 gigs of ram?

We've yet to optimize the game, and at the moment all previously visited areas are loaded into memory (If you have it to spare) so changing back to previously visited areas is much faster. You can add -gc1 to the end of your Path of Exile shortcut if you don't want it to do this.