How can I install/copy PoE onto a USB drive?

All you really need to do is install the game, and move the Path of Exile folder to your flash drive. You'll have to run the installer on every PC you use the game on, but you won't need to patch it, and the installer itself is very small and doesn't add much to the PC. Once the installer has run, you can play the game off your flash drive.

Note that your settings and login details are saved in a file in your My Documents/My Games/Path of Exile folder. You probably want to remove that after playing on a public PC! The password is hashed, but someone else could install the game and log on to your account.