Disconnect from the game when playing with friends

When playing on my own the game seems fine, I don't usually experience any latency issues or d/c's but when two other friends are playing POE on the same network then the game flat lines for all three of us until we relog. I've called my ISP and there don't seem to be any issues.

Note: This only seems to happen when all three of us are playing together in the same area.