Party invite fail

When I invite someone to trade, the invite instantly fails even if I invite the same person multiple times. This happens every 3 or so trades, meaning I lose out on the chaos or divine. This is not through the trade site, but once I get or send out a whisper to someone and invite them.

It just says that the person joined, then failed to join, then after a minute or so, they are kicked or leave the group. They do not appear in the social tab, and I cannot invite them again.