Respec bug

I tried to respec a few points just now and i ended up permanently losing 3 points. Those points look like they're still allocated on the tree but they're actually removed. I'm not using the dual spec option either.

I have a similar problem. Respect doesn't work when I try to remove passives and add new ones simultaneously. Sometimes, changes are reverted after a zone change.
To work around it:
1. First refund nodes and confirm your choice.
2. Return to the zone you were in before (like use town portal etc)
3. Allocate the rest of the point, then confirm.
At least it worked for me.

Steps I did to reproduce the bug:
1. I got 2 points unallocated.
2. teleport from zone to town.
3. refund some nodes (don't confirm!), then add all remaining points to a different tree passives. Confirm choice. (At this point tree may still highlight nodes and paths that were refunded)
4. return to zone via town portal
5. passives allocation and refund are reverted (possibly some points missing)

edit:
I am using controller on PC

Hope this will be helpful

I ran into this too. It happened when I tried to remove a couple of nodes that were required for later nodes by creating an alternate path of prereqs. In my case, I tried to swap the minion life nodes for spell and minion damage nodes in the third and fourth move away from witch's start.

After this, the game said the nodes were unallocated, but they were still highlighted in the tree as if they were. I was able to fix this by logging out and logging back in (then they reset to allocated and I was able to refund them).

I think behind the scenes multiple refunds and re-allocations are applied sequentially rather than atomically. The bug is triggered when the game chooses a sequence order that attempts to refund a node that is required as a prerequisite (and therefore cannot be refunded) before it allocates skill points to the alternate prerequisite nodes.