Skeleton Warrior weapon unequip bug

When I unequip my scepter (thereby loosing 100 spirit) I am freely able to allocate as many skeleton warriors as I want (I can spam click "+"). Then, then I equip my scepter again, I have too many skeletons, but not the 30 I tried to allocate.

I am using Skeleton Arsonists and I have the Hellhound as well. Doing the unequip / reequip made me have 4 skeleton warriors and 2 arsonists, even though I am only supposed to have 2 warriors and 2 arsonists with my current spirit