spirit cost when changing wepons

i have wold pack + skeletal snipers, when it forces you to change wepon set when interacting with things, on swaping back to my wepons, disabling all the minions but the wolf pack leaves me on 143/243.

turning wolf pack on and off then i have 193/243d

It doesn't happen everytime not too sure what exsactly causes it, but it happens somewhat fequently.