Swapping weapons with minions confuses minion count / spirit reservation

My minions keep getting busted. The system thinks I'm out of spirit when it was working fine before. This happens sometimes when swapping between weapon sets with minions / different minion configurations.

You'll see in this screenshot how borked it gets. The gem configuration is two brutes, there are three on screen, and the buff icon says I have 5.

Normally in weapon swap #1, I have three brutes. However, I observed that my sceptre skeletons and lone priest wasn't being summoned, and I had four brutes. When looking at the gems, the priest said it no longer had spirit reservation. I decreased the brutes by 1 (to 2) and the summons that had 4 brutes then because 3 brutes, 2 warriors, 1 priest. (Side note: why does the skeletons not summon on spirit problems, they don't have a spirit reservation on the sceptre.)

Can confirm, this happens to me as well!

This is what is shown after a few weapon swaps: