Weapon swap minion cooldown

So, this only seems to happen while in combat, but not all the time.
With minions alive, I swap to 2nd weapon to cast skill, despawning my minions. Swap back and minions are on cooldown.

This happens whether the spell I cast is Conductivity or Sigil of Power.

A bit more on this... Please note, this has nothing to do in combat.

I decided to add some melee minions on to my weapon slot 2 (WS2) because I was finding myself in trouble having no minions while casting Conductivity and Sigil of Power.

What I found was that, when swapping back to weapon slot 1 (WS1) I was only getting a number of my main minions equal in value to whatever spirit the system decided I had left (see below). After I swap back, I have a bunch of spirit untouched, yet my gems say I don't have enough spirit. I have to reduce a minion and bring it back up to correct this.

Below, WS1 shows the spirit I'm left with when I swap to it and no longer have any of the minion in question, whereas WS2 shows the amount of spirit I'm using with the max number of that minion. Perhaps someone smarter can notice a pattern. All the minion gems have base level 15.

Total Spirit - WS1 - 369/ WS2 - 206
Reavers - 220/369 | 196/206
Brutes - 185/369 | 180/206

-Decided to check other ranged minions:
Snipers - 220/369 | 196/206
Arsonist - 185/369 | 180/206
Frost Mage - 185/369 | 185/206

-Spectre: Terracotta Soldier - when I swap into my 2nd set, all the soldier show up, then almost immediately drop to 1.

Since I was used a sceptre that gave +3 to minions I decided to try one without minion levels. The result was the same.

I tried swapping the weapon slots around. The result was the same.