Persistent skills not always enabling properly after weapon swapping

I have a weapon swap setup that uses different persistent skills. I have set it to use the correct ones for each weapon set (scavenged plating in 1 and herald of ice in 2). Sometimes when I switch between my weapon sets or travel between areas my herald of ice does not turn on. This can be fixed by removing the gem and putting it back in. I believe the issue is that I am using the weapon swap points on the reduced attribute requirements passive nodes at the bottom of the tree. I raised my attributes to the required amount without the passive points and I was unable to replicate the issue, however, as soon as I reduced my attributes I ran into the problem again.