Safe way to gem swap without losing spectres?

I recently got up to 4 spectres with a new +1/+1 wand, but I want to get some of my gems off of my backup weapons without losing any of my spectres. Is there a safe way of doing this? Can I do it in town safely? I've already lost a handful of them in different ways this season, and I don't want to screw up yet again. Thanks in advance.

Yes do it in town.