Corpse Spectre Question

If you replace the raise specter gem in town with a level 21/20 in the same slot, will you lose your corpse specters? Are they attached to the gem itself?

Thanks

Attached to the gem.

just swapped out a 20/20 gem in town with a 21/20 and I kept my corpse specters. So it's definitely not attached to the gem

https://www.poewiki.net/wiki/Corpse_item

"Raise Spectre provides conditional Loss Protection if you need to move your gem to another piece of equipment."

Not sure how it works exactly, but probably something to do with this. It might also extend to swapping the gems as you have done.