Accessing Derelict Mansion

So I've spent the best part of a day trying to access one of the derelict mansions. I found multiple activation stones (Can't remember the name) to still not be able to access it. Then it labeled maps I had completed around as attempted. I had to then redo them all, to finally access the manor. About 7/8 Stones.
Finally I run the mansion and I get one shot exploded by nothing (I watched the video back)
Even Xest can't one shot me. So I wasted all that time and can't even replay the damn mansion!
Also the vaults, out of 3 I've only ever found one idol as apparently they aren't even guaranteed.