Can't go back to Ogham village from manor ramparts (bug or design overlook?)

If you accidentally click on the manor ramparts entrance after killing executioner before talking to the prisoner, then you can't go back there again and have to warp to the waypoint all the way in the beginning and walk back to talk to her.

Yeah there's no way to go back... it's so dumb. This just happened to me.