POE 2 Ziggurat Refuge slow down(?)

Last two days, every time I'm in the Ziggurat Refuge, the game slows to a crawl and stutters. I am also unable to interact with anything in my inventory specifically and because of this, I can't use waystones to do more of the endgame. I am officially locked off and can't do anything about it.

Anybody got a reason for this happening and what I could do to fix it?