Game crashes on specific maps (latency raising until client crash)

Anyone experiencing their crashes in maps?
What happens for me is that game will just freeze and then i can see my latency just keeps raising without stopping ever. I can still pause the game, but latency will keep going up.
If I wait the game will give me the disconnect message and then I lose all portals. If I exit the game and re-open I also lose all portal.

Wierd thing is it only happens on juiced maps. Might be everytime I run delirium.

Specs:
GTX 4090
Ryzen 9800X3D