Abyss holes not advancing?

I have had several occasions where on 3rd of 4th abyss hole it would not generate reward or next hole despite me clearing all enemies around it. After defeating rares and other initial spawns of enemies the hole barely spawns a few more enemies then times out. What is going on?