Some ritual altars are impossible to activate

This happened to me at least three times in the last hour - only one altar can be activated, and the rest remains impossible to activate.

Example:


The icon on the minimap in the top right shows that the altar should be available, yet the actual pillar cannot be clicked.

Example2:
Another altar from the same map, after I cleared the single altar that worked.



I noticed that every time this happened, part of the altar area was impassable terrain, which makes me wonder if some mobs that are part of the ritual group spawn in those locations and cannot be killed.