Lost vaal outpost entry after using waypoint

While delving a vaal outpost I reached the destination, killed everything, clicked "start the encounter" which immediately finished and dropped all the loot, and then backtracked along the now-lit line to pick up some loot I'd left behind along the way.

When I reached the starting position I waypointed back but couldn't enter the vaal outpost anymore, even though I'd never entered it.

The issue was that you used the waypoint to return. Doing so creates a new instance, while the encounter is only available in the instance you reached it in.

If you had returned to the encampment to empty your inventory, you would have been able to return to the instance with one of the icons in the top left (I forget the labeling) at the point you left from.

From your story, the button would have returned to the node you started the trek from, and if you walked back through the now-lit path, the encounter would have still been available.