Can't progress to next difficuly

To reach the next difficulty, you need two things:

• To either kill the boss yourself or be in the instance when a party member does (you can even be dead),
• To either place the Combined Amulet into the Dark Altar yourself or have in your inventory.

Elafar, it sounds like you didn't have the Combined Amulet with you at the time when you killed the boss, and so couldn't progress to the next difficulty level.

If you did indeed manage to get credit for killing the boss that first time around, you should be able to walk past the Dark Altar and into the next difficulty now that you have the Combined Amulet. If you didn't actually make it back in time you may need to kill the boss again.

Other people entering your party can legitimately happen when people change parties and revisit the same world areas. If it happens, one (or both) groups should leave and force a new instance using Ctrl-Click, otherwise people may miss out on quest states.