Refunding Ascendancy on Cruel

I wen't through the game once, did the labyrinth, and spec'd into the wrong Ascendancy class. It's my understanding that I can refund all the points in that class ( for 5 refund points a piece ) and then go to the labyrinth guy again and switch classes completely.

My question is, I'm about to do the labyrinth on cruel ( because that's where I am in the game ) can I wait until I get to the end room, refund my points there, then switch and now have 4 points to put in my new Ascendancy class?

I'm pretty sure you can do that, since it seems to check whether you have any ascendancy skill point allocated every time you click the ascendancy thingy at the end of the Labyrinth.

If you have spent ascendency points when you touch the altar it will tell you that if you want to change class you first have to unlearn the previous spent points first. Once you do that, you are allowed to change class as long as you don't leave the room before doing so.

Thank you guys so much for the help!
I think my question has been answered =P It was totally worth going back and doing all those quests to get the 2 refund points and later today I will do the labyrinth on cruel and swap ascendancy classes... Also thank you Path of Exile for allowing players to do this... This is my first time playing PoE and I made it to level 62 hardcore perandus so far on my first character and I'm still not sure what i'm doing so being able to go back and change mistakes like this is amazing!