How to refund skill point from bandit?

Eramir's favour will grant you the passive point, which will mean that the vendor recipe you need to use is 20x Orbs of Regret + 1x Onyx Amulet. If you purchase the incorrect book you can sell it back to the vendor and get back your Orbs of Regret, but the amulet in the recipe will be lost.

Make sure you use the vendor recipe in the difficulty that you wish to change the reward from.

The wiki page has all of the information here.