Portals for Sekkemma (POE2)

I was playing Trail of Sekkemma and in the reward room there are portals nearby the chest which I'm open the chest and looting Time-lost Jewels, which accidently click to portal and when I comeback I cannot go to reward room I have to continue running on next floor

I don't know that is anyone having this problem, but I think it will be good to move all the portals possible to different location as the chest in mouse and keyboard I don't think this is any problem but in controller right now, I'm kind of aware for it every time

PS. Controller using