Keybind issue

On XB1 if you bind anything except regular attack to A on the first page, you can not retrieve items from the ground.

I'm playing a witch and I never use regular attack so I'd like to use that slot for a skill I do use, but I can't because if I try I can't pick anything up. Instead of picking the item up she just casts the spell.

It should actually work with regular spells, only problems I have seen are spells that have corpse targeting, that prevents you from looting.