Dark Effigy Totem icon inconsistently showing

I'm running 2x dark effigy totems and the buff icon at the top which shows the totem will sometimes show the one or two totems that have been summoned, and sometimes not show any even though they're clearly there right next to me.