Volatile Dead Spellslinger Latency Spikes (caused by map loading?)

I am getting unpredictable latency spikes when using VD Spellslinger. It seems to be linked to loading or unloading a map, as traversing a portal back to the hideout can cause/fix the issue.

The latency spikes only happen on casting, particularly when hitting monsters.

If I leave the map and return again, it might fix the issue, but sometimes it doesn't.

Here's a video where I documented an example of the problem: https://www.youtube.com/watch?v=_ckkQeI2wPE

Everything is fine for the first 5 portals of the map - consistently low latency while moving and attacking/casting. During the last transition you can immediately see/feel the difference - massive latency spikes when casting and attacking, especially when hitting monsters. This lasts for the remainder of the map (or until I portal out). Garden portal doesn't affect it, standing still returns latency to normal temporarily.

This video was recorded with dynamic resolution disabled. Before that, it was enabled and I still found the same issues (though not as bad FPS drops). Currently using Vulkan for the renderer, though I don't think it's a graphical issue.

Internet specs: DSL line, 25/10 Mbps, wired connection

PC specs:
CPU: AMD Ryzen 5 2600X, six core, 3.60 GHz
RAM: 16 GB
OS: x64 Windows 10
GPU: Radeon RX 580, 8 GB

PoE is on an SSD also. Thanks for reading and checking this post!