Game always stutters AFTER finished compiling shaders / no shader bar activity

everytime the shaders bar are green - the game run really smooth, but as soon the shader green bar disappear the stuttering start and cpu graph goes crazy

- playing youtube in background does help reduce stutter
- disabling multithread also help but reduce framerate

stutter happen regardless cpu or gpu usage (most of the time both are not maxed but still stutter really bad)

tried all kind of setting and renderer options, same result stutter only happen when no activity on shader bar, there something at the end of shader compiling process that make the cpu goes crazy

cpu : i7 4790
nvidia 1660 super
16gb ram
win 10
game, poe cache, nvida cache are moved to ssd
cpu gpu temp 55c ish

Well, I got heavy performance issues with TOTA, but after the small investigation and experiments boiled it down to shaders cache as well. Tried to move shaders cache to the SSD(where PoE already resides), but that didn't help, just like OP states, game starts lagging after all shaders compiled/loaded - immediately after the shaders bar on graphs goes to zero. I also noticed that my CPU is not loaded properly, i.e. I have 4 physical cores, and PoE runs on all of them, but still tries to stay under 50% CPU load (don't know why) - every time it goes above 50% - CPU load drops to ~45% and I get CPU spike on graphs.

my specs:
Intel core I5 3570
DDR3 16GB
GeForce 1660Ti 6GB
Win 7 Ultimate

Game settings:
Windowed Fullscreen
1920x1080
Vulkan - changing to DX11 makes it only worse
VSync Locked - changing does nothing
Tripple-buffering Enabled - changing makes worse

All Detailed Settings set to lowest possible option

Dynamic culling Disabled - changing does nothing
Dynamic resolution Disabled - changing makes it only worse and my eyes bleed
Engine Multithreading Enabled

Here is screenshot:
https://imgur.com/a/4sMrAu1