3.11 Legion FPS drop

Using Vulkan (Beta) at RX570 with 20.7.1 drivers.
I have deleted shaders cache and since then completed ~5 Legion Encounters but every damn time my FPS graph looks like this:



I'm pretty sure 5 encounters should be enough to recache all the shaders required, yet the system lags just like first time.

I've been using Vulkan since release, prior to 3.11 it was constant 74 (locked) FPS now it works just as bad as Dx11 version.