Performance DX12 Question

Hi, I recently started playing again and I think I might have noticed a performance issue. After playing for about an hour, the game starts dropping to around 40–60 FPS. However, if I restart the game and load the same map, the FPS goes back up to 80–140.

I'm using DX12 and DLSS, and my system specs are: AMD Ryzen 7 5700X3D and an RTX 3060 12GB.

I’d like to know if anyone else is experiencing this issue—I tried searching for it but couldn’t find anything.

edit: I also tried switching to Vulkan, but in my case, performance was even worse.