DirectX 12 causes driver timeout

With the new patch I was eager to try out the DirectX 12 beta.
It works flawlessly, but every 20 minutes the game just shuts down and a message from the AMD Software pops up to tell me that a driver timeout occured (Whatever that's supposed to mean)
I deinstalled drivers and software completely and reinstalled the up-to-date version, but it keeps happening.

My graphics card is an AMD Radeon RX 5700 XT.

Now that I switched back to DX 11 I Will edit this post if it happens with DX 11 too.

I was semi-afk looking through my stash just now and I also got a similar crash on DX12. The line at the end of the client log is:

[D3D12] graphics_queue->Signal failed

Just as a short update:
It also happens with DX11 now, but less frequently. About every 2 hours.
Kind of annoying, because I have to restart PC before I can open the game again. Will try to have a look in the log next time it happens

Happened again with D11 just now while sitting in my hideout. Here are the client log lines:

2022/05/19 11:55:02 232823968 172ac405 [CRIT Client 15472] [D3D11] CreateBuffer: Die GPU-Geräteinstanz wurde angehalten. Verwenden Sie GetDeviceRemovedReason, um die erforderliche Aktion zu bestimmen. (GetDeviceRemovedReason: Die GPU antwortet auf keine weiteren Befehle mehr. Dies liegt wahrscheinlich daran, dass von der aufrufenden Anwendung ein ungültiger Befehl übergeben wurde.)

2022/05/19 11:55:02 232823968 55712961 [CRIT Client 15472] [EXCEPTION] CreateBuffer: Die GPU-Geräteinstanz wurde angehalten. Verwenden Sie GetDeviceRemovedReason, um die erforderliche Aktion zu bestimmen. (GetDeviceRemovedReason: Die GPU antwortet auf keine weiteren Befehle mehr. Dies liegt wahrscheinlich daran, dass von der aufrufenden Anwendung ein ungültiger Befehl übergeben wurde.)

I have no idea why this is in german, but it roughly translates to:

GPU instance was stopped. Use GetDeviceRemovedReason to determine the necessary action. (GetDeviceRemovedReason: The GPU did not respond to further commands. This is probably caused by an invalid command sent by the application.)

(it says the same in both lines)

Hope someone in the team can figure out what's causing this :)