I have an AMD 5700 XT. I have had this issue for the last few years in both PoE and LoA. I have found that limiting the FPS, either in-game in PoE or via the AMD Adrenaline software with AMD Chill, removes all crashes.
Personally, I set mine to 80 where the game will run at 110-140 unlimited. I have zero display driver crashes while FPS is limited. If I fat-finger f11 and turn chill off I’ll crash within 20mins usually, requiring a hard reboot.
AMD Chill is the only way I can play the game and not have to get a new video card. 80FPS on a 3440x1440 display is perfectly fine with me.
Good luck, I hope this is helpful to those struggling.