AMD Performance issue

As the name said, i’m having fps problems playing with an AMD setup. My game runs a 120-100 fps, but when i’m travelling or enter to a dungeon my fps goes down to 50-40, it’s kinda hard to play dungeons like that. The wierd part is that if i use the “auto settings” option i will put everything on very high (meaning my setup can run the game very good) but i can barely play having everything on low. I have tried every solution you find on forums, reddit, etc. But nothing seems to work. My setup is:
-RX 480 8 gb.
-ryzen 5 1500X.
-16 ram.

This is with High settings. Even if i change it to Low settings its pretty much the same.