Button stop working

I play with controller, set “B” as normal attack (and it is also skip button). Not always, but most of times skipping a cutscene “kill” B use, as it doesn’t work anymore (neither in skills). Usually, after another cutscene skip, it works again. It happens mostly (but not only) in dungeons.