Unfortunately I don’t have the same setup: I use the xboxdrv kernel module to use the PS5 controller for Steam games without native PS5 controller support. I deactivated Steam input for this game (Elite Dangerous, btw.).
I connect (USB) the controller before starting Steam. After connection I immediatly unload the hid_playstation module and start xboxdrv as root (I needed to create a custom mapping for it). Only after that I start Steam and can use the PS5 controller flawlessly in-game.
You might ask, why I am using a PS5 controller instead of an xbox controller. It’s all about ergonomics. The PS5 controller is simply better for me.
I am using gentoo-sources-6.12 . Idk, how mainline that is. It is pretty upstream with some Gentoo patches, I guess.
To increase the responsiveness of the system I changed the default setting of the scheduler to prioritize user input over system background processes (I don’t remember the exact config name in the kernel). Other than that, I compiled it very close to Gentoo handbook recommendations: selecting only what I need and carefully choose between compiling drivers and features as a module or builtin.