The simple solution would be to only have the controller you want to use plugged into the system. The next best thing is to go to controller settings > player 1’s controller each time you switch the system you’re playing. The last one is to use RetroArch’s override feature to permanently lock in your controls for that emulator.
The fundamental problem is that you’re trying to use a plug and play system as though it were a set up and configure everything system. If you remove Batocera’s configuration for controllers by using a RetroArch override, then you lose the ability to plug in new controllers and use those with the system.