I solved the problem. Apparently Batocera has kept track of all 3 controllers I have used on this installation. When I go into CONFIGURE A CONTROLLER it detects 3 controllers. Even though only 1 controller is connected at a time.
After configuring a controller you have to go to INPUT P1 and specify which of the 3 installed controllers #0, #1, #2 you have connected for that controller to work. Leaving it at DEFAULT does not work if Default controller #0 is not connected or you have multiple controllers setup.