Upon further testing it only seems to happen when using multiple XBOX 360 wifi dongles. As soon as an additional controller connects to the second dongle it messes up the order of the controllers in emulationstation. After that the order in emulation station no longer matches the order in any of the emulators you launch.
I play 5 player titles with the family and it is kinda annoying trying to figure out what the controller order is every time.