N64 controllers cannot be used as expected with Batocera due to issues with the mapper. This can be worked around by using a custom mapping created by manually editing config files: https://wiki.batocera.org/systems:n64#controls
Arcade encoders, especially ones that accept multiple “players” at once and only connect via a single USB cable, usually cheap out and only have one USB controller on them, resulting in Batocera only seeing the each player as a single controller. High quality encoders do not have this issue, so you don’t know if this is the case until you try it out yourself.
If this sounds like too many issues for what should be a plug and play system, join the club.
USB SNES controllers should be plug and play though.
