I think Pi4 GPU driver support/compatibility with certain plugins is just not there yet. On my Pi4 glide64mk2 is the fastest plugin and will even upscale some games internal resolution depending on what is set in the .ini. This plugin is old and outdated however so glitches are rampant when using this in certain games and compatability is lower also.
On my Pi3b+ I use glideN64 which is the most accurate and updated plugin of all the available options. All games on my Pi3b+ are playable full speed with the resolution set to 640×480. However on the Pi4 this plugin struggles even at the lowered resolution settings. Either due to poor driver support/compatibility or due to some setting that is causing this plugin to upscale to a much higher resolution and the downscale back again. I cant really figure it out even though I’ve dug through loads of the internal config files that should set the resolutions for this plugin on Mupen. Guess we’ll have to wait for bugfixes for good N64 on Pi4 in Batocera.