for the flashing borders, it seems to happend sometimes ; i've not yet understand why. i think that it's only on retroarch. can you confirm that the n64 has not this issue ?
for the keys, can you access via ssh and try :
choose your keyboard
and press the volume keys
on my side, it's still working, so because i've upgrade the linux version, perhaps that there is a change on your keyboard (the VOLUME key is perhaps renamed), the evtest, if you know how to do it would help to understand.