nadenislamarre Thanks for this reply. It's a bit sad but there is a solution.
Instead of writing the settings in retroarchcustom.cfg you can create system specfic setupfile there you write the keylayout and now it works! But is it a thing that you can change on your own coding skills or is it a retroarch thing?
So you create a file /recalbox/share/system/configs/retroarch/gb.cfg and in this you put only
# Keyboard input. Will recognize letters ("a" to "z") and the following special keys (where "kp_"
# is for keypad keys):
#
# left, right, up, down, enter, kp_enter, tab, insert, del, end, home,
# rshift, shift, ctrl, alt, space, escape, add, subtract, kp_plus, kp_minus,
# f1, f2, f3, f4, f5, f6, f7, f8, f9, f10, f11, f12,
# num0, num1, num2, num3, num4, num5, num6, num7, num8, num9, pageup, pagedown,
# keypad0, keypad1, keypad2, keypad3, keypad4, keypad5, keypad6, keypad7, keypad8, keypad9,
# period, capslock, numlock, backspace, multiply, divide, print_screen, scroll_lock,
# tilde, backquote, pause, quote, comma, minus, slash, semicolon, equals, leftbracket,
# backslash, rightbracket, kp_period, kp_equals, rctrl, ralt
#
# Keyboard input, Joypad and Joyaxis will all obey the "nul" bind, which disables the bind completely,
# rather than relying on a default.
input_player1_a = "num2"
input_player1_b = "num1"
input_player1_y = "num3"
input_player1_x = "num4"
input_player1_start = "num0"
input_player1_select = "num5"
input_player1_l = "num9"
input_player1_r = "num8"
input_player1_left = "left"
input_player1_right = "right"
input_player1_up = "up"
input_player1_down = "down"
input_player1_l2 = "num7"
input_player1_r2 = "num6"
#input_player1_l3 =
#input_player1_r3 =
#input_enable_hotkey = "num5"
input_exit_emulator = "num9"
