Ok. Following the tips in the thread I solved the issue as well on my Elitedesk 705 G5 mini. So as previously mentioned in the thread there is a solution.
However! The mentioned cause of error is misleading as it’s not about legacy boot, but rather the problem lies in HPs stupid bios settings for secure boot. For some reason the secure boot settings as well as legacy boot settings are grouped together in bios.
To solve the problem, you need to change the secure boot options under Advanced > Secure Boot Configuration > Configure Legacy Support and Secure Boot and select Legacy Boot Disable and Secure Boot Disable.
By doing this. You disable Secure Boot witch solves the problem, but the solution has nothing to do with the legacy boot changes. It just happens to be the only way to disable Secure Boot in bios.
This solution work with Batocera on external drive as well as Batocera on internal drive (as I have).