There are often two places to set the boot options in BIOS.
In one of them you set the priority of the options ... make sure that all options are allowed and that the internal HDD is the last one while testing.
Do you have an option to display boot options during first switch on?
It may be that your BIOS is set not to display them ..... I usually make sure that no graphic is displayed during POST (BIOS boot) and that the display of options is allowed.
With those options set, insert a bootable USB drive and then power on the PC.
During the time before you get to the Grub page of your normal OS, pressing a 'special' key should bring up the boot options list.
This key is often the F12 key ....... but could be one of many ..... I have seen Del, Esc, F1, F2, F8 & F10 also used.
If your PC can be booted by USB drive it should show on the list .... usually as the last entry.
If you know your motherboard and BIOS version it is also possible to get the info from a manual available on line.