You can commonly deploy the device/server with the client's public key.
[0] For example, you could let the user write their key to an SD card and then mount it ro on the device.
Is that uniform across all devices, or device-specific?
Practice I've seen for some years now is to have a label on the device with admin/root password, which is presumably neither uniform across devices nor trivially-determinable from device characteristics (e.g., MAC address, sequential serial numbers, etc.).
I'd still consider that practice reasonably tolerable, though you should be keeping better tabs on assets and credentials.