I would argue that it should scare you. This is targeted towards people who are working in the AI world and are likely not embedded experts. Also, the bigger problem is closed source libraries and drivers that choose not to support arm64.
This is a totally different world than the open source world that you are referencing, likely that embedded product will also not be open source. Commercial licensing is your only option in that case anyway, unless you are just looking for FOSS stuff with permissive licenses, but in that case those won't be closed source to begin with...
So the problem you perceive simply does not exist. The biggest questions will revolve around commercial viability, proof-of-concept and time to market. Rarely around such details as closed source libraries or drivers. Though, in case your supplier goes belly up those could become factors, but for that you have escrow agreements.
arm64 is very common (Android!) and Xavier runs stock Ubuntu. If your camera manufacturer doesn't ship a driver for arm64, you should speak to them. It's extremely likely that they have one already.
The problem is that chips like that are about as hacker unfriendly as they could be. But in industry hardly any of that matters.
Being in the middle - hundreds to tens of thousands of units to ship - is the toughest place of all. No vendors will talk to you and you're going to be out of options if someone decides to EOL that chip your design depends on.
In that case I would advise to only use open hardware and to take the associated performance, size and power requirements hit. At least your product will live.