Also, your story is quite fascinating, can you tell us more?
I am very, and I mean VERY interested in working on something like that, but not sure how to transition out of the generic CRUD web programmer to your area... :(
The difficulty comes in configuration, where things are much more complex than the cloud, since every device is "special". Different customers, keys, settings of all sorts. So we allow individual environment variable settings for each device.
The real complexity is in ensuring you can reach every device as long as possible, and that the devices behave well when they can't reach the cloud. We set up a VPN to ensure the former, and have a pull-based architecture where the device is responsible for "catching up" to the fleet when it comes back online to handle the latter.
There's a bunch of other things we've solved, like container deltas to improve download times by something like 10-70 times (and reduce bandwidth costs) but honestly the biggest difficulty has been in integrating cloud/hardware/developer/network/OS etc seamlessly, so people can succeed at building fleets, and fast. It's like there's 10,000 papercuts moreso than one or two big problems to solve. It's a long chain of things that all have to work right to make the system work, and each one has to be done in a way so that it doesn't break after some time passes (see the OP for an example).
To answer your last question with (yet another) plug, you can probably run through our getting started guide in an hour and have your first device that you can deploy a js project to pretty fast. If you spend a few days, you can start to have some real accomplishments. It's not that hard to make the jump these days if you have a web background (hell, most of the founders didn't have IoT experience when we started) but there are things you need to learn that relate to hardware and linux if you want to do something more advanced. You can learn at your pace though, and I thoroughly recommend starting with a project you enjoy, something like, but not necessarily https://www.balena.io/blog/make-a-web-frame-with-raspberry-p...