NativeAOT is on the way, so that you can compile small binaries without any dependencies.
If that does not work, you can compile your project as "ReadyToRun" with "Trimming", which will treeshake all unneeded runtime stuff and prepare an acceptable small binary.
One way to overcome the problem with missing drivers is to setup a python API (e.g. Flask) and triggering things through the Avalonia UI. Or you could wrap a binary execution via CliWrap, an excellent library for .NET to run external processes.
I once wrote a little eventhub[1] to run on the Raspberry PI, it is just an experiment but worked ok.
There is also .NET IoT[2], which exactly targets these platforms.
Although I had to rewrite the software, because the original was WinForms, it was a pretty simple application.