Ok. One thing I'm worried about is that the tooling is made for hobbyists only. Is there a way to do everything from the commandline with FOSS tools and without installing e.g. the Arduino-IDE?
Yes. In fact, the official getting started guide[0] is all command-line based, with an optional chapter later on about using VS Code if you wish.
I have a low volume product in production using the RP2040 and I've never once opened up any sort of GUI for developing the software or programming it.
[0] https://datasheets.raspberrypi.com/pico/getting-started-with...
Probe-rs [2] works perfectly well with GDB and the CLion debugger.
[1]: https://embassy.dev/book/
[2]: https://probe.rs/