You can't program on bare metal or arbitrary hardware I/O with just a browser. And running the OS off a SD card makes recovering from screw-ups easier + less costly, so less barriers to tinkering vs. a "real" PC.
You're right, there's some programming you can't do inside of a browser, I was just forgetting all this low level stuff which happens behind the browser :-)