Modern systems ? I dunno. I wouldn't use anything like Linux, Mac or Windows - there's just way too much indirection and the system as a whole is just too opaque for a beginner. I think the A500 could make a fantastic platform for systems programming teaching. I mean, it's probably going to be AmigaOS 1.2/1.3 so very basic compared to later releases but there's enough in there to hold attention and create something interesting. Plus it's always nice to have something like the original beige "wedge" to hack on, it's a lot more interesting and "real" than working on an emulator.
I think the thing that would hold you back on an un-expanded A500 would be choice of programming language; a C compiler is probably just too heavyweight to use on a basic machine, particularly if you don't have a hard drive fitted. Assembler would be OK, although possibly a little too low-level. Something like AMOS would fit the bill perfectly though: Would run find on older systems and very quick to create something that would hold the attention like a game or demo. Although that's not really "systems programming" as such.