The O'Reilly book Understanding The Linux Kernel is fantastic. Starts you off in the deep end with memory management/addressing and fans out from there. Notably excludes networking, as the book is long enough without it.
As far as kernel architecture, the rest of the system can work with the assumption "writes go into a buffer, reads come from a buffer, and the networking subsystem does the right thing in between", plus some stuff about DMA optimizations.