I read both Computer Architecture and Computer Organization and Design in college and I really enjoyed them.

I would recommend COaD to any beginner who wants to learn some basic concepts of computer design (and if working with FPGAs why not build one).

CA deals with more advanced concepts but doesn't overwhelm you with math and circuit theory (as you noted) so it's a natural progression (from COaD). I think something more advanced and "hard science" should be part of a post-graduate curriculum