Semiconductor theory is more interesting, but don't even consider it unless you love math and can do standard integrals without having to look up tables.

Also, the mistake I made when I went to grad school: Semiconductors seemed to be a "new" field compared to the rest of EE. One of my undergrad professors said "They still haven't figured out what a standard textbook should contain."

In reality, from a research standpoint, it's a very mature field. Don't expect low hanging fruit. If you're going to focus on theory, expect it'll take a number of years of dedicated study before you get to the frontier. You'll need to know quantum mechanics and statistical mechanics, and some electromagnetics, just to begin studying semiconductor theory. Then a whole bunch of specialized solid state courses. Then you start studying the specific subtopics (reading key journal papers).

> But later I'd want to work as compiler engineer

Why are you wasting time with semiconductors...?