Not sure if I would recommend building a career in embedded but you will start learning transferable skills doing it in your free time.
Currently, I am an embedded system developer and I concur, I've learned a lot about data structures and how computers work because "everything is C". Especially true as I graduated as EE with more interest in software. I've been enjoying my short career so far, and like the way my thinking has changed from all learned experience. But I can't lie that I worry about whether I should be focusing on web tech and that embedded stuff will fall off. You not recommemding building a career in this area and actually making the switch intrigues me a lot.
Sorry for the wall of text, hope to hear back!
One thing that happened in embedded, and bear in mind that it ain't just low-end microcontroller stuff, is that it tends to be tightly bound to hardware. Hardware tends to be built in Asia (which I remember ramping up in the mid 1980's). Successful/higher volume products tend to be built in Asia. Engineering tends to follow manufacturing over time.
For fun, it's hard to beat specialty companies that build complicated gizmos with software in some sort of sexy business, but they are hard gigs to track down.
Can you expound on that?
After talking with some friends who do webdev (Node, Python) I learned JS in a couple of weeks and got hired into a consulting firm where I worked for 6 months building a document search engine in C# on top of ElasticSearch. I now lead a team at a small firm that builds specialized search applications for different parts of the internet.
I now know of places that have the software culture (and software money) that do embedded systems well but I love what I do now with no shortage of cool problems to work on. Not sure if I would go back into embedded but I still dabble with Arduino's and Pi's.
While they're never going to be as plentiful as web jobs, it seems unlikely embedded jobs are going to be decreasing anytime soon. The market for industrial and consumer IoT and wireless mobile devices just keeps on growing and growing. Unless you actually prefer webdev over embedded, there's no reason to jump ship and every reason not to waste your specialized education.
If anything, it's more likely that web development will hit a peak and then start declining. As people on HN are fond of proclaiming, much of a college education isn't needed to be a successful web developer (very much unlike embedded development). So every year, more and more people keep flooding into web development and that is bound to depress wages and demand at some point.
People get quite invested in their physical things, and I think that's going to translate in to ongoing software updates, and making interfaces between older things and new things (I'm currently working on an interface to old-school telephone gear, working in Rust). More things are being made with computers in them, those computers are getting more sophisticated, people are caring more about software quality and security. And, as a sibling post mentions there's always pressure to (re)engineer things to cost less or earn more money.
Sidetracking a bit, I think one of the best pieces of career advice is to be comfortable at the intersection of a couple different areas/fields. While it may be easy to find/train a developer or EE to some particular skill level (maybe not the best example given the closeness of these fields), it's much harder to get someone at that level in both fields. It sounds like you're already doing that by working as a developer but having an EE degree - you'll do well to stay sharp in both.
I have to say, the embedded stuff feels way more like you have control over the machine, I guess for obvious reasons. The tooling is specifically designed for your hardware, so what you need to do to achieve your goals is more obvious. I've found native app development similar to that. There's nothing quite like just focusing on a specific problem and plugging away at it and not having tools get in your way.
On the other hand, web development is kind of a chore in that there are so many tools, frameworks, and just different ways of doing things. For focused work, I think I prefer embedded development, but it's kind of limited in the type experiences you can gain for sure.
But I have realized that I have some gaps in understanding how a processor works (bad idea to not pay attention in your Comp Arch class), I am trying to cover that up by reading up on Comp Arch (Onur Mutlu lectures).
would working on Embedded (in my spare time) help in understanding the processor at a deeper level ? Any pointers on how I can accomplish that ? I am well aware that modern processors are complicated beasts, more complicated than Embedded systems for sure not sure how transferrable those skills are.
Embedded can certainly help with both, but I would go in the opposite direction from hobbyist-oriented systems like Arduino: learn to set everything up yourself. Bare metal programming, particularly with C and assembly, will require you to understand things like stack pointers, setting up your clocks, initialising hardware.
While 8-bit micros are simpler and you could start with one of those if you wanted (AVR is probably the most accessible), I'd probably go straight to a 32-bit Cortex-M4, get a cheap dev board like [0], and get the sample programs running and try to understand every part of those.
[0] https://www.st.com/en/evaluation-tools/nucleo-f401re.html
I think a good approach to this is to take working examples, look up the used registers in the datasheet, understand why setting up the peripherals that particular way accomplishes the goal of the example, and then start experimenting with changes to those registers.