As an embedded programmer, I wholeheartedly second this.
I'm in an area where there are relatively few opportunities to do close to the metal work. The ones that do, don't pay any more than backend web developers make. In fact I've come to assume that they can offer less due to either the intrigue of the work, or because you often compete with computer/electrical engineers who aren't expecting outsized developer salaries.
Likewise, while I have loved the skills I've learned in embedded and low-level work, working at an embedded shop has taken a lot of the joy out of the learning. You trade off an inherent rewarding development environment with the realities of developing against hardware, where project cycles are long, you're often dependent on horrible vendor APIs and support, and where everything moves much slower.
The counter to that is the type of work I believe you're describing, which sounds like HFT optimization on Intel. I got into embedded because I wanted to eventually end up there, but again, it's an extremely limited market. And as someone who has indeed dabbled in the Fog optimization manuals, you quickly learn to realize how much the ROI of assembly isn't worth it. The future of speed isn't going to be going lower down the programming stack. It's going to be in new hardware: FPGAs and ASICs, writing RTL for system specific CPUs. And that is highly exciting, though it necessitates a pure programmer has to learn hardware at a deep level as you imply.
As for me, I concur with the statements others are making about using my skillset for security research and reverse engineering. Working in embedded development has bored me to tears, and taken the magic out of learning to love the skills themselves. I'd much prefer to work at a pace that isn't limited by the pace of a cross functional team of software and hardware engineers.
Learn the skills, yes, completely. The knowledge is worth it. But for the love of God, don't get expect to get joy out of working in it.