In my experience, low level security careers tend to be held by people with a very strong background either in exploitation, code review and reversing or hardware and electronics. Quite a few of us have been around for a "very long time" and passed through many roles before focusing on this subject.
To get a better grip on the topic, I would suggest the following pointers:
- challenging CTF, either exploitation (pwnable.kr, overthewire, ...), reversing (flare-on 6 upcoming), assembly (microcorruption) or hardware (RHME2)
- buy a development board for a popular chip, e.g. i.MX6, STM32 or experiment with raspberry pi
- read datasheets and technical reference manuals for the SoC on those boards, experiment setting up security features such as secure boot, or using a hardware cryptographic accelerator
- experiment with U-boot and read the source code
- get awesome at C and assembly (x86, ARM, ...)
- learn how to extract contents from non-volatile memory
- learn reverse engineering of raw binaries
- learn to work with common hardware debug interfaces (JTAG, SWD)
Just realize you're choosing a hard topic here and you won't be done anytime soon. Persistence will be required and whatever you've done up till now is a good start but will not be enough.