What Are the Tricky Parts of Embedded Development?
embedded.fm
embedded.fm
* You're developing a controller that controls something. Now you have to have the hardware it controls. This can be a sizable piece of industrial equipment. You may need a simulator for it.
* I've seen auto engine controls developed. Phase I was run connected to an engine simulator. Phase 2 was run connected to a test board with the auto components, including one spark plug, and an analog computer to simulate the vehicle power train. Phase 3 was run on a test stand with a real engine. Phase 4 was run in a car with a debug system plugged in. Yes, you need all that stuff.
* Safety issues. You may need a separate safety system to monitor the primary system and shut it down. Traffic lights have a simple hard-wired device which has inputs from all the green lamps, and a PC board wired with diodes to indicate which can't be on at the same time. If the checker detects a conflict, a relay trips, all the lights go to flashing red, and the CPU can't do anything about it. Some systems allow the CPU to try again after 30 seconds of flashing red, but usually it requires someone to come out and replace the electronics.
* JTAG is your friend, but JTAG is so low-level that it's a huge pain.
Yes, JTAG is definitely your frenemy. Hopefully you have a fast version or even better a full trace probe but those are usually vendor specific so more cost
An example are SEGGER products -- their ordinary JLink products (JTAG/SWD) vs their J-Trace series (dual Cortex Trace/JTAG) -- https://www.segger.com/jtrace-pro-cortex-m.html -- The difference is somewhere in the realm of $1000, in this case (plus some other CPU support differences). Their tools are even fancy enough to reconstruct the instruction trace back into live code coverage/profiling reports (using the DWARF information). Your IDE "populates" the report live as the trace comes in.
(As a side note, SEGGER makes an excellent and affordable JTAG probe if you're new to embedded stuff like me and using ARM systems. It's a standard probe but comes with a _lot_ of very useful features and is standard GDB-compatible -- the RTT features are much nicer and faster than printing over UART/SWO, for example. The non-commercial, basic version is only $60 too, so it's very affordable. I have one along with my BusBlaster for OpenOCD setups.)
(Arduino driven ignition and fuel injection for a motorcycle a bunch of years back. Gave up on it 'cause though it worked well, I never got it reliable enough. Fun way to waste a year or so's worth of weekends/evenings...))
Support was pretty good for custom sensors, only problem is your knowledge has to run damn deep with these things before you can get the engine to start :)
I had wideband autotune for O2, idle control, knock retard, coolant and air temp compensation, basically every original sensor from the motor and a few extras I bolted on worked great
Of course, you need to know how to get an engine running from a blank fuel and spark map...And blank enrichment/retard and aux maps for all these sensors.
Anyways, was good fun. Lot more interesting than the toy stuff people are doing with Arduinos for the most part
* Just run unattended, bug free, 24/7, for a year. Very tricky.
* Not giving up when your errors do lead to physical damage. Repair the heat exchange radiators that froze because you didn't test what happens when outside temperatures drop. Change the timing belts that snapped because of off-by-one mistake when estimating position of part on conveyor. Straighten the door bent by air pressure when your pressure/temperature regulator oscillates because it was tuned without taking sudden disturbances (opening the said door) into account.
* Staying motivated during the long product development cycles. Just when you think you're finished when manufacturing starts- there is still a lot of grunt work to do.
* Designing a logging system that stores enough information to pinpoint issues yet be lightweight enough.
* Going back to V-model / Waterfall development.
You've first got to figure out what kind of SOC you want, and no it's likely not Arduino/RPi or whatever is popular for hacking projects. And no it's not the EE responsibility to choose these things; they just route them. You have to work with vendors to figure out what performance characteristics, lifespan, etc you'll need for your system along with preferences such as OS support (first class, 2nd class, community/experimental, etc, because no, you don't typically just download and install the latest Ubuntu and call it a day; drivers etc are often custom and proprietary, so unless you want to write your own driver stack...). It all doesn't "just work out of the box" like a SAAS service. You've got to put a lot of thought into "other stuff" and make some big decisions before you even write a line of code.
Anything less than equal-and-honest cooperation tends to lead to lopsided or quirky designs.
Just like, past a certain point, it's impossible to do good embedded development on the software side without at least some basic knowledge of EE, there's no way you can come up with a good hardware design if you're completely ignorant about software development. It's no coincidence that the best engineers are the ones who can do both.
Sadly this seems to be a common problem; either the teams are siloed, or one of them is in fact outsiders. There's not enough of the tight iterative refinement loop work to put features in the right place. This seems to be why so many companies put out hardware which is handicapped by terrible drivers.
* Develop a capture program for the input feed/sensors, and capture as much as you can.
* Develop a software model to recreate the input for a 'target' embedded system
* Write the embedded system against the captured input. That will get you 95% there.
* Run it 'live', check anything wrong (you will). Usual debugging & tweaks.
* Do a feedback loop if problem input comes in, and keep /that/ preciously for your test unit sequence.
* Once software is done, every time you make a change, run your simulator with all the test input you have and check your output for divergence.
I very, VERY rarely need to JTAG into a board, I'd rather spend the time on the simulation model and get it accurate as I can than spend time 'debugging' on the target.
That's why I wrote simavr for example [0], but I also use qemu lot for bigger systems. Unfortunately it's next to impossible to get anything upstream in qemu, so most of the work there just is dropped eventually [1].
What is the performance you're usually seeing for simulations like that?
For example in my last opens source project. I fed the signal I was receiving to a plain linux program, because I was more interested in the algorithm than the pure embedded bit. THEN I transposed it all into the embedded firmware as is (See [0], there are still remnant of that in the rather shotgun-style linux program).
So 'applicative' simulation is as good for many cases, while the instruction level is good when you want to validate the /true/ embedded responses of the real CPU...
The idea is primarily to be able to isolate the problems, and be able to develop/test them in a feedback loop individually if possible.
As for instruction level speed, well, these days QEMU is likely to be at least as fast as a good ARM CPU without any problem (very often, a lot quicker). And simavr is several hundred times quicker than a real AVR when running on a x86* host. In fact, it's often more of a problem trying to simulate 'real time' like input/timers than the other way around...
This is how I develop, too.
The strategy is essentially to develop the algorithm/software outside the embedded platform, then port it to the embedded platform. I, too, record input data streams and process them via desktop. Then I stream the recorded data through the embedded system.
In the worst case, updates can be made in the embedded system, but in the best case, they can be developed and proven outside then the benefits carried out into the embedded system.
These days, I firewall as much code as possible into "flat", "pure" C/C++ files that contain no low-level calls or libraries or includes whatsoever. Even my tasks under RTX are wrappers to the actual tasks. I can run my multi-task programs under linux just fine with a pseudo task manager. My build system uses the same exact C files for both the desktop and embedded code. There are layers of API calls, but the compiler just optimizes them all out so I don't even sweat it.
You might think that makes it less "realistic," but in practice, it makes it more robust, since the tasks are (mostly!) coded to run at any frequency or pattern of switching.
You might think this massive front-loading takes time, but it's never wasted. Things come out so clean at the end. Especially because lots of documentation can be written between the desktop and the embedded porting.
I was inspired to do this by projects like MAME and MIDIBox. Both are multi-media real-time system across numerous platforms. They benefit from repeated porting. It shakes the bugs out.
Once I submitted an issue with a workaround that had some rather nasty side-effects, and asked for a better solution. Weeks went by. They were promising a code sample. More weeks, more "Well, we're still working on this." The solution I got back, about a month later, was exactly the workaround I'd included in the support request, with some of the serial numbers filed off.
One issue took months to resolve. We finally told $VENDOR "Look, we've got to ship. But we can't if we can't get this (critical component) working, and we'll be forced to ditch you." That got all kinds of political bullshit out of the way, and in an hour we were talking to the guy who had designed the circuit we were having trouble with.
"Oh," he said, "You have to do (three simple steps)". And I tried it while one of my cow-orkers kept him on the phone, and it worked.
I never want to deal with $VENDOR again.
Like this 850 page behemoth that I wrote a driver for:
http://web.archive.org/web/20110701163842/http://www.broadco...
So many undocumented pieces and workarounds for silicon bugs, etc. Luckily there is a Linux driver to refer to, but still...yuck.
"Adjust [setting] until the sun shines most brightly upon the screen."
Fortunately, the setting did not have any effect on our friendly neighborhood ball of fusing hydrogen, and a bit of fiddling around eventually revealed that the instrument had turned its gain way up, allowing one to easily determine the zero point by adjusting the bias until the screen filled with noise (rather than pegging at the top or bottom). If one were to squint a bit, I suppose it might look like the sun shining on the screen?
I've worked embedded systems for years but every year I tell my colleagues that I'm switching to IT so if my hardware doesn't work I can just throw it away and buy a new workstation.
Also, recently having received an Arduino project to work on, it was a bit of a garden path to get to the point where I could get it running under GDB.
Now I make sure I always have two working pieces of hardware, and I verify the hardware functionality to the extent possible before trying to implement software features on top of it.
Always read the errata, folks.
Case in point: Microchip ENC28J60 (http://www.microchip.com/wwwproducts/en/en022889). Less than half of the advertised features actually work.
The ENC28J60 did at least four more silicon revs after the errata were released, so... yay Microchip?
In general, I now avoid Microchip like the plague. Which means I now avoid Atmel, Micrel, SMSC, etc. <cries>
Brands? A lot of TI and ST.
1)Writing device drivers for a chip that I can't get to work. Has to be my code, this is a basic function of the chip; why am I so stupid??!!? Finally call the vendor: oh, yeah we know about that bug, the next revision fixes it. Why isn't everyone else screaming about this problem? Well, you guys are only the second company to sample this chip. The first one found the bug!
2) System has to autoload a tray when operator closes the door. Works fine right up until it doesn't. Turns out some of the (visually opaque) trays are transparent at the infrared wavelength of the sensor that detects them.
3) (on a call with Field Service). Intermittent problem that seems to be software: a moving carriage is reporting errors, but only during certain moves. Field tech has replaced every relevant module and problem still happens. On a hunch I have him disable the safety interlocks and open all the doors to watch what happens. Problem is that there is a weak spot in an cable in an unrelated assembly that causes it to momentarily collide with the carriage, making the carriage fail to get to its destination. By the time they both stop moving, all hardware tests fine, so no one suspected the other assembly.
4) I love telling this one: hardware caught on fire because a jammed assembly caused a motor driver to overheat. Tester filed a bug against the software because it didn't report an error to the user. The CPU was fkn toasted!
So each "cycle" can be as long as, flash the thing, wait for it to warm up, get the device into the right state, test the thing you wanted to test (which might involve another device etc).
Also a lot of the time you don't really have good specs or there are closed source bits e.g. firmware blobs for wifi.
That said, there are some cases in embedded where there is just no possible substitute for the real thing. I've heard of physical actuators and webcams being used to drive devices from test harnesses in those cases. I think a popular example of this that's open source is published by one of the pay tv companies.
Many of the Android app test facilities use webcams to report what's actually on screen rather than relying on software hacks (which are different for every device).
The HAL layer for device must be tested separately. Ideally do it in an automated systems test jig, which has the target hardware+software, and allow programmable input stimuli and output verification. You really should have such a setup for your final (per-board) QA anyway, so better to start developing it sooner rather than later.
For instance, on-device testing is a first-class citizen in the microcontroller programming framework I've developed. Shameless plug: http://www.jonnor.com/2017/02/data-driven-testing-with-fbp-s...
I recently ran into an issue where an interaction with the DMA controller and the SDRAM controller in a MCU lead to invalid reads from the SDRAM. I still don't know what is actually going on with the MCU (SAME70, in this case).
Using a HAL lets you debug your application code more easily, but I don't think I've ever had a hard problem in an embedded system where the underlying cause wasn't related to some odd hardware behaviour under a corner case stimulus. There's no way to simulate that without a full model of the entire device (which isn't available).
The virtualization helps you ensure that A is doing what you intend to do, but not as much with B) knowing what your intent should actually be... and even if you can build a reasonable model of 80% of the behaviour, the really critical stuff tends to be out of the envelope type constraints like, oh the sensor gives this bogus reading under conditions X, so you need to ignore it or avoid condition X.
Edit: case in point (an adjacent story in HN)
https://news.ycombinator.com/item?id=13700798 "UPS Showcases New Delivery Drone, Fucks Up"
Debugging a quality control machine I inherited:
The heat sensor would trigger randomly, about twice a week.
It was a software problem, as swapping out for a brand new one didn't fix anything, despite doing it three times.
Everything seemed above board, and the review didn't take that long, it was only 8000LOC, simple C.
I found the answer on a test device: The vendored library that we could link against, but not look at, was reading the temperature from a sensor over I2C... And then reading it into the main onboard chip via an analog pinout. The docs said digital.
Which meant, when an inspector walked passed the machine, and looked at it, their own electromagnetic presence would mess with the readout. It wasn't sealed properly against this, because we believed the documentation.
The machine was several hundred thousand £s worth of lasers, optical table and stepper motors moving a stage that must have weighed half a ton. The purpose was to print digital holograms, line by line, on glass plates. Between each line there was a delay of some seconds, during which time various calculations took place; even the smallest 3"x3" holograms took half an hour to print. Larger ones could run overnight. Once printed, the (expensive) plates needed another half an hour of darkroom wet processing, after which they could be briefly viewed (the image disappeared again until the plate was completely dry - uneven shrinking destroys the interference pattern).
The byzantine hardware control software had been written long before my time - my job was merely to write the imaging software that fed this machine as it ran. There was an elaborate startup procedure to follow, and just before you'd hit "start" you'd run my little C program, which would synchronise with a timing signal from a serial port. Then, as the stage whirred back and forth, it would feed images to a little display - 30 Hz, pixel by pixel, line by line.
One day, just as we were getting confident of the rhythm of operating the machine, a larger hologram came out "back to front" in a strange way - one of the parallax axes was reversed. Perplexed, we carefully tried again with the same file - and it came out fine the second time. Unable to reproduce the issue, the matter was dropped.
Some weeks later, just as our confidence had begun to wax again - boom. Another inverted hologram. Another large plate wasted. Another careful investigation, another failure to reproduce. A Heisenbug!
This continued happening, off and on, most vexingly - and of course, never when I was there, and only with the larger, more expensive plates. Eventually one day I came in and it had happened again, and this time I noticed something peculiar - the hologram had finished printing, but my little C program was just sitting there, waiting on seemingly the last line, for a timing signal that would never come. And the light began to dawn.
My C program had the basic structure "buffer line, wait for signal, display images, go to start". It turned out that what was happening was people were hitting "start" too soon after launching it, before it had had a chance to properly buffer the first line. It thus missed the "start line" signal of the first line - which caused it to print line 1 as line 2, line 2 as line 3, etc. And because it was a rastering stage, this meant that every line was printed in the opposite direction to what was expected.
The really treacherous thing about it was, it only manifested when I wasn't there because I knew instinctively to allow the program enough time to buffer. It only happened with big, expensive plates because they had longer lines which took longer to buffer. And it only happened at all when the operator of the machine was sufficiently confident, or in a hurry, to perform the final two tasks of the checklist within seconds of each other.
The fix? A note in the checklist.
I miss the job, but I don't miss debugging that darned machine.
By far the most frustrating part was waiting to test it. I didn't want to run the washing machine (empty) just to test it, so I had to schedule debugging and testing times when we were doing laundry for real.
The moment it worked for the first time was definitely a happy one though!
* If you or your coworkers are not organized, you can waste a lot of time looking for proper sized wrenches, proprietary screw heads, speciality crimpers, oscilloscope probes, etc. etc. The more hardware the company makes, the worse this problem can be.
* Most internal connectors are not meant to be constantly plugged and unplugged. In a testing scenario where you have to change connectors or test harnesses frequently, it is common for the connectors to break or wires to become loose. Then you have to waste time figuring out why your hardware stopped working.
I spent a about 15 years mostly writing server code for UNIX machines in C (before ditching it in favor of Java, and 13 years later: Go). Since embedded programming is a bit of a specialty field where things like predictable performance and robustness is important, I expected the embedded world to be pretty professional. Because any time people start using words like "guarantees" or "real-time", you tend to assume that they do some pretty amazing stuff.
I can't really say that's what I found. A lot of code is brutally ugly, many lack understanding of even the most basic defensive programming techniques and there's a lot of superstition around abstractions by people who don't really seem to understand what comes out of a compiler (Having the compiler "compile away" abstraction layers was something we often obsessed over on projects I worked on in the 90s and early 00s).
Code is often badly organized, badly formatted, badly documented and amateurishly maintained (eg bullshit commit logs -- if the source is even kept in a version control system). As a result, you constantly fight the urge to rewrite stuff because the code is just so damn hairy. Of course, any talk of rewriting code makes people nervous ("we invested a lot in order for this to work and now it does! Don't touch it". Yeah, I'm not surprised it took a lot of work)
And all of this was code by serious companies whose brands you have heard of.
I'm hoping the IoT craze is going to accomplish at least one thing: educate embedded developers. Sure, a lot of us "regular" software people are going to run around like a bunch of flatfooted morons because it is unfamiliar territory, but the embedded world is in _dire_ need of some software culture and discipline.
Although coming from the microcontroller world there is practically nothing in the STL which gives me the guarantees I need to use it in ISRs and most devs have no chance at reviewing the STL piece to see if its going to deadlock in ISR or something.
Another problem is that although the abstraction compiles away in release mode the debug build usually still has to fit on the chip and/or meet realtime deadlines. Tooling is still super bad at optimizing part of a build and not other parts even though we have an optimize pragma (I brought this up in the SG14 working group but concluded that its more of a tooling than a language issue).
In short I think at least the drivers should be written in C++ with proper abstraction but for the most part those abstractions have not been written yet and we can't just borrow from other domains because we have to be deterministic in timing and RAM use and also usually use other threading models (RTC event based) and memory management models (pools, state local storage) at least in drivers.
- Odin Holmes
And for embedded in high value (hundreds of millions of dollars) or human life, I think there has always been lots of proper discipline and engineering process.
As an EE working in software, I volunteered. One of the comments I made was that the many magic numbers in the code should be replaced by definitions that explained what they meant.
I got back code for re-review that contained the line:
#define ZERO 0
To this day I'm not sure if the author didn't understand or was just irritated at having me review his code. Probably both, come to think of it.
The IoT craze has forced a lot of software on hardware companies: everything from ZigBee, Thread, and Bluetooth stacks to RTOS's to manage all of those stacks, to IDEs that are actually usable.
The companies embracing software as a vital part of their products are doing well, and will continue to do well at the expense of the companies who treat software as an after thought.
I have seen this a lot, specially like caring about bounds checking or virtual method dispatch without measuring if it really matters.
Even some modern microcontrollers are quite powerful versus those mainframes running Lisp, yet were we are still preaching Assembly and C as if it was the 80's, early 90's.
* Documentation? HAHAHAHAHAHAHAHA LOL
* Debugging can be extremely challenging in real-time systems. Things like JTAG printf will slow things down enough to wreck your timings.
* You have to at least know the basics about the hardware, especially if you're doing control systems and meddling with GPIOs and such.
Currently working on a thing (approaching EOL thankfully, but still have to support and add new features until the manufacturer says "no seriously we're cutting you off") that uses a gcc cross compiler from the 90s, no gdb for me.
Side channel debugging is a life skill. Oh it's crashing weirdly with no diagnostic output on the UART terminal? Rewrite a timer interrupt that's still running to print your diagnostic information when you toggle a switch. It's actually fun if there's not deadline pressure or a line down situation at a customer.
I was taking over development of a new display driver for a small hand-held instrument. The display and drivers were both new to the organization, so there was no experience in-house. And we had these strange "color-flowing" bugs, that nobody could understand. Fields of greens and blues that bled across the screen in weird ways. Of course everybody thought it was a driver (=software) bug.
Weeks passed, my hair got thinner, then finally I looked once more on the schematics, traced a signal back to the CPU, and said to the hardware designer "hey, isn't this a 3.3-volt signal?" Turned out we were backfeeding the display driver from its reset line, causing it to power up due to the voltage overpowering the input circuitry and flowing into the power rail, enough to power it up but not to make it behave correctly. Yikes that was frustrating (but fun to catch, of course).
I've come to realize software almost always gets the initial blame. Software is almost always the one painting the error screen. So even if it says "Voltage out of range", someone is going to accuse software of not working. So they pull you into the lab, watch you open the box, probe it with a multimeter, show that the voltage is out of range, and then get the hardware engineer.
I once spent over a month (nights & weekends) tracking down a memory corruption bug. Everyone accused software of course. It turned out to be poor signal integrity on the memory bus (hardware problem). It was horrible.
Yea, "fun"
There's more than one project that fixes chip issues via errata and curses at software engineers.
RTFM is a nightmare, the time you look at another page for a related device, you have forgotten what you read previously.
Besides the undocumented proprietary protocols which isn't embedded specific as a backend engineer I used to struggle heavily with development environment setup. As a JVM and Python guy I'm not used to fucking with weird compiler tool chains at all.
I also found it a challenge writing Linux device drivers for chips with an external 32 bit bus and an internal 16 bit one. The 32 bit data had to be split into two, set the data pins for the first half, set a bit saying the data was available, wait for it to be read in and repeat with the second half. Do the reverse for reading.
Also challenging was setting the bits in a register for a chip with multiple commands per register. My program checked the input data for an error, e.g. trying to set three bits to 128, read the register, masked out the bits to be changed, changed them, wrote the value, read the register, masked out the bits and checked them to make sure they had been changed.
Then there was the time the board manufacturer changed the memory map of the board without telling us. Boards made before April would boot, newer ones didn't.
I also found that some device drivers from SOC chip manufacturers had to be debugged before I could use them.
I've successfully made code that worked just fine with the online MBed IDE and gcc-none-arm-eabi, I don't recall anything having to be done differently, just more to setup locally (linker scripts and so on).
IIRC, most of the stuff they've added are surrounding libraries and such.
It's basically a canonical example of why the Affero GPL exists.
Furthermore, you don't even need to use their online IDE to do anything. The online IDE has an option to export your project such that it can be built with arm-none-abi-gcc and make[2], or you can use their mbed-cli tool, which is a python based command line tool, and is also on github[3].
The mbed guys have way embraced open source and working with existing tools, and I wished the community would embrace them more, since I think their API's beat the snot out of Arduino.
[1]: https://github.com/ARMmbed/mbed-os [2]: https://developer.mbed.org/handbook/Exporting-to-offline-too... [3]: https://github.com/ARMmbed/mbed-cli
I'll have to re-evaluate it the next time I need a little mcu board!
Simple challenge: the system must run 24/7, without interruption, for at least a year. Bug free.
Quite the tricky thing to achieve.
It seems to be not very visible. I'm under the impression that some "not very visible" work is well paid and not going away any time soon. I'm thinking of some cobol or pascal developer called out of retirement at great expense.
Or is this like being at the edge of new/exotic web 2.0 where you are continually looking for the next job, re-learning tech over and over.
I know this is a big topic and perhaps there is no correct answer overall.
If anyone has any resources to job sites or further articles I'd love to read them.
The trick is breadth of knowledge. If you can just program a microcontroller, you've got a lot of competition. If you can program a microcontroller AND an FPGA, your competition dropped by orders of magnitude (this is the biggest step and probably the easiest to tack on--Verilog isn't that hard but you WILL foul up until you get race conditions and concurrency properly beaten into your head--do everything synchronously and synchronizers are your friends). If you can program a microcontroller, program an FPGA, AND design the board--you are in rare company. If you are good at debugging these boards, people will worship you as a god. If you have domain knowledge on top of that, you should be starting a company.
> And is it relatively easy to get new work once you've broken into it?
Well, it's networking like anything else. If you've been doing it for 10 years, things seem to just drop into your lap. Otherwise, you have to beat the bushes.
However, if you do a good job, the good people around will notice quickly. And those good people are often at capacity so they will throw stuff over to you.
Obviously, being near a tech hub helps.
> I'm under the impression that some "not very visible" work is well paid and not going away any time soon.
The problem with "not very visible" is that it also means "executive level may not appreciate it". I've seen medical device companies trying very hard to get rid of the single person who actually understands their hardware.
> Or is this like being at the edge of new/exotic web 2.0 where you are continually looking for the next job, re-learning tech over and over.
No and yes. :)
ARM is dominating the embedded space currently. So, especially at the Cortex M end of the spectrum, the base programming environment looks pretty much the same. This hasn't changed for quite a lot of years.
In addition, 8-bit development is mostly going away. The delta between an 8-bit micro and a 32-bit micro is now so small that it makes no sense to use anything other than 32-bits unless you have a very specific use case.
For FPGA, the tools are similarly stable over long time frames.
The peripherals, on the other hand, completely differ from manufacturer to manufacturer. And have bugs from revision to revision. So, that is like learning stuff over and over :(
However, what the peripheral are supposed to DO is stable. Once you understand that, you will be very good at ferreting out the small differences and your life gets some easier.
> If anyone has any resources to job sites or further articles I'd love to read them.
You can read a lot, but doing is better. Go get a Nordic BLE development kit for $39 and build something.
http://www.digikey.com/product-detail/en/nordic-semiconducto...
Getting up and running on an Eclipse/ARM/Segger toolchain sucks. No doubt. But you have to chew through it.
After that, it's about 3 lines to get an LED to blink.
From what I could see pay was on par with most enterprise development. Well above basic web stuff though.
Like the other poster said, if all you do is code, then your position is pretty weak. I have an EE background and I'm pretty good with mechanical things, so I could debug the electronics and figure out issues with mechanisms, etc. (most of my programming career has involved writing code for things that move) My skillset pretty much goes from low-level assembly code up to writing simple web apps, so I was generally able to handle anything thrown at me. I'm also pretty good with creating documentation, understanding the business and built up my domain knowledge at any company I've worked for.
In this field, the more you know, the more valuable you become because it tends to require a wide range of expertise. IME, the people who have done well are always generalists.
When you're developing a product, hardware iterations take most of the budget. As 'the software guy' your job is to be handed a piece of hardware and some vague requirements, and to make it work the way the end user (who you probably never get to meet) expects it to. The product will already be over budget and behind schedule, so you don't get funding and you're probably already late in delivering the product (no, your deadline does not move due to this).
It's hard to find embedded work because the people who 'need a software guy' for their hardware product don't know any software developers so you have to be lucky to meet potential employers.
That said, there is opportunity for some COBOL-style big bucks later on, when suddenly they need to make another production run of the product you worked on back in the day, and they want a few tweaks, and you're the only person on the planet who has the faintest clue how to build the software and program the hardware, so you get to pull your hair out all over again but this time for a more reasonable wage.
Memory allocation, compilers, threading, libraries etc. - you have to be careful at each and every step.
You're probably going to end up working with C or C++ - which is fine, and they are common enough, but you don't get a lot of nice 'VM-ish' things along with them.
You end up often having to know specific things about chipsets etc..
It's great because you 'learn a lot' and will come out a better Engineer - but sometimes that knowledge can be arcane.
You'll spend a lot of time doing things that 'everyone else' in software takes for granted and it can make it feel like you're moving slowly.