Low-Level Programming University – A roadmap to becoming a low-level programmer

Also: to say its a calling is accurate. I love what I do because what I create is physical and interacts with the physical world. I can hold my projects in my hands and say "I made this". I've done web and desktop stuff and can say that this work is significantly more fulfilling to me.

I'll add that while full-time native work may be in short supply, it's an awesome skill to have in your belt. You many not need it often but when you're suddenly up against limits where current technology doesn't suffice(Java/Python/Ruby/etc) it can seem like magic to break down into the lower stack and get 10-50x performance boost.

Having that escape hatch available is incredibly valuable and for me distinguishes the engineers I hold in high regard.

It's similar to learning a functional language. I have no idea if I'll ever use Haskell professionally but learning a bit of it has been a good way to see problems and logic differently. I think both assembly and Haskell have made me a better programmer, even if I never become truly proficient in either or use them directly in my job.

I'm in an area where there are relatively few opportunities to do close to the metal work. The ones that do, don't pay any more than backend web developers make. In fact I've come to assume that they can offer less due to either the intrigue of the work, or because you often compete with computer/electrical engineers who aren't expecting outsized developer salaries.

Likewise, while I have loved the skills I've learned in embedded and low-level work, working at an embedded shop has taken a lot of the joy out of the learning. You trade off an inherent rewarding development environment with the realities of developing against hardware, where project cycles are long, you're often dependent on horrible vendor APIs and support, and where everything moves much slower.

The counter to that is the type of work I believe you're describing, which sounds like HFT optimization on Intel. I got into embedded because I wanted to eventually end up there, but again, it's an extremely limited market. And as someone who has indeed dabbled in the Fog optimization manuals, you quickly learn to realize how much the ROI of assembly isn't worth it. The future of speed isn't going to be going lower down the programming stack. It's going to be in new hardware: FPGAs and ASICs, writing RTL for system specific CPUs. And that is highly exciting, though it necessitates a pure programmer has to learn hardware at a deep level as you imply.

As for me, I concur with the statements others are making about using my skillset for security research and reverse engineering. Working in embedded development has bored me to tears, and taken the magic out of learning to love the skills themselves. I'd much prefer to work at a pace that isn't limited by the pace of a cross functional team of software and hardware engineers.

Learn the skills, yes, completely. The knowledge is worth it. But for the love of God, don't get expect to get joy out of working in it.

I know a hell of a lot more about what our code is actually doing up and down the stack than my colleagues, including most of the managers I've had and "Principals" I've worked with and so on. It helps, a whole lot, in fact, and I find that it's personally rewarding, but it doesn't amount to much more than being the person who gets asked the challenging profiling/performance/optimization questions.

Genuine question: what would you quantify “not by much” as here? In my experience, it's not uncommon to see 2x+ speedups from well-written software pipelined hand-optimised assembly in hot loops. (Especially so for in-order processors, like you might see in embedded applications or as a LITTLE core in your smartphone.)

Assembly is useful for understanding early initialization code for embedded platforms (startup file). Depending on how aligned the vendor's board support package is with your actual target, you might have to touch this, and since this code runs at a time where there is neither stack nor heap (can't have a stack unless the stack pointer has been set up), this is easier in assembler than in the stackless C variant that you might coerce your compiler to support. The same applies to debugging issues bootloaders, where you'll have to inspect how the reset vector looks.

1. Writing embedded software. You're not going to get very far if you don't know how to set up your memory, text section, etc.

2. Debugging embedded software. Good luck interpreting your JTAG output if you don't have a good knowledge of assembler. Even if you get to use gdb/Wind River Debugger/etc., you will want to be able to tell what is happening. Symbols only go so far. A backtrace will get messed up and you'll need to look at the stack to figure out what happened. An access violation will occur and you'll need to figure out what line of code it corresponds to.

3. Debugging without source code. Same as #2, even more difficult.

3. Reverse Engineering. You are trying to figure out what is happening in a proprietary driver binary blob (such as NVidia's Linux driver). You are trying to figure out how to get your software to work with an ancient piece of software whose author has long since been out of business. Even if you have the cash for Hex-Rays decompiler or can readily use the Hopper decompiler, this code is simply not high-level enough to interpret without knowing assembly and having a deep understanding of the stack and memory.

4. Vulnerability research / hacking. You are writing an open-sourced client for a proprietary protocol, looking for a memory corruption vulnerability, etc. You can pretty much forget it unless you know assembly.

http://blog.timac.org/?p=1707

Still, I have to say that there is indeed an opportunity for a few low level people there.

I"ll go on and make another bold claim. Focusing on lower level stuff and systems concepts lay an excellent foundation for designing complex systems regardless of the language used. Once you understand how a program is executed from the ground up i.e. right from the program counter to page table lookups to cache hits to cache coherency, all abstractions are easier to deconstruct and understand. I work up and down the stack and this has been my, quite possibly just anecdotal, experience.

When you know what the stack and the heap are, when you know how memory locality affects performance, when you know the full cost of allocations, you can write better Java/C#/OCaml/Haskell, you can check on the assembler they output, you will be ready to occasionally write small C libraries if portions or slow, etc.

Ignoring this stuff completely leads to text editors made in electron.

I'm studying (at my own pace) the book, practicing the problems, and watching the lectures--I even downloaded the Panopto iPhone app to watch lectures while on the bus ride into work.

The course is very rewarding: x86, memory systems, systems programming, linking, socket programming ... all packed in a single book, providing a holistic view of systems, from a programmers perspective.

My point is that your job market is not representative of the whole world and you're acting as if it were.

The most obvious example would be the video game industry were you're not straying very far from C++ anytime soon. Performance, speed and native compatibility are a big deal there.

There's definitely a valid career path there.

I can't say I've ever lacked for work and I make $125k in a very low cost of living area, and if I wanted something new I could have interviews arranged tomorrow both where I currently live and pretty much any major US city.

It's probably not on the same scale as web dev, but there's also a lot fewer of us working on this side of things.

He replied that he'd have better job security than any of us in IT doing stuff like PHP or Java. He said the reason is he had to be on-site to do his work. Impossible to out-source. When I pointed out in-sourcing (H1-B, etc), he said they preferred people with both strong command of English (avoid costly misunderstandings) and experience (avoid costly mistakes). The experience often comes from working locally in colleges or companies.

So, I definitely encourage people to explore coding in C and C++ for local, embedded systems at the least. There's other jobs that don't require local or embedded where those languages are used. They're not outsourcing-proof w/ consistently good pay, though. ;)

in a capitalist society, small/rare usually means high-paying! don't make it your career but it pays to know these things when writing your high-level code.

You have to wonder what percentage of devs are jumping on these trains out of naivete and what percentage are jumping on out of cynicism, as both are good explanations. The explanation that rarely applies is "this is the best way to solve our problem".

None of us should be so naive to believe that the developer market, or even subsets of the developer market, are a cozy little corner for engineers and tinkerers where rationality wins the day. It's big money (even if you don't pay anything for your tools), and that means that advanced marketing tactics are at work. Hackers are not invincible, though they like to think they are.

"[buzzword] systems need lots of training data. To process that data quickly, systems must parallelize and distribute computation. Concurrent queues are a basic building block in parallel and distributed systems. They are used both internally in [buzzword] systems and directly by engineers building ingest pipelines and serving systems. Engineers need to know the trade-offs of different concurrent queue implementations to avoid introducing unnecessary bottlenecks in [buzzword] systems."

https://www.coursera.org/learn/build-a-computer/

Like an arduino kit (https://www.arduino.cc/) or ARM mbed (http://mbed.com/).

Either one is fairly low-level (C, or C like), but has plenty of beginner-friendly documentation. And if they progress beyond that, moving down to assembler is supported for both.

- For concepts as well as execution, from the bottom up (hardware level etc.), I've read that nand2tetris is good. It gets mentioned once in a while here on HN. Check hn.algolia.com for threads about it.

- any good book on assembly language programming. I've looked at Randall Hyde's book earlier, seemed good. There is also another which I think is good, by an American professor called Paul Carter, IIRC.

Googling should find all of those and more.

Maybe that will change down the road but it doesn't look like it right now.

It's really quite depressing.