- Know how your hardware actually works, especially CPU, storage, and network. This provides the "first principles" whence all of the below are derived and will allow you to reason about and predict system behavior without writing a single line of code.
- Understand the design of the operating system well enough to reimplement and bypass key parts of it as needed. The requirements that would cause you to to build, for example, a custom storage engine also means you aren't mmap()-ing files, instead doing direct I/O with io_submit() on a file or raw block device into a cache and I/O scheduler you designed and wrote. Study the internals of existing systems code for examples of how things like this are done, it is esoteric but not difficult to learn.
- Locality-driven software design. Locality maximization, both spatial and temporal, is the source of most performance in modern software. In systems programming, this means you are always aware of what is currently in your hardware caches and efficiently using that resource. As a corollary, compactness is consistently a key objective of your data structures to a much greater extent than people think about it higher up the stack. One way you can identify code written by systems programmer is the use of data structures packed into bitfields.
- Understand the difference between interrupt-driven and schedule-driven computing models, the appropriate use cases for both, and how to safely design code that necessarily mixes both e.g. multithreading and coroutines. This is central to I/O handling: network is interrupt-driven and disk is schedule-driven. Being explicit about where the boundaries are between these modes in your software designs greatly simplifies reasoning about concurrency. Most common concurrency primitives make assumptions about which model you are using.
- All systems are distributed systems. Part of the systems programmers job is creating the illusion that this is not the case to the higher levels in the stack, but a systems programmer unavoidably lives in this world even within a single server. Knowing "latencies every programmer should know" is just a starting point, it is also helpful to understand how hardware topology interacts with the routing/messaging patterns to change latency -- tail latencies are more important than median latencies.
The above is relatively abstract and conceptual but generalizes to all systems programming. Domains of systems programming have deep and unique knowledge that is specific to the specialty e.g. network protocol stacks, database engines, high performance graphics, etc. Because the physics of hardware never changes except in the details, the abstractions are relatively thin, and the tool chains are necessarily conservative, systems programming skills age very well. The emergence of modern C++ (e.g. C++17) has made systems programming quite enjoyable.
Also, the best way to learn idiomatic systems programming is to read many examples of high-quality systems code. You will see many excellent techniques in the code that you've never seen before that are not documented in book/paper form. I've been doing systems work for two decades and I still run across interesting new idioms and techniques.
I learned about MIPS CPUs in college, 20 years ago, with Patterson and Hennessy, and built one from logic gates. It was basically a high-powered 6502 with 30 extra accumulators, but split across the now-classic pipeline stages.
I understand that modern CPUs are much different than this. I understand that there are deeper pipelines and branch predictors and superscalar execution and register renaming and speculative execution (well, maybe a bit less than last year) and microcode. Also, I imagine they're not built by people laying out individual gates by hand. But since I can only interact with any of these things indirectly, I have no basis for really understanding them.
How does anyone outside Intel learn about microcode?
- Agner Fog's instruction tables for x86[1], which has the latency, pipeline, and ALU concurrency information for a wide range of instructions on various microarchitectures.
- Brief microarchitecture overviews (such as this one for Skylake[2]), that have block diagrams of how all the functional units, memory, and I/O for a CPU are connected. These only change every few years and the changes are marginal so it is easy to keep up.
Knowing the bandwidth (and number) of the connections between functional units and the latency/concurrency of various operations allows you to develop a pretty clear mental model of throughput limitations for a given bit of code. People that have been doing this for a long time can look at a chunk of C code and accurately estimate its real-world throughput without running it.
[1] https://www.agner.org/optimize/instruction_tables.pdf
[2] https://en.wikichip.org/wiki/intel/microarchitectures/skylak...
Thanks for sharing the above very useful resources, blocked my Saturday for first scan!
I don't disagree, but at least anecdotally a lot of the shops I've been involved with/worked with are really excited about Rust and Go. A previous employer that used C++ exclusively has even shipped a few Golang based tools, and is planning to introduce it into the main product soon. No new projects are being started in C++ either.
Definitely recommend learning C, but having Rust (or Golang) exposure will likely be helpful in the near future.
Great post btw.
Know your requirements and then you can decide on the runtime.
- Understanding how to write purely event-based programs using e.g. epoll since that will avoid lots of complexity and potential problems compared to throwing threads at everything.
- Floating point representation and gotchas ("What Every Computer Scientist Should Know About Floating-Point").
- Unit-testing as part of designing interfaces to get low coupling, sane ergonomics, and forcing some thought to error handling instead of just working in the happy-path case.
- Interrupt- and thread-safety (races, mutexes, deadlocks, orderings etc.)
- Storing state in maintainable ways (i.e. not just dumping C-structs to disk).
- Understanding basic computer architecture concepts like virtual memory, caches, and pipelines.
- Know what can be achieved with OpenMP and GPU programming using OpenCL/CUDA etc.
- Write great commit messages (What is the problem being solved, why this way, etc.)
- Basics of database and distributed systems theory like ACID properties and Lamport timestamps.
- Using regular expressions instead of lots of a complicated set of if-statements.
- Understanding networks regarding error sources and latencies.
With that being said, your list of things to know are all things I have to know for my job (these are things almost all programmers should know, in fact) and I would not consider myself a low-level or systems programmer, just an application programmer.
This takes in the definition of a regular language as a set of regular expressions, and generates C code for finite state machine to parse the language. You can visualise the state machine in Graphviz to manually verify all paths, making it much easier to spot hidden corner cases, while being a lot quicker to code than a big pile of if statements.
The other points are solid, although I have to admit I have never used regular expressions on a microcontroller
But then someone has a bright idea that e.g. a global cache to share some state between all threads would be a good optimization, or that threads need to be able to store some global settings, etc. and complexity related to threads start to creep in to your application.
And I would also add fuzz-testing to your list.
1. Get a platform you can tinker around with, and recover quickly in case of disaster. Raspberry pi is a good example.
2. Learn how to find and read detailed register-level manuals of the underlying chip you're working on.
3. Learn what a programmer's memory-model is.
4. Learn what peripherals are, and what they do. You may have to dig a bit into basic electronics.
5. Learn what user-space vs kernel-space is.
6. Learn what device trees are, and how to prevent kernel from taking over a peripheral.
7. Based on above information, write a user-space driver for a simple peripheral (e.g. GPIO or UART) in C (don't try to learn assembly, yet). You will not be able to use the peripheral interrupts in user-space.
8. Learn how to verify hardware functionality, and see your driver in action. (Warning: high dopamine levels are reported at this point).
9. Repeat steps 7 and 8 for all peripherals, and see how far you can go.
I'd recommend something cheaper and simpler like the STM32F4 discovery boards. You won't get linux and will have to program via JTAG but the documentation for the STM32F4 is not as overwhelming as Broadcom's.
In reality every vendor's peripherals, and more importantly HAL and associated drivers and libs, will be different. But if you've done it once learning another vendor's way of life is a bit like learning a new programming language; it's just syntax.
Broadly most (IE communication, ADC, watchdog, etc) peripherals will be similar but there are definitely places where the differences will be larger. I'd expect those to be largely focused on clock trees, interrupt architectures, power modes, etc.
So short answer I agree with parent, get a cheap board and learn some stuff. If you like it, go from there. Do NOT choose a Cortex A though unless you want to really dig into embedded Linux. Systems at that level are way, way more complex and if the goal is to learn "everything about this processor" that will be especially difficult.
General ARM programming--yes.
Specific peripherals--no.
General peripheral knowledge--yes.
An ARM is an ARM withing certain limits so your knowledge transfers. Sadly, every chip has a different way of programming the periperhals. However, every chip has roughly the same core peripherals (I2C, SPI, UART, etc.), and those electrical specifications and how you use them for protocols doesn't change.
General peripheral knowledge--yes.
Except for the Kinetis DMA, I never understood that one!!
For a related twist the OP might check out the Nordic Semi's M4 with radio SoC offering, the nRF52840DK. It has an onboard, breakout-able segger JTAG programmer and is easy to get going with Segger's freebie take on a GCC IDE.
Hardware is physical and can be quirky. Signals take real clock time to settle down and your CPU is probably much faster than the transition time. Hardware can and does have bugs just like software. A bad ground pin can make your hardware do “impossible” things.
You need to have much more awareness of the context your code runs in the further down the stack you go.
OTOH, there is a danger that I've seen frequently that embedded systems guys don't seem to be able to make the conceptual leap to fully paged, SMP programming. Having sort of come up this way myself (via mircocomputers without MMUs and just a single processor) it doesn't make any sense to me, but it seems common.
People who have been exposed to a more common multi-threaded and virtualized environment seem to be able to pick up those pieces easier.
If you go that path, bear in mind that you will not be involved in creating clever algorithm. When safety is involved, you need to program very simple and easy code. The complexity lies in how all the parts and ECUs are interacting with each other. On the upside, you do not need to constantly learn new languages and library, and you accumulate expert knowledge which is interesting for companies.
But that's not the point. C and C++ are in practice used completely differently, so writing C++ using C concepts with some additions rarely happens in the wild.
But the major point is that there are almost always safer or more ergonomic ways to do things using C++ features that are not present in C.
Because sometimes that's annoying, things can get less type safe, but in your head, there's a C++ program that you're representing.
But in reality, why C is still the best programming language for large projects (IMO) is exactly that the programmer is allowed to choose a suitable structure, such that the program can fulfill the technical requirements. Other languages force the project into a structure that somehow never fits after a couple thousand LOC.
What good programs are written in C that don't have well-structured memory management the way C++ does it with RAII?
Edit:
"But in reality, why C is still the best programming language for large projects (IMO) is exactly that the programmer is allowed to choose a suitable structure, such that the program can fulfill the technical requirements. Other languages force the project into a structure that somehow never fits after a couple thousand LOC."
Yeah, this doesn't make any sense. The reason is, C++ doesn't impose anything on your program structure that C doesn't, while C, with the limitations it has, imposes a tax on all sorts of ways of structuring your program.
For example, you can't practically write a program using a futures library (such as the Seastar framework) in C. And every program you write in sensibly written C can be translated to C++. The exception might be really really small-scale embedded stuff that doesn't allocate memory.
MISRA C standards, popular in embedded projects especially automotive, ban the use of memory management altogether.
The whole point of RAII is that the compiler manages it for you as far as it can. This is impossible in C because you have to do it manually. You might end up writing malloc() at the top and free() at the bottom of functions but that's the opposite of RAII.
If you're running in an automotive environment, you're probably real time; that is, you have to finish your processing before, for example, the next cylinder comes into firing position. You have to hit that, for every cylinder of every rotation of the engine, for any rotation rate that the engine is capable of reaching. You can't be late even once.
Now in the processing you have a malloc call. How long will the call take? Depends on the state of the heap. And what is that state? Depends on the exact sequence of other calls to the heap since boot time. That's really hard to analyze.
Yes, you can get a memory allocator that has a bounded-worst-case response time, but you also need one that absolutely guaranteed always returns you a valid block. And the same on calls to free: there must be a guaranteed hard upper bound on how long it takes, and it must always leave the heap in a state where future allocations are guaranteed to work and guaranteed to have bounded time.
And, after all of that, you still have a bunch of embedded engineers scratching their heads, and asking "explain to me again how allocating memory at all is making my life easier?"
So embedded systems that care about meeting their timings often allocate the buffers they need at startup, and never after startup. Instead, the just re-use their buffers.
Then all the implications like exceptions and needing to implement copy constructors, move constructors, etc. in each little structure.
As to what C project doesn't just emulate RAII: Take any large C project and you will likely find diverse memory management strategies other than the object-oriented, scope-based one. Also, other interfacing strategies than the "each little thing carries their own vtable" approach. The linux kernel is one obvious example, of course.
But I also want to reference my own current project since it's probably written in a slightly unusual style (almost no pointers except a few global arrays. Very relational approach). https://github.com/jstimpfle/language. Show me a compiler written in RAII style C++ that can compile millions of lines of code per second and we can meet for a beer.
> The reason is, C++ doesn't impose anything on your program structure that C doesn't
Of course you can write C in C++ (minus designated initializers and maybe a few other little things). What point does this prove, though?
It wasn't really implemented for performance, and maybe the language is more complicated -- no doubt it's a lot slower. On the other hand, I can look at any function and see what its inputs and outputs are.
I've mostly optimized for clean "mathematical" data structures - basically a bunch of global arrays. This approach is grounded on the realization that arrays are just materialized functions, and in fact they are often the better, clearer, and more maintainable functions. If you can represent the domain as consecutive integer values, of course. So I've designed my datastructures around that. It's great for modularity as well, since you can use multiple parallel arrays to associate diverse types of data.
But anyway, your language looks impressive I must say.
I don't know enough about D to really comment on it (I read the black book on it 9 years ago, and the examples crashed the compiler), but... it has better compile times, right? There's a module system? I'd have to look at the D-with-no-GC story, figure out how hard it is to interop with C++ API's. I think I'd have better feelings about D (or C++) if it didn't have class-based OOP.
Treating C as a sort of "portable assembler" is a lot better, although it runs into UB problems (see DJB on this subject).
I think there is a world of wild and crazy C++ (like, boost::spirit-grade, or std::allocator-using) that you're imagining, that is not what I am thinking of. If you took C, and added vector<T> and hash_table<K, V>, added constructors and destructors so you don't have to call cleanup functions, you'd get a language which most sensible non-embedded C programs would map to, which then maps upward to C++.
Maybe some templated functions like std::min<T> and std::max<T> and add_checking_overflow<T> would be nice to have too.
Edit:
An example of where I think properly written C diverges from portable assembler is things like here: https://github.com/Tarsnap/tarsnap/blob/master/libcperciva/d...
It depends on whether you think ELASTICARRAY_DECL is within the scope of portable assembler. (It gives you type safety!) (And I don't know what advanced assembly languages can offer in terms of that -- maybe they do too.)
#include <stdio.h>
typedef enum EFoo {
FOO_A = 1,
FOO_B = 2
} TFoo;
int main()
{
TFoo foo = FOO_A | FOO_B;
printf("foo = %d\n", foo);
}
But it does in C: https://godbolt.org/z/zdTKiE
https://blog.directededge.com/2009/05/21/c-and-c-are-not-the...
Json is not exactly a subset because it supports representation of numbers of any precision. JavaScript will convert to an IEEE 754. But it's up to a json deserializer to decide how many decimals to use. I think.
Then pick whether you want to start from the "front" or "back" of C++; i.e. learning the language in chronological order or in reverse. C++17 idiomatic style is very different from the 99 style that most existing C++ is written in.
I would suggest picking a codebase to work on and learning its style and subset of C++ first.
EDIT. The book also comes with a collection of excellent projects, which the authors have made freely available online at http://csapp.cs.cmu.edu/3e/labs.html
>before you rewrite your C book, perhaps you should take the time to actually dig into it and learn C first (inside and out.)
which is uninformed, as Zed wrote Mongrel and Mongrel2 in C. Saying he doesn't know C is ludicrous. He might have a different approach to C, but then argue this view instead of claiming your way is the only way. The author of that blog post is saying the book is bad because it is not the way he writes C. Not because it is objectively bad.
Also, replies to that post like "Just for info K&R stands for "Kernighan" and "Ritchie", please, don't compare the bible of C with "just another book on C". It is a blasphemy." are hilarious. People are just parotting "read K&R!!" off of each other. The term Stockholm syndrome is overused but it is very appropriate for people who think C is actually good.
I think that C is actually good, and that C++ is a Scooby-Doo sandwich of fail. But this is an opinion concerning a particular domain (low-level programming) that doesn't carry over to anywhere else.
Also 21st Century C has been recommended, and it goes into tooling as well.
> as a self-respecting software engineer
Then you probably already know more about C than you realise: if, while, for, etc. all work the same as most other languages.
What you're probably not used to is needing to define functions in advance, pointers, and memory (de)allocation.
If you learn by doing, then with google and github, you can create a few simplified versions of unix utilities (cat, ls, grep -F).
If you're on Windows, I recommend using a Linux VM, WSL, or Cygwin - it's easier to setup than the C tooling in Windows (and you're stealth learning something else).
Once you know C, you can then move onto C++ (I stopped at C).
Agreed. APUE is perhaps one of the best book available covering SUS, POSIX layer in detail.
Nice. I'll check it out.
But all the fun stuff happens when you start talking with your OS, so get a book about that too. If you are planning to develop on Linux, Michael Kerrisk's The Linux Programming Interface is excellent. Much of it will be familiar to someone used to shells and the terminal, but there will be plenty of new ideas too, and even the stuff you know will get a much deeper perspective.
I'm french and studied in the UK. I'm sometimes lost when my colleague use french technical terms, i have to ask the concept behind it to be able to identify the english term which i learnt during my studies.
> However, as an English speaking engineer, I found many of the discussions rather clumsily written. I'm guessing that it was translated from the German by someone who doesn't thoroughly understand the subject matter.
It's worth picking up some basic gdb skills. Use of ddd can help with this. On windows you can use VS for most of this of course. Picking up the basics of valgrind will also help you.
Get comfortable with the preprocessor. Get comfortable with Makefiles. Get comfortable pulling in library headers and binaries as needed.
Errr....
I was (mostly) a C programmer for over a decade, but that's about all I can think of right now!
--edit--
And someone below has just triggered me - FFS use stdint.h!
You absolutely need to know in C++ too. Modern C++ just gives you tools to express ownership in code. Rust goes way further and gives you compile time correctness of your lifetime handling
Pointers, stacks (one in ever 23.7 bugs is a stack smashing bug), bit bashing and endianness, types and coercion at the byte level (see also: pointers, bit bashing), C strings, the stupid rules about when a variable's value is actually written to memory that need to die in a fire, memory allocation/clearing/copying/ownership/freeing, ALWAYS CHECK RETURN CODES, what the heck an lvalue is.
This book is fun:
- runtime/space complexity and general algorithmic heuristics to know how the system will perform, or to be able to fix scale issues in the code.
- network characteristic (eg fallacies of distributed computing)
- CAP theorem and applying it to real world designs
- A co-ordination tool like zookeeper
- Understanding of the technological lay of the land (eg compression like snappy, different tools and approaches for communication like zeroMq, kafka
- Metrics and alerting - eg get how to instrument with statsd and know how and where to alert.
- Concurrency concerns and scheduling - you can get away with understanding futures and the actor model and can steer away from threading fortunately. (eg elixir/OTP or Akka)
Embedded programming is all about forgetting all the really complicated algorithms you might have learned in school, because they usually don't matter, and when they do it's more important to be able to gather performance data than it is to do something fancy.
Traditional concurrency is still a very real concern because DMA engines, specialized coprocessors like the TI N2HET, and offload engines like modern audio codecs all have their own internal firmware that your system must interact with. Even getting the system to boot and to transition to low power states requires you to understand clock trees, clock gating, power supply states, and how to interact with any PMICs. Getting a real-time clock working is a similar story.
Raspberry Pis are good for dipping your feet into a Linux system that uses a specialized bootloader, but if you want to do anything truly embedded you're going to have to go deeper than that and work with a system like the PIC32 or even the RTUs on the Beaglebone Black.
If you're not using a JTAG at least occasionally then you're probably not close enough to the hardware to be considered "embedded."
[0]: https://people.freebsd.org/~lstewart/articles/cpumemory.pdf
Jokes aside, the guy asked how it's been useful and it's not terribly convincing that your first (and only?) thought was interview questions.
I, honestly, don't know if it will help me in terms of my career because at my current job, it's not something I can put to use. However, I want to learn this because it is fun (for me) to understand things close-to-the-metal.
And yes, the title isn't hyperbole. You can skim the parts describing specific tooling, but the fact that the description of the memory hierarchy and how it works hasn't been internalized by every programmer is a travesty.
* put a cap between power and ground
* get some nice ESD-birkenstocks
* don't listen to anybody telling you how it's done
The start of the art is BS and we need a revolution. There will be so many people who tell you that you can only do it in C and if you don't, you can't be taken seriously. The result is that everybody is keeping to C and nobody invests time in bringing new ideas to the field. The chip vendors stubbornly ship you really, really bad C SDK's and have no interest in doing better, for reason beyond my comprehension.
As you mentioned Rust - there is a Rust working group for embedded targets and they do cool stuff, but it's hard and a lot of work. Also, hardware is basically a huge block of global mutable state, so that is a problem to wrap ones head around.
But eventually we have to get rid of "C is the only serious option" which is an argument made by people who know how to write "safe" C and perpetuated by people who can't, but act like it.
[Before you react - I know this is an "extreme" statement and it's not 100% accurate and there is much more nuance to it - it is exaggerated for comic effect ;)]
Many chips, like a bluetooth chip, don't draw power continuously, but there are spikes. These spike in current can lead to the voltage dropping to low. A filter cap acts against that.
Sometimes, especially in prototyping circuits, where nobody took care of properly designing a power supply, filter caps are the first cheap shot at fixing weird behavior.
If you wanna know more, google the term, there is way better explanations then mine, form people with way more profound knowledge.
So depending on what frequencies you want to filter out you can change the capacitance C.
People already explained the reasons. I just wanted to add that point.
Could be a case of unfit business model leading to wrong incentives.
Only reason to go lower in stack is if you really ~like~ love hardware and exploring nitty gritty of how things work. In that case, pick any OS book and get ready to go down the rabbit hole :)
I am bored of the churn associated with stuff higher up the stack. I would prefer to master a craft rather than chase new fads every few years.
Also, you won't be required to work on pet side projects, push them to Github, blog about them and post Show HN comments to build up street cred for future job interviews.
But I have no idea how one breaks into this field. Maybe that's worth an Ask HN or a Stack Exchange post.
I learned enough to know when to call IBM for support, but with that experience it wouldn't be too difficult to find another job managing one.
I remember 12+ years ago, my old boss bitching about working on an AS/400, then he'd hire his brother (who worked on them for years) to come out to do very simple stuff for quite a high premium.
I just wish they weren't so proprietary, but at the same time, I am glad they are. It's been a love hate relationship.
Now I work with an AIX system. Unix under the hood, but IBM still has a stranglehold on it. Both tanks though.
- on memory access/caches/memory hierarchy: https://www.akkadia.org/drepper/cpumemory.pdf
- on DSO: https://www.akkadia.org/drepper/dsohowto.pdf
- Agner Fog on CPU optimization, latencies, vectors etc: https://www.agner.org/optimize/
Surprised that this hasn't been exploited to build a C compiler targeting the platform. (Unless I needed to Google something other than "pico-8 C compiler"...)
When is it appropriate to allocate memory from the heap? If you're in a rendering or audio processing routine in realtime context, avoid it at all cost.
Think about which parts of the code could profit from optimization. Does learning an assembler for a specific platform pay off or can the compiler do a sufficiently good job with -O3? Use profilers to identify performance bottlenecks.
Think about portability. Does the code have to compile with ancient C89 compatible compilers [Mine has to, and I would be excited to see a Rust to C transpiler]? Can you choose your compilers by yourself? Are they provided by your customers?
There are many different types of jobs. I will assume you mean low level development on an otherwise normal operating system like Linux.
I would start by learning how Linux userspace is constructed. Dig up old Linux From Scratch docs, build your minimal system, experiment with it. Try to understand how different stuff is put together. How elf/static/dynamic libraries works, etc. Look at your running processes (there shouldn't be many if you do LFS) and try to explain everything you see (what this process does, how it does its job, etc.)
Learn how kernel communicates with userspace. What are devices? What are syscalls? What syscalls are available? How filesystems work? How devices work? Etc.
Learn what is the job of Linux kernel. How memory management works, what is virtual memory, what is the difference when programming in user/kernel space.
The best way to learn system programming is definitely not getting up to your ears in a single open source project. You need variety of knowledge because you want to understand in breadth how the system works. Once you get a job or you figure out something interests you more there will be good time to specialize (say you want to specialize in security or containerization etc.)
I would never ship a product today without remote signed upgrade ability.
All devices that plugs into a windows PC can use Windows Update for example to upgrade firmware. Any device that sits at a remote location probably ought to have a GSM modem in (adding only $1 to the cost of the device) for tracking uptime and firmware updates. Any device which offers an API to other devices should have 'provide firmware update' as part of that API.
Obviously security is a concern with updates, but by signing update files, and making sure downgrades aren't allowed, the security benefits of being able to patch vulnerabilities outweigh the disadvantages of the device manufacturer being able to produce evil update files.
Those who say that your lightbulb/toaster/USB hub don't need automated software upgrades are naive. Software on them will typically consist of many libraries totaling perhaps hundreds of thousands of lines of code. Security vulnerabilities will be found in that code, and even if the security of this particular device isn't of concern, it can be used as a jumping off point to attack other networked devices or for data exfiltration.
What about the $1-2/mo network charge? Or battery life considerations? I've got sensors that run for 2-3 years on a CR123a battery. That power profile isn't going to support a GSM connection.
￡0.96 | DIY KIT GSM GPRS M590 gsm module Short Message Service SMS module for project for Arduino remote sensing alarm https://s.click.aliexpress.com/e/b4tPrIy0
Lots of providers are happy to provide worldwide service for free, as long as you pay $10 per gigabyte. I generally budget 1 check-in per day, of about 250 bytes, so a coin cell can easily power it for a few years with a total service cost for 2 years of just a few cents.
You have to watch what compiler flags you use, like someone turning off stack cookies, not using clang's sanitizers. Check out https://clang.llvm.org/docs/AddressSanitizer.html it would have prevented the Heartbleed vulnerability if it existed at the time.
You need proper bounds checking everywhere! You should fuzz your code with something like AFL and if you don't have the time to setup test cases for it, just send your program random junk and see if you can get it to seg. fault.
Multithreading is hard, and detecting multithreaded bugs is even harder. Random monkey testing can sometime's help find these, but they are very hard indeed. Monkey testing is just literally if a monkey was smashing your keyboard and your program was open, what would happen?
Know the most vulnerable function calls by taking a look at banned.h from Microsoft's SDL. It looks like it's no longer on the official website, but the author put it up here Take a look at https://github.com/x509cert/banned/blob/master/banned.h. Sometimes you can't avoid using these functions but know why they can be considered bad.
A proper Makefile is your team's best friend. The same can be said with one build machine for your whole team. It is easier now than ever to build and distribute it between your team with Docker. This is just a personal opinion, but I think downloading all the exact library versions you need once and putting them in a Docker image will save your team some pain in the future.
If that was true, then so would Valgrind have, and Valgrind was in wide use at the time.
It was however more complicated than that. If my memory serves me right - OpenSSL had its own memory management.
Being able to read and write C programs goes a really long way.
Beej's guides are a good resource. I learned a lot from them.
Guide to network programming: https://beej.us/guide/bgnet/html/multi/index.html
Guide to Unix IPC: https://beej.us/guide/bgipc/
The Unix Programming Environment (Kernighan & Pike) is also very good.
http://files.catwell.info/misc/mirror/the-unix-programming-e...
A good exercise would be to write a simple IRC-like chat server and client in C.
I think it can be summed up with some questions to ask at different points in a project:
- Should I use a microcontroller or a processor? If a microcontroller should I use a simple 8 bit or more featured 32 bit?
- Do I need an operating system like Linux, RTOS like FreeRTOS, or bare metal?
- Are there existing code modules out there to help kick start the project? Like SD card libraries, ethernet middleware, etc
- Should I design a custom PCB or look at development kits, or off the shelf electronics?
- Where should I design in flexibility in the project? What requirements can be solidified to simplify the design?
Knowing about what is out there helps pick appropriate solutions to problems, which will save the most time in the future.
Also sometimes you may have existing expertise, tooling, or firmware already for a processor such as a PIC, AVR, or STM8, so why reinvent?
Some examples may be things like a custom serial to input/output expander, small motor controller, power controller, data logger, or other small and simple machines.
If for your own learning, do you want a full-powered environment? Or do you want a simple system that you can learn all of, even if it's more of a toy?
If it's a shipping product, do you care more about ease of development, or about total parts cost? (The difference is often quantity that you expect to ship - 10 cents in part costs matters if you expect to make 100 million of them.)
This comes up incredibly frequently, and having it be second nature will benefit you.
A lot of the time, this gives you a starting place for how your inputs and outputs should look, what your method signatures should be. For example, if your data can can fit in a uint8_t, why waste bytes on a uint32_t?
This isn't the be all and end all, especially since modern architectures use larger word sizes/RAM is plentiful/bandwidth is cheap, but it can be a helpful way to frame the conversation when you're architecting software (especially network protocols).
I'm an ME who went into embedded systems, so I'm not sure what qualifies as low level for a web developer and if checking timing using an oscilloscope or saleae is feasible for the type of work OP wants to do. But even just looking at the assembly would have made things obvious in my example.
EDIT- This made me realize that a rudimentary understanding of the assembly language for your architecture will be very valuable. Doesn't have to be enough to actually write any code in it, but it's great to be able to take a look at what was generated when things are acting weird.
Without that though, performance is really really bad. The inner loop of a motor controller is no place for them...
Anyway a bit toggle tells all, and if you're making speed optimizations you should be checking that things actually got faster.
So I agree, knowing how data is stored is critical.
- become good at either C or C++ or both. Lots of fun jobs/projects involve codebases that happen to be written in those languages.
- become fearless and systematic with assembly. You don’t have to be great at it. You just have to have had enough experience looking at it and writing it that you can hack it if you have to.
- learn to read code quickly and accurately. The only way to do that is a lot of practice. It’ll be hard at first but as you practice your reading speed will go up by >10x and eventually it’ll feel like second nature.
- become great at working with large code bases. What makes “systems code” so interesting is really just how big “systems” are. This kinda goes along with the bit about reading code - reading is how you survive in large code.
A project I worked on that was also an interesting challenge was a RAMless monitor ROM. Write a program that will boot up the monitor (not the OS) and provide an interactive console that uses no RAM at all. Commands are to poke or dump RAM, read/write IO ports, probe the bus for devices etc. You can’t use the CALL instruction because you have no stack. Test your code by pulling the RAM modules.
Then you can have "low level" as in close to the metal, for example drivers that talk to a hardware. That being said even these are (and should be written mostly) at fairly high level (especially in the user space) and it's only a relatively small part that has to mess with HW specifics.
And then of course there's some embedded IoT stuff that has also different layers of software although for some people working anywhere in the embedded stack would qualify as "low level".
Generally speaking you'll want to lean on a few core technologies and APIs such as POSIX (on linux), have a decent understanding of the von neuman architecture and a solid grasp of C/C++. Having a decent idea of how different subsystems such as network or TCP/IP stack works and such can be useful depending on the domain.
That's exactly my point as well. Well said - ha at least better than I would be able to explain myself.
I grew up with Tannenbaum for networking, though most people went toward Comer and Stevens. Maybe there's something even more current.
I honestly don't know anything as good for storage, which is funny since it's my own specialty. I can try to cobble together a reading list if you'd like.
Something on operating systems. Vahalia for an overview, Bach for System V, gang of four for BSD. Yes, study something besides Linux ... but do study Linux as well. Not sure which books are good for that.
Something on compilers. Is the dragon book still the go-to reference here?
Even if you don't work in those specific areas, that should give you the grounding to study more. I'd also throw in something on databases, but don't know a specific book/author. Distributed systems much like storage: definitely learn, can't think of a good single source, can create a list if you want (I think I did one for a colleague not too long ago).
Good luck!
For compilers the dragon book has fallen out of favor (especially as a first compiler book/source). Modern Compiler in ML (don't touch the Java or C versions) and Engineering a Compiler tend to be the goto books now.
The things you talk to will fail, you should be prepared to deal with those failures.
Electrical Engineering is basically being able to read drawings and choose components for your application.
Signals I found fun, but I can imagine doing a digital application and this being difficult.
FMEA comes with the job IMO, but a good tool to have.
This is the best advice.
For example, why not build a home automation system?
Start with a Raspberry pi and start hooking up peripherals...light switchtes, timers, sensors, etc. You'll stumble across buses like I2C/SPI, you'll learn about networking. You'll figure out what registers are and learn what interrupts are and what they mean.
You'll get lazy rewriting communication code and stumble across messaging frameworks like MQTT to communicate with devices on your network. You'll run out of money using Raspberry Pis for each new device you build, and you'll find cheaper ways of doing things like desiging devices using MSP430 or ESP8266s.
You'll make mistakes, and you will learn. Best of luck on this new adventure!
I prefer to generate them, so I would add to that "tools for generating C code and state machines".
(Tangent: A guy who worked on Verilog and FPGAs told me that the "whole idea" is state machines... and they are hard to get right.)
- Essential books and papers are good as a reference to learn and improve but focus first on hands-on stuff.
- Get involved in an open source project that fits the 'system level' category, learn how it works and try to contribute to it (again, put your hands-on)
- Start with development in C language, a good tool to build would be a TCP client and server (kind of 'echo' program). The goal of this will be to learn: how to compile a C program, basics of memory management (malloc, free, realloc), networking (create a socket, bind, listen, connect), transfer data (write, read) and so on.
- If possible, learn using a native Linux environment.
All of the suggestions above will pay off if you are persistent and keep trying learning by building; of course, getting involved in an open source project is a must if you want to have a successful career.
Focus on C and follow up with an understanding of the underlying assembler code. Haven't played with Rust for about 1 year, so I don't how how mature it is these days.
Also have an idea of how to debug the code. Debugging on embedded is very different to debugging a web/js app.
Low-level systems programming is hard and can be a bit boring at times. I agree that debugging is very different from debugging on higher level systems. Most of the time an oscilloscope or an LED is the best debugging tool. Also, in well written low level code you tend to have more code making sure that everything is okay than you have actually doing stuff, which can be tedious. It takes a certain kind of personality to enjoy it.
[edit] I have the LDD and https://www.amazon.com/Linux-Kernel-Development-Robert-Love book. I got confused about the authors name :)
The author Robert Love has done a tremendous job on Linux Kernel Development book. Every aspiring "In-betweeners and Low-Low Level engineers" [0] should read this book. It's awwwwespiring.
And please read LDD too!
2. Learn about data structures and memory handling (pointers) https://www.geeksforgeeks.org/data-structures/
3. Try to understand how an OS works (e.g. linux) https://github.com/tuhdo/os01
4. Buy an Arduino or alternative with some I/O. https://www.arduino.cc/
Linked lists, bitcounting problems (which is covered in the first few chapters of K&R), etc. are all still popular.
The biggest issue you might have is the difference in salary and available positions you're going to see going from FANG Backend Dev at 5 bazillion a year or whatever it is now to 80-120K.
Just make sure you don't buy the `international edition` of the book, it's unreadable due to the amount of errors in the exercises.
Nand to Tetris: https://news.ycombinator.com/item?id=18519883
Nand to Tetris II: https://news.ycombinator.com/item?id=14526344
I asked a question in a here before that might have some relevant answers: https://news.ycombinator.com/item?id=17282466 (not exact topic)
Any recommendations or popular reading?
I realize some choices are for proprietary reasons, then compression, but that aside, there seems to be thousands of choices available.
Compared to the web, using json etc, and releasing a schema, but not necessarily creating a new file extension.
OTOH, you have to know when no to use it and step up (down?) to something that is text editor hack-able (XML!?) or has barn burner I/O abilities (yah actually just dumping raw buffers with regularized binary data to disk). Or for that matter is used to exchange data with other apps with other services (JSON, and the long list of other data dependent formats, although for at rest exchange I have to point at XML again).
[0] https://en.wikipedia.org/wiki/Mass_spectrometry_data_format
Then you should learn Unix. From an understanding point of view, I think it's probably better to learn something like FreeBSD, NetBSD, Xv6 ... Linux is very pragmatic, and very general, and so it doesn't have the purity that smaller, more focused or curated systems have. Once you have a handle on Unix, look at other OSes: Plan9, Minix, FreeRTOS, L4, etc.
Then networking: I suggest starting with IP, TCP, ARP; then step down to the physical layer: Ethernet, Token Ring, 802.11, hubs, and switches; then static routing, RIP, OSPF, and BGP; maybe look at mesh routing. Then some application layer stuff: DNS, NTP, SSH, NFS, LDAP, HTTP, etc. Reading the RFCs is really valuable, and they're remarkably accessible up until say 2500 or 3000 or so.
Security: symmetric and asymmetric crypto, Kerberos, SSL, SSH, OAuth, etc. Read up on pen testing, social engineering, defensive programmings, fuzzing, etc.
Databases: both relational and otherwise. SQL. Wrap your head around how a filesystem and a database are the same and how they're different.
Messaging: some sub-set of protocol buffers, Cap'n'Proto, Avro, Thrift, XDR; brokers vs. p2p; pub-sub vs. directed. There are hundreds of system you can look at here: pick a few that look different.
Learn about complexity analysis, distributed consensus, locking, concurrency and threads.
so far as tools go, you need to understand a debugger (how to use it, and how it works), packet capture and analysis (Wireshark is good), profiling and performance analysis.
That's probably a decent coverage for the software side. The exact focus will differ depending on embedded/real-time vs enterprise, etc.
From the hardware side, I think it's worth starting with an 80's or earlier, 8 or 16-bit system, and learning how it works to the digital logic level. What a simple microprocessor actually does: fetching, decoding, execution units, etc. A Z80 or 6502 or similar system is a pretty simple circuit, and it's worth really grokking how it works.
From there, you can move forward to more complex CPUs, newer memory architectures, newer buses, etc. But it's much harder to dive straight into a modern x86 or ARM CPU and try to understand how it works.
It's a this point that reading Drepper's memory article, and the "everything you should know about latency" article(s), etc, really start to be useful, because you've got a solid grounding in what's underneath them.
You don't need to do this all at once, or before you start working more on backend or systems level code: I'd guess it took me close to 10 years to feel like I had a decent grasp on most of it.
- Embarrassingly low-level (chip developers): OS, C/VHDL/Verilog, MA, DS, GPH, EE, Apathy, SE
- The in-betweeners (kernel, device, storage developers): OS, C, MA, DS, GPH, Apathy, Crypto, SE
- Low-Low-level systems: OS, C, DS, GPH, Apathy, Crypto, SE, FTDS, MSG, NW.
- Low-High-level systems: C/Rust, DS, GPH, Apathy, HDS, Crypto, SE, FTDS, MSG, NW, TL.
- Unicorn rock-star systems hacker (perhaps only a handful of these creatures exists :-): Multiple OS, C/Rust/JVM/Erlang/Haskell, Multiple MA, DS, GPH, EE, Crypto, FTDS, MSG, NW, HDS, SSE, TL.
Abbreviations used:
GPH = Good Programming Hygiene
EE = Electronics and Electrical Engineering
OS = Operating System
MA = Modern Assembly
DS = Data Structure
FTDS = Fault-Tolerant Distributed System
MSG = Messaging System
NW = Networking
TL = Toolings (the darlings of UNIX)
HDS = High-Density Systems
SE = Security Engineering
That and documentation for the platform you are working on, in my case intel/amd manuals but not sure what system you are working on, i presume one of those though.
it's also interesting to look into assembly/ assemblers, disassemblers, perhaps look at capstone / keystone frameworks to learn about that. linkers & loaders is also interesting.
it really depends on where you want to go with low level/ system things what is relevant though, as a lot of it involves platform specific details.
C
You may not need to do new work in it, but the large body of existing stuff is in C and that's not changing for a while.
If you want to move into the mid-stack then Java is pretty good these days, python too.
If you don't know what ASLR is, you're not ready for systems programming.
The resources I'm linking are supplementary to the above, and you'll likely encounter them in the wild. But they'll help you build a base of knowledge, and give you terminology to search for, and work with.
- What Every Programmer should know about memory: https://people.freebsd.org/~lstewart/articles/cpumemory.pdf
- Cache Obviousness: https://www.youtube.com/watch?v=bY8f4DSkQ6M (the suggestions, and terminology are important)
- Parallelism: (This is a good primer, there are a lot of complementary posts linked on the site) https://preshing.com/20120612/an-introduction-to-lock-free-p... https://preshing.com/20120913/acquire-and-release-semantics/
- If you plan on working with linux these is an excellent reference: http://man7.org/linux/man-pages/dir_section_2.html remember there is no magic in Linux, everything eventually has to go through a system call. So if you learn the systemcalls, you can can learn how things work :)
- Fog's optimization resources are awesome: https://www.agner.org/optimize/
- MIT courseware: https://www.youtube.com/watch?v=ytpJdnlu9ug&list=PLUl4u3cNGP... https://www.youtube.com/watch?v=HtSuA80QTyo&list=PLUl4u3cNGP...
- This cheat sheet is worth committing to memory: http://www.bigocheatsheet.com/ the reference links are also great for building up knowledge
- I highly recommend CMU DB open course ware: (intro) https://www.youtube.com/watch?v=vyVGm_2iFwU&list=PLSE8ODhjZX... (advanced) https://www.youtube.com/watch?v=poEfLYH9W2M&list=PLSE8ODhjZX...
This should give you a good primer on Concurrency, and DB's. For networking likely a basic TLA+ certification class will be 99% review, but for the things it isn't will offer great insight.
The pay sucks, but it is much more enjoyable work than web development. There's a lot less BS to deal with.
I really can't tell whether an entry-level systems programmer gets paid better than an entry-level web programmer. But it seems to me that in web programming, you hit a wall at about ten years, where more experience quits translating into more pay. In embedded, you can find at least some jobs where 30 years experience gets you more pay than 20 years experience.
For me, low-level is assembly, not C.
You don't have to write it (much). You should be able to read it (with the instruction set manual if needed), and maybe be able to write a little bit of it in a pinch.
But be warned, earning potential-wise you may lose out(unless you get lucky) to the latest whiz-bang Web/Mobile technology/framework brouhaha. That is the nature of the market. However i consider that the satisfaction of learning and "knowing" how things work, more than makes up for the small loss in earning potential. This technology is also more fundamental and stable and thus will not go away anytime soon.
I have found the following papers/books (somewhat different from the most commonly cited) useful in my study;
Languages: Fluency in C is a must. It is THE "portable assembly" language and is available on everything from 8-bit MCUs to multicore servers. You can also link C modules to everything under the sun thus allowing you to extend almost any other language. C++ is also needed; but judicious usage as a "better C" rather than heavy-duty OOP/Generic/Template-mataprogramming madness.
- Computer Systems: A Programmer's Perspective 3rd ed. based on x86-64. You might also want to get the 1st ed. which is based on 32-bit x86. These books cover the HW/SW interface and thus includes almost all the topics under "Systems Programming".
- The C Companion by Allen Holub.
- Inside the C++ object model by Stan Lippman.
Tools/Toolchain: Knowledge of the GCC toolchain is a must.
- The Definitive guide to GCC by Hagen
- ELF: A Programmer's perspective; paper by H.J.Liu
- C++ Under the hood; MSDN article by Jan Gray
- The Practice of Programming by Kernighan and Pike.
OS/Kernel/Drivers:
- UNIX Systems for Modern Architectures: Symmetric Multiprocessing and Caching for Kernel Programmers by Curt Schimmel.
- Linux Kernel Development 3rd ed. by Robert Love.
- Essential Linux Device Drivers by Venkateswaran.
- Embedded Linux Primer 2nd ed. by Hallinan
- The Unix Programming Environment by Kernighan and Pike
- Advanced Unix Programming 2nd ed. by Rochkind
Computer Architecture/Organization
- Modern Processor Design: Fundamentals of Superscalear Processors by Shen and Lipasti
- Computer System Design: System-on-Chip by Flynn and Luk
Parallel/Distributed processing:
- Foundations of Multithreaded, Parallel, and Distributed Programming by Andrews
- The Art of Multiprocessor Programming by Herlihy and Shavit.
Bare-metal Embedded: Where the "rubber meets the road"
- Embedded Systems Architecture: Explore architectural concepts, pragmatic design patterns, and best practices to produce robust systems by Lacamera
- Patterns for Time-Triggered Embedded Systems by Pont.
but, a traditional trial by fire is to write an operating system.