Low-level is easy (2008) | Hacker News Reader

Low-level is easy (2008)

yosefk.com

93 pointsZephyrBlu24 comments

Low level code is also easier because it's rarely user-facing. There isn't as much pressure in tech to deliver low level results quickly. Probably also because management doesn't understand low level as much, so engineers have more freedom. You get the time you need to do things right.

The most arduous task for me as an engineer is the endless hotfixing of high level code many tech companies engage in, never taking the time to rewrite and solidify the code as business needs are always evolving. They call this "agile" these days (no relation to the actual agile practices).

> There isn't as much pressure in tech to deliver low level results quickly.

This isn't my experience. Companies producing hardware or devices tend to think hardware-first. Any firmware or software that needs to be developed is often over-trivialised. The way management sees it is that driver/firmware development eats into profits, so it needs to be done quickly. "It's just software". I'd like to hear what stories other people have.

Interesting counter-point about companies where low-level software is user-facing. Embedded systems probably would be another one. I was talking more about the software tech, closer to "big tech".

In my experience, it depends wildly on how much of the special sauce is in hardware vs. software. Companies that work on mature technology (ex. ECG) but have to continuously improve the signal processing tend to view software as the hard part. Companies where you're building fundamentally new hardware tend to view software as the trivial part.

> Companies producing hardware or devices tend to think hardware-first

This was my experience. I worked as a firmware developer for a company producing computer hardware. The attitude of management was “the product is perfect then firmware breaks it”. Never mind that the thing didn’t work without firmware.

The concrete result was that it was impossible to get support from hardware teams. You would chase some weird bug for weeks, be ghosted by your supposed hardware support contact. Then finally chase the guy out to his car at 6:00 pm and find out, “oh yeah, we changed the address of that register, guess we forgot to update the software interface doc. I’ll update the errata tomorrow.” (obviously he never did).

"We'll fix it in software" was a common statement from management at an early job. It's a very frustrating thing, hilariously (or sadly?) they even put "will be waterproof" under the software requirements for one system. That was an actual mistake, but given the general theme of the place it was conceivable at first that it was deliberate.

This is in line with my experience of lower pay offers for embedded SW development compared to business applications.

>There isn't as much pressure in tech to deliver low level results quickly.

I wish there was less over-generalised "Oh, these other guys, they've got it easy, unlike us!".

From my time in embedded, I've gathered some rather different experiences. Deadline are often tight, and as software integration and testing are by necessity the last step in the development process, it's always the software guys who're left holding the bag if anything goes wrong at the earlier stages of the project. The overall deadline must be kept, after all.

So when the hardware guys slip on their deadline (for whatever reason, it doesn't have to be their fault), it's of course up to the software guys to make up for the delay. How hard can a bit of embedded software be, after all? So you're left writing lot's of stuff on the dry, as without access to the hardware, you have no way to test everything in advance.

Then there's the cases where bugs in the layout or docs are only found after you insist forcefully that either the documentation or the hardware itself must be at fault (after hours of futilely hunting for a nonexistent software bug).

So, no, embedded is not necessarily as easy as some people like to make it out. As with most everything, it's very much a case of "it depends".

A lot of the points also seem to apply to algorithms or mathematical programming, just because the libraries do what you expect and mathematics isn't messy the way web framework du jour is. At least until you get to tweaks that seem to work but there are no good proven bounds for their performance.

Anecdotally, I've always felt that what makes low-level seemingly hard is that documentation is either sparse, or written in an extremely dry way that is not engaging for a lot of people. Lower level stuff is prone to what I like to call "man-page syndrome"; i.e. when the resources for a software tool or conceit are presented in a kind of mathematical/formulaic format that average people can't be bothered to parse.

While the documentation can be hard for a novice, it's actually one of the reasons I like low-level programming. If the docs exist, they are generally clear and concise: Instruction mnemonic, parameters, result, flags, maybe number of cycles. That's about all you need to use the information effectively. It also makes browsing the documentation easy if you're a nerd like me who flips open specs and standards to read random sections and learn something new.

One of my favorite experiences was programming for a TI MSP430 uC. My memory is that the reference documentation was outstanding, though that was true of all of the chips I worked with in embedded.

I have never worked with that one for real, but I did look up the docs while playing around with Micro Corruption years ago. And yeah, it had good documentation.

https://microcorruption.com/

Its like the difference between using cake mix or learning to bake cakes using raw ingredients. Cake mix is easier, but if there is something you don't like about it then it is harder to fix. But still, nobody would say that cake mix baking is hard.

I think this metaphor is a bit misleading. If you wanted to contrast work at different levels like the author does, the metaphor would be "cake mix baking" vs. "producing cake mixes". The metaphor above contrasts two different ways, but both are to make cake (output at the same level), where the one using raw ingredients spans multiple levels and the one using cake mixes stays at a higher level. One of the authors points is, that you cannot solely stay on this higher level.

> you cannot solely stay on this higher level.

Lots of developers stays on that higher level, they don't even try to do things that can't be done at that level.

So, in this case, the author is referring to "low-level" as being assembler written for bare metal. I actually agree with his position, but my reference for work of this nature is restricted to the Zilog Z80 and the Intel 8088, and it's also recreational. The ease for me is in two different parts. The tooling is minimal. I don't need to learn much. I can sit down with an editor, an assembler, and a debugger and do whatever I want. The code base isn't large. My version control can be handled in any manner I wish. It's pleasant.

The second I work on something higher level, I am normally talking about work. With work, I am forced into whatever tooling the rest of the team uses. As there are serious deadlines, frameworks come into play to leverage someone else's code as much as possible and reduce how much needs to be done. Higher level stuff also means at least 3 APIs are normally involved (our own, a database/storage API, an OS), but this number can grow if we end up using a service of some sort. There are also multiple arenas. Client, server, and usually also service. There is also the interaction of the team involved, which adds complexity. Debugging takes significantly longer, and it isn't just the OS, the frameworks, and all of that jazz. More code does mean more bugs, but the biggest time sink in debugging is that it isn't all my own code. I don't know it quite as well as I do my hobby stuff. I think this is all rather natural honestly.

Requirements are more constrained the further down the stack you go. I think this is why the engineering types find it more appealing. I successfully ejected from front end development years ago. From my perspective, front end development always tended to devolve into an argument of what the better color is for the button, blue or red. It just became so tiresome for me.

I think you nailed it's and it's the same reason I fled data work to backend and now systems work.

Endless bickering about which metric goes on which page and whether we're correctly capturing the concern of the month was driving me nuts. I don't miss it.

The author thinks like a low level engineer by saying (paraphrased) “that’s a lot of spec to learn”, but in high level we don’t have to know the entire spec to be productive.

Same author, different day: http://yosefk.com/blog/i-cant-believe-im-praising-tcl.html

> Remember how I told low-level programming was easy? It is, fundamentally, but there's this other angle from which it's quite a filthy endeavor.

>but in high level we don’t have to know the entire spec to be productive.

yeah, because most of the time there isn't even a spec to follow or it changes so often you might as well update your code based on which new errors you get

I find the term "low-level" to be an ambigious one. In my experience it partitions into two forms that I call low-level, and low-high-level. Low-level is anything directly touching the hardware, such as device drivers, OS kernels, firmware, and microcode. Low-high-level is anything just a level about that, including the lower levels of graphics programming, compiler backends, linkers, systems libraries, software runtimes, network code, and so on. I think low-high-level is an appropriate characterization in e.g. computer graphics since even using say Vulkan, one programs against an abstract model of a GPU, instead of touching the GPU directly. My mental model of the lower levels of computer graphics in that sense is that the platform APIs are different compiler backends for implementing rendering models, and the domain ontology is buffers, shader binaries, and locking primitives, with the system being built on those two or three ideas.

In either case, in low-level code and low-high-level code, the number of ideas one deals with is many fewer than in the higher levels of the stack. At the end of the day, computers are really fancy calculators, and the ontology at that level of the stack only consists of a handful of things. A consequence of it is that one reuses the same handful of ideas in myriad different ways. It's part of what makes the lower levels of the stack both fun, and monotonous at the same time for me anyway.

I cannot say whether low-level software or low-high-level software is easier than higher level code. One deals with fewer ideas, but the picky details matter a lot more. Hence why e.g. systems languages generally do not have the abstraction power that higher level languages do, but they allow you to directly manipulate data. With the higher levels, there's an explosion of ideas (far more different problem domains and business domains to consider up there) but one gets to take advantage of good abstractions and good infrastructure to help. I do find that the difficult/time spent per LOC ratio is a lot higher with low-high-level code than high-level code, but low-high-level code is also a tiny fraction of the entire ecosystem. One side benefit of experience in the low-high-level region in particular is that I have yet to run into a software system where having a sense of how that stuff works wasn't hugely beneficial. Since there are relatively few ideas that turn up everywhere in computer software, grokking them has a massive power to weight ratio for working with the higher levels of the stack, especially when debugging and stuff. At least I have yet to find time spent down there to be time wasted.

With good hardware and software tools low-level programming is a pleasant experience. But the tooling is most often lacking, so either you have to make do with what you have or invest into building support tools. Both of which can be frustrating.

Reply on news.ycombinator.com