Hacker News Reader

In practice, the demarcation line for "systems programming" is when the software directly controls and manages the hardware resources it uses. Software like this takes on some of the characteristics of an operating system because it is required to provide similar functionality, even if not an operating system per se. In application code, some other piece of software, usually the operating system, takes care of this for you.

Database engines are an excellent example of this spectrum. Some simple databases contain no systems programming at all whereas advanced databases are essentially complete operating systems that run in user space. If you walk the code bases along this spectrum, you will see a shift in the construction, character, and design of the code base as more of it transitions from "application" to "systems" code.

The distinction is important because systems programming is a different skillset than applications programming. For example, there is often a requirement to do a lot of sophisticated scheduler design that is integral to the correctness of the software in systems programming. Application programming rarely concerns itself with this, it relies on threads, locks, etc and has the OS sort out the scheduling.

Outside of actual operating systems where systems programming is unavoidable, the primary motivation to do systems programming in a user space application is it can greatly improve performance, scalability, and robustness.

Under your definition, would embedded programming fall under systems programming or do you see them as distinct areas?

I think embedded is a distinct subset of systems programming. It definitely deals with hardware resource management, but is a bit different because it tends not to build the high-level resource management abstractions on top of the hardware for other application code to use e.g. what a database kernel is to a database.

Rabbitmq is a good counterexample. It is overwhelmingly used to provide services to other programs, deal with resource/performance constraints every second, but is an application-level program. Written in non-system programming language and not part of operating system and never will be.

It's interesting in that the RabbitMQ example helps illustrate how fuzzy and subjective this all is. Some people would definitely take the approach of "it's not part of an operating system so therefore it's not 'systems' code". Personally I would disagree. I've always considered "middleware"'ish type stuff to be "systems level" by virtue of the "it mainly provides services to other software" aspect of the definition seen above.

Web browsers are another interesting item that don't fit neatly into this classification either. I mean clearly a browser is something that provides services directly to the user. You use it to navigate the web, view and download documents, store bookmarks, and all sorts of stuff. It's 100% "application level." Except... the browser also provides a runtime, including memory and process management, networking services, graphical rendering, etc. to other software. In fact, a browser is basically a poor man's Operating System. So it's 100% "systems level." Wait, what?!??

>I've always considered "middleware"'ish type stuff to be "systems level" by virtue of the "it mainly provides services to other software" aspect of the definition seen above.

The middleware-programming is also systems-programming was also reinforced by Rob Pike when he originally presented Go as a "systems language". However, he now admits[1] it confused people and Go might be better categorized as a "server" apps language -- i.e. writing Google's backend infrastructure code that doesn't need to be written in C++.

In the 1980s, "systems programming" was usually a synonym for "bare metal programming" because "systems programming" was just a shorter way of saying "_operating_ systems programming". (One writes an operating system to boot up on bare metal.) And systems programming was typically assembly/C/C++ instead of business programming like COBOL or dBASE. In that mode, writing kernel drivers or a UEFI boot loader is also "systems programming". In contrast, "applications programming" was something else.

That said, if today you surveyed 100 random programmers in various domains, I'd have no idea if there's a dominant view of what systems programming is.

[1] from the article: Rob Pike: When we first announced Go, we called it a systems programming language, and I slightly regret that because a lot of people assumed it was an operating systems writing language. What we should have called it is a server writing language, which is what we really thought of it as. Now I understand that what we have is a cloud infrastructure language. Another definition of systems programming is the stuff that runs in the cloud.

I think based on the parent posts choice to go with the "who is your customer" definition and the fact that we intuitively think of RabbitMQ as some sort of systems program, it probably fits the modern definition of systems program that we all kind of agree with but do not necessarily put into words.

I don't think RabbitMQ being written in a "non systems language" is relevant. Its a systems program even if it was written in shell (if it was it might be a bad systems program, but a systems program nonetheless because of it's intended customers and use case).

Which brings us to browsers, which are not systems programs (and I think we tend to not think of them as such) almost entirely due to their primary customers not being programmers. This is despite the fact that they fit the "complexity/size and close to the metal" definitions a lot better than many other programs that are commonly considered to be systems level.

Which is quite interesting when we then talk about microkernels and hypervisor based workloads.

> whereas systems programming aims to produce software and software platforms which provide services to other software

We have the word "middleware" to describe that.

If you look how the word is used, "systems programming" does clearly not mean that (personally, I don't remember ever seeing it used that way). It's more used as an architecture paradigm, where you don't expect a lot of things to be abstracted, for whatever reason. The word choice is obviously because the underlining system that provides those abstractions is architected this way.

The distinction is actually relevant in a lot of contexts. As we get better languages and compilers that provide lower cost or more expressive abstractions, it is getting less and less relevant, but it still has a lot of relevance. And yes, it's a fuzzy concept like any other software architecture one.

A somewhat better dichotomy, faulty as all dichotomies, I suppose would be: No Magic Programming and Plenty of Magic Programming (or even All the Magic Programming). Where by magic one understands the expectation of the developer for the system to "just work": from the CSS developer writing a line such as `color: red;` and having no second thoughts to the browser engine developer knowing/caring about all the layers, from parsing to composition, to actually display a red colored text.

But even here, there is also a spectrum: should the browser engine developer know about the actual pixels of the display, and the "electrocoagulation of Ag nanoparticles between silicon nanostructures that support Mie resonances" [1], and so forth: the layers never stop, hence maybe it's more important actually caring about the product as a whole, in all its effects and side effects, a sort of omoiyari [2], as the Japanese would say.

[1] https://www.degruyter.com/document/doi/10.1515/nanoph-2022-0...

[2] omoiyari: 'give your thoughts and take an action on it', English: https://youtu.be/KJ5kcEQi934?t=71 Japanese: https://www.youtube.com/watch?v=16P_t4ApyhI

Or Copy-Pastable from StackOverflow Programming and Not-So-Copy-Pastable from StackOverflow Programming?

Just brainstorming here xD.

I think this is a different dichotomy - declarative vs imperative. Generally, the declarative languages have all the magic implemented in an imperative engine.

Sure, but the context is very important: you can use Rust to control the DOM from WASM [1] and then you can use Haskell to write a load balancer [2].

[1] https://rustwasm.github.io/wasm-bindgen/examples/dom.html

[2] https://thewagner.net/blog/2019/01/30/load-balancer (off-topic: Rob Pike's Concurrency is not Parallelism https://www.youtube.com/watch?v=oV9rvDllKEg)

It is a meaningful distinction. A mobile, desktop or a web app solves a problem for the end user and should be under their control via commonly-understood UI metaphores and IO mechanisms.

A systems program is a background service or a tool that provides the underlying abstractions for applications to run. An application programmer operates with well-defined APIs and, in general, should not be concerned with storage mechanisms, concurrency issues and resource management.

It's still a continuum, not distinct categories. People can't deal with that, and given the fact that there is really zero point talking about this continuum (it's not even in job titles) I would say we should just ditch the term "systems programming" entirely.

Kubernetes is clearly a different sort of program compared to Microsoft Word. Even if we don't have a rigorous ontology that sorts them into two distinct boxes neatly, they're just so different at the extremes that it's useful. We make a distinction between pulmonologists and cardiologists even though they both know anatomy and deal with the human body, and as we mature as a discipline, specialization is only natural. Denoting the area someone works in as belonging to one group or another is useful for determining peers that share your concerns, vs people that work in totally different areas from you. That doesn't make them better or worse than you, just different.

To me, one difference often is how much you have available to build on. At least for some "systems programming", the answer is: Not much.

Take an OS, for instance. You can build on the other parts of the OS (if they exist yet, and/or if your layered architecture permits you to), and the bare metal. That's it. That's all you have.

Contrast that with writing an app on top of, say, a POSIX-compliant OS, and there's a huge difference in how you program. Sure, as other comments point out, there's a continuum of points between the two, but there also is a pretty stark difference between the ends of the spectrum.

Guessing a bit here, but the difference may more generally be that in application programming, you are more constrained by the logic and purpose of the application, and in systems programming, you are more constrained by the system. The app is more stand-alone, and the system is defined by the interaction of the components. From this perspective, it may be fair to think of a microservices architecture as "systems programming".

Every software has a user. Other programmers are users too. In addition, all software is performance constrained, as no user has an infinitely fast computer or infinite time.

Of course it's fuzzy and unimportant, they're just words for categories we've made up, and didn't even make a very good job of it.

Even if we did make it clear which is which, what's the gain from being able to make a clear distinction? Is it any more productive than fighting over who got the best text editor?

Even if we did make it clear which is which, what's the gain from being able to make a clear distinction?

Exactly. I mean, what will you do differently at work tomorrow if your job title is changed from "Systems Programmer" to "Application Programmer"? Approximately nothing.

Is it any more productive than fighting over who got the best text editor?

The text editor discussion is probably more productive!

>Other programmers are users too.

System programming is about a program being used by another program unlike an application which is being used by a person. A programmer too will not use that program but use an application to develop a program which will then use that program. This makes system programming distinct from application programming in purpose, philosophy and reality.

What's the significant difference between having users, and having a chain of users, in purpose, philosophy and reality?

if we simplify it to being an OS (systems) vs a gui (applications) programs, we can see their purpose, philosophy, and reality. The OS' purpose is to run other programs, and thus that interface, Elf, is full of technical implementation details fitting the philosophy of a OS that allows multiple processes to run, and reality - Linux (OS) does not compete with Airbnb (application).

Contrast this to a GUI program a user interacts with. Its purpose is dependant on its relationship to the user. The philosophy is that the user could click on anything on the screen at any time, so the program had better be ready! The reality is that users live in their own world, free from all concerns of the silicon world, except one - power. They may perceive memory pressure in the form of slowness, but they don't feel it the way the operating system does.

This appears both incomprehensible and wrong, but let me check if I understood.

Systems:

- Lacks a GUI.

- There are technical details.

- Doesn't compete with applications.

- Memory pressure is an issue.

Applications:

- Has a GUI.

- Need to be responsive.

- Power usage is important.

- Memory pressure results in slowness.

I don't think we got any closer to a useful answer.

It can't have been that incomprehensible if you got that much out of it, but I mean if you're going to call me wrong, what's your take?

I don't think there's a significant difference and I don't see the importance of making the distinction.

UX?

The distinction goes to use case, and helps the hammer and nail programmers not get into arguments with the screwdriver and screws programmers over which is better - a hammer or a screwdriver? Of course, we as an industry love to argue, but I do think there is some use in having a distinction. Systems programmers have other programmers as users, application programmers have people as users. B2B vs B2C.

Got it. Businesses heavily employing a B2C marketing strategy should employ Python, Java and JavaScript programmers, while the B2B equivalents should use C and C++. Very useful to be able to make these decisions on a higher level, so that programmers don't need to argue about which tools would be well suited for a particular product. Allowing sales to make these decisions for us will improve our time to market.

My personal distinction is that application programming is more selfish, not interested in most other parallel applications whereas systems programming needs to take a more global view to ensure the system serves sufficient resources to all applications

Well, arguably any library provides services to other software and not directly to the user. The definition needs to be more specific than that. Is implementing left-pad systems programming?

Systems programming : application programming :: B2B :B2C