Hacker News Reader

Comment by Areading314 | Hacker News Reader

Out of curiosity, what are the main reasons to need to actually write assembler in 2018? Compilers? Games? Genuinely curious

View on HN

1. When you have no choice in languages: Homebrew development for, say, the Nintendo Game Boy happens by and large in assembly language.

2. Optimization: Some critial code sections or cryptographic operations can be massively sped up with hand-crafted assembly[1].

3. Obfuscation: Sometimes, you have to drop to assembly for obfuscation. This is especially the case if you need to/want to fool the Hex-Rays Decompiler. Applications include malware and digital rights management (some might argue the latter is just a sub-species of malware). This also includes writing your own custom assembly language for a custom virtual ISA for the purpose of code obfuscation.

4. Embedded platforms: Some very obscure microcontrollers may only have support for pure assembly, though at least C89 compilers being available seems to be the norm.

5. Education: It can be very enlightening to understand how things work below your "home" layer of abstraction. Some parts of C seem to be much easier to understand once you have a firm grasp on a common assembly language.

6. Compilers: Self-evident, you kind of have to understand the code you emit.

[1] See e.g. this benchmark https://monocypher.org/speed -- authenticated encryption on libsodium gets an improvement of over 300% from hand-crafted assembly over the portable C implementation

Another common use case is for writing shellcode and other exploit-related work.

Low-level system code (particularly when you're doing things like servicing interrupts) needs to be written in assembler. The very early boot phase is even more annoying because you're starting out in one of the 16-bit modes and working your way to changing to x86-64 and virtual memory.

Outside of system code, the main use for assembly is trying to maximize performance on hot loops. Your optimized matrix-multiply routine, or media decoding kernel, could well be written in assembly. I've seen a few cases where people do things such as manual calling conventions in assembly as well.

>you're starting out in one of the 16-bit modes and working your way to changing to x86-64 and virtual memory

"ontogeny recapitulates phylogeny"

https://en.wikipedia.org/wiki/Recapitulation_theory

The book reasons that learning assembly also teaches the fundamentals of computer architecture.

1.3.1 Gain a Better Understanding of Architecture Issues 1.3.1 Understanding the Tool Chain 1.3.1 Improve Algorithm Development Skills 1.3.1 Improves Understanding of Functions/Procedures 1.3.1 Gain an Understanding of I/O Buffering 1.3.1 Understand Compiler Scope 1.3.1 Introduction to Multi-processing Concepts 1.3.1 Introduction to Interrupt Processing Concepts

The book uses the 1.3.1 heading for each and I'm too lazy to change them.

Reasons to (maybe/arguably) write assembly:

1) You're bringing up a new board, your bootloader is partially written but you need some customization for the real-time OS you're using. It can be advantageous to do this in assembly 2) You're dealing with some particularly old hardware and (ab)using it for some commercial purpose

Of course, I can imagine that for each you'll have someone obstinately state there's no need to use assembly because of some gcc feature. More than one way to get things done, and most use the tools they're comfortable with.

Two use cases I've had in recent years (technically not 2018, but well…): - optimization of simple functions that are called a huge amount of times (the only viable alternative to assembly being intrinsics that come with their own portability constraints), - just in time compilation.

Additionally, knowing assembly can help a lot when debugging weird crashes in code written in higher-level languages. gdb's asm layout is an awesome resource if you know how to use it, but someone who's never used assembly before will probably not even consider using it.

One practical "high level" example would be analyzing function/type specialization in Julia and understanding the resulting code that can be easily inspected.

See eg (from 2013, but at a glance seem to outline the ideas well):

http://blog.leahhanson.us/post/julia/julia-introspects.html

For Julia - it's fairly easy to work "high up" most of the time - and drop down inspecting the code (and profiling) - and unlike many other languages, even high pref libraries will often be julia all the way down (unlike tcl/perl/ruby/python + c/fortran etc).

https://docs.julialang.org/en/stable/manual/profile/

https://docs.julialang.org/en/stable/devdocs/reflection/

Similar for sbcl (common lisp) or various languages that target/use llvm. And obviously for looking at output of optimizing compilers for c, c++, rust, d and similar "medium to high" level languages (Pascal, Ada, crystal, etc).

Assembler is still popular for IBM mainframes. The current version has been around since '92 and is called High Level Assembler.

It's popular partially because people have codebases that they started writing in the 70's or 80's in assembler that they maintain to this day because it's cheaper than switching it all over to a new language. Pretty much the same reason that COBOL is still around.

z/OS (the OS that runs on IBM mainframes) also exposes a lot of it's functionality through HLASM, so it's far more convenient to use than x86 assembly.

For whatever reason, C also never really caught on as ubiquitously as it did in the PC world. Probably because IBM themselves generally used their proprietary PL/S language instead back in the 70's and 80's.

https://en.wikipedia.org/wiki/IBM_High_Level_Assembler

UNIX was to C what the browser is to JavaScript.

Naturally it didn't caught on mainframes, already using better system languages.

It is also barely used on Unisys ClearPath MCP.

This is fascinating, I didn't know there was a such thing as a high level assembly language, but IBM High Level Assembler has IF/ELSE/ENDIF, and several types of built in loops. I wonder how similar it is to writing in C. One thing this page doesn't mention is structured data types, I suppose these would still have to be implicit like in other assembly languages.

The IF/ELSE stuff is similar to the preprocessor macros people write in C. They basically generate HLASM code on the fly based on certain flags being passed to the program and what not.

If you're curious what a simple program ends up looking like, I've got one I wrote that copies the contents of one file into another file up on GitLab. Lots of loading registers and what not.

https://gitlab.com/thisisnickwilson/zos/blob/master/IO

Thanks, this is pretty interesting to read through, and your comments are very helpful. I didn't realize this language has no comment syntax, but I guess it makes sense since each opcode probably has a fixed number of parameters and anything after that can be safely assumed to be comments. Neat stuff.

Yes, it was quite common in the 70 and 80's.

MS-DOS and Amiga assemblers (TASM, MASM, DevPac) had quite powerful macro capabilities, many of which gas still doesn't support.

Here is the documentation for MASM high level constructs, it is still distributed with Visual Studio.

https://msdn.microsoft.com/en-us/library/8t163bt0.aspx

NASM also has macro capability, though I'm not sure how it compares to the others you mentioned (EDIT or to gas). On the plus side, it's available on Linux.

https://www.nasm.us/doc/nasmdoc4.html

Also not sure, they are supposed to be quite good, but by the time NASM came around, my focus was no longer on pure Assembly programming, so I never used it in anger.

I used to write assembly language programs back in the 70s while working on process control computers (Texas Instruments 990, TI 980, TI 960 etc). At one point I was using an assembler that supported complex macros (macros that could be expanded into other macro definitions and supported counters and so forth) so I developed a library of macros that supported nested it-then-else and loops. They made the code a bit easier to read, but it was probably not worth the trouble.

The problem with a high level assembly language is that it really isn't very high level; your program still rests right on the hardware for a reason, and usually that reason is a concern about using registers and instructions very carefully for performance or interacting with hardware at ring 0 level where you are managing the virtual memory page table or handling network device interrupts or system IPC and so forth.

In my experience (as an IBM AIX kernel architect, virtual memory architect, and distributed file system design), sometimes one needs assembly language, but it was always a relief to get up to the level of C programming where the programming teams were much more productive. Much OS development has been done with C and it really was the best choice for most of the kernel work going on back then in my opinion.

AIX was an interesting project. The hardware didn't exist in final form while AIX was being developed. The challenge for our group was developing/porting a whole OS, the kernel and user space code, that would run on hardware being developed at the same time. IBM's language PL/1 was an important mainframe language, but seemed a poor fit for systems programming. However, IBM had state of the art compilers for it and a strong research interest in compilers for RISC machines (like the POWER processors, the first of which outside of IBM's research processors would run AIX 1); so they took the 80% of PL/1 that seemed useful to systems programming and wrote a compiler for PL.8 (.8 of PL/1) to run on the hypothetical RISC system my group was developing.

We were developing a Unix system on the RISC hardware, but we didn't have a stable target (page table sizes, floating point hardware traps, etc.) and couldn't afford to wait for the hardware before starting development. The approach my group took was to write the lowest level parts of the kernel in PL/.8 so that as the hardware changed the compiler could be tweaked to take advantage of it easier than rewriting low level assembly language code. The high-level parts of the kernel (coming from licensed Unix code) could then be mated to the low level code and wouldn't be affected by the changes in the hardware that happened over time.

I wasn't in charge of these decisions, so I don't really know enough about them to say that this was better or worse than just using C and assembly language as is normally done in most OS development, but I do see some of the trade offs that had to be made.

An aside on higher level system programming languages, I know that some on HN say that C is a terrible choice for OS development. Perhaps there are better choices (now), but I see things a bit differently. At the time there were not obvious choices that were better. We didn't have Rust or even C++. We had C, Pascal, MODULA, PL/1, and a few other unlikely choices (e.g. ALGOL-68, LISP, JOVIAL). C is a big improvement over assembly language, but it isn't clear to me that PASCAL or MODULA, or LISP or the others available back then were better choices than C. Unix became a kind of proof of C's suitability as a OS development language. Before that, PL/1 had been used to develop Multics, but Multics failed as a commercial OS (despite it's subsequent influence on OS design). C was simpler than PL/1. Algol had been used by Burroughs, but it was a non-standard version of Algol specially designed to work with the rather novel hardware.

C is flawed but none of the other candidates for a language higher level than assembly language for system programming was without flaws and they hadn't produced something like Unix. The C used in the Unix kernel was the real K&R C; it was the same language that ran on many platforms. Other attempts at a high level systems programming language based on Lisp, Smalltalk, Pascal, Algol, and IBM's proprietary subsets of PL/1 were all languages modified for the hardware they ran on. C seemed to be just low enough to work for most of the kernel's requirements without special extensions.

I always appreciate pjmlp's comments reminding HN readers about Pascal or Modula. I liked those languages; I'm very familiar with them. I still think C was the correct language for system programming in the past. Today, I'm more interested in seeing what happens with Rust for kernel development and Go for non-kernel systems programming.

Thanks very much for the insightful comment.

Also interesting to learn that PL.8 also had a shot at the Aix kernel. I got all the PL.8 papers I could get my hands on.

Regarding UNIX and C's adoption, I think that had Bell Labs been allowed to go commercial from day one with UNIX and history of C's adoption would have been quite different.

If you are developing a compiler you will need to understand the output assembly. Learning to program in it is one way to get to know it. There is also plenty of little code snippets around run-time libraries and OS that are written directly in assembly (e.g. crt0.o, crt1.o & friends).

Basically: somebody has to provide the infrastructure between assembly and high level languages, so that everyone else can write in HLL.

Optimization of games or HPC workloads is also a valid use-case, but today you probably want to use intrinsics in C instead.

Not necessarily regarding the latter. Intrinsics only help with the part where your code doesn't fit the instruction set because the latter can encode operations the former can't natively, and the compiler might not always be able to detect such structures.Als Also, if you use assembler you can take better care about scheduling instructions around loops in a way to get very high throughput, because you might see that k-ary search happens to be faster than binary search because you can hide some of the latency you'd otherwise have. Also some other issues requiring higher-level restructuring once you see that your first implementation stalls the processor too hard. Once you reach GPU cores and other non-OoO cores you need to be careful with memory load stalls, and try to use some staging memory or so and e.g. load the data for the iteration after the next iteration into staging, then fetch it into registers in the next iteration, so it is ready in the one where it will be processed. You need such optimization to reach e.g. high throughput in matrix multiplication, because you need to minimize the instructions that use the ALU but don't do actual result-affecting calculations (and instead just e.g. loop/array index manipulation and such). The ability to be good with this is reduced in OoO machines with register renaming, sadly, and I'd like a core that is fast for sequential code due to low-delay high-clock operation, yet exposes the whole register set to the assembly programmer... One of the recent chip startups seems to target such a processor. Obviously also the Mill, but that's vaporware since forever...

Some mainframe and workstation architectures never had direct support for Assembly instructions, only intrinsics on their system languages.

Burroughs, Xerox PARC workstations, ETHZ workstations as a couple of examples.

For me, because it is interesting. It is neat to understand how things work at a lower-level.

Knowing assembly also helps out when doing capture the flag challenges, specifically reverse engineering and binary exploitation.

Lots of reasons, but one would be processor specific optimizations and/or extensions for performance reasons.

OpenSSL, for example, has a fair amount of ASM in it for various processors to speed things up.

See https://software.intel.com/en-us/articles/improving-openssl-... for some examples of what they specifically did for x86/64.

I've seen it used a lot in I/O from micro-controllers where speed/timing is important. The arduino C++ library for instance does some weird things under the hood when reading from an Analog-Digital-Converter (read the voltage at some pin), whereas with assembly you know that you can read in exactly 1 instruction. When timing or speed are important, that level of control is useful.

Knowing assembly made me a better C programmer. I still remember this tech interview I had with a Seattle based company (no, not MSFT) many years ago. The interviewer was trying to twist and turn all kinds of C-pointer questions, but knowing how the code translated to assembly made the whole thing a breeze. I got the job but didn't take it, and I somewhat regret that.

Perhaps wanting to know how the computer works.

Firmware and driver development. Not in all cases, but you're very likely to encounter it in those domains.

I work on compilers and I hate assembly.

shellcode, maybe, or really tight inner loops that need to use weird vector instructions.

It's not the vector instructions, it's the careful scheduling of instructions to spend just enough time manipulating pointers when you want to crunch actual data. All while respecting dependency chains and memory stall times. (Hyperthreading helps a lot with the latter, see Nvidia Maxas (nervana systems now) for details on how flexible number of threads benefit weighing of memory load stall hiding vs. register pressure causing more data shuffling.

Take a look at projects such as pixman [0]

But other than highly opimized graphics manipulation I can't really think of any other good application for assembly.

[0] http://www.pixman.org/

Not necessarily "write" assembler, but understand it, and, as always we are not at the end of history. New dominant CPU architecture could arise, people understanding and writing assembler will always be needed.

Compilers, embedded systems and similar low level work, reverse engineering where you don't have the binary (e.g. malware), and that kind of thing.

Tasks involve reading assembly are more common, and it's hard to understand it very well if you've never written any.

Cryptography, not only for speed, but also for ensuring implementations are constant time.

Embedded devices?

I have never had the need to write assembly on any embedded device. There are C (or other high language) compilers for just about any architecture. For very specific instructions (i.e. interrupt jumps) there is usually available as a compiler alias/feature.

If you want to turn an AVR/AtMega32 into a spectrum analyzer, you need to massage the x, y and z registers that support implicit incement/decrement adressing. C fails hard for such code.

better performance and millions of dollars of monthly savings in cost (in a bank) - not x86 though

reverse-engineering ?

Maybe not that much to write it, but it's very valuable to understand the disassembly when debugging.

To learn what programming really is and how systems really work.

You can't really understand programming without having built a compiler. Or at the very least having done assembly programming. I don't think you can truly understand any basic programming concept like variable, pointer, reference, etc without learning compilers or assembly. Yes, a pointer "points" to something. But what does that really mean? When you realize that it's all just higher level concepts of specific assembly mechanisms of accessing data, you'll have a eureka moment. Don't see how you do that without digging into compilers/assembly.

And ultimately, it'll make you a better overall programmer since compilers/assembly is fundamental to every programming language.

> You can't really understand programming without having built a compiler. Or at the very least having done assembly programming. I don't think you can truly understand any basic programming concept like variable, pointer, reference, etc without learning compilers or assembly.

This is just elitism with a little bit of "back in my day" thrown in.

I can assure you that I fully understood the concepts of pointers, variables, and references without knowing anything about assembly language or compilers. The concepts are not that difficult to grasp.

Hmm , this is great. But to understand where variables are placed in the stack and how stack is managed your need to play with Assembly. Pointers are pointers but when you play with using Assembly then your know why some languages exposes only reference and lack pointer arithmetic. It sounds easy but it help you to build compilers or whatever.

Pointers deliberately hide these details from you by obscuring where things are actually located; presenting a unified memory model. To really understand what's going on you need to know how the stack is laid out, how structures work, and how functions are called.

> This is just elitism with a little bit of "back in my day" thrown in.

That wasn't my intention and I apologize if it came off that way. Also, I'm not that old and I certainly didn't start programming with assembly. My programming experience began with OOPs ( Java, C#, C++ ). Didn't really touch assembly until I went to college. Also, compilers is a basic core course you have to take to get a CS degree. How is something everyone has to learn elitist?

> I can assure you that I fully understood the concepts of pointers, variables, and references without knowing anything about assembly language or compilers. The concepts are not that difficult to grasp.

The concepts were fairly difficult for me. Especially pointers. I thought I understood it but then later on realized I didn't. Maybe other people have their eureka moments sooner. For me it happened when I built a compiler and did some assembly programming.

I just think knowing how things work under the hood will make you are better programmer. Especially something like assembly since all programming languages gets translated to assembly whether you are using C# or Haskell or Lisp or ML. I wasn't trying to offend anyone. But I'm sorry if others were offended by my comment. That was not my intention.

For context, I agree with you that understanding compilers is very important to a deep understanding of the tools used by programmers. Knowing how languages are implemented sheds light into why certain design decisions are made and why certain language features behave the way they do.

That said...

> Also, compilers is a basic core course you have to take to get a CS degree. How is something everyone has to learn elitist?

Not everyone is lucky enough to get a CS degree.

Some people come from backgrounds where CS was not in their worldview growing up. Or they were discouraged from pursuing CS despite the subject being interesting to them. Or they tried taking CS, but they had to deal with external factors that prevented them from continuing.

To discount these individuals from making meaningful contributions to a programming endeavor is short-sighted. Their particular experience may allow them to be better at implementing the correct solution, even if they are not as strong of a programmer. Or they may be a very fast learner, and given more time and mentorship, they will learn about compilers, but they can be strong contributors until then.

> Also, compilers is a basic core course you have to take to get a CS degree.

I'm not sure if this is true. It wasn't required for my degree.

I wasn't offended by your comment, though. I just disagree with you. No hard feelings about that.

Anyways, I agree that knowing how things work under the hood will make you a better programmer. I just disagree with how you phrased that sentiment in your original comment.

Reversing the NSA's latest bad idea to figure out how to stop it from taking over your network.

Reply on news.ycombinator.com