Learn x86-64 assembly by writing a GUI from scratch
gaultier.github.io
gaultier.github.io
It wasn't until I created a sinple 8086 emulator where you take the raw machine code instructions and translate those into not only the assembly instructions but actually emulate what those instructions do that I finally felt like I REALLY knew assembly.
My suggestion to others that want to learn assembly is to skip any assembly books. Using whatever language you want first start with a translator from machine code into assembly instructions, and then do an emulator. You only need to implement a small subset of the instructions, check out godbolt and translate some simple programs to know which instructions you need to implement.
Other than that all you really need is the 8086 manual, it has all the information there. I also found this site useful when implementing the flags https://yassinebridi.github.io/asm-docs/8086_instruction_set.... This takes less time than finishing a book and you learn a LOT more.
The goal is not to program in assembly at all but to truely understand the cost of everything and what you can expect from your hardware.
I learned assembly by reading the assembly listings from the C compiler. It is extremely interesting to be able internalize how high level constructs are compiled and optimized.
It was quite a long guide, but I would recommend it to anyone starting out. I don't have it in my bookmarks it seems, but I will try to update my comment tomorrow when/if I find it.
Edit: damn I guess it was https://beginners.re/ before it became pay-walled. Web archive still has copies of the book, but if you like it you should consider buying it even if it means signing up for patreon m-( I still got a few versions of the book somewhere as well. Have to dive in again to see if it is as good as I remember
eg
https://gitlab.com/hal88/junkcode/-/blob/master/c_template.c
The great thing is it still works if you include a bunch of local headers that are a hassle to supply to godbolt. Latency locally is a win.
The #if 0 trick for the compiler lines and #else for the code is a good one. Got it from Rusty Russell of iptables fame iirc. Write your script in C, why not?
Page Not Found
Make sure the address is correct and the page hasn't moved.
Please contact your GitLab administrator if you think this is a mistake. #if 0 //instructions to build and run
THIS_FILE=$0
BIN_FILE=/tmp/$(basename $0)
gcc -std=c11 -O0 -g -march=native $THIS_FILE -Wall -Wextra -o $BIN_FILE
if [ $? -ne 0 ]; then
echo "Bug in your C code or there is something wrong with your operating system's c compiler..."
exit 1
fi
# run it
$BIN_FILE "$@"
retval=$?
# uncomment below to examine the generated machine code
#
# objdump -DC $BIN_FILE | less -p '<main>'
# uncomment below to examine the assembly language the compier
# thinks it is generating
#
# gcc -S -std=c11 -O0 -g -march=native $THIS_FILE -Wall -Wextra -o ${BIN_FILE}.s
# vim ${BIN_FILE}.s
# clean up
rm $BIN_FILE
exit $retval
#else // c program
#include <stdio.h>
#include <stdint.h>
int main(int argc, char **argv)
{
for(int i = 0; i != argc; ++i) {
printf("argv[%d] = %s\n", i, argv[i]);
}
}
#endif //end c programThe “homeworks” only require implementing basic data transfer, arithmetic, and logic instructions, but I enjoyed it so I implemented everything except the interrupt handler stuff (into, etc.), and the BCD stuff (aaa, etc.).
I agree that it’s a good way to learn, and Casey provides a reference implementation.
Knuth's reasoning seems to be that higher-level languages go in and out of fashion all the time, but hardware and its associated assembly is quite sticky, so it's more "timeless". It's also a smaller set of primitives, so less overwhelming for the learner.
MMIX assembly is easier to understand than x86, having been designed specifically with learners in mind, and GCC even has a backend for MIX, so you can write C code and see how GCC would translate it to MIX assembly.
In for a penny, in for a pound.
> The goal is not to program in assembly at all but to truely understand the cost of everything and what you can expect from your hardware.
Entirely this. Also, to help you understand more deeply how computers really work.
That said, being able to program in assembly is still of great use to me. I do it to this day, usually on ARM processors -- not entire programs anymore, but critical parts.
Step 1. Implement a simple calculator
Step 2. Create a file format that encodes sequences of instructions and operands for your calculator
Step 3. Create an interpreter for that file format that runs your file. Add an accumulator and flags that represent overflow and such. And a instruction pointer.
Step 4: Add comments support to your file format (optional)
Step 5. Add support for logical operators, comparisons to your file format and interpreter
Step 6. Add support for labels and jumps to your file format and interpreter
Step 7. Add support for a stack, memory and related operators to your interpreter
In the end you should end up with something like
In order to read it, I suppose? There's no reason for memorizing binary encoding schemes. I mean, you will learn that 0x90 is NOP on x86 but that doesn't help you a whole lot.
[0] https://www.bartlettpublishing.com/site/books/learn-to-progr...
[1] https://download-mirror.savannah.gnu.org/releases/pgubook/Pr...
Seems very handy, since most assembly tutorials nowadays are Windows/Linux-specific.
This sounds like another one of those common "learn Asm by acting like a compiler" articles, which IMHO completely misses one of the best reasons to learn Asm: you can beat the compiler on size (relatively easy), speed (often harder), or both, precisely by not acting like one. I suspect the author, like so many others, also learned from only reading (some) compiler output. The complete lack of any use of static initialised data is shocking.
mov rdi, rdi
lea rsi, [rsp]
Stripped and OMAGIC (--omagic linker flag, from the man page: Set the text and data sections to be readable and writable. Also, do not page-align the data segment): 1776 bytes (1 KiB)
Besides being a very notable date (was that deliberate?), 1776 is closer to 2k than 1k. I suspect if you wrote it in C with inline Asm for the syscalls, it wouldn't be much bigger (and may even be a little smaller.)
If you want to see what Asm can really do, the sub-1k categories in the demoscene are well worth looking at.
These are pretty non-specific, but these are area I know about already for others who may have the same question as me:
1. Compiler development
2. Security research (malware analysis/reverse engineering) - although not much if any writing assembly, just reading
3. Kernel development - again mostly just reading assembly, not writing it. Bulk of code written in C (or potentially a very recent development, rust)
4. Driver development - mostly C but some devices can involve assembly
(Also, if you are dealing with something with mass deployment, it's good to recognize the single-bit flips that are hallmarks of bad RAM. But don't assume too much; bit flips are also the sign of bit flag manipulations.)
[0] https://en.m.wikipedia.org/wiki/Basic_Linear_Algebra_Subprog...
Do people go further than using instrinsics for let's say AVX?
Portions of the algorithm have been translated into assembly for ARM and x86. Shaving even a couple percent off something like motion compensation search will add up to meaningful gains. See also the current reference implementation of JPEG: https://github.com/libjpeg-turbo/libjpeg-turbo/tree/main/sim...
Why wouldn’t it be? Compilers haven’t advanced tremendously in the past two decades in terms of optimizations and don’t have much new to add to high performance SIMD numeric kernels.
[0] https://github.com/xianyi/OpenBLAS/issues/1968
[1] https://github.com/xianyi/OpenBLAS/blob/develop/kernel/x86_6...
[2] https://github.com/xianyi/OpenBLAS/blob/23693f09a26ffd8b60eb...
https://github.com/golang/go/blob/master/src/crypto/md5/md5b...
eg:
x264 https://code.videolan.org/videolan/x264/-/tree/master/common...
ffmpeg https://git.ffmpeg.org/gitweb/ffmpeg.git/tree/HEAD:/libavcod...
if you go up, you will find folders for other architectures (ARM, MIPS, SuperH, ...)
Not always the case. You're not writing it all the time but you still have to write it. For example the trampoline I use to jump from the boot stage to the kernel entry point is common-mapped between the two memory spaces and performs the switch inside of it, and then calls the kernel. That's all in assembly.
The kinds of utilities that come built into ROM, or that you run from a CD or USB drive, where you test memory and disk by writing different bit patterns to them, reading them back, and checking if they match, probing the hardware, processor and peripherals, etc.
I use assembly on the regular in my embedded systems work. It lets you get away with using a lower-spec (and therefore cheaper) microcontroller than you could otherwise.
But I'm interested in reading about malware written in assembly and was hoping for a diving board into that particular pool.
Indeed. It's also useful to differentiate between malware and exploits (although the former often includes the latter). Exploits it's common to use assembly when finding and developing the exploit, but unless you're severely byte constrained you're just gonna use tools to generate your shellcode instead of hacking it out by hand. Even then there are tons of pre-written shell code snippets you can reuse from places like metasploit. The number of jobs where you're paid to write an exploit are small unless you can get on an elite team in a government agency (or contractor). Malware on the other hand is mostly just written in higher level languages like C
High level hacking requires assembly because you're trying to reverse engineer opaque APIs that aren't meant to be interfaced with. What other way is there to do that other than trying to examine what things are being moved to which memory addresses
Why is Linux singled out there? No OS can use rcx for that, since the syscall instruction itself overwrites rcx with the return address.
in any case, there's a good discussion of registers and syscalls here
https://stackoverflow.com/questions/53290932/what-are-r10-r1...
> Following the System V ABI, which is required on Linux and other Unices for system calls, invoking a system call requires us to put the system call code in the register rax, the parameters to the syscall (up to 6) in the registers rdi, rsi, rdx, rcx, r8, r9, and additional parameters, if any, on the stack (which will not happen in this program so we can forget about it). We then use the instruction syscall and check rax for the return value, 0 usually meaning: no error.
> Note that Linux has a ‘fun’ difference, which is that the fourth parameter of a system call is actually passed using the register r10.
That makes it sound like it's only Linux that uses r10 and every other OS uses rcx as the 4th parameter with syscall, but really no OS uses rcx with syscall.
The app is an X11 client and will run under an OS, meaning you’ll learn to make system calls and other library calls to get things on the screen. Very educational, and not scary-deep.
http://www.afturgurluk.net/documents/Info/Win32ASM/Iczelion%...
https://www.masm32.com/download.htm.
Nowadays I mostly work on Linux and Mac, and wonder why there's no equivalent project exist. Perhaps those Unix coders are satisfied enough with C...
https://learn.microsoft.com/en-us/cpp/assembler/masm/microso...
Regarding UNIX, due to macro assembler nature of K&R C, and the spread of UNIX source tapes, there was never an Assembly culture like on the home computers where we had an integrated experience, of hardware and OS, that defined the platform.
It is similar to how John Carmack describes the IDE culture from those of us that grew up with those platforms, versus UNIX.
If you don't like MS assembler, the good thing is you can use open source ML-compatible assemblers like:
- https://github.com/JWasm/JWasm
I think a lot of the win32asm scene overlapped with defeating copy protection and making mods/cheats for games, not really Linux and Mac territory (at the time at least)
This is not correct. Only Linux has a stable kernel-userspace interface which allows you to depend on these numbers. In pretty much every other operating system, you are required to use the system libraries they provide. Compiling a program with these numbers hardcoded into them will cause them to break when the OS developers change the syscalls.
I wrote an article about this with more details and lots of citations:
The documented procedure for upgrades is update the kernel, reboot, update userland, restart deamons as needed. That only works if the old userland can run on the new kernel, so usually it can. Certainly there have been exceptions, but in my experience, they've all been fixed.
FreeBSD documentation includes a guide on assembly programming [1], which means libc isn't the only acceptable interface to the kernel.
Staticly linked executables are supported by the included compiler, those are expected to continue to work after kernel updates, which requires syscall stability.
Syscalls that take structures at risk of changing sizes generally include size as a parameter or within the structure, and the transitions to different sizes haven't always been easy (cpuset increases were rough in the past, but the next one will be better), but the syscall interface didn't change really.
[1] https://docs.freebsd.org/en/books/developers-handbook/x86/
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
Other systems have the same "usually works but no promises" disposition which means they can't be relied upon. Things work fine until they don't. When breakage occurs they take no responsibility for them because they didn't promise anything to begin with.
Torvalds says today's Linux can run binaries from the 90s, that's how serious this guy is about ABIs. I've never seen other operating system developers claim anything of the sort. Not even Microsoft.
Should probably be something like "Writing a Linux X11 application in assembly".
Is that true? I remember ~20 years ago I was looking at the i386 syscall ABIs (since amd64 wasn't big then), and there, Linux syscalls passed arguments by register and FreeBSD passed them on the stack. Maybe for amd64, FreeBSD switched to pass by register on Intel, but I wouldn't assume a syscall ABI is such a quick and simple substitution.
But, the BSD syscalls use the carry flag to indicate error, rather than the returned value of rax being negative. If your syscalls always succeed, and never return values within what would be a negative range as a signed value, then the code would run; but that's not exactly "portable".
* I would like to understand the assembly used for exception handling. Does anybody know how exceptions work at an assembly level? (I am interested in algebraic effects)
* Need to create a closure in assembly.
* I have some assembly ported to GNU assembly based on a blog post whose website is down that executes coroutines.
I'd suggest to start with the paper "Aspects of implementing CLU" from 1978 that covers both CLU's early type of exception handling and Iterators, which are a form of closures. To find out how modern C++ - style exception handling is done, read "Itanium C++ ABI" (yes, Itanium !), which most of the Unix world used as template for x86-64 and AArch64 later. Then look up "Zero overhead deterministic exceptions" for a proposal for C++ that didn't get picked.
> We would have been well-served to spend more time with CLU and C++ concepts earlier.
From https://go.googlesource.com/proposal/+/master/design/go2draf...
Another feature is the checked exceptions that everyone blames on Java, which were already present in Modula-2+, Modula-3 and C++, all of them inspired by CLU.
Assembly doesn't really have a concept of exceptions. System defined exceptions and handlers exist, like if you're on x86 and run in protected mode, you can get a processor exception if you access a memory address that's not mapped for the type of access you do; that functions more or less like an interrupt; if you're running in an operating system, the operating system will handle that in some way, and maybe pass that information to your program in some way (or maybe just kill your program), but again, that'll be defined by the system you're on, and we can't talk much generally. On some systems you can get an exception for math errors (divide by zero, overflow, etc), on others you have to test for them, some systems will generate an exception for unaligned data access, some won't, etc.
> * Need to create a closure in assembly.
Again, this isn't really an assembly concept. You've got to define what a closure means to you, and then build that however you like. In my mind, a closure is more or less a function plus a list of variables, in assembly, I'd model that as the address of a function that takes several addresses as parameters, but passing parameters is up to you --- if you're calling your own functions, you don't need to follow any particular convention on parameter passing, it just needs to make sense to you, and be written in a way that does what you mean: the computer will do what you told it to, which isn't always what you meant.
I would like to know how high level concepts map to assembly so I can understand how to compile to it.
I feel low level assembly gives so much freedom to decide how to do things.
I should probably get better at writing assembly so that I have inspiration on how to solve the high level things. But it's generations of technical ideas, solutions, implementation details and understanding I have to go through. I would like to understand exception handling to implement algebraic effects.
I also think structs are extremely useful and that it's amazing that sum types were invented.
and upstream:
> Need to create a closure in assembly.
"I would like to know how nuclear reactors work so I can build my own. I'd like to skip the Schrödinger Equation, differential equations, and linear algebra. And I want the nuclear reactor to run on thorium."
> I should probably get better at writing assembly
Yes.
Exceptions are not hard to understand once you know assembly language (any one of them). There are lots of blog posts you can look at. Algebraic effects are rather new and haven't been widely implemented yet ("thorium"). You most likely won't be able to find a pre-written document that spoon feeds you the implementation details.
A simple exceptions implementation uses a stack of "handlers". The code pushes and pops as it enters and leaves scopes. When an exception is raised, this stack is searched from top to bottom for a suitable handler (or maybe just the top handler is used). Precisely how depends on the implementation. The problem is that it's kinda slow to do all that work if exceptions are rare. Another implementation strategy is to have a table: the code address where the exception was raised is looked up in the table. That gives you the handler info. A bit more cumbersome if you have to handle linking. More so with dynamic linking. Even more so with run-time generated code.
Closures can be implemented in a billion and a half different ways. A common way is to allocate whatever local information ("captured variables") that the closure needs in a block on the heap. When the closure code is invoked, it gets a pointer to this block as a hidden parameter.
Of course, there are all sorts of code transformation tricks to complicate things...
Some you might run into often are transformations to and from SSA and CPS (+ the optimization transformations you can do on those):
https://en.wikipedia.org/wiki/Continuation-passing_style
https://en.wikipedia.org/wiki/Static_single-assignment_form
If you are interested in these things, you should really know some semantics and type theory:
https://en.wikipedia.org/wiki/Operational_semantics
https://en.wikipedia.org/wiki/Denotational_semantics
https://en.wikipedia.org/wiki/Type_theory#Technical_details
You do not have to know all the myriad variants, of course.
Learning a little assembly language is easy compared to semantics and type theory. Just learn your linear algebra and stop looking for shortcuts. It's like wanting to learn calculus while still being uneasy about fractions.
https://en.wikipedia.org/wiki/Microsoft-specific_exception_h...
https://learn.microsoft.com/en-us/cpp/cpp/structured-excepti...
Yes, Windows NT (and up) has a cross-language exception handling mechanism, separate from and in addition to whatever C++, Modula-3, etc have.
Steve Yegge worked there and tells an interesting story. 15 million lines of hand-written x86 assembly!
http://steve-yegge.blogspot.com/2008/05/dynamic-languages-st...
"OK: I went to the University of Washington and [then] I got hired by this company called Geoworks, doing assembly-language programming, and I did it for five years. To us, the Geoworkers, we wrote a whole operating system, the libraries, drivers, apps, you know: a desktop operating system in assembly. 8086 assembly! It wasn't even good assembly! We had four registers! [Plus the] si [register] if you counted, you know, if you counted 386, right? It was horrible.
"I mean, actually we kind of liked it. It was Object-Oriented Assembly. It's amazing what you can talk yourself into liking, which is the real irony of all this. And to us, C++ was the ultimate in Roman decadence. I mean, it was equivalent to going and vomiting so you could eat more. They had IF! We had jump CX zero! Right? They had "Objects". Well we did too, but I mean they had syntax for it, right? I mean it was all just such weeniness. And we knew that we could outperform any compiler out there because at the time, we could!
"So what happened? Well, they went bankrupt. Why? Now I'm probably disagreeing – I know for a fact that I'm disagreeing with every Geoworker out there. I'm the only one that holds this belief. But it's because we wrote fifteen million lines of 8086 assembly language. We had really good tools, world class tools: trust me, you need 'em. But at some point, man...
"The problem is, picture an ant walking across your garage floor, trying to make a straight line of it. It ain't gonna make a straight line. And you know this because you have perspective. You can see the ant walking around, going hee hee hee, look at him locally optimize for that rock, and now he's going off this way, right?
"This is what we were, when we were writing this giant assembly-language system. Because what happened was, Microsoft eventually released a platform for mobile devices that was much faster than ours. OK? And I started going in with my debugger, going, what? What is up with this? This rendering is just really slow, it's like sluggish, you know. And I went in and found out that some title bar was getting rendered 140 times every time you refreshed the screen. It wasn't just the title bar. Everything was getting called multiple times.
"Because we couldn't see how the system worked anymore!"
...I have to say, the "140 redraws by accident" part sounds like an ordinary day in web UI development using 2023 frameworks. The problem of not seeing the entire picture of what's going on isn't limited to assembly programmers. You can start from the opposite end of the abstraction spectrum and end up with the same issues.
https://en.wikipedia.org/wiki/RollerCoaster_Tycoon_(video_ga...
Besides the early computing days, the 8 and 16 bit home computers were mostly Assembly.
Even if Amiga, Atari and Archimedes had their share of BCPL, C and Modula-2, MS-DOS was fully implemented in Assembly.
Both were for 32 bit assembly, not 64 bit, IIRC.
Paul Carter was a professor or lecturer at a US college.
I think his book was available online.
Thanks for making it available online.
http://pacman128.github.io/pcasm/#
Scroll down the page for the PDF book.
This takes me back about 30 years as a youngster discovering the magic ASM incantation to efficiently draw to the screen in DOS mode 0x13.
Of course that won't be true for every program, but it's worlds away from asm.
Meant it more in the same that asm to rust is a jump and then there is another jump from say rust to python. I’m sitting on the rust rung and wondering whether that was the right choice so surprised people go one step further. (To each their own ofc)
Or even less recently...whoever wrote the first Rust, Zig, or insert <new compiled language> here?
Because don't you ultimately have to know how to make your own syntax translate into efficient assembly code?
Or is there someway these days for programming language designers/creators to avoid it entirely?
Another popular way to _sort of_ avoid assembly is to target the LLVM IR (intermediate representation), in which case LLVM takes care of optimization and producing processor-specific machine code for a bunch of CPU types. But LLVM IR is basically a fancy assembly language.
But in general, yes. To generate assembly you need to know assembly.
...
cmp BYTE [rsp], 1
jnz die
How can I diagnose the issue? The article didn't dive into the matter of reading error codes.
> In 64-bit mode, still use `xor r32, r32`, because writing a 32-bit reg zeros the upper 32. `xor r64, r64` is a waste of a byte, because it needs a REX prefix.
> just 600 lines of code