C Portability Lessons from Weird Machines
begriffs.com
begriffs.com
Of course it is wonderful that you can make programs that are indeed portable between this huge range of computers; just that not every program needs to do so.
Java sidesteps this, of course, by defining a JVM to run on and leaving the underlying details to the implementation.
Isn't most code that would behave differently on different architectures subject to undefined behaviour, however? The signed overflow case mentioned, for example.
Sure, some of it is implementation-defined, but in practice you need to write ultra-portable code anyway in order for your compiler not to pull the rug out underneath you.
Many "correct" programs rely on undefined behaviour for optimizations while the authors knowingly assert (or assume) that the undefined behaviour doesn't actually occur.
So the question is, if you are writing non-portable code for a specific environment, is a UB that actually triggers on your environment "non-portability" or is it a "bug"? I think most people would define that as being a bug.
Edit: Okay, bad example cos you could maybe use int_leastN_t or something in that example to document the assumptions... But you don't always have control over types, and there may be multiple constraints, etc..
It's not a fetish if you have ever ported legacy code that was not written with potability in mind.
"ints are always 32-bit and can be used to store pointers with a simple cast" may have worked when the legacy program was written, but it sure makes porting it a pain.
A program with 32bit longs will probably work fine most of the time, and save memory while doing it (which can matter for cache hits/misses).
If the spec was to make it work under certain conditions and not to worry about others, then the additional work for portability just adds cost.
Clearly if it it would have cost 2N dollars to obviate N dollars of work, then cost would not have been worth it. This can occur if either a) portability complicates things a lot or b) it complicates things a little, but over a very long timespan during which maintenance work is being done.
http://archive.opengroup.org/public/tech/aspen/lp64_wp.htm
"By far, the largest body of existing code already modified for 64-bit environments runs on LP64 based platforms."
This.
You can code portably and take advantage of platform-specific features or performance advantages.
I used to work on a product line that supported dozens of different UNIX platforms (and, eventually, even DOS protected-mode). The entire build process was portable. We had a primary configuration file for code/libraries and one for server internals that provided for:
. shared memory configuration (if any) and other limits
. BSD sockets vs System V streams
. byte-swapped vs regular processors and little- vs big-endian
. 16- vs 32-bit processors (later 64 bit)
. RISC vs CISC
. vagaries of C compilers, normal flags to pass for optimization, gcc vs cc, static vs dynamic links, libraries etc
. INT / LONG bytecounts, floating point, etc.
. ... and many others.
The goal was to abstract everything possible and have platform-specific code limited to the configuration files. Then, porting becomes fine tuning the configuration files, adding platform specific code to the mainline (with appropriate DEFINES) only when absolutely necessary, test, finalize configuration, test again, post the final config files and any changes back to the mainline.
So the question is are you contesting the POSIX defined behaviour over using more logical interfaces from the OS or the wild west where people were hacking around broken, platform-specific features and often broken or awkward system libraries or just the more modern case where people use the standard interface instead of a more performant one. In the latter case I agree I wish there was a more 'nice' way of doing more dynamic and efficient feature detection without making simple C programs crazy complex.
And newer C standards code may even unbreak such code. Ints already are two’s complement in the latest version of C++ and will be in the next version of C, for example.
Be careful about undefined behaviour still. Even if your architecture have defined semantic, the compiler might still not assume them.
That is a pretty bold claim. I would be surprised if any nontrivial "modern" program would be bug-free on a platform with nonzero NULL, non two's compliment ints, big endian an non IEEE floats.
The assumptions we make about the platform just don't feel like assumptions anymore because every machine is the same nowadays.
I fully agree, but the real problem is not the limitation to platform-specific logic but the conflation of fundamental requirements and incidental requirements. There's no good way to know when you meant "int" as just a counter of a user's files, or "int" as in a 32-bit bitmask, or "int" as the value of a pointer. For the former, it probably doesn't matter if someone later compiles it for a different architecture, for the latter, if you mean int32_t or uintptr_t, use that!
Because if you make an Electron app it's only logical that because it is platform-independent it can only be run on macOS
Your code includes your build system and dependency management. If they are not portable, your code is not portable.
I'm a language descriptivist. A standard is an imperfect attempt at describing a language. When the compiler and the standard disagree, the compiler is right, because it decides what the code will actually do.
OpenMP is an essential part of C/C++. Compilers that don't support it can't compile code that claims to be C/C++.
There is no point in pretending that you support systems you don't regularly use. In my case, portability means supporting Linux and macOS, x86 and ARM, GCC and Clang, but not in all combinations.
Portability, simplicity, and performance form a triangle. You can choose two and lose the third, or you can make compromises between them.
Pointers are 64-bit. If they are not, you need a lot of workarounds for handling real data, because size_t is too small.
Computers are little-endian, because that enables all kinds of convenience features such as using data from memory-mapped files directly.
Compilers should warn you if you use integer types of platform-dependent width, unless the width is required to be the same as pointer width. And except as argc and related variables.
To pick up some of that, compiler maintainers obviously disagree about the compiler always being right (which one?), and I'm baffled by the requirement for a compiler to support OpenMP (which version) to be considered C, especially if you're dealing with, say, embedded systems for bioinformatics data acquisition, for instance.
I've successfully claimed to support systems I'd never used in high-profile projects (particularly when the architecture and operating system landscape was rather more interesting); I couldn't just tell the structural biology users there was no point supporting their hardware. I currently support a GPU-centric POWER9 system, but I hadn't actually used POWER in the years I'd been building packages for it (and ARM).
As I said, I'm a language descriptivist. The compiler I'm using right now is right, because it builds the binary. If another compiler has a different opinion, then I may have to deal with multiple dialects. And because the default compilers in many HPC clusters and Linux distributions don't get updated that often, the dialects may continue to be relevant for years.
The situation with OpenMP is similar. A compiler does not have to support OpenMP to be considered a C compiler but to be useful as a C compiler. The bioinformatics software I need to build often uses OpenMP. If a compiler can't build it, it's not doing a very good job of being a compiler. OpenMP versions are not particularly relevant, as the usage is usually very basic. The only version conflict I can remember is from the C++ side: some compilers didn't support parallel range-based for loops.
The issue is when platform-specific code ties your business logic to a particular vendor API, and then this vendor milks your company for profit.
btw: a good indicator is GCC support - AVR, also an 8-bit µC - is perfectly well supported by GCC. 8051 and 6502, you need something like SDCC [http://sdcc.sourceforge.net/]
I only ever hear glowing/nostalgic reviews of 6502 programming, I guess from the retro/8bit gaming scene, curious what you find so atrocious.
What makes the 6502 atrocious for C is:
- internal CPU registers are 8 bits, no more, no less and you only have 3 of them (5 if you count the stack pointer and processor status register).
- fixed 8-bit stack pointer so things like automatic variables and pass-by-value can't take up a lot of space.
- things like "access memory location Z + contents of register A + this offset" aren't possible without a lot of instructions.
- no hardware divide or multiply.
Many CPUs have instructions that map neatly to C operations, but not 6502. With enough instructions C is hostable by any CPU (e.g. ZPU) but a lot of work is needed to do that on the 6502 and the real question is - will it fit in 16K, 32K, as most 6502 CPUs only have 16 address lines - meaning they only see 64K of addresses at once. Mechanisms exist to extend that but they are platform specific.
IMHO Z80 is better in this regard with it's 16-bit stack pointer and combinable index registers.
Also, the stack isn't addressable. The only stack operations the CPU natively supports are push and pop. If you want to access values anywhere else on the stack (say, if you're trying to spill a couple of local variables to the stack), you're going to have a bad time.
In the context of the original post, it's a bad target for C — I have no clue about other 6502 use :)
The great quality of 6502 was that it allowed a very cheap implementation, resulting in a very low price.
A very large number of computers used 6502 because it was the cheapest, not because it was better than the alternatives.
For a very large number of programmers, 6502 was the 1st CPU whose assembly language they have learned and possibly the only one, as later they have used only high-level languages, so it is fondly remembered. That does not mean that it was really good.
I also have nostalgic happy memories about my first programs, which I have entered on punched cards. That does not mean that using punched cards is preferable to modern computers.
Programming a 6502 was tedious, because you had only 8-bit operations (even Intel 8080 had 16-bit additions, which simplified a lot multiply routines and many other computations) and you had only a small number of 8-bit registers with dedicated functions.
So most or all variables of normal sizes had to be stored in memory and almost everything that would require a single instruction in modern microcontrollers, e.g. ARM, required a long sequence of instructions on 6502. (e.g. a single 32-bit integer addition would have required a dozen instructions in the best case, for statically allocated variables, but up to 20 instructions or even more when run-time address computations were also required, for dynamically-allocated variables.)
A good macroassemmbler could simplify a lot the programming on a 6502, by writing all programs with a set of macroinstructions designed to simulate a more powerful CPU.
I do not know whether good macro-assemblers existed for 6502, as in those days I have used mostly CP/M computers, which had a good macro-assembler from Microsoft, or the much more powerful Motorola 6809. I have programmed 6502 only a couple of times, at some friends who had Commodore computers, and it was weak in comparison with more recent CPUs, e.g. Zilog Z80 (which appeared one year later than 6502).
They certainly did. I don't know about the communities that grew around Commodore, Atari, or other 6502-based computers, but in the Apple II world, there were multiple macro assemblers available. Possibly the most famous was Merlin. As a pre-teen, I used the Mindcraft Assembler. Mindcraft even sold another product called Macrosoft, which was a macro library for their assembler that tried to combine the speed of assembly language with a BASIC-like syntax. The major downside, compared to both hand-coded assembly and Applesoft BASIC (which was stored in a pre-tokenized binary format), was the size of the executable.
Edit: Speaking of simulating a more powerful CPU, Steve Wozniak implemented the SWEET16 [1] bytecode interpreter as part of his original Apple II ROM. Apple Pascal used p-code. And a more recent bytecode interpreter for the Apple II is PLASMA [2].
at least that's the part I remember
On the real low end PADAUK is king, at under 3 cent retail. They don't even bother with a C compiler and instead have a macro assembler on steroids called Mini-C (there is also sdcc but I never used that having needed every last 13 but word of ROM). Programming these, with 64 bytes of RAM, is strangely refreshing - no fancy abstractions, microsecond-precise instruction timings.
I pointed out that any programs that needed to manipulate byte data would require very special coding to work. Special enough to make it pointless to try to write portable code between it and a regular machine.
It's unreasonable to contort the standard to support such machines. It is reasonable for the compiler author on such machines to make some adjustments to the language.
For example, one can say C++ is technically portable to 16 bit machines. But it is not in practice, because:
1. segmented 16 bit machines require near/far pointer extensions
2. exceptions will not work on 16 bit machines, because supporting it consumes way too much memory
3. ditto for RTTI
(SCNR)
Actually this concerns more pre-80286 processors, since 80286 introduced virtual memory, and the segment registers were less prominent in "protected mode". Moreover I wouldn't say it was a pain, at least at the assembly level, once you understood the trick. C had not concept of segmented memory, so you had to tell the compiler which "memory model" it should use.
> One significant quirk is that the machine is very sensitive to data alignment.
I remembered from school time about a "barrel register" that allowed to remove this limitation, but it was introduced in 68020.
On the topic itself, I like to say that a program is portable if it has been ported once (likewise a module is reusable if it has been reused once). I remember porting a program from a 68K descendant to ARM, the only non-obvious portability issue was that in C, the char type is that the standard doesn't mandate the char type to be signed or unsigned (it's implementation-defined).
> Actually this concerns more pre-80286 processors, since 80286 introduced virtual memory,
86 had segments, 286 added protected mode, 386 added virtual. I would agree, though, 286 wasn't as bad as people make it sound. In OS/2 1.x it was quite usable.
Here's a summary of how it worked in protected mode for those not familiar with 286.
In protected mode the segment registers no longer held segment numbers. They held selectors. A selector contained three things:
1. An 13-bit index into a "descriptor" table. A descriptor table was a table describing memory segments, with each entry being an 8 byte data structure that included a 24 bit base address for a segment, a 16 bit size for a segment, and fields describing access limits and privilege information,
2. A bit that told which of two descriptor tables was to be used. One table was called the global descriptor table (GDT) and was shared by all processes. Typically the GDT would be used to describe the memory segments that were used by the operating system itself. The other table was called the local descriptor table (LDT), and there was one LDT per process. Typically this described the segments where the code and data of the process resided.
3. A 2-bit field contain access level information, used as part of the 286's 4 ring protection model.
To translate a selector:offset address to a physical address, the index from the selector was used to find a descriptor in the descriptor table the selector referred to. The offset from the selector:offset address was added to the segment base address from the descriptor, and the result was the physical address.
(The 80386 was similar. The main difference, aside from supporting 4 GB segments, was that the base address in the descriptor was no longer necessarily a physical address. The 386 included a paged memory management unit in addition to the segment based MMU. If the paged MMU was enabled the base addresses in the segment descriptors were virtual address for the paged MMU).
Here is how they packed the 13-bit index, 1-bit table selection, and 2-bits of access level into a 16-bit selector:
+--------------------------+--+----+
| INDEX | T| AL |
+--------------------------+--+----+
If they had instead laid out the fields in the selector like this:
+----+--+--------------------------+
| AL | T| INDEX |
+----+--+--------------------------+
So why didn't they do this?
One theory I've heard is that because the descriptors in the GDT and LDT are 8 bytes each, to get the address of a descriptor the processor has to computer 8 x INDEX + BASE where base is the base address of the descriptor table, and by having INDEX in the selector in the to 13 bits, it is already in effect multiplied by 8 saving them having to shift the INDEX before feeding it into the adder for the address calculation.
I've talked to CPU designers and asked about that, and what they have told is that shifting INDEX for this kind of calculation is trivial. The segment registers would already be special cases, and it would be likely their connection to the adder used for address offsetting could simply be wired up so that the input from the segment registers was shifted. Even if they did need a little more hardware to handle it the amount would be very small, and almost certainly worth it for the tremendous improvement it would make to the architecture as seen by programs.
My guess is that simply no one involved in designing the selector and descriptor system was a compiler or OS person and so they didn't realize how they laid those 3 fields out would matter to user mode code.
Took a bit to get used to but really the only way to get to the good stuff was by writing assembly code and hand-tuning all the pipeline stuff.
Too bad programmer laziness won and most current hardware doesn't support this.
As a teenager I remember getting hit by this all the time in assembly language programming for the IBM S/360. (FORTRAN turned it off).
S0C8 Fixed-point overflow exception
I don’t think this is a age problem. Plenty of adults are lazy and plenty of kids aren’t.
The question is rather: was it really a good idea to bake 'signedness' into the type system? ;)
That’s an extremely disingenuous line of reasoning, the situations where it’s a feature is a microscopic fraction of total code: most code is neither interested in nor built to handle modular arithmetics, and most of the code which is interested in modular arithmetics needs custom modulos (e.g. hashmaps), in which case register-size-modulo is useless.
That leaves a few cryptographic routines built specifically to leverage hardware modular arithmetics, which could trivially be opted in, because the developers of those specific routines know very well what they want.
> signed vs unsigned are just different views on the same bag of bits […] The question is rather: was it really a good idea to bake 'signedness' into the type system? ;)
The entire point of a type system is to interpret bytes in different ways, so that’s no different from asking whether it’s really a good idea to have a type system.
As to your final question, Java removed signedness from the type system (by making everything signed). It’s a pain in the ass.
That's not removing signedness. Removing signedness would be treating integers as sign-less "bags of bits", and just map a signed or unsigned 'view' over those bits when actually needed (for instance when converting to a human-readable string). Essentially Schroedinger's Cat integers.
There'd need to be a handful 'signed operations' (for instance widening with sign extension vs filling with zero bits, or arithmetic vs logic right-shift), but most operations would be "sign-agnostic". It would be a more explicit way of doing integer math, closer to assembly, but it would also be a lot less confusing.
And in x86 land there's the into instruction, interrupt if overflow bit set, so you're left with the same options.
There were discussions around this a few years back when Regher brought up the subject, and one of the issues commonly brought up is if you want to handle (or force handling of) overflow, traps are pretty shit, because it means you have to update the trap handler before each instruction which can overflow, because a global interrupt handler won't help you as it will just be a slower overflow flag (at which point you might as well just use an overflow flag). Traps are fine if you can set up a single trap handler then go through the entire program, but that's not how high-level languages deal with these issues.
32b x86 had INTO, and compilers didn't bother using it.
BTW. The Mill architecture handles exceptions from integer code like floating point NaN: setting a meta-data flag called NaR - Not a Result. It gets carried through calculations just like NaNs do: setting every result to NaR if used as an operand... Up until it gets used as operand to an actual trapping instruction, such as a store. And of course you could also test for NaR instead of trapping.
I've done C on half of those platforms. 8051 is actually more complex than described. There are two different kinds of RAM with two different addressing modes. IIRC there are 128 bytes of "zero page" RAM and, depending on the specific variant, somewhere between 256 bytes and a few kilobytes of "normal" RAM. Both RAM types can be present, and the addresses can both be the same value, but point to different memory, so the context of the RAM type is critical. The variants usually have a lot more ROM than RAM so coding styles may need to be adjusted, such as using a lot of (ROM initialized data) constants instead of computing things at run time.
6502 has a similar "zero page" addressing mode to the 8051.
I never encountered any alignment exceptions on 68k (Aztec C). Either I was oblivious and lucky, or just naturally wrote good code.
I do remember something about PDP-11 where there was a maximum code segment size (32k words?).
C on the VAX (where I first learned it) was a superset of the (not yet) ANSI standard. I vaguely remember some cases where the compiler/environment would allow some lazy notation with regard to initialized data structures.
They left out some interesting platforms (such as 6809/OS9, TI MSP430 and PPC), which have their own quirks.
PDP-11 had a weird RAM overlay scheme of squeezing 256KB RAM into a 64KB 16-bit address space.
IBM System/360 also had a weird addressing scheme with base register and up to 4KB offsets.
https://en.wikipedia.org/wiki/IBM_System/360_architecture#Ad...
With the imminent C23 spec I'm really amazed at how well C has held up over the last half century. A lot of things in computers are very 'temporal' (in that there are a lot of things that are all available at a certain point in time that are required for the system to work) but C has managed to dodge much of that.
The Texas Instruments' C40 was a DSP made in the late 90's. It had 32-bit words, but inefficient byte manipulation. The compiler/ABI writer's solution was simple: make char 32 bits. So sizeof(char)==sizeof(short)==sizeof(int)==sizeof(long).
I remember writing routines for "packing" and "unpacking" native strings to byte strings and back.
It may be easier to fix the code than if you didn't think about the Unisys, but that effort has to be weighed against the incredibly vanishing odds of it ever being required.
Also you're very limited as to what you can actually do in ANSI C. You're going to need to start poking directly at the hardware which is not going to be portable. Hell, even stuff like checking if a letter is within a certain range in the English alphabet might not work between machines. Letters might not be contiguous in the machine's native character encoding.
People who say everything is little endian presumably don't maintain packages for the major GNU/Linux systems which support s390x. I don't remember how many big endian targets I could count in GCC's list when I looked fairly recently. The place to look for "weird" features there is presumably the embedded systems processors.
I am no expert in HTML video delivery and haven't tried it out, but maybe setting the preload attribute to "none" or "metadata" might help?
I can't read the model number on the frame of video it statically gives me, and I'm not going to play the video to find out...