Hacker News Reader

I mentioned this in another article of mine. Robert Seacord closed the loop on "how do I print numbers the non-shit way?" in C23 with N2680:

https://thephd.dev/c-the-improvements-june-september-virtual...

https://www.open-std.org/jtc1/sc22/wg14/www/docs/n2680.pdf

Great, thanks! But that only solves printing. How do I do the reverse operation? I.e. parsing a string to an integer type of an unknown size, while checking for overflow?

¯\_(ツ)_/¯

Someone's got to write a paper to get it in, hopefully based on some sort of existing practice.

I think this would work: first, determine the range. For unsigned variants, we have whatever_max = (whatever)(-1), if it isn’t already defined.

That makes parsing the unsigned variant relatively simple. Undefined behavior would be end-of-game, but unsigned addition and multiplication will not invoke that (if you need he,p, look at your favorite multiple-precision library for ideas about detecting wraparound)

For parsing signed integers, derive the min and max values of the signed variant from the max value of the unsigned one ((whatever_max - 1) / 2 or something like that

Then parse your input to the unsigned variant first, then check whether it fits in the unsigned version.

I think it's best to regard intmax_t as a failed experiment. E.g. GCC has long supported __int128, clang has _ExtInt, other compilers have similar extensions, but they refuse to increase intmax_t due to ABI concerns.

OK, fine, but how, then, can the two problems I described be solved?

Well, what did you do before intmax_t was added to C99?

- POSIX defines that in_port_t is equal to uint16_t, so for that type there's no problem at all ;)

- Don't use printf/scanf for this, roll your own

- Refuse to print/read uid_t that are outside intmax_t range

- Use an autoconf test

My program was first written when intmax_t already existed; AFAICT, intmax_t was introduced in C99. I also need to parse and/or print uid_d, gid_t, and pid_t, so I guess that in the future I’ll have to implement my own integer parser and printer, like user “orlp” suggested elsethread.

Until then I guess I’ll have to do something like

  #if sizeof(pid_t) > sizeof(uintmax_t)
  #error Nope
  #endif

…repeated for all the types I need to parse and/or print.

You can't use sizeof in #if, but you can use C11 static_assert instead.

Also this doesn't just apply "in the future" -- integer types larger than intmax_t already exist, C23 just updates the standard to match reality.

You can do this:

  #if SIZEOF_PID_T > SIZEOF_UINTMAX_T
  ...
  #endif

Where you detect these sizes from the toolchain and deposit them as #define constants in some "config.h" header.

There are ways to detect sizes by compiling a source file to an object file, and then analyzing the object file (no execution), so things work under cross-compiling.

I've used a number of tricks over the years, and settled on this one:

https://www.kylheku.com/cgit/txr/tree/configure?h=txr-278#n1...

The basic idea is that we can take a value like sizeof(long), which is a constant, and using nothing but constant arithmetic, we can convert it to the characters " 8". These characters can be placed into a character array where they are delimited by some prefix that we can look for in the compiled object file with some common utilities.

This is quite durable in the sense that it can be reasonably expected to work through a wide variety of object formats.

In that test program, I have a structure with some character arrays. The larger character arrays hold the informative prefixes. The two-byte arrays hold decimal digits for sizes. Two digits go up to 99 bytes so we are safe for a number of years.

The DEC macro calculates the ASCII decimal value of its argument, expanding to a two-byte initializer for a character array:

  #define D(N, Z) ((N) ? (N) + '0' : Z)
  #define UD(S) D((S) / 10, ' ')
  #define LD(S) D((S) % 10, '0')
  #define DEC(S) { UD(S), LD(S) }

E.g.

  DEC(42) -> /* the equivalent of */ { '4', '2' }

But actually

  DEC(42) -> { UD(42), LD(42) }
          -> { D((42) / 10, ' '), D((42) % 10, '0') }
          -> { ((42) / 10) ? ((42 / 10)) + '0' : ' ',
               ((42) % 10) ? ((42 % 10)) + '0' : '0' }

The space instead of a leading zero is so that if we pull this into shell arithmetic, it isn't confused for an octal number.

That problem could be a really good application for a _Generic macro, which is another newish language feature in C.

If people use intmax_t and then they expect the ABI to stay stable, than they are at fault, not libc, LLVM or GCC for changing it when the actual "max int" changes.

I feel the fault instead lies in the operating system for wanting to run old binaries so much that the operating system pretends that the system architecture (and associated ABI) has not changed, even when there exists larger integers than what the previous architecture had set as intmax_t.

The problem is not old binaries, it is old code that assumes an upper bound to the size of intmax_t.

I’m not sure what you mean. I agree that the problem is not the binaries, but I believe the problem to be the operating system. The source code may not have assumed a specific upper size limit on intmax_t, but the binaries were compiled in good faith to a specific machine architecture, which had a specific width on intmax_t, namely the largest integer type which that architecture could support. If then new CPU models come out with larger integer types, but the operating systems still runs that same binary straight up (without any emulation layer), this exposes the old binary to larger integer types, breaking the binary. The binary is not at fault, it’s the operating system which committed the sin of running the binary on what is effectively a different architecture without any considerations on what architecture the binary was actually compiled for.

The OS could do this for old binaries but no for old source. A lot of FOSS code has baked in assumptions about the max size of intmax_t and the compiler and the OS cannot do much about broken logic.

> A lot of FOSS code has baked in assumptions about the max size of intmax_t

Well, that code is not, and was not ever, standards compliant.

And it turns out that doesn't matter much in terms of adoption and popularity.

After all, if predictable behavior in the future across all possible configurations were a high priority requirement for the tools people use to make computers do things, neither C nor C++ would have gotten off the ground. Instead, people don't memorize the entire standard. They write code that works on their machine, publish it, fix the bugs when people say it doesn't work on their machine, and maybe converge towards something that is standards compliant but, more likely, they converge towards something that works on the standard implementation on 90% of the operating systems available.

> A lot of FOSS code has baked in assumptions about the max size of intmax_t

Would you be able to name three examples?

I realize I used FOSS improperly, I had no reason to single it out; my comment reads like a critique of open source which was not my intention.

My intention was to make a good case for old code being compiled today by third parties.

But to my understanding intmax_t was introduced in C99, before widespread dominance of 64 bits platforms. So it would not be surprising that there exists code with lower assumptions about its size.

But actually I made also another mistake, the major problem had little to do with faulty logic and more to do with dynamic linking [0]

I will see myself out today.

[0] https://thephd.dev/intmax_t-hell-c++-c

But all major C compilers promise that they won't make ABI changes. So I'd rather say that standardizing intmax_t was a mistake.

If you don't standardize intmax_t, programs will invent the concept for themselves through some combination of preprocessing directives and shell scripts that probe the toolchain.

If you want to know "what is the widest signed integer type available", you will get some kind of answer one way or another.

Or even worse, if they don't know their build tools, some kind of union based monstrosity...

GCC changed long to 64-bits and x86 codebases had to accept that. The base integer types are designed to be variable sized and you code accordingly based on the minimum size guarantee. Assuming types are frozen in time is how you get things like LLP64.

Long is 32 bits on x86, 64 bits on x86_64. x86 code bases broke because they made non-portable assumptions.

No. It's 32-bits minimum. It can be anything larger. That's what the language says. Assuming otherwise is nonportable.

Why are you saying “no” and then agreeing with what I wrote? Am I misunderstanding something?

Because "32 bits or greater" != "exactly 32 bits"

Right. One is what the C standard says, and the other is what the ABI standard says. Maybe I’m missing something here.

Obviously, when someone says “type X is size Y on architecture Z”, they’re not talking about the C standard, they’re talking about some particular ABI.

> the other is what the ABI standard says

There are multiple ABIs for x86_64. I think the major compilers use 32 bits for int and 64 for long, but there could very well be a compiler that used different sizes, with a different ABI.

Yep, the x32 ABI on Linux is based on the amd64 instruction set, registers, etc. but uses 32-bit long and pointers (which in my experience is a nice performance boost, I measured up to 20% in some cases)

Because it isn't "64-bits on x64". It's 32-bit minimum on all platforms. Assuming larger sizes than the minimum is how brittle code gets written. A compiler vendor can set the sizes to whatever they want regardless of the target architecture and claim conformance if they meet the minimum.

Talking about the SysV ABI. There are multiple standards involved, not just the language standard itself.

> A compiler vendor can set the sizes to whatever they want regardless of the target architecture and claim conformance if they meet the minimum.

Compilers claim conformance with other standards besides just the C standard.

You are confusing ABIs with what the C standards mandate. A compiler emitting code for a particular architecture need to conform to that architecture's ABI. Or else it won't be able to interface with syscalls or other libraries. This goes for other things too like calling conventions and setup of stack frames. As klodolph states, long is 64 bit on x86_64, (32 on x86_64-win) and 32 on x86.

No, gcc added support for ABIs that defined long as 8 bytes. They didn't change the size of long for any ABI.

Right but the reason we don’t have ILP64 is mostly legacy code.

The mistake was not properly versioning the ABI

intmax_t can rationally be regarded as being outside of the ABI concept. It should give you the maximum integer available, not the maximum integer that your compiler had 15 years ago for the sake of compatibility.

Anyone who uses intmax_t in a durable API (public function argument, member of public structure) is just a goof.

Lotta goofs write a lotta code, though. I'm not sure that I disagree with you, but this feels like the sort of thing that could really hurt to break.

> It should give you the maximum integer available

How would you compile that to machine code which would, when run, still give you the maximum integer on next year’s CPU with all-new super-duper wide integers?

It is more reasonable to think of a binary to be compiled to a specific architecture triplet, and if the CPU changes, the architecture must change, and therefore the ABI, and if the OS wants to run a binary from an older architecture, some emulation layer is needed.

Of course, this would be a lot of work for the operating system people if they want to run binaries from lots of what are now different architectures, and it was apparently easier to just force the C standard to abandon the entire concept of intmax_t being the largest.

In C, you get what you pay for. When the platform changes, it's not unreasonable to recompile in order to take advantage of new hardware features. The OS is certainly different, even if it's still called "Whatever OS", it had to be recompiled in order to support the new hardware, unless, of course, it doesn't. We're all lazy when it comes to our own work, whether it's supporting new hardware in a compiler or OS or super web service on some new cloud platform or a large system vendor.

intmax_t by intent is the maximum integer available on a given platform. Abandoning that idea is a disservice to compiler writers, programmers -- especially scientific programmers -- everywhere.

Oh, I agree completely. Distributions should have bit the bullet and announced a new architecture when the new CPUs with new integer types were released.

> if the CPU changes, the architecture must change, and therefore the ABI

That would add a lot of friction to changing CPUs, and making distributing software more complicated.

There are a lot of goofs out there running goofy binaries for which they have lost the goofy source code. These people are unfortunately also the committee’s problem.

To the first problem you can use a classic mod 10 divide by 10 loop to convert into a string digit by digit.

Similarly for the second problem you can read digit by digit in reverse order and create the number using x = 10x + d while detecting overflow.

I'm not saying that this is a great thing to need to do, but the mentioned problems aren't particularly fundamental and the solutions are CS101 stuff.

You may be right that it’s the best solution. It’s horrible to contemplate, but it might be so.

Oh I agree, it's bad.

The best solution is to say that uid_t et al are probably no larger than uint64_t, and introducing a static_assert that makes sure that the compiler will fail if it's not.

Sure, someone might come up with an architecture where that's not true. But even in an intmax_t-is-really-the-largest-integer-type world [which hasn't been true for the major C compilers for the past decade or so!], I can almost assuredly come up with some compilation modes that still conform with whichever C version you use that still causes your code to fail horribly. I don't think there's any architecture where uid_t et al would exceed uint64_t, and until such an architecture comes to exist, it's quite frankly not worth caring about.

> "To the first problem you can use a classic mod 10 divide by 10 loop to convert into a string digit by digit."

But the "physical" size of the number as represented in memory can change across platforms, no?

Maybe C just needs to provide a way to print/format any built in type without specifying it. They got stdc_popcount to work with any integer type (looks like function overloading in C through macros), so can same be done for printing integers?

Don’t forget the other side of the coin; we also need to read a string into any integer type (while also detecting overflows).

> But now, when any type might be larger than an uintmax_t? What do you do?

Complain to POSIX so they add printf macro defines for all their implementation-sized integer types by the year 2100.

For printf(), that would work. For sscanf(), POSIX could add macros there, too, but sscanf() does not detect integer overflow. That’s why the only previous solution was strtoimax() which does detect overflow.

scanf not detecting integer overflow is arguably another problem. It's usually the easiest way to parse numbers out of simple strings, but the lack of overflow detection is a problem. (Actually we use sscanf for this and ignore the overflow problem, as long as it doesn't cause any security or correctness concern.) IIRC glibc actually does overflow detection, so there's not anything inherently problematic with adding it.

> "How do you portably printf() an integer type that you don’t (and can’t) know the size of"

"Portably" across what? You need to narrow down the problem space and better define it.

If it's portably across a myriad of processor architectures, hardware profiles and operating systems then you would need to take the intermediate representation route (like Java). There's very little else that can be done in the face of infinite, non-compatible ABIs at every layer.

I mean portable over all platforms which claims to support standard C.

> you would need to take the intermediate representation route (like Java). There's very little else that can be done in the face of infinite, non-compatible ABIs at every layer.

No, I don’t think so. There’s an obvious solution, dictated by the architectual choices made by C (i.e. every architecture which introduces a new larger integer type, by necessity redefines intmax_t, and creates a new ABI), but the community (for reasons of their own) did not want to do it that way, and instead broke the promise of intmax_t always being the largest type.