C’s Biggest Mistake (2009)

In C you can declare pointers to arrays, the syntax is just somewhat strange. You can even declare it as a pointer to a variable sized array with c99, eg:

  void foo(size_t length, char (*x)[length]){
      size_t size = sizeof(*x);
      assert(size == length);
      printf("sizeof(*x): %zu\n", sizeof(*x));
  }

Yes. As I've long said, C does not have arrays. It has pointers and notations for initializing memory, but it does not have arrays.

There are actually two problems here. One is the absence of bounds checking. The other, which is related but technically orthogonal, is the hole in the type system: an array is not an object. It's been true since Unix v7 that you can pass or return a struct by value, but you can't pass an array by value unless you wrap it in a struct.

The type system also makes no distinction between a pointer to a single object and a pointer into an array of objects. I've worked on static analysis tools that try to find potential buffer overflows, and this turns out to be a surprisingly big problem. One has to do a global dataflow analysis just to discover which pointer variables could ever point at array elements.

Previously:

https://news.ycombinator.com/item?id=17585357

https://news.ycombinator.com/item?id=1014533

I’ve been writing C code for 30 years. What I like about it is code from back then is pretty much similar to what you’d right today so there’s limited need to chase the train. Sure it has security issues but if you don things ‘right’ it’s still untouchable, imo. Most other languages are heavy and slow compared to C. A lot are not even backwardly compatible (python) or suffer from being kitchen sinks (C++). My own opinion is that python is the only other ‘must know’ language as you can do RAD. For speed you resort to C. There are so many toy languages in between with some nice features (coroutines, native JSOn support, etc). But lots are just meh. All personal opinions.

This was probably the most confusing thing about C when I first started learning programming back in the day. When you call a function you pass the value, except in arrays where it gets converted as a pointer. It was explained back then to me that the reason is because copying the whole array was not efficient so it was better to pass the reference.

Unfortunately working with dynamically-allocated buffers is still a thing; adding array syntax just favors one form of size specification without solving the other case.

Although C is usable on many types of hardware, interesting things could be done, e.g. on desktop OSes if certain hardware-specific extensions were made.

One of the things I wish we could do with our now-absurdly-large pointers (64-bit) is to reserve a handful of bits for other information such as the size. Sure it means we can’t store anything at location 2^64-1 but it wasn’t that long ago we only had 32-bit pointers and the 33rd bit is twice as many addresses all by itself so I think we can lose a few.

For example, if all allocations were rounded up to buckets of a certain size, the precise byte count would not need to be encoded in the pointer (just the number of buckets, requiring fewer bits). There could be a couple bits to give pointers a type for other interesting scenarios, e.g. perhaps a pointer identified as an “immediate value” that isn’t actually allocated at all, and it is “dereferenced” by treating its “address” as the “stored” value. There could even be a couple of bits to track use of common allocators (it would be so nice to simply know that a pointer was allocated by "malloc" vs. "new" for example).

In high-level languages, then, the syntax change would be not to identify arrays specifically but pointers with encodings that are “complete” (e.g. "char const complete*" or something), covering both stack arrays and dynamic buffers.

Something I see as wrong with C outside of the context of the Linux kernel is mostly something wrong with us the developers. We are far too content to live in filth.

In addition to unsafe/irregular buffer handling, I also constantly see poor data structure choice, presumably due to a lack of default choice of library. It is very common to see code scanning linked lists when they should be doing map look ups. (And often even the linked list operations are ad-hoc and repeated for every type of struct with an embedded next/prev pointer.) Everyone always defaults to linked listing it up because they never have to pay the up-front cost of finding a library or investing in re-inventing the wheel. I think this is also why you see so much sketchy buffer code - no one has bothered investing in safer buffer abstractions.

Perhaps some of this is caused by the difficulty of taking on dependencies in a portable way. (CMake/Autotools can make this better, but it is a far cry from NPM.)

Niklaus Wirth would concur - that was similar to his argument for pascal - strings contain a size.

C's pointers wouldn't be an issue if the world had used the Intel iAPX 432 processor instead of the 8086. The iAPX 432 included bounds checking for every array in hardware (among many other features), so it was impossible to make an out-of-bounds access.

Unfortunately the iPAX 432 was delayed, so Intel introduced the 8086 as a stopgap processor and computers have been using the x86 architecture ever since. It's interesting to think that if history had gone a bit differently, whole classes of security problems would not exist.

https://en.wikipedia.org/wiki/Intel_iAPX_432

I don't see a mistake here, certainly not a "biggest mistake".

This is C, not C++. Keep it simple.

Here, the idea is that there is no special type for "pointer+size" (what the author proposes as an array). Ok, let's add one and see the implications.

- How do I get the size, the number of elements? A "sizeof" like operator?

- Can I resize the array? If yes, how? If no, why?

- What happens if I overflow? Undefined behavior?

- A memcpy-like would be an obvious function to implement, what happens if sizes differ?

- What is the relationship between static arrays (ex: int a[5]) and "pointer+size" arrays? Are these completely different types? Is there an implicit cast between the two?

- About casting, how can I go from a separate pointer and size to an array and vice versa? If it is possible at all.

- What if I do a bit of pointer magic to access the internal representation of the array? Probably undefined behavior.

It is much more complex than "just add array[..]", I expect more tradeoffs, more undefined behaviors (C wouldn't be C without them). Adding complexity to the language can actually make things worse.

As for zero-terminated strings, they have advantages and drawbacks. They are preferred by the C library, but you can do pointer+size if you want by using mem* instead of str* , or %.*s instead of %s in printf (not sure about this one).

I agree that missing bounds checks are the biggest problem, with the unhealthy attitude of dismissing the Annex K for political reasons. My Safe C library has only adoption with the big players.

But even weirder is the total lack of a proper string library. Nobody but Microsoft uses whar, and they are the only ones with the proper whar_t size. Everybody else was wrong with size 4. But nowadays it should be clear that only u8 is the only way forward, C++ even adopted now char8_t for it. But they all still ignore the unicode problems with an overly simplistic, glorified null-terminated memory buffer library. These are not strings anymore nowadays. Strings have multiple representations of characters in unicode, strings need the unicode version to be exposed which changes every year. They need a proper fold case and norm API, otherwise you cannot compare them, so you cannot search for strings. Grep would be happy to find unicode strings, but it still cannot. coreutils still cannot do unicode in 2020.

Also the complete lack of security, esp with names, ie identifiers. Such as pathnames. Most filesystems just ignore security, spoofing, bidi changes, mixed scripts as if this problem does not exist at all. Strings are not normalized, not properly fold cased.

The _l locale mess, it still relies on global runtime state, which is not compile-time optimizable, in opposition to _l or simply just a new u8 API. Not reentrant. Not compile-time optimizable. It's a huge mess.

gcc cannot do compile-time constexprs checks, only clang can, leading to up to 200x faster libc code. gcc cannot do user-defined warnings of errors.

glibc, FreeBSD libc, musl, none of it fixes anything.

It's probably worth remembering the history of C - it didn't appear fully formed, lots of stuff evolved as people used it - for example in V6 Unix += used to be =+.

In particular you used to use structures in a weird way, essentially field names within structures lived in their own global name space, any pointer could be used with any structure - there weren't unions yet so this was used for good effect in Unix kernel drivers (there was a standard buffer queue header you could add your own stuff at the end of.

I think this was kind of descended from the BCPL/Bliss world view where explicit pointers were a relatively new thing in languages and their typing was pretty simple (there was a limit to the number of indirections allowed) - fully orthogonal typing systems were only just becoming a thing then.

Also I suspect that the idea that a[i] was the same as a+i was an idea with legs, this is still legal C:

  char *x()
  {
        int i; char *p;
        return &i[p];
  }

Stupid question, but how do I access the size of an array using this fancy new declaration if it were to be added? It doesn't seem like any sugar is there to provide "range based for loops."

Wait I would just use `sizeof` but then I'm still doing pointer math then?

The C standard doesn’t forbid fat pointers. You could have a compiler that implements fat pointers (and crashes on out of bounds access attempts) without violating the standards in any way, since our of bounds accesses are undefined behavior.

To me, i think C is a powerful language that is weakened by 2 things:

1- Trying to find the proper style and methods to write few lines of code. The reason for this is because C is an old language that kept changing. Thus, you can read a book, yet find someone to tell you “you shouldn’t do it that way”.

2- compilers made C into different flavors. Microsoft C compiler provides scanf_s with the old scanf being deprecated. In the other hand, gcc has different approaches without the scanf_s that Microsoft has. This can be so annoying to use.

The proposed change has another important aspect to it: it would help standardize a way of passing arrays across language boundaries. Currently when using FFI, you often have to decompose fat pointers into pointer and length before calling a foreign function, or compose a fat pointer from a pointer and length in extern functions. This could mean languages like rust, d, etc. Could just pass arrays directly.

I for one would love to see this proposal become a reality.

Personally I like it the way it is. If you want to copy an array when making a function call you can define a struct with a array in it, and pas the structure.

If C did pass array lengths it still wouldn't matter since C doesn't (and in my opinion shouldn't) check for overflows.

That is "modern" C++ is frequently being used as "C with classes and basic safety".

I wonder if a better C can be made by just stripping-down the bloated C++ and introducing the "unsafe" keyword for dangerous features like directly using arrays, etc.

    typedef struct fat_ptr_t {
        size_t size;
        void * start;
    } fat_ptr_t;

    extern void foo(fat_ptr_t a);

if it's a good idea to use a fat pointer, why do we need new syntax to sell it? What am I missing here?

It’s not a “mistake”. This article is complaining about a misinterpretation of C’s functionality. Arrays are not “real” data structures in C: there’s no such thing. The array-ish syntax that’s available is just a some syntactic sugar on top of pointers. You could say that having the sugar at all is a mistake. Or that C is incomplete without first-class array types. This is a cute hack, but at this point (far more than 10 years ago) it’s probably better to move on to Rust if you don’t like this aspect of C, rather than proposing to hack the language.

You can if you make them fixed-size arrays inside a struct. I agree with the thesis of the article.

I remember my, "Modern C as an Oxymoron" joke went over pretty badly. Buuuuuuuuut...

The "fat pointer" is the std::array in C++.

Have you ever proposed a paper for this to WG 14?

>"What mistake has caused more grief, more bugs, more workarounds, more endless hours consumed, etc., than any other? Many people would say null pointers. I don’t agree.

Conflating pointers with arrays.

I don’t mean them using the same syntax, or the implicit conversion of arrays to pointers. I mean the inability to pass an array to a function as an array, even if it is declared to be an array. C will silently convert the array to be a pointer, and will rewrite the function declaration so it is semantically a pointer:

[...]

This seemingly innocuous convenience feature is the root of endless evil. It means that once arrays leave the scope in which they are defined, they become pointers, and lose the information which gives the extent of the array — the array dimension. What are the consequences of losing this information?

An alternative must be used.

For strings, it’s the whole reason for the 0 terminator.

For other arrays, it is inferred programmatically from the context. Naturally, every situation is different, and so an endless array (!) of bugs ensues.

The trainwreck just unfolds in slow motion from there.

The galaxy of C string functions, from the unsafe strcpy() to sprintf() onwards, is a direct result. There are various attempts at fixing this, such as the Safe C Library. Then there are all the buffer overflows, because functions handed a pointer have no idea what the limits are, and no array bounds checking is possible."

PDS: The root of all of this -- is that C, being a low-level, close-to-the-hardware, designed in the 1970's programming language (some in academia pejoratively call it a "glorified assembler"), was not designed with a proper string storage class as we know them in programming languages today; instead, arrays of characters were substituted for this purpose, and those arrays were not implemented containing total size (length) and dimensionality information.

Basically an array in C -- is a set of contiguous memory, which has a starting address (the pointer passed), and a stated element size that the compiler knows about, but not the length (aka, total size, element count, etc.) of that array, nor its dimensionality.

Observation: C's arrays need length information signalled in an out-of-band fashion (that is, this information cannot exist as a zero (0) -- somewhere in the array).

The irony of all of this is that C was invented at AT&T, and AT&T for the longest time had difficulty with phreakers exploiting 2600hz signals to gain access to its long distance trunk lines, from which they could call to anywhere in AT&T's system for free.

But, that's what the engineering error of in-band signaling generates...

C, by using arrays to implement strings, and letting the zero terminator (information about string length) exist in the memory space of the string, made exactly the same engineering mistake -- just in software -- and that is the mistake of in-band signaling.

Now, that being said, hindsight is 2020, and it couldn't be expected that Dennis Ritchie, who invented C in the 1970's would have foreseen the consequences of that engineering "mistake" (AKA, "act which generated quite the education for a future populace". <g>).

Such is the price of being an innovator.

On the one hand, he advanced computer technology greatly -- far beyond the technology advancements created by most of his contemporaries of his day...

On the other, that advancement gave us this highly educational engineering "mistake" -- that we can all learn from!

Such is the price of being an innovator -- and pressing the "bleeding edge" of what is possible...

Humanity could not be advanced without such innovators, and the occasional future flaws (and the wisdom that comes from examining them in hindsight!) that their innovations generate...

> Conflating pointers with arrays.

AND Strings.

FTFY.