Some Were Meant for C (2017) [pdf]
cs.kent.ac.uk
cs.kent.ac.uk
Anyway, the author makes a few good points that are important but often overlooked, but I don't think I'd agree with him on the safety aspect in section 6. Making a safe C implementation doesn't really appear possible. If you allow casting integers to pointers, how do you implement these saftey checks he's talking about?
@aap_, I asked a similar question some time ago and got some answers, you can check the thread here: https://news.ycombinator.com/item?id=19302581
The direct answer I got was:
> I'm guessing because an off-by-one or an extra skip might mean you miss the end of the string and go off into la-la land feeding whatever garbage happens to be in memory to your parser? That would mostly be a C issue (as it has no string abstraction at all).
Well, if someone is not familiar, why do they read a subthread on the matter?
Shouldn't they better start with a tutorial on C/C strings?
Even if people on this thread gave arguments, how would they (not familiar with C and C strings) would evaluate them? They could be totally bogus.
At least in C it's quite obvious that strings are not trivial if you want both an intuitive way to work with strings, and high performance. The C++ std::string type is neither intuitive to work with, nor does it allow to write high-performance code.
For string processing it's really better to use another language with different trade-offs.
As for string processing in general, I do agree that other languages are better suited.
I don't think is a huge problem per se, though, you can just use a string library.
See the confusion here for an example: https://stackoverflow.com/questions/308695/how-do-i-concaten...
snprintf looks like it is the easiest way out.
How does utf8 make that difficult to answer?
When was the last time you iterated over a string of unicode points and said you know what would be handy right now? If these code points were split up into arbitrary and unusable bytes of memory.
If what you say if you want mutable strings, many languages have those and you don't need anything like "NULL" to have them (and you can use a bytearray of the string in Python, though Unicode complicates this).
>The NUL character is in my experience not so terrible, you typically have null pointers at the end of a linked list or whatever as well and nobody complains about that
That's not the same thing at all. The linked list is comprised of structs with next fields, that can be null or point to something. Your program can handle either just fine, as both are valid cases (a linked list expects to find the NULL guard at the end but also expects a non-NULL next pointer if the node is not the last one, so will handle both).
OTOH, if an incoming string doesn't have a NUL byte your program will crash/corrupt memory/worse. On top of that, you need to remember it to add it/make space/for most string manipulations. Strings are not expected NOT to end with NUL, and when they don't there's no way you can mitigate it, except to set arbitrary limits to how many characters you consider.
how are linked lists different? if the last node contains garbage for its next pointer, the outcome will be exactly the same. it's a bit more rare to encounter an "unterminated" linked list, but I've seen it happen plenty of times deserializing a linked list from disk or if the programmer just forgot to initialize the pointer. c strings basically are linked lists with an implicit next pointer.
It has string literals, so yes it does.
> The C++ std::string type is neither intuitive to work with
Many would disagree.
> nor does it allow to write high-performance code
True, but only due to backwards compatibility with C - std::string operations have to add a null terminator for no other good reason.
Many would also agree, that means nothing. Personally i dislike C++'s strings... and the rest of STL, which i view as one of the worst standard library APIs in wide use.
That's about the least reason why std::string is inefficient.
2 - no enforcement that a null terminator actually exists in the string
3 - C brags about performance and is probably the slowest language to compute string length
4 - manipulating strings requires very carefull handling of buffers, usually forcing everyone to use the heap as easier way out
All the listed weaknesses also have benefits. For instance it is easy to get a substring without need to copy.
But yes, many bugs in C software originate from spring buffer overflows.
That substring won't remain valid without copying it.
Unless it's not modified? And unless, when modified, that shouldn't be its new value?
C Strings (nul-terminated) are the right approach for static storage of small static strings (like strings literals in the source code) since they have low overhead, and "substrings" aren't second-class citizens.
For dynamically allocated strings that won't be modified after creation, the right approach is using a large memory chunk that is shared between many such strings, plus two indices for offset / length (or just offset if it's text that can be terminated with a sentinel).
Having a short string of about 10 characters allocated as a dynamic object in its own allocation is wasteful. Slow to allocate and has about 2x to 3x overhead. This approach isn't good for applications that store a large amount of data.
> O(N) is not the same as O(1).
Don't call strlen() in situations where the strings are large and you need to know the length ahead of time, and running time is paramount. Instead, store the length.
I can tell you that in one of my programs, the difference between garbage collected strings and optimized ones (actually, strings converted into unique integer handles immediately) for one of my projects (SAT solver in Java), handling a few million variables, was something like a second until completion vs a couple of minutes before the garbage collector finally dies due to lack of oxygen, losing all data computed up to that point.
To make substring in some other languages, you need to store pointer to beginning of the substring and length of substring.
To make substring in C, you need to store pointer to beginning of the substring and put '\0' into original string.
Plus a pointer to the beginning, plus a reference counter as the user expects it to manage lifetime. In C this is the user's job. Where they know the life time is guaranteed they can optimize.
Those are not practical? Billions of dollars have been wasted on issues stemming from this...
Or is the insinuation that we are hand-wavy about it, and you doubt the existence and scale of the problem? It's a well researched, well established problem, known for almost half a century.
https://en.wikipedia.org/wiki/C_standard_library#Buffer_over...
https://security.web.cern.ch/security/recommendations/en/cod...
http://www.informit.com/articles/article.aspx?p=430402&seqNu...
https://randomascii.wordpress.com/2013/04/03/stop-using-strn...
https://courses.cs.washington.edu/courses/cse341/04wi/lectur... (null termination)
https://www.geeksforgeeks.org/why-strcpy-and-strncpy-are-not...
https://www.owasp.org/index.php/Reviewing_Code_for_Buffer_Ov...:
And let's not even get into format string issues...
If you are going to mention Assembly as possibility, check ClearPath where there is no Assembly, NEWP has full control over the hardware stack.
Obviously, since many other languages that do so are implemented in C.
I have a feeling that of all axes of performace C cares the most about memory overhead. Then the obvious idea is to have it at exatly one byte per "simple" string, and you get to pick the class of programs that can't get away with that default string type:
• One-byte terminator: complicated text-handling application with a lot of (longer than a couple of pointers on average) string slices.
• One-byte length: anything that needs strings longer than 255 chars.
And then of these two solutions you pick the obviously more general one. What could possibly go wrong?
Character arrays with open length, bound checked.
Naturally it requires better compiler support than C authors were willing to implement.
Which is still the case with many things in Go, a language of close origin to C (though this time not about strings).
I guess security is not relevant as infrastructure work.
> I guess security is not relevant as infrastructure work.
...
And our data through some JS monstrosity.
So?
Strange argument. Javascript is widespread, COBOL was widespread, Windows is widespread, X86 is widespread. Widespread doesn't mean good. UNIX was a disaster, and the whole family of UNIX-like OS spent decades just for mitigate its errors and faults.
And nowadays C code is considered legacy, with C#, Rust and constrained C++ as the road to the future.
Length not known, so prone to overflows at anytime, atrocious standard library, ... (and let's not even go into the Unicode situation).
From the first chapter, the Go code starts using strings as a short-cut to represent tokens . In other languages this is trivial because strings are very easy to create, resize, change, etc. Using C, this became an issue though, as using strings became a roadblock where I started having to implement different solutions rather than focusing my attention on the contents of the book.
You will find my nick on those archives at the C++/Pascal trenches side.
Mainframes were written in a mix of Assembly and safe system languages (IBM i, z and Unysis ClearPath are the survivors).
8 and 16 bit home micros OSes were largely written in Assembly, and all system languages were in equal footing as platform guests.
That's not what I remembered. Back in 1989, I bought the very popular Microsoft C 5.1 compiler (in shrinkwrapped box) from the Egghead retail store. Egghead was basically a smaller version of CompUSA.
Also, non-UNIX publications like PC Magazine and BYTE had ads for Borland Turbo C. Watcom C was also a competitor.
In the late 1980s and early 1990s, if one wanted to write COTS software for DOS on IBM PC compatibles, it was either assembly or C Language. Other languages like dBASE/Clipper/4GL were for LOB apps instead of COTS. In 1985, both Lotus 123 v3 and MS Excel were written using C.
It was more like the early 1980s before the personal computer revolution when the C Language was mostly tied to UNIX.
Back in the Iberian Penisula, Turbo Basic, Turbo Pascal, TASM, NASM reigned on the PC for COTS.
Amiga demoscene was all about AMOS and 68000 Assembly.
I code since 1986, touched my very first C compiler in 1992.
By then I already had used Timex 2068 Basic, 48+ Basic, CP/M on +3 A, Z80, Turbo Basic, Forth, Turbo Pascal 3 - 6, Dbase III Plus, 8086 with TASM.
Thankfully also in 1992, I got hold of Turbo C++ 1.0, and I was settled on my opinion about C.
And yes, apart from the myriads of usual C++ leaky abstraction bugs that "just" cause wrong results, this includes segfaults.
I guess that doesn't only apply to C++, but also C. :)
But that doesn't mean you can't have fun with it, or write good computer programs.
It just means that user inputted text is always going to be a pain.
If you want support for unicode etc, there are libraries.
I'm not sure what you mean by compatibility issues in this context?
This tends to be important for strings you're going to show to the user (or input you gather from them), maybe most of the C in the world isn't aimed at interfaces like that...
For other encodings, well you need to decode explicitly (or just go for wide chars). Shouldn't come as a surprise, and not a big deal either.
I'm not aware of any. I've never encountered a language that is more frustrating than C to manipulate strings in.
> Are you saying other languages don't have security exploits?
No, I'm saying they have fewer. Far fewer.
> And in 35 years of writing in C, I have only had a very few segfaults while writing new code but never in production
Good for you. In 25 years of writing C and working with dozens of other programmers writing C in that time, I've lost count of how many segfaults and memory corruption errors I've had to debug.
I don't miss those days and never want to go back.
> unless your knowledge is only from reading reddit headlines.
It is not. See above.
I'm glad because C is an idea you can hold in your head. I can look at old C, I can look at new C, and it's all pretty sensible. (Excepting macro abuse, deliberate obfuscation, or gross incompetence.) It's hard to say that about most any other language.
That makes it the most natural way to call into the operating system, without any concerns that the chunk of memory returned by mmap has to be somehow managed by the language runtime.
Not only in embedded systems. It is a huge advantage when interfacing with the operating system from within a userland process.
Lack of a runtime means that a call to a kernel function like, say, `sched_setscheduler(2)` would not interfere with any such thread scheduling policies of the host runtime. Or that the runtime is grabbing file descriptors out of your control, making it hard to reason about resource usage.
> After publishing the Friendly C Proposal, I spent some time discussing its design with people, and eventually I came to the depressing conclusion that there’s no way to get a group of C experts — even if they are knowledgable, intelligent, and otherwise reasonable — to agree on the Friendly C dialect. There are just too many variations, each with its own set of performance tradeoffs, for consensus to be possible.
Because making C safer would introduce a lot more complexity in the runtime environment and instrumentation, and that is even hard to achieve correctly or has varying performance implications across all ISAs that C compilers are currently targeting.
Consider out of bounds indexing. To determine that an instruction is touching a memory region that is not, in abstract terms, a C array or a memory allocation by malloc and friends, now you need to insert long traps in memory, and even then, there is nothing stopping you from accessing array `b` through, for example, `a + 42`.
I don't understand the "undefined behavior" bandwagon.
1. We have a perfectly defined list of "undefined behaviors".
2. Said list also happens to be relatively small and scoped.
3. "Undefined behaviors" exist because the language can't make certain runtime guarantees which are largely dependent on compiler/os/platform/hardware-specific promises. If you have to, just roll in your own runtime checks. C won't force those on you...
Or evaluation order - `i = ++i;` could easily be defined in some way. But might prevent some niche optimisations by the compiler.
Of course by C's nature there are limits (C won't be able to detect use after free or similar without changing language notably) but there is room where UB could be reduced, if it was seen as neccisary.
> "Of course by C's nature there are limits"
You seem to acknowledge the fact that most of the undefined behaviors in C are essentially born out of compromise. Those compromises were driven by principles such as "keep the power in the hands of developers", "don't impose unnecessary restrictions", "keep the language clean", "avoid hidden runtime magic". The end results reflect that.
As I've mentioned in my previous comment, there's no "one size fits all", so the language makes it trivial for you to roll out your very own runtime magic (à-la Zig/Nim) which suits you best. Why is that a bad thing?
An easier approach would be to put a bound on what is permissible undefined behavior.
Sounds a bit like an oxymoron, doesn't it? After all, it is "undefined", right?
However, the C standard does exactly that!
Permissible undefined behavior ranges from ignoring the situation completely with unpredictable results, to behaving during translation or program execution in a documented manner characteristic of the environment (with or without the issuance of a diagnostic message), to terminating a translation or execution (with the issuance of a diagnostic message).
(section 3.4.3)
In fact, in the first version of the ANSI/ISO C standard, this was a binding part of the standard. In later versions, it was made non-binding, though the language is still in the standard.
Yes, the standard has language that says what permissible undefined behavior is, but you are free to ignore it and still call yourself compliant. Which is what just about everybody does nowadays.
Make it binding again and most of the mess disappears.
https://blog.metaobject.com/2018/07/a-one-word-change-to-c-s...
Setting "the compiler" = "the environment" and therefore anything the compiler does is part of the environment and thus legit seems at best the type of sleight of hand the compiler writers use to justify their actions.
When defining the behaviour of the compiler, "the environment" obviously cannot be "the compiler".
It's a lot of work.
> Is such a thing impossible? Have people done it?
It's possible; Ada has a really good take -- in that standard, there's a class of errors called "bounded errors" which on the surface look like "undefined behavior", but are a lot different in that they list out the possible results and thus preclude the "nose demons" problem of C -- see: http://www.ada-auth.org/standards/2xaarm/html/AA-1-1-5.html
It's much too easy to produce buffer overflows in C, be it due to the design of C strings or just because of the manual handling of dynamic memory.
The big difference was that those OSes versus UNIX costed real money.
Any idea why there was no popular safe array or safe string library for C? Maybe at that time there was no internet and everyone had to create his own abstractions?
Bell Labs was forbidden to sell their research, so they offered UNIX for a symbolic price to universities (vs what other OS used to cost), alongside source code tapes and a liberal license.
This gave birth to several startups that tried to create a business using UNIX instead of the alternatives, given the authors experience with UNIX at the university, e.g. Sun and SGI among others.
Later when the US goverment dictated AT&T split, AT&T was allowed to charge for UNIX and that is when they decided to go after BSD, because 10 years later, under such conditions, UNIX was slowing eating mainframes, given the success of Sun, SGI, Aix, ....
There are no safe array or safe string libraries, because they all fall appart under scrutiny, given C's approach to secure code.
I do not understand, maybe give an example.
About the history I am not sure that proves that UNIX and C advantage was only because of that, it could be a factor but there are merits to C and UNIX that if you want to disprove you can;t just do it by mentioning that history. I mean Windows API used C.
Because since those libraries are not built-in types, you always need to convert back to a ptr and length values at some point when interoperating with other C code.
And given the lack of bounds checking, you are back at square one.
> I mean Windows API used C.
Windows API used C, because by the time Windows came around UNIX was already well established in the enterprise.
C spread outside UNIX, because many of us were using C dialects, e.g. Small-C, on personal computers when bringing work home from job/university.
And that's exactly why C wins, in my opinion. Because that means interoperability.
Storing a pointer + length pair in a fixed layout struct is bad from a normalization standpoint. They are independent data. If you don't separate them you will end up with data redundancies as soon as you have parallel arrays. If you use dynamic vectors and in C++ or similar languages and you ask yourself which is the object that you should call .size() on, that's when you notice that it's morally wrong to bundle pointer + length.
Or more precisely, 68% of Linux kernel exploits according to Google's talk at Linux Kernel Summit 2018.
You can't rip them out and replace them with something better without massive code legacy issues. And then you still haven't dealt with other sources of buffer overflows in C.
The practical Pascal variants like Turbo Pascal and Quick Pascal, which compiled to native code, are essentially on par with C regarding the features. That includes access to absolute memory addresses, port I/O and the speed of the resulting code. I've done over a decade of Turbo Pascal coding, and going on three decades of C, and I agree they feel like essentially much the same thing.
https://www.embarcadero.com/products/delphi
It's just as capable as any other language. I don't know what your definition of "low level" is, but I suppose you mean inline assembly, manual memory management, and pointers.
Pascal has inline assembly: http://docwiki.embarcadero.com/RADStudio/Rio/en/Inline_Assem...
Pascal has manual memory management: http://docwiki.embarcadero.com/RADStudio/Rio/en/Memory_Manag...
Pascal has pointers: http://docwiki.embarcadero.com/RADStudio/Rio/en/Pointers_and...
Having a language that's simple, has a low runtime-overhead and can be implemented in a simple compiler is a quality that quite obviously is still valuable. C would have vanished like a lot of other languages that got out of fashion.
I don't think UNIX has much to do with it, otherwise C wouldn't have survived the 80's and 90's when other operating systems (Windows!) ruled supreme and UNIX operating systems were the underdog.
That place always belonged to mainframes and later UNIX, due to its "cheap" price versus mainframes.
Your story seems to be that C won because of UNIX, which won because of the server room. But the server room was a small chunk of landscape, and code was being written for the whole landscape, not just for the server room. And C won all over the place - the server room, the desktop, and the embedded space. It didn't win on the desktop because of influence from the server room.
> You might ask, “Why would someone write code in a grotesque language that exposes raw memory addresses? Why not use a modern language with garbage collection and functional programming and free massages after lunch?” Here’s the answer: Pointers are real. They’re what the hardware understands. Somebody has to deal with them. You can’t just place a LISP book on top of an x86 chip and hope that the hardware learns about lambda calculus by osmosis.
[0]https://www.usenix.org/system/files/1311_05-08_mickens.pdf
The author doesn't deal with the problem that the "direct access to memory without abstractions" style they love so very easily drifts into undefined behaviour. It also optimizes very poorly due to lack of aliasing information, unless you enable type-based alias analysis in which case accidental undefined behavior is catastrophic.
The "dynamically checked C" they propose as a solution to C's safety issues is no solution at all until you can get it to work in practice. A lot of really great people, including the researchers he cites, have tried really hard to make it work and have failed in practice. He doesn't seem to understand why they failed or have any insight into how to overcome the problems.
While some were meant for sea, in tug-boats ’Round the shore’s knee,
(Milling with the sand,
and always coming back to land),
For others, up above
Is all they care to think of,
Up there with the birds and clouds, and Words don’t follow.
—Tiny Ruins, from “Priest with Balloons”Anyone ever try to build "C the good parts"?
First they made "C the good parts" by gathering all the various C bits then in use to make something portable and called it C89.
Then they made "C the good parts" by removing all the parts that made C slow to write and called it Perl.
Then they made "C the good parts" by making C easier but still fast and called it Java.
In the meantime, we've gotten Objective-C and C++, which were attempts to make C better while preserving backwards compatibility. C++ has gone on to spawn its own legacy of "No actually, these are the good parts" with D, Clay, Rust, and various half-steps along the way that want some of the features of C++ but not all of the features of C++.
I don't think they'll stop, because as it turns out, people use tools for different reasons and it's very rare that a general purpose tool solves your very specific problem perfectly. I'm inherently skeptical of any "X the good parts" because the "good parts" are domain specific.
Actually I really don't get this endless "silver bullet" discussion. C has its place and for very good reasons. Also the author makes really good points about the integration aspects of C.
There are examples of OSes that don't work like that, despite targeting x86. For example, bare-metal Forth.
https://github.com/Microsoft/MSRC-Security-Research/blob/mas...
They are also very clear that C is done on Windows, and compatibility is only to the extent required by ISO C++ and a couple of key FOSS projects.
https://herbsutter.com/2012/05/03/reader-qa-what-about-vc-an...
UNIX is C's platform, there is hardly any reason to use it outside of non-UNIX OSes.
Plenty other languages offer system programming features, with better type safety and equal portability.
There is a continuum between the possibilities of a programming language and the possibilities of a configuration file. As soon as one says that a programming language is suitable for a particular purpose one has moved a few steps towards the configuration file end of this continuum.
This is, of course, a personal preference but I very much prefer to enjoy the power of programming language as opposed to the lack thereof of configuration files. Therefore, I like programming languages that attempt to be useful for any purpose. My favorite language is still C++ and if I were to switch to something else I would be inclined towards Rust.
C++ is too complicated. "Nobody" writes C++, but a subset of it. Select your subset.
Dynamic typing in Lisp (and the like) is really nice for quick prototyping where program correctness is not key. You are exploring what you want to achieve with the program, be it algorithm level or architecture. C++ is not well suited for that, since type checking and memory worries are slowing you down, too many details to drag along. Hence no "silver bullet".
Quick prototyping is not something that I do. Nor am I very much interested in it. At my place of work I have seen it done around me after which it was my task to turn the python prototype into C++. The most surprising thing there was how far the prototype turned out to be from what was actually needed, to the point where I very much question whether the prototyping exercise was useful at all. YMMV regarding this, of course.
That one doesn't feel right. More like "figure out how to nudge the C++ crowd into the general direction of Smalltalk without them noticing".
No way. C++ was always much more into the functional programming paradigm. (See the STL, for example.)
The connection between C++ and OOP is because OOP was the insane hype at the time when C++ was being invented. The OOP lipservice was mandatory in order to be taken seriously by the fashion-driven programming industry, but real C++ programmers always looked down on OOP and considered it a code smell and crutch.
Not really, the C++ OOP features were directly modeled after very similar features in Simula. C++ was specifically designed as a way of bringing these sorts of features to C, although it did include other improvements to the language as well. Templates and the STL as we know it were a relatively late addition to the language.
It's also worth mentioning that there's only a handful of things about OOP that could be genuinely considered "a code smell"; in fact you could restrict that concern to one feature, viz. implementation inheritance. Object-based programming which follows the "composition over inheritance" guideline can still broadly tap into the improved-modularity benefits that 'objects' are generally known for.
Always? The STL was a last-minute addition to the standard library before C++98. Just a few years before, template implementations in compilers were buggy. There's a reason Qt has its own containers: because it's that old.
> The connection between C++ and OOP is because OOP was the insane hype at the time when C++ was being invented
No, it was because Bjarne wanted features from Simula whilst still generating fast code.
> real C++ programmers always looked down on OOP and considered it a code smell and crutch.
Absolutely not. Again, look at Qt. Look at CERN's ROOT. Java looks a lot like it does because that's how C++ code was written at the time. Even in the early 2000s I was getting funny looks from people when I told them to default to putting variables on the stack.
C++ without a standard C++ library is not really C++.
> No, it was because Bjarne wanted features from Simula whilst still generating fast code.
Yeah, but Simula is somewhat its own thing, before the OOP madness.
> Again, look at Qt. Look at CERN's ROOT. Java looks a lot like it does because that's how C++ code was written at the time.
Only because C++ was the only thing available at the time, so people twisted it into 'OOP', despite the fact that C++ was very a poor fit for 'OOP'.
> C++ without a standard C++ library is not really C++.
You seem to be missing the point, which is that there was a time (two decades!) when the C++ standard library existed, but didn't include the STL.
> Only because C++ was the only thing available at the time, so people twisted it into 'OOP', despite the fact that C++ was very a poor fit for 'OOP'.
You are completely mistaken on your history here. C++ was intended to be "C with classes" from day one.
"Classes" is a low-level thing that you'd need for implementing many language features. Including things like 'abstract data types' of the ML kind.
Good C++ style has always viewed "OOP" as something highly suspect and hacky.
(This didn't apply to "classes" in the C++ vein, which are mostly about pre/post-conditions and RAII.)
That sums it up quite well.