Moving the Linux Kernel to Modern C
lwn.net
lwn.net
> You are not "introducing" a new macro for this, you are modifying the existing one such that all users of it now have the select_nospec() call in it.
> Is that intentional? This is going to hit a _lot_ of existing entries that probably do not need it at all.
> Why not just create list_for_each_entry_nospec()?
Let's ignore whether this patch is needed or works for now - I don't feel competent to comment on that. But this is such a bad suggestion. Instead of fixing the default to be safe, and possibly having a _unsafe_but_fast() variant for the places where it makes sense they want to keep the broken version and require user to explicitly opt in into safety.
Same as the infamous PHP mysql_real_escape_string (don't make the mistake of using mysql_escape_string!) [2][3] or whole host of C stdlib footguns like strcpy/strncpy [4].
The default, easy, obvious option should be safe. The unsafe but faster option should be hard to use by accident and obviously marked as such from the name.
[1] https://lwn.net/ml/linux-kernel/Yg6iCS0XZB6EtMP7@kroah.com/ [2] https://www.php.net/manual/en/function.mysql-escape-string.p... [3] https://www.php.net/manual/en/function.mysql-real-escape-str... [4] https://en.cppreference.com/w/cpp/header/cstring
TBF that is really a straight bridge to the mysql C API, which is why it’s also in mysqli.
MySQL actually has a third one called mysql_real_escape_string_quote, because if the sql mode NO_BACKSLASH_ESCAPES is set the escaping function needs to know what context it’s used in, and thus what quote to double up, and mysql_real_escape_string will fail.
The unsafe version can be deprecated and eventually removed, but that is down the road.
> tools for mass renaming that try to guarantee that there are no regressions
beyond sed or whatever a fancy IDE already does? What kind of voodoo magic are we talking about here? Did Google solve the halting problem and I just haven't noticed yet...
As for the actual renaming, yes, definitely beyond what sed does but probably around what the fanciest IDE does. Imagine a big global symbol dependency graph produced by the entire build toolchain all at once and cached, I think?
Also, this isn't the halting problem unless your codebase allows for fully dynamic invocation :)
Given that compilers can be smart-enough to detect "non-dynamic" function-pointer invocation (e.g. when an execution-trace proves that a function-pointer parameter always points to the same function address), it's not safe to say that "all" function pointer invocations are dynamic.
Another case to consider is when one implements (Smalltalk-style) OOP with message-passing: in many cases it's possible to build that without needing to use function-pointers at all.
So you can exclude many things that would in-principle be possible.
I'm not 100% sure, but i remember that i could have have both a two and three function argument be called with the same function pointer, its probably UB however i think that as long as any dereferenced functions adhered to the standard argument behavior of the platform specific calling convention it worked, and i dont think gcc, clang or msvc complained, but my memory might be wrong.
You want there to be two states: State A where the thing was called old_name and State B where the thing is called new_name. State AB where some code thinks it is called old_name but other code thinks it is called new_name is broken, and must not exist.
† This might seem outrageous to modern programmers, but CVS thinks in terms of files, so from its point of view it's fine if out of sixty files you tried to commit, 48 of them succeeded and 12 failed. Good luck fixing the resulting mess.
SVN, baz/bzr and friends are what you get if you add networking before atomic commits. Git is what you get if you start with the proper data model and only then add networking.
as you might imagine, it’s probably a pretty involved process to touch thousands of lines of code in a complicated system.
Bors works for git. Google has similar tooling for their system. (I think it's called the 'train'.)
struct foo *iterator;
list_for_each_entry(iterator, &foo_list, list) {
do_something_with(iterator);
}
list_for_each_entry_v2(iterator, &foo_list, list) {
do_something_with(iterator);
}
Mind you, there are not a ton of people around who have the time to learn how to write custom clang-tidy rules. It might be done for 15,000 changes that potentially fix kernel vulnerabilities, though.
http://bbannier.github.io/blog/2015/05/02/Writing-a-basic-cl...
Yes, it involves forking. Forking isn’t that big a deal. We even had checks that were specific to internal libraries.
Really… I have worked at two companies that had extended clang-tidy to add custom checks for their code base. The whole point of clang-tidy is to automate code fixes in other parts of your code base. When done well, use of clang-tide pays down technical debt faster rather than adding to it.
You can brows the clang-tidy source code yourself. There are existing checks specific to the Linux kernel in there already. Well, there’s one check. But if you use clang-tidy, you’ll discover that automatic refactoring of extremely large code bases is within reach.
https://github.com/llvm/llvm-project/tree/main/clang-tools-e...
The only question is whether you would want to spend the staff hours working on a clang-tidy check. Large code bases are exactly where the tradeoff makes sense.
1. https://en.wikipedia.org/wiki/Coccinelle_(software) 2. https://en.wikipedia.org/wiki/Sparse
Typed languages let you know whats broken at compile time and aides in refactoring code you can see. It doesn't help you refactor code you can't see.
Cool concept, but once you've dealt with any legacy codebase that has used it extensively, it feels like an anti-pattern/footgun. Ultimately you just want the language to do those things directly and for the compiler to be smart enough to optimize it later.
Essentially it is too clever for its own good.
When I first learned C++, using cfront in the late 80's, lots of the language was implemented as C pre-processor macros.
You can even check out how it was done: https://github.com/seyko2/cfront-3/blob/master/src/template....
#define BASE hey
#define HEADERS foo bar baz
#append HEADERS bad bal bah
#push OSSUFFIX
#ifdef WIN32
#define OSSUFFIX win32
#else
#define OSSUFFIX unknown
#endif
#foreach H HEADERS
#eval include "$(BASE)/$(OSSUFFIX)/$(HEADERS)"
#endfor
#pop OSSUFFIX
Even though some stuff can be done via language features, it is often necessary and more flexible to work with the source code itself.
Its also surreally poorly designed, encouraging worse habits than C itself. Avoiding definition collisions and lack of namespaces alone make it horrific for any moderate sized project and up. Combine that with the near complete lack of static analysis tools, and its a recipe for disaster.
I don't claim to be an expert in knowing what is/isn't in the various standards; I just look at the build errors and static analysis alarms. That said, I think the argument is that in the vast majority of cases it's not actually possible to change the type of a variable; off the top of my head, I can only think of pointer co-erosion. Otherwise, if you define an unsigned int, it stays an unsigned int.
Now strong/weak typing isn't necessarily the same thing as type safety. C has always seemed astonishingly bad on that front. It's like they try to trick novice developers into thinking they have a robust type system sometimes.
If you define functions implicitly they will link to any symbol, EVEN A VARIABLE... this is bananas. I think I sort of understand why compilers work this way, but it really feels like a bug in the language. Under some compilers you might not even get a warning for implicit function, either! TI had a compiler that hid them by default.
Enums are garbage in C. Again, you can misuse them, and may not even get a warning. You can pass an enum for color to a function that takes an enum for kittens and the compiler will be happy as a clam. The way they're often used can cause constant implicit conversions every time there's an assign/compare. May not be a problem for most positive values, but it's annoying if you're trying to develop standard compliant code. MISRA defines an "essential type system" and normal enums usage violates it.
I'm probably forgetting a TON of deficiencies. Please add them or correct me where I'm wrong.
It’s the C preprocessor that causes a mess. Tooling for C and C++, like automatic refactoring, lags behind tooling for languages like Java, C#, and Go. Refactoring tools have to deal with macros, conditionals inside #if/#else blocks, and header search paths.
In this case, the refactoring involves removing a variable from the enclosing scope of a macro invocation. The most likely way to automatically refactor it would be to write a custom check in clang-tidy.
Eg C tracks whether a variable is an int or a char. Haskell also tracks whether a function causes side effects or not. More sophisticated systems also track whether a function has to return eventually or could run forever.
1. Have 3 versions of the thing to change: select_nospec, select_nospec_safe, select_nospec_unsafe.
select_nospec_unsafe is identical to select_nospec. Do the mass replace. All stay same.
2. Delete select_nospec
3. Start migrating to select_nospec_safe. You can still do this in small steps, because you old thing stay.
4. Where select_nospec_unsafe must be, keep it and maybe mark with a comment or similar to not delete it later
5. You finish and all is changed.
Maybe replace select_nospec_safe to select_nospec if wanna.
It's relatively easy to do the mass replace in master. You leave the burden of the replacement in the other branches to people who own those branches.
Set up an automated script, to keep the 'unreplaced' version out of master (and people can also use that script to check their own branches).
Not sure if that counts as non-trivial, yet?
There are probably some other trade-offs involved. The people running kernel development ain't idiots.
Assuming perfect tooling and infrastructure doesn't sound like it necessarily have to be a big deal.
But, how far into fantasy land are we? How much effort would be required to get there? Is this and all similar problems large enough to justify it?
Is adequately tooling and infrastructure even possible?
Does that make proposed changes, such as moving to C11, order of magnitudes more difficult?
The only major issues is that the change is let half-way. But that is only possible with MAJOR undisicpline.
---
The major point, to me, is that this EXCESSIVE fear of breakage is not good. Yes, making upgrades is pain, but you CAN make the pain tolerable with some planning.
Refactoring is like exercise for code: Everyone dislike the idea of excessive, but is GOOD.
And lets be honest, among all the things that could requiere a refactoring, this case is on the most simples of the simples scenario..
The unison language offers an interesting take on the problem of code evolution backward compatibility. https://www.unisonweb.org/
In all of those, the enhanced function takes more paramaters. Switching the existing name to require a new parameter will break existing code, and people will not want to update. Making the new parameter optional is possible, but IMHO is messier than a new function that has required parameters and then deprecating the old function (and eventually removing it).
On userspace that's the time when you deprecate an entire module and switch everything into a new name. The kernel does that once in a long while, but it's a bit harder for them.
Like most things introduced in early php, the mysql extension was just wrapping c functions.
The concept of “real” escape comes from mysqls’ C API: https://dev.mysql.com/doc/c-api/5.6/en/mysql-real-escape-str...
void fun(int x)
{
struct foo = { x }; // not allowed in C89
int bar[3] = { 0, x }; // ditto
}
struct list_head {
struct list_head *next, *prev;
};
struct foo {
int fooness;
struct list_head list;
};
struct foo *iterator;
list_for_each_entry(iterator, &foo_list, list) {
do_something_with(iterator);
}
Ah -- That does seem like what they do:
#define list_for_each_entry(pos, head, member) \
for (pos = list_entry((head)->next, typeof(*pos), member); \
&pos->member != (head); \
pos = list_entry(pos->member.next, typeof(*pos), member))
#define list_entry(ptr, type, member) \
container_of(ptr, type, member)
#define container_of(ptr, type, member) ({ \
const typeof( ((type *)0)->member ) *__mptr = (ptr); \
(type *)( (char *)__mptr - offsetof(type,member) );})(You can also build with clang, but only because clang deliberately aims to support most gcc extensions.)
The ladder's being moved a millimeter.
Ken Thompson, Reflections on Trusting Trust.
EDIT: I don't think I've seen 5 nearly simultaneous replies sharing the same link before.
LOL I was searching for a non-PDF link and delayed the reply. It would have been 6 simultaneous answer.
How do you prove your stack is secure?
[1]: https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_Ref...
[0]: https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_Ref...
https://www.win.tue.nl/~aeb/linux/hh/thompson/trust.html
The paper's a pretty entertaining read:
> First we compile the modified source with the normal C compiler to produce a bugged binary. We install this binary as the official C. We can now remove the bugs from the source of the compiler and the new binary will reinsert the bugs whenever it is compiled. Of course, the login command will remain bugged with no trace in source anywhere.
is the benefits of "modern C" worth compile times 2-3x times longer?
Are you able to give hints that would guide us in thought exercises?
You have to be very large to notice it, but the likes of Facebook, amazon, and Google have massive warehouses almost entirely filled with computers. It doesn't take much to see how their power bill can add up.
The Distribution collectors, they brag about speed, and optimization is very important to them. They want to be able to test changes very quickly.
"Jun 30, 1987 — The Cray is part of a $20m installation used by Apple's Advanced Technology group and consists of four CPUs operating at 9.5nS per cycle,"
Now we have distcc, that can both massively parallelize compilation, and optimization.
Googles team probably shows some slight improvement, just to justify their work, but in the long run, its just as sloppy as industry wide coding is.
Microsoft is the absolute worst, along with Apple.
For example, Rust is frequently said to have long compile times. There are a number of interesting reasons why that is often true, but if we ignore the pathological cases then what we find is that borrow checking takes a significant fraction of that compile time. This is a big trade–off between features and complexity that the Rust language made very deliberately: the advantages of the borrowing rules are what makes Rust such a great language. The cpu time spent checking that those rules have been followed are a small price to pay, but not a negligible one.
I mean, you get more security by default and security is pretty darn important for kernels...
I would guess that you're referring to either "How ISO C became unusable for operating systems development" ([0]) or "How One Word Broke C" ([1]).
[0]: https://arxiv.org/abs/2201.07845 , most recent HN discussion at https://news.ycombinator.com/item?id=30022022
[1]: https://web.archive.org/web/20210307213745/https://news.quel... , HN discussion at https://news.ycombinator.com/item?id=22589657
So “old C” in such a case would need to mean “an old C implementation” (or possibly a new one, but simple or configured to behave like an old one), something like GCC 2.8 maybe, and nobody’s using that on desktop. So the language standard version should be mostly immaterial, and it’s not like the C89-to-C17 difference is anything like the yawning C++98-to-C++20 chasm. (This is a carefully phrased statement: C99 had complex numbers, which are annoying, and variable-length arrays, which are a significant change, but C11 demoted both to optional features.)
With that said, we still have options even when moving to a newer standard. Many new language features can be machine-translated to C89 if needed (similar to how we have protoize / unprotoize, though not always as seamless). If we need to keep a bridge to C89 the kernel authors could hold to a subset of new language features that are amenable to machine translation.
As compiler technologies evolve, it becomes not only a necessary evil but rather a better course of action to trust compilers rather than humans tiptoeing around security risks masquerading as language idiosyncrasies.
Erm. I was always under the impression that he had actually done it, and that his presentation was a historical anecdote. Is that not the case?
It's more work - you have to figure out a sneaky bug and write a legit looking patch - but anyone can do it. You don't have to be in a position of power already (e.g. being the Debian GCC packager) so overall it is much easier.
Thompson's hack relies on the Halting Problem, and the space for deviousness within the Halting Problem is infinitely large.
For example, I run Gentoo Linux, and haven't reinstalled my OS since 2004 or so. That means that, modulo a few binary packages, I have a direct source lineage to the state of Linux in 2004. If you want to pull off that attack against my system (and you didn't already back in 2004), you'd have to tamper with source archives. That would both imply changes that are easy to analyze (more than binary patches), and it would involve changing the archive hashes in the Portage tree. That tree is in Git, which means that it would create an immutable public record of what happened (Git is the original blockchain, remember), modulo forced pushes which people would, again, notice all over the place.
In practice, if you want to persistently backdoor a new system (supply chain attack), it's usually easier to do that in hardware or firmware than trying to do a RoTT attack on the distro and its compiler. In fact, it's users of binary distributions (or proprietary OSes) that should be more worried, as it is much easier to do a binary-based RoTT attack that self-updates to handle new versions consistently when all your users run the exact same binaries. Source code users should be more worried about compromise upstream than local persistence. And those attacks are a review / auditing issue, unrelated to RoTT.
In the end, if you are worried about being personally targeted, it's easy enough to make that impractical by re-bootstrapping your computing from an unpredictable source (e.g. walk into a random shop and buy a PC, walk into a net cafe and download your favorite distro and check the hashes there). And if you are worried about large-scale attacks, RoTT style ones aren't practical without someone somewhere noticing; you should be worried about traditional compromise instead.
C11 added insecure Unicode identifiers, and you won't find them in reviewing them manually via email. You need a special linter to detect such Unicode attacks. Or a proper development environment.
Compiler development usually evolves into more bugs, not less. Just now they caught up with the hundreds of bugs they added with gcc-9. Not talking about const, restrict, and strict aliasing, and the still not existing -Oboring for the kernel. Would you dare to use -O3 and -flto and -fstrict-aliasing in the kernel?
Heck, Linux already uses GCC plug-ins to implement some fancier security stuff; it is silly to think they can't handle forbidding Unicode identifiers.
Actually, as a compiler writer, I'd go a little bit further and point out that Linux itself isn't even written to the gnu C89 very well; it's often written to a "C is portable assembly" view of the language, which results in nasty grams and invective being hurled at compiler writers if they compile the C specification correctly and not according to the "proper" assembly the code author thought they were getting.
One of the benefits of more modern language revisions is that they actually tighten the wording on a lot of the more ambiguous parts of the specification--C11 in particular adds a much more comprehensive memory model that's very shrug in the older revisions of C.
Their hesitancy with newer standards is understandable, when viewed against that backdrop
More to the point, though, the only changes to undefined behavior in the C specification in newer versions (compared to C89) are either clarifying things that were already undefined behavior (e.g., INT_MIN % -1) or actually making some undefined behavior well-defined (e.g., allowing type punning via unions).
[1] http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2464.pdf
Older standards said "it is implementation-defined whether the old object is deallocated". This didn't really work well:
- if realloc(..., 0) returns NULL if it freed the object, then you have confusion with error cases. strtol already has this kind of interface and it's unusable
- if realloc(NULL, 0) returns NULL and does nothing it does something different than malloc(0). Some chose to make it do something different, some chose consistency with malloc.
- if you choose consistency with malloc then realloc(NULL, 0) likely will end end up inconsistent with realloc(ptr, 0) where ptr is not NULL. On BSDs the two are consistent but also different from any other platform, so portable code could not rely on realloc(..., 0) doing something known: either you had possible double-free bugs on some platforms, or you had a memory leak.
Man I so vibe with that. But the new Cs do have a ton of new features and, more to the point, I trust Linus to make this kind of decision more than just about anyone.
I was always under the impression that the "message" of Reflections on trusting trust was that you have to trust someone at some point.
Why does the complexity of the OS, or even its ability to be compiled by multiple compilers, matter for "trusting trust" attacks?
As I've always seen it, the problems and solutions all exist at the compiler level. The whole premise relies on starting with a binary compiler that you are expected to trust. The source is also assumed to be safe and un-tampered for purposes of this discussion because that's an entirely different issue.
The solution, of course, is to build the compiler itself with different compilers. If you build the same compiler with two or more different compilers, then use that to compile itself, you should be able to with the right options (see the work that has been done on reproducible builds) get binaries that are equal or close enough to easily compare any differences.
At that point if they are the same then you know either it's good or both of the upstream compilers were also compromised.
If for whatever reason that's not practical at the top level compiler the same concepts apply going further back in history until you get to some early compiler a bored grad student wrote in the '80s in pure ASM.
As a result from a practical sense I don't really see "trusting trust" attacks to be that big of a concern. It's always possible to work your way back down the tree of software until you get to a point where you can actually trust a compiler and then build your way back forward from there.
If a compiler starts depending on its own tricks and gets to a point where it can only be successfully compiled by itself, then there are reasons to be suspicious. Even then you'd just have to have the last version to be able to be built by other packages as one more stop along the way to trust.
for(int i = 0; i < 10; i++) {
// ...
}
{
int i;
for(i = 0; i < 10; i++) {
// ...
}
} {
list_for_each_entry(...)
}At that project scale, wouldn't it start making sense to start solving problems like "If it were possible to write a list-traversal macro that could declare its own iterator [...]" by adding the functionality you want to GCC?
If some shenanigans were to happen upstream in GCC it would not be the end of Linux.
> The Linux kernel has always traditionally been compiled with GNU toolchains such as GCC and binutils. Ongoing work has allowed for Clang and LLVM utilities to be used as viable substitutes. Distributions such as Android, ChromeOS, and OpenMandriva use Clang built kernels. LLVM is a collection of toolchain components implemented in terms of C++ objects. Clang is a front-end to LLVM that supports C and the GNU C extensions required by the kernel, and is pronounced “klang,” not “see-lang.”
It doesn't seem useful to make a "C+" instead of switching to a language that just has the feature set the kernel needs. Like Rust, which is gaining some support within the Linux kernel already.
But that's exactly why I think a customized language for the kernel is an interesting idea. You could get pretty much exactly what you want for the kernel. Add features you need, remove any undesirable behavior or features.
For most programs that would be too much complexity, but the kernel has very particular needs. And I think a similar in spirit approach has worked very well with Qt.
This is probably one of the biggest reasons I may turn to rust over C++ - simply less features creep due to less time being around.
Can you comment on how wrong I am about my feelings this way?
C++11 and newer are all largely one "category" in terms of recommendations. The CppCoreGuidelines is a good place to cover all that, but it's not a from-scratch introduction by any means
C++(17+) template based metaprogramming far more powerful and generic than what you can do in eg rust. Converting eg a rust or go or python, or julia, etc library into c++ is pretty straightforward, just use appropriate overloads, and a few template tricks, and you can mostly copy the code directly. But copying between these, or from c++ is much harder.
The solution is to learn one approach suitable for the problem you have right now, then widen over time. The many languages of c++ as you describe is less of a problem, the problem is that some approaches are deeply and fundamentally flawed, but still in common use.
C++ allowed the Serenity OS people to produce an entire OS with a GUI stack able to play Diablo and their own web browser in something like two years. It's depressing to think where we could be today in terms of OS if the Linux and GNU people weren't as insistent on their hate of C++
Simple question, simple answer. Using C is purely an act of risk-aversion, contrarianism, and fashion.
I'm not necessarily advocating that Linux should switch to C++, just that there's probably not a good reason to invest in a new C/C++ hybrid language at this point in time. Not when C++11 is honestly pretty good, but also Rust, Zig, or even D's betterC all already exist.
In Rust, allocation lives in a library, alloc, and so the Rust for Linux project did all the work to offer alloc (it's full of useful stuff and it isn't like the Linux kernel can't allocate memory) but without implicit allocation.
If you use Rust to write say a Linux command line program, you can write
greeting += " and welcome traveller";
However the most critical reason Linux doesn't have C++ is that C++ proponents didn't do the work. Now, that will probably be because "reform the entire language to suit Linus" wasn't a viable plan, but the fact is that Linus can't accept patches that nobody writes, so even if you're sure C++ would be viable without drastic changes, you didn't write the patchset that does it. Likewise if people don't send Linus patches to do C11 it probably won't happen.
Ah, but see, it doesn't. Linus was wrong about many things in his rant. This being one of them.
Now the standard library does indeed have things that do implicit allocations, such as std::string. But these, like with Rust, are distinct from the language & replaceable. Would it be effort to make a kernel-safe std:: replacement? Yes. Would it be a lot of work? Not really. And it's the kind of thing Linux has been doing for decades with C anyway. It's not like they use a standard libc implementation (much less a rich libc implementation like glibc), for example.
And it's something game devs have been doing with C++ for decades without any issues, too.
So for this one you don't even have to ban anything. Just don't pass a standard library implementation to the compiler. It doesn't come with one, after all. You have to add it. So you could just... Not do that.
As for nobody did the work... That's true. But Linus also pretty much nuked the entire concept of using C++, regardless of the what or how. Time seems to have changed his mindset on some of those things, hence his reaction to Rust, which largely makes all this moot. But we shouldn't confuse short term politics with technical issues, either.
Its not what is being discussed here. C++ has an implementation defined memory model. That just cannot work with Linux out of the box. You need an OS and compiler designed from the ground up to handle this.
Only since C++11, and it's not a full blown memory model.
There is nothing when can do to change political views.
And lets not pretend that Rust in the kernel won't suffer from creative uses of macros, and there is still a big laundry list of issue to fix before it actually makes it.
Or that for the time being, Rust compilers need to link to C++ code to actually work, at least until Cranelift is a match in code quality against LLVM or GCC.
So in the end Linus gets C++ into the kernel, even if indirectly.
GCC itself has been written in C++ since 2010.
If they switched to c++17, and banned everything from throw and RTTI to the standard library, they would still get for each, auto, and templates and inheritance which does not have to be emulated in C.
I tend to agree, as a C++ developer. There are many core issues in the language that haven't been resolved and that are unacceptable for kernel code.
My personal pet peeve: C++ is unable to reallocate a new[] region. This makes basically all structures (vector, hashmap, trees...) unusable for large data handling.
But this is assuming you have a malloc implementation that does something other than implement realloc as just malloc+memcpy+free. Which not many do, not unless the allocation is so large as to be in its own dedicated mmap or similar.
That aside, sure would be great if you elaborated on these unsuitable, unresolved for kernel language issues? Exceptions and rtti are the only two I'm aware of and both have had off switches for decades.
Sure, but basically it means rewriting all structures that rely on a bucket of stuff.
By the way maps often use a large bucket, and rehash in-place can be preferable.
> Which not many do, not unless the allocation is so large as to be in its own dedicated mmap
Do you know a modern operating system that does not have a mremap equivalent ?
On Linux you pretty much use it as soon as you reach large blocks.
std::unordered_map (what I'm guessing you meant by a hashmap) uses a linked list for the nodes. There's no movement in the first place to worry about being realloc'd.
> Do you know a modern operating system that does not have a mremap equivalent ?
You have to be very large before most mallocs will put you on a dedicated mmap that can even be mremap'd at all.
If you're working with stonking huge data inline in a std:: vector... Yeah just make a container for that usage, not really an issue. There's tons of examples out there, typically to add SSO but doing realloc would be the same basic thing.
It really, truly is not comparable. list_for_each_entry does a well known operation that's just moving some pointers around. Safe to do while holding a spinlock, or in interrupt context, etc.
std::for_each is a generic iterator implementation. What code runs in order to iterate? Will it transparently take locks, risking a deadlock? Will it need to schedule()? You don't need to ask those questions about the standard linked list macro, and that has a lot of value when every instruction the compiler generates might matter.
All your other stuff about for_each is just... Wrong? It's all well defined what it does. The substitutions that for_each makes and exactly what it calls are all defined.
You could give it a container that does something stupid, but you could also give list_for_each_entry the wrong field or accidentally use it when it's no longer valid (literally the bug being fixed)
I suppose the thrust of my opinion is that a lot of code depends on the fact that it iterates over a linked list, which is safe in many contexts. The idea that callers could substitute container types which break its assumptions seems unwelcome to me. Nobody provides the wrong fields to list_for_each_entry because they get very nasty compiler errors if they try. But anybody can plug a new container type into a for each statement and get new behavior that silently breaks the assumptions of the original code.
I suppose that could be protected by a better type system though. There's a lot to improve on C even for kernel dev, and "C is greatest" isn't a hill I'm willing to die on :)
How would it be "silent"? You're changing the container at the point where you're making any assumptions about it. This would just fall under normal code review purview, just like code reviews are how Linux enforces that anyone even used this linked list macro in the first place. C certainly doesn't care if you use list_for_each_entry for your container iteration, after all.
But it'd be a lot easier to replace these linked lists with a vector that has the same observable contract if C++ was used, just would be way faster to iterate on.
{ iter= xs.begin(); while(iter++ != xs.end()){L} } Not exactly hard to see what happens. In particular as iter is typically just a raw pointer.
Regardless of if you use one or the other you must answer the question, Should I take locks, etc.
With the macro you also have the concern that if any of the headers you include happens to change order of inclusions, the macro called may change without being noticeable. That is a harder problem.
If you started writing a major project in C now you'd rightfully get sacked, but open source projects like this are often dominated by the opinions of those who only work on that project. That doesn't mean they're automatically wrong but just that they're often massively detached from any feedback apart from disaster.
Deathtrap situation-- maintain an operating system and a fork of a compiler. Benefit: you can get more control over the compiler. Cost: you still cannot always get the compiler to behave as you think due to time constraints. Death cost: your first cost is multiplied by the fact that you're now maintaining a goddamned compiler.
I rankly speculate Linux is an extant project at its scale because it has refused to fight on two fronts like this. (And yelling across a border isn't the same as crossing it.)
Edit: clarifications
https://gcc.gnu.org/legacy-ml/gcc-patches/2009-07/msg01556.h...
Static keys are a kernel API that allows code modification at runtime to achieve zero cost feature flags, like tracing:
https://www.kernel.org/doc/html/latest/staging/static-keys.h...
I would not be surprised if there is some niche environment somewhere that needs c89. Not any mainstream desktop, server or phone though.
I think C99 introduces some bad things that should be keep out of any code base. Variable length arrays being the biggest one. Generics is another one. Being able declare variables anywhere doesn't make the code better but it makes styles diverge and that's a problem.
If Linux wants to adopt C99 they should define what parts of c99 should be adopted in their style guide.
It does, though. The ability to declare variables "anywhere" and allows a reduction in the portion of the code where a variable exists uninitialized, or initialized with a dummy value which should never be used, or holding an obsolete value which is not intended to be used. It also allows more variables to be declared `const` (which requires them to be initialized in the declaration, which must occur after the initial value is computed) which helps to detect accidental assignment and also makes the code easier to review.
It's worth noting that you can more-or-less declare variables "anywhere" in C89 too—you just have to wrap the variable's scope in a compound statement. From that point of view, C99 doesn't change where variables can be declared. It just lets you remove some syntactic clutter.
stuff like this: [0]. I mean, what's the point of declaring variables like cp or err at the beginning when they're assigned first only halfway down? What is the point of having path around for the whole function, when it is a super temporary assign-once variable with a scope of 3 lines? I'm having a really hard time coming up with a justification for this, other than "this is the way we've always done it and we like it" (you can find actual rants in defense of this style from about 10 years ago).
For-loop declarations are another instance of this that make the code more fluent IMO. And they help reducing the scope of variables to a strictly smaller block. I would find it hard to argue that this does not remove some bugs or improve readability.
The only thing that declare-first has going for it is that the variables that are in use can be seen at once, a little bit like in a struct declaration. Looking at how optimizers butcher those variables depending on liveness makes this feature seem less valuable, though.
[0] https://github.com/torvalds/linux/blob/2729cfdcfa1cc49bef5a9...
The kernel already used VLAs as an extension, and is currently in the process of removing them from all the code using them.
https://www.phoronix.com/scan.php?page=news_item&px=Linux-Ki...
It is unwise to build a bridge on something untested with unknown failure modes, and it is equally unwise to do a rewrite or create new core infrastructure in a language without knowing it as well as a civil engineer knows concrete.
What years were you a consultant reviewing C applications?
Mind you I was mostly reviewing cryptographic-related applications, but most C applications contained bugs that had nothing to do with the logic (lots of memory corruption bugs) while most Golang applications contained logic bugs. Or at least I would find logic bugs because Golang was both rid of most memory corruption bugs, and also an extremely readable language (so easier for me to understand the code and find logic bugs). Although Golang still had nil dereference bugs (happens a lot when people used protobuf), because they don't have sum types. Today I think a great language would be a mix between a readable language like Go (with good defaults, toolings, stdlib) and a safe languages like Rust.
We can't even agree on safe string functions for C, half a century later. You shouldn't have security bugs baked into the standard library and you shouldn't have to do a mountain of research to know which functions are safe and in which cases.
However, for most things non-string, non-pointer, and non-array, I agree with you.
And for insecure standards, the committees rather want to eliminate the safe functions, than fixing the spec bugs or add u8 support.
What prevents us from having one?
>Unless you're being wildly negligent and reckless with your programming, C is really not that scary
They are moving that direction because of the safety promise that Rust offers and it is a "hot" language with a bunch of momentum behind it.
I've seen time after time people saying "nah, C is fine, you just avoid this and that, use these tools, etc." and this turning out to be insufficient. I've heard many times "maybe you're just a bad programmer and can't handle C, but I'm a good programmer and have no problems" and their code not surviving 5 minutes of fuzzing. I've seen people conclude that multi-threading beyond simplest constructs is just infeasible to get right, and think that's an inherent property of threading, and not fragility of C.
Have they not worked on large projects with other developers? Have they not seen the myriad of ways things can silently go wrong for seemingly no technical benefit? (although I know there's often less obvious reasons, for eg. UB, performance, platform specificity, etc.)
All that, said I do think the following might be reasonable: > "nah, C is fine, you just avoid this and that, use these tools, etc."
I guess you mean they write off the risks entirely? You should never be this "handwavy", and should always take the risk seriously, especially with a language like C. However, I think it's fair to say, that C is a good choice, many of the risks can be mitigated, and it's not THAT big a deal. In which case, the above doesn't seem that absurd.
Following basic common sense and making an effort to identify and eliminate some of the sketchier situations, backed by some really good integration testing, can really help. I feel reasonably safe under those circumstances (I mean not really, but other languages can be "unsafe" too). A huge chunk of the really evil things I've seen have been the result of taking absurd risks, and/or disregarding the rules entirely. If you were paying attention and "trying" to write good C code, they would never happen; these aren't just individual developer things, but project wide. Eg. I had a compiler that didn't even warn for implicit functions... jerk move by TI, but that should be flagged and dealt with. Instead, the team just thought "great no compilation errors".
I will say there's a lot of developers that are just uninterested in any of this and will deliver some really, really sketchy C code. In their mind they're smart programmers and their code will just be right, and they don't seem to understand any of these issues. Just plow ahead and patch around the bugs, then move on to the next gig.
Lots of developers out there are widely negligent. C provides them with enough rope to hang themselves. To be honest, I'm a little surprised to find veteran C developers that AREN'T scared. I guess they just see every disaster as the fault of "negligence and recklessness". If you're not scared of your code (mistake), you should certainly be scared of other peoples.
> Most of our memory bugs occur in new or recently modified code, with about 50% being less than a year old.
> […] we’ve found that old code is not where we most urgently need improvement.
https://security.googleblog.com/2021/04/rust-in-android-plat...
It's worth giving it another try if you haven't in awhile, if for no other reason to understand what's going on behind the scenes of your abstractions in Java/C#/JavaScript/etc.
It is as unsafe as you let it be, consciously or by mistake.
long* p = malloc(sizeof(int));You mean, aside from Fortran, COBOL, and Lisp?
I suppose in the case of list handling there's not really a good alternative though. If C had lambdas we could use inline functions instead but that's not a thing. Still think it ought to have been uppercase though.
If you are using a 0 day old kernel. Why would you be using GCC 5.x still. 5 1 is almost 7 years old now. 8.1 is almost 4. Can't people just use apt / yum / rpm / whatever to upgrade to the latest gcc? Is that really too much to ask of the community?
[0] https://access.redhat.com/product-life-cycles/?product=Red%2...
Even if all distros have long ago moved to GCC 8+, you still have other chips and systems running, and who knows what buried somewhere depends on GCC 5.
If it works, why spent money ‘fixing’ it after all.
It would be like running OS/8 (for the PDP-8, popular in 1965) on a modern machine. No time sharing for processes, no network stack, only a TTY for I/O... what would you even do with it if you could compile it?
If someone made a product which uses it, and people keep buying it, most companies aren’t going to bother with any of those things if they don’t need it.
That is of course completely different from what Linux is doing (as in outside of that context). It may be at version 510, but that doesn’t mean someone won’t be running version 1.0 somewhere. And that’s fine.
It makes sense to play it safe and do this in smaller incremental jumps.
I do think they could move up a bit further, but it's useful to be able to just build kernels with the distro compiler and repology thinks that for instance Debian stretch (still an LTS supported version) is only gcc 6.3.
It's totally meaningless and usually dominated by the fact they've been asked to migrate, but it's still true unfortunately.
It's similar to but not quite the same thing as the Y2K problem.
That's a very "legacy code" way of avoiding the problem, it's just a great option for a centenary frozen language.
"Oh, no, look, C98 is from 2198. Do not confuse it with C97 that was settled in 2097."
For the .c/.h part, it's because of use (or intended use). Header files are meant to be specifications (though a lot of implementation ends up in them, annoyingly). It actually is an example of DRY thanks to C's need to forward declare things. You can't call a function the compiler doesn't know about, for instance, so you declare (not define) it in the header if it's defined in an external module. If you didn't have the headers, you'd need to copy/paste or handwrite the declarations everywhere.
Why do you think there needs to be a declaration beyond the definition? The definition is enough information for the compiler, and having both is a DRY violation.
int foo(int n) {
char c = bar(n);
...
}
char bar(int n) { ... }
I don't understand your point about using a function that hasn't been declared. Of course it won't work!
I understand your point about using header files as interfaces to third-party code, but there are ways of doing that that don't involve duplicating all functions, structs etc.
int main(int argc, char* argv[]) {
printf("%d\n", foo(10));
return 0;
}
int foo(int n) {
return n - 1;
}
But that's C. You can't fix it without fundamentally altering the language.
I still don't understand; why does the implicit declaration assume int bar() is the signature when bar is being passed an int and returning a value to a char?
Friends of mine who are actual CS majors say with C it's not fixable without changes to the language. With Java or C# the compiler can do a pass and extract the object definitions. But because of ambiguous syntax and majorly because of the preprocessor it's not possible with C as it stands.
Also, this explains all the prefixing that OP was complaining about lol, or at least that's one reason. I saw a particularly bad bug, due to poor naming decisions. Less is not always more, even if it annoys you.
I understand why the linux kernel folks aren't moving to the bleeding edge (Linux is old, you have to do these ports incrementally), but I'm curious why the pace of language is so slow. I guess it's because C has more or less stabilized and thus further revisions are less necessary?
1989, 1995, 1999, 2011, 2017.
The 1995 one was an "amendment", not a full revision (and is mostly additions to libc, which Linux doesn't get to use; the only language change is the addition of digraphs).
C17 was a "bugfix" revision; compilers will have applied these fixes to their C11 implementations, so in effect the only difference between C11 and C17 is the value of __STDC_VERSION__. So for most intents and purposes C11 is still the most recent revision.
That is a big part of it. C is an old, stable, complete language.
There is also the existence of gnu extensions, which bridge the gap between language revisions. For example, anonymous unions were added in C11 but they have been a gnu extension since forever.
This is a strength of the language.
If you've ever done CI/CD for a large project in a fluid development ecosystem (e.g. nodeJS), you can understand why it might be refreshing to develop your operating system in a language where standards are measured by the decade.
That said they do seem to be discussing both C99 and C11 so it does seem like going with something that would qualify as modern (especially in light of Linux's need to be conservative) is actually on the table.
Writing C89, without variable declaration in a for loop, and without // comments, etc, feels ancient. C99 on the other hand is what defines how C looks today.
This is also why the attitude is: if they jump to c99, they may as well jump to c11 because they are basically the same.
> the objective was not to rewrite the kernel’s 25 million lines of code in Rust, but rather to augment new developments with the more memory-safe language than the standard C normally used in Linux development.