Hacker News Reader

Comment by defen | Hacker News Reader

This is why we can't have these discussions...everything gets interpreted literally, and it becomes some sort of ego contest. I'll take your claim at face value, that your team wrote 2M lines of C for a commercial project, with no memory corruptions, crashes, integer overflows, etc. That's a shockingly impressive achievement, one that I think has probably never been done before.

But how is that a useful or replicable result, that will work in the real world for other projects? You're basically saying that everyone just needs to be a top 0.001% C programmer. This is the "abstinence-only sex education" of programming advice. It's super simple to not get STDs or have out-of-wedlock pregnancies - just don't have sex with anyone but your spouse, and ensure they do the same. It's 100% guaranteed to work, and if you don't follow that advice you're stupid and deserve whatever bad things happen to you.

View on HN

> You're basically saying that everyone just needs to be a top 0.001% C programmer.

A more constructive/charitable phrasing of the parent's point might be: you should only be allowed to code in C if you're already a top 0.001% programmer. Then, obviously, all C code will be good—not that there'll be much of it.

That's actually a more interesting point than it seems. Programmers generally tend to reject the idea of guilds and "professional" licensing—mostly because, AFAIK, most programs aren't critical and most crashes don't matter. But what if, instead of licensing all programming, we just licensed only low-level programming? What if, as a rule, everyone who was going to be hired to write code in a language with pointers was union-mandated to be certified as knowing how to safely fiddle with pointers? That could probably split on interesting dimensions, like a bunch of regular coders working in e.g. a Rust codebase, with a certified Low Level Programmer needing to "sign off on" any commit that contained unsafe{} code, where the actual failure of the unsafe{} code would get the Low Level Programmer disbarred.

Not all C developers need be a top 0.001% C developer to write high performance and rock solid code. Why?

I'm a very experienced C programmer and one day my boss came to me and said that the sales guys had already sold a non-existing client side module to a house hold name appliance manufacturer. The deal was inked and it had to be ready in only 3 months. Even worse, it had to run in the Unix kernel of the appliance and therefore be rock solid so as to not take the whole appliance down. It also had to be ultra high performance because the appliance was ultra high performance and very expensive. Now the really bad news: I had a team made up of 3 more experienced C developers (including myself) and 3 very un-experienced C developers. We also estimated that in order to code all the functionality it would take at least 4 months. So we added on another 4 less experienced C developers (the office didn't have a lot of C developers). The project was completed in time, a success, and almost no bugs were found, and yet many developers without much C experience worked on the project. How?

(a) No dynamic memory allocation was used at run-time and therefore we never had to worry about memory leaks.

(b) Very, very few pointers were used. Instead mainly arrays. And not just C developers understand array syntax, e.g. myarray[i].member = 1 :-) Therefore we never had to worry about invalid pointers.

(c) Source changes could only be committed together with automated tests resulting in 100% code coverage and after peer review. This meant that most bugs were discovered immediately after being created but before being checked in to the source repository. We achieved 100% code coverage with an approx. 1:1 ratio of production C source code to test C source code.

(d) All code was written using pair programming.

(e) Automated performance tests were ran on each code commit to immediately spot any new code causing a performance problem.

(f) All code was written from scratch to C89 standards for embedding in the kernel. About a dozen interface functions were identified which allowed us to develop the code in isolation from the appliance, and not have to learn the appliance etc.

(g) There was a debug version of the code littered with assert()s and very verbose logging. Therefore, we never needed to use a traditional debugger. The verbose logging allowed us to debug the multi-core code. Regular developers were not allowed to use mutexes etc themselves in source code. Instead, generic higher level constructs were used to achieve multi-core. My impression is that debugging via sophisticated log files is faster than using a debugger.

(h) We automated the process of the Makefile so that developers could create new source files and/or libraries on-the-fly without having to understand make voodoo. C header files were also auto generated to increase developer productivity.

(i) Naming conventions for folders, files, and C source code were enforced programmatically and by reviews. In this way it was easier for developers to name things and comprehend the code of others.

In essence, we created a kind of 'dumbed down' version of C which was approaching being as easy to code in as a high level scripting language. Developers found themselves empowered to write a lot of code very quickly because they could rely on the automated testing to ensure that they hadn't inadvertently broken something, even other parts of the code that they had little idea about. This only worked well because there was a clear architecture and code skeleton. The rest was like 'painting by numbers' for the majority of developers who had little experience with C.

The same team went on to develop more C software using the same technique and with great success.

I'm pretty you would pass whatever sort of bar exam for a "Low Level Programming" certification they might deign to thrown at you. You're the 0.001%. (Although 0.001% might be the wrong number—I'd hazard 0.1%, or maybe even as high as 0.5%.)

Which is to say, your story matches my hypothetical pretty well. You effectively created the same structure as Rust has, where there are safe and unsafe sublanguages, and the "master" developers thoroughly audited any code implemented in the unsafe sublanguage.

Which makes my point: there exist a small set of people qualified to write in "full C", and a much-larger set of people who aren't; and the only way—if you're a person who isn't qualified—to write C that doesn't fall down, is with the guidance and leadership of a person who is.

Though I think maybe your thesis statement agrees with that, so maybe I'm not arguing with you. Not all the developers on a given project need to be from the qualified set, no. But at least some of the developers need to be from the qualified set, and the other developers need to consider the qualified ones' guidance—especially on what C features to use or avoid—to be law for the project. As long as they stay within those guidelines, they're really working within a DSL the "master" developers constructed, not in C. And nobody ever said you can't make an idiot-proof DSL on top of C; just that, if you need everything C offers, your the only good option is to remove the idiots. :)

We do agree that the experienced developers are necessary along side the less experienced developers. At least when the project is getting off the ground and the less experienced developers are still learning the ropes.

Congratulations on the project's success.

Unfortunely your case is the exception that confirms the rule.

I never saw a company using C like that.

On my case the teams used to be composed from circa 30 developers, scattered around multiple consulting companies with high attrition.

Thanks! And I have not seen any teams doing this kind of thing either.

Another tidbit: The developer pairs were responsible for writing both the production code and associated automated tests for the production code. We had no 'QA' / test developers. All code would be reviewed by a third developer prior to check-in. However, at one stage we tried developing all the test code in a high level scripting language with the idea that it would be faster to write the tests and need less lines of test source code. However, because we did several projects like this, we noticed that there was no advantage to writing tests in a scripting language. The ratio of production C source lines to test source lines was about the same whether the test source code was written in C or a scripting language. Further, there was an advantage to writing the tests in C because they ran much faster. We had some tens of thousands of tests and all of them could compile and run in under two minutes total, and that includes compiling three version of the sources and running the tests on each one; production, debug, and code coverage builds. Because the entire test cycle was so fast then developers could do 'merciless refactoring'.

As another software developer working in the "embedded systems" industry, I can tell you that using C like that is not unique. The companies I have worked for and several of my friends have worked for all use C similarly. Sometimes we do use the dangerous features of C. When we do, additional reviews are done to ensure the use is both sufficiently justified and properly done.

Do you happen to know the company names where they work?

That's really an awesome achievement, and your post is chock-full of very practical tips. Thank you!

Thanks!

I also forgot to mention about performance. Whether you code * foo or the easier to comprehend and safer(?) foo[i] then the compiler still does an awesome job optimizing. However, it's much easier to assert() if variable i is in range rather than *foo. And it's also easier to read a verbose log variable i (usually a 'human-readable' integer) than to read a verbose log pointer (a 'non-human-readable' long hex number). At the time we wrote a lot of network daemons for the cloud and performance tested against other freely available code to ensure that we weren't just re-inventing the wheel. NGINX seemed to be the next fastest, but our 'dumbed down' version of C ran about twice as fast as NGINX according to various benchmarks at the time. Looking back, I think that's because we performance tested our code from the first lines of production code, and there was no chance for any type of even puppy fat to creep in. Think: 'Look after the cents and the dollars look after themselves' :-) Plus NGINX also has to handle the generic case, whereas we only needed to handle a specific subset of HTTP / HTTPS.

It was at this point that I realized that the performance of the user-land code in general (including NGINX) could be much better if it wasn't for the legacy network stack in the kernel, which does a bad job at some things such as slow performance accept()ing new sockets, and slower performance because every packet read has to be another system call unnecessarily copying memory; why not use shared memory? That's when I started to get interested in C10M... :-)

Well, there's a trick to writing C++ with no crashes and no memory corruption: never allocate memory using pointers. If you store all your data in stack variables and STL containers, C++ becomes a _much_ more forgiving language (especially if you also avoid templates and operator overloading).

I imagine that this poster and his team took a similar approach with C: avoiding the dangerous features, or just finding ways to factor them out.

Even if you do all the things you said, you can still crash a C++ program. Iterator invalidation is very easy to do on accident, for example.

I'm gonna have to steal this analogy. Its impressive how often these dick-measuring arguments pop up.

Sorry to side-track, but the percentage of people who abstain from sex until marriage or death is far higher than 0.001%, so the comparison isn't really fair.

The "everything gets interpreted literally" though, that the parent wrote, is still spot on.

"out-of-wedlock pregnancies"?

What is this, 1953?

There are a great number of couples choosing to have children in stable, committed - yet unmarried - relationships.

This sums up internet commenting in 2016. Congratulations on entirely missing the point because you chose to focus on an irrelevant and pointless detail, which I'm sure will spawn a stupid subthread that changes no one's mind.

HN needs "collapse whole thread" feature. Badly.

An official solution would be great, but in the meantime there was a thread last year with lots of options:

https://news.ycombinator.com/item?id=8706496

+∞

Way to hijack a useless subthread and do something useful with it. Great meta-point. Honestly. This feature alone makes the comments on Reddit bearable for consumption.

And the parent thread will?

Yeah, that comment is way off-base.

Should be 1967.

</s>

The great thing about threaded textual commenting is that you can have a sub-thread to discuss a side point, without removing the ability to debate the main point in parallel.

The comment struck me as incredibly anachronistic, bordering on offensively so.

> bordering on offensively so.

please don't.

The parent commenter is saying that there is a type of person who believes that everyone should just abstain from sex until marriage, and there should be no out-of-wedlock pregnancies. Those people are as ridiculous as someone who believes that everyone should just write perfectly secure, bug-free C code.

Nice troll

> This is the "abstinence-only sex education

For the record, I don't think your stupid if you have a child accidentally.

However, once you do accidentally have a child, the results are often not pretty, and I will feel bad watching as you deal with the consequences of your poorly thought out decision.

Regarding programming, I can't see the relationship you make at all.

If you write a good program, well good job. If you write a bad program, you are probably going to write a better one next time. (Ever look back at your old code?)

It takes practice to get good, and the more we work at it, the better we'll get. Just keep trying.

I think defen's point was that if a team produces truly high-quality C code, it's similar to a couple that both abstain until marriage. Yes, it's technically possible, but it's rare, and it's not something that should be the expected outcome (that is, most C code will have some problems, and most couples will have at least a little bit of regrettable sexual history before they're married).

It's as if I say: "Step 1 of writing good code: Don't make mistakes. If you make mistakes, you deserve what's coming to you!" Some mistakes are expected, and essentially inevitable; they're the default, not the exception. Blaming the programmer might seem like the right thing to do, but there's more practical benefit to improving the tools and teaching programmers how to handle errors, so that we can decrease the negative impact that bugs will have.

The point of the analogy is that it is very easy to prescribe behavior that is guaranteed to work if followed 100% to the letter, but fails catastrophically if there is even 1 mistake. There are two responses to this - you can either design systems that are easier to follow & fail less catastrophically in the real world (don't use C for new code if you don't need it; teach people about safe sex practices), OR you can double down on the sermonizing about how if people had just been smarter / had more willpower, they wouldn't be in this bad situation.

Yes, but the analogy isn't accurate, in the sense that when things do fail in a C system, it's probably not that hard to fix.

> However, once you do accidentally have a child, the results are often not pretty... Regarding programming, I can't see the relationship you make at all.

I have two words for you: MAINTENANCE PROGRAMER.

Ha! Yes, aren't we all.

It's either that or the life of the grasshopper ;)

Now that I think of it, there's another parallel with unwanted pregnancies. There are men that stick around, and there are boys who run and let others clean up after their mess.

Would you let your daughter date a contractor???

Sure, if she was over 25 and fully aware that it is "no strings attached".

Reply on news.ycombinator.com