Hacker News Reader

Comment by tptacek | Hacker News Reader

This is pretty silly. Being fluent in C has not much at all to do with language quirks. Since (I assume) the 1980s, the comp.lang.c crowd has been reminding us that compilers are fundamentally unknowable and that sizeof(char) could very well be a random walk averaging 1.35. Meanwhile, in the real world, network and kernel programmers have been using some variant of u_int32_t since forever to ensure that they can parse network packets and driver data structures by copying or even casting things into structs.

Also, why would anyone write ' ' * 13? For most of these questions, if your first instinct is "that doesn't look like good C code", your answer is better than the "right" answer.

View on HN

> Meanwhile, in the real world, network and kernel programmers have been using some variant of u_int32_t since forever to ensure that they can parse network packets and driver data structures by copying or even casting things into structs.

And compiler developers in the real world are also increasingly nervous at this state of affairs. The subset of C that works in practice is unspecified. There are plenty of optimizations that compiler developers would like to do, but they don't because they're worried about breaking some code somewhere that relies on undefined behavior (like the code you're talking about). At the same time, performance competition pushes compilers to exploit undefined behavior more and more heavily. There is (IMHO, well-founded) worry across the board that this situation is not sustainable. Experience with this problem motivated the design of languages like Swift, which try to clamp down on UB drastically—which also has the happy side effect of making articles like this one inapplicable.

Undefined behavior has already had practical consequences, such as security-sensitive VM escapes: https://bugs.chromium.org/p/nativeclient/issues/detail?id=24...

Isn't that code wrong anyway? If the bundle size were 16 bytes, the shift would be well defined but still give the wrong result, 64K alignment. Not a great example of undefined behavior causing a bug.

Actually, since C99, sizeof char is defined to be 1 (byte). The number of bits in a byte is implementation-defined.

However, "it's implementation specific and it's most probably this, or that if you use such and such a compiler flag" is different from "I don't know".

As a rule of thumb, don't assume things about C's abstractions, read the standard instead, or ask friendly humans who do if they check out.

Yes, but the term "byte" doesn't tell you much. Most people assume that a byte is 8-bits which isn't true. If the definition the of `char` was `1 octet`, then you could make safer assumptions about "What's the result of `' ' * 13`?".

From ISO/IEC 9899:2011:

  3.6

  byte

  addressable unit of data storage large enough to hold any
  member of the basic character set of the execution
  environment

  NOTE 1 It is possible to express the address of each
  individual byte of an object uniquely.

  NOTE 2 A byte is composed of a contiguous sequence of bits,
  the number of which is implementation-defined. The least 
  significant bit is called the low-order bit; the most
  significant bit is called the high-order bit.

Edit: Reading further into the C11 spec, `5.2.4.2.1 Sizes of integer types <limits.h>`, it says that `CHAR_BIT` must be 8 bits (or larger). A google search suggests there exists some processors that have 16-bit bytes and others that have 32-bit bytes.

I've also personally encountered 24 for C. That's not the problem with (' '*13) here, though — C doesn't mandate ASCII.

char is also allowed to be a signed type, in which case overflow is undefined behavior.

It’s not because of the signedness of char—not only do character literals have type int, but even if this weren’t a literal, the char would be promoted to (signed) int before the multiplication. But of course the multiplication of two ints can still overflow.

> Most people assume that a byte is 8-bits which isn't true.

It is worth remembering that POSIX (and Windows probably too) mandate 8bit chars so there is no point being defensive about it on these particular platforms. And I kid you not, I have seen people who are. Because "ISO C99 this and that".

> read the standard

Casey Muratori, who's a better C programmer than me, is of the opinion that you should check your compiler's behavior rather than read standards.

(Not that you should take his word as gospel, I don't fully agree myself, but it's worth considering.)

The problem with this idea is that, if you're relying on undefined behavior and don't know it, your compiler's behavior might change depending (at a minimum) on the code you use in production but not in your test case and depending on the compiler version.

It's much better to avoid undefined behavior entirely. The problem with this idea is that it's hard to learn and remember all the different kinds of undefined behavior.

>> read the standard

>> Casey Muratori, who's a better C programmer than me, is of the opinion that you should check your compiler's behavior rather than read standards.

In Casey Muratori's case, he's the type of person who cares to understand what the compiler is actually generating under the hood. He doesn't exactly trust the compiler to always do the right thing.

There was an episode of Handmade Hero where he caught an optimization #fail of the compiler and tried to figure out what was going on (live). https://twitter.com/cmuratori/status/596775025023287297

Casey Muratori also has been developing in Visual Studio for a very long time. Visual Studio has a long, tortuous, history of being non-stanards compliant and buggy. And Microsoft until very recently, never cared to fix anything. (And it still doesn't conform to C99, let alone C11.)

Microsoft already stated multiple times that the focus of Visual C++ is C++ (hence the name).

Whatever C related updates you might see, are the ones required by the ANSI C++ for C compatibility.

For C99 and C11 support there is the clang frontend to C2, the Visual C++ backend, currently taking an LLVM role at Microsoft, being shared between clang, VC++ and .NET Native.

The Visual C compiler is only for backwards compatibility. If you are looking for improved C compilation you'll be waiting a very long time. The only improvements they've made were those also required for C++.

So, how exactly do you check your compiler's behavior?

Given: 1) There are various optimizations that may kick in or not depending on UB. 2) The cleverness of the optimizer may and does(!) change between versions. 3) There's no "-warn-on-optimizer-changes" flag for any compiler.

Do you think it's realistic to compare compiler output whenever you upgrade your compiler? Do you think it's realistic for 1M SLoC projects? If not, do you take a statistical approach with random sampling? If so, I'd be very interested. If not... what exactly are you sure of and how?

> is of the opinion that you should check your compiler's behavior rather than read standards

Do both.

Disassemble your compiler's output to see what horrors the optimizer inflicted on your code to "break" it. Read the standard to understand why it thought it was an okay optimization to do in your case, to avoid running afoul of the same mistake again. Ask what other optimizations are enabled by this undefined behavior, the better to spot them when the code your coworker's former boss's roomate, who interned for a month or two, invokes undefined behavior.

Repeat.

(To be fair, I haven't read the standards cover to cover - perhaps I should, I'm still occasionally learning about new and exciting forms of UB...)

> you should check your compiler's behavior

There is one thing I have noticed in gcc and clang which is done against the standards: The sign of bit sized int in a struct.

eg: In a struct with member 'int a:8', The standard says that 'a' can be signed or unsigned (based on the machines default of the sign of char). But In gcc and clang, this is signed always regardless of the machine.

Similarly, msvc guarantees signed overflow, it's not UB when targeting only that compiler. And gcc/clang have a compiler switch to do the same, I believe. Those things are probably what they meant when they said to look at what the compiler does.

> eg: In a struct with member 'int a:8', The standard says that 'a' can be signed or unsigned (based on the machines default of the sign of char). But In gcc and clang, this is signed always regardless of the machine.

Wait, are you saying gcc/clang's behaviour is permitted by the standard, or not?

> Wait, are you saying gcc/clang's behaviour is permitted by the standard, or not?

It's permitted. The same way as a char can be signed or unsigned: Implementation defined. But in this case, it's always signed regardless of the architecture (against how a char is handled).

From c99 draft (n1570), J.3.9 (Implementation-defined behavior): Whether a ‘‘plain’’ int bit-field is treated as a signed int bit-field or as an unsigned int bit-field (6.7.2, 6.7.2.1).

But say, another compiler (ARM compiler) treats this differently [0]: Untill version 5 of the compiler, such an int was unsigned by default. Later versions defaulted to signed (as do gcc/clang).

[0] http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc....

For the record, I disagree with this. You only need to check the implementation if you're straying from the standard or the compiler is. Neither should be... standard.

Exactly. I immediately thought "implementation specific" on the first question but then didn't see that as an option. By the second question, I had caught on that 'D' was the intended answer throughout.

Minor point: sizeof(char) was defined to be 1 even in the original C89. "char" has always been a misspelling of "byte" in C.

sizeof(char) has to be 1, but char doesn't have to be 1 byte in size. It's more correct to say that sizeof(foo) returns a result relative to sizeof(char).

> but char doesn't have to be 1 byte in size

It is, but not required to be an octect. Quoting from the standard (n1570, 3.6):

byte ==== addressable unit of data storage large enough to hold any member of the basic character set of the execution environment

Later in a comment (page 44 in pdf, note 49): A byte contains CHAR_BIT bits

char doesn't have to be 8 bits in size, true, but C basically doesn't give you any way to address memory smaller than a char.

> C basically doesn't give you any way to address memory smaller than a char

It does, via bit-fields.

Or via the bitwise operators. Still, it is not possible to take the address of a bitfield.

The difference may be 100% semantics, but when I am using a bit field, I am addressing the particular bits I am interested in via a name, and when I am shifting and masking bits, I am directly manipulating a piece of memory until its value represents the bits that I am interested in.

I prefer to think of "byte" as unsigned. Although I've never quite figured out why char is signed on most implementations, it would most naturally be unsigned too.

I suspect because signed has more undefined behaviour. C compilers will always choose the option that gives them most optimization freedom since that's what they're judged on.

What optimization opportunities does it give?

Add 128 to a signed char and the compiler is free to assume it is zero/false (because undefined behavior) OR assume the value is always greater than 127 (because undefined behavior). Or if it compiles it into machine code, the result may depend on register width since it may or may not store it back into memory. Resulting in a value either larger than 127 or mod 128 depending on register pressure since the compiler isn't obligated to AND 0xFF because Undefined Behavior.

> Add 128 to a signed char and the compiler is free to assume it is zero/false

To be honest, I hope compilers don't do such things. I would vastly prefer to see it run Tower of Hanoi simulation in Emacs at this point.

But evading bound-checking of 8bit math done in 32bit registers is totally reasonable (by the standard of usual UB optimizations), thanks.

>Actually, since C99, sizeof char is defined to be 1 (byte). The number of bits in a byte is implementation-defined.

(edit: i misread, it's always 1) To be pedantic, a size of char is defined by CHAR_BIT in limits.h. As well as all of the min and max values and lengths in bits for integers.

To add to the discussion, many high level languages don't have a hard min and max values (haskell, that is popular in these woods, i see has minBound/maxBound).

From the "So you think you know C?" test i answered... probably correct for the "standard" x86 implementation, while knowing that they are implementation specific (for 4 i'm not sure). The authors "Don't know" is his own, i would have answered the questions as "It depends", but that was not offered.

In the end i agree with the general notion here, that this is ugly C.

PS Integer wrapping is undefined as well, so "good" C code should work around it. (i miss this from assembly)

> To be pedantic, a size of char is defined by CHAR_BIT in limits.h. As well as all of the min and max values and lengths in bits for integers.

To be even more pedantic: it's the number of bits in a char that is defined by CHAR_BIT. However, sizeof returns the number of chars needed to store a type, where sizeof(char) is defined to be 1.

Yes, was just about to correct myself, sizeof(char) is always 1.

Nevermind me.

> PS Integer wrapping is undefined as well, so "good" C code should work around it.

Signed integer overflow is undefined. Unsigned integer overflow is defined.

> Actually, since C99, sizeof char is defined to be 1 (byte). The number of bits in a byte is implementation-defined.

Ever since, reading introductions to C that say things like "a char is usually one byte in size" make me cringe. I never bothered to check, though, what C89/C90 had to say on this.

Is this true? I thought the C99 standard just specified that the guaranteed minimum size is 1 byte. I've been using CHAR_BIT to define this.

sizeof(char) is indeed 1. It's the CHAR_BIT that fluctuates (and we have a Cray compiler to prove it).

Note that POSIX defines CHAR_BIT as 8.

sizeof(char) is inherently 1 because sizeof()'s unit is chars, not bits or bytes.

Better yet, write a test program and check the answer yourself.

yeah

I guess you don't spend too much time looking at the typical C code written on most average enterprise shops.

Back when I used to code C professionally on early 2000's, these type of issues were quite common.

Specially since regardless of what one reads on HN, or hears at most conferences, most of those companies, which tend to see IT as a cost center, usually don't employ any processes regarding code reviews, static analysis or unit tests.

Which in languages like C this means an heavy price when things blow up, or the way things are going with IoT.

So given the increase in CVE entries, this still matters a lot.

I think that this might be true in the early 2000's, I hope that we have moved on significantly by now. I know that the automotive folks are getting a lot better at code review, static analysis and unit testing.

If you have time, enjoy this talk by Herb Sutter at CppCon 2015.

CppCon 2015: Herb Sutter "Writing Good C++14... By Default"

https://www.youtube.com/watch?v=hEx5DNLWGg

When he asks the audience how many use such type of tools regularity, he gets around 1% as result.

I see this with the other languages as well, such practices are still seen as extra costs, shuffled to the end of the projects.

Unless the customer has software development as core business, there the attitude is quite different.

Yeah, and keep in mind that the developers who even attend CppCon 2015 are probably in the top 10% (at least) in terms of interest in these types of things.

Very sad.

It might be that they're in the top 10% of interest in ramping up in these types of things. I'm sure I'm not alone in having little interest in attending a technical conference that mostly covers things I already know - I'm there either to learn, in which case I want new content - not rehashed content - or I'm there for other reasons. Low rates of already using these tools, then, may make a certain amount of sense. If they were already putting them to practice, they may be less likely to attend.

Or I could be way off base - I've never attended CppCon. And I do love me some ASAN when I need it ;)

Indeed. If you contrast it against other engineering disciplines, even a closely related one like hardware engineering or FPGA engineering, the mindset is totally different. The amount of checks, reviews and many other practices is astounding.

... which is even more surprising given that many of these practices (in software) are almost-free. Granted, there aren't compile-time guarantees for most of these things, but with ASAN/UBSAN/etc. you'll certainly find the top 90% of your problems pretty fast.

In my experience diligent programming and going to conferences (or giving talks at conferences) are inversely correlated. Many conference pundits do little to no actual programming in the first place.

I'm not sure exactly what you're driving at, but for conferences like CppCon (and similar), the driving factor is usually technical interest and not just a way to waste time away from work (while drinking every night or whatever[1]).

So... could you explain what you're insinuating so I could respond to that instead of just guessing?

[1] Not sure about CppCon, but all "technical" conferences I've personally been to did have a lot of late-night drinking, but that was secondary to the technical stuff.

Thanks for the tip, your link is missing an A at the end though...

https://www.youtube.com/watch?v=hEx5DNLWGgA

I'm intimately familiar with an in-production, business-critical code base and there's not a single best-practice it doesn't breach. If the compiler, on it's most lenient setting, allows it, then it has been done. It's fortunately not in C, but it tries to compensate by being pretty big.

"I guess you don't spend too much time looking at the typical C code written on most average enterprise shops"

The list of qualifiers and disclaimers in that sentence made me dizzier than taking that quiz on dark corners of C.

As non English native, I gladly accept corrections to improve it....

Perhaps "Really? Most C I've seen is actually that bad."

Less is more.

However, I think that your original sentence is fine too. GP is being a bit pedantic.

Fairly new to HN. GP? Best guess is grand-poster?

Nailed it.

I'm non-native English speaker/writer too. The qualifiers and disclaimers that I'm objecting to are not grammatically wrong. The point I'm making is that your statement loses all force due to the style.

The problem with all these language lawyets is thay they present themselves as the keepers of the Truth and if you can't cite the standard for every line of code you write, you "don't really know the language." While in reality, language lawyering is an obscure hobby, almost orthogonal to real world programming.

Except in this case the OP presented a situation where it really made an actual difference.

Oh, but I'm sure that someone somewhere once ran into each of these things - maybe that guy who once ported some SMS messaging platform from the IBM mainframe to Solaris and had to use some specific commercial compiler, or the other one who wrote some software that had to work on every version of Linux from the 1.0 kernels up and had to link to specific binary libraries so it had to work on versions of gcc spanning 15 years.

What I'm saying is that knowing these minute details doesn't make one a proficient developer with the language, nor does getting every one of this list wrong a bad developer. Sure, with 10 years of experience most people would run into one or two on the list and know about it; still doesn't make the premise that not knowing or caring about trivia competitions makes one not 'know' the language.

The amount of exploits caused by such issues, is not as irrelevant as you hint at, after all if Apple developers get it wrong [1], what about all those that work in boring enterprises 9-5 and don't care one second about mundane stuff like static analyzers, warnings as errors, unit tests, code reviews.

The moment one needs to work with more than one compiler, including different versions of the same compiler, one needs to become a language lawyer to keep some sanity.

[1] I know the source of error wasn't related to these exact questions.

Or you could just not worry about it and assume the language is well-defined with precise methodologies grounded on common sense like Ada.

https://gcc.gnu.org/onlinedocs/gnat_rm/Value_005fSize-and-Ob...

On second thought, who cares about reading its for squares and nerds. I would rather spend 2 hours Google searching what is an octet and finding out how enums are represented... :p

Yes, and corollary: you'll find people that have memorized the C standard and can answer the more subtle questions which are actually unable to use the language to create a decently sized program.

Is the slanted apostrophe symbol ( ‘ ) that he uses in #3 even treated the same as the normal single quote ( ' ) in C? Note that it's not even a backtick ( ` ), it's a third symbol that's not on a keyboard.

That may be a typo. Macs have a feature that automatically converts ' and " into ‘ and “ called "smart quotes." It's innocuous most of the time and maybe even welcome, but can be a real pain when trying to share a command with someone (who is probably going to copy and paste it, receive an error and not know why).

Microsoft Office has this "feature", too.

LaTeX (and probably TeX alone, but I never use it without LaTeX) supports something similar, but AFAICT only in appropriate contexts (matching pairs of `` and '' are converted to the appropriate quote, as are ` and ', but they don't seem to affect "typewriter text", so the smart quotes stay out of my code snippets). Or maybe I just haven't noticed LaTeX munging all over my document.

Actually my experience with quoting in LaTeX has not been fun.

Fun fact: PowerShell recognizes smart quotes just like normal quotes, so you can delimit a string with “, ”, or " if you want. Same with en-dash and hyphen-minus.

One might argue, though, whether it's really a good thing to be resilient against crappy WordPress blogs, since it encourages running copy-pasted code from the web. But I guess that would happen anyway, regardless of language designer intentions and a little frustration with copy-pasting won't cause people to actually read the code they're running.

This is a good example of where Apple should have followed the KISS principal.

To be fair, as a Mac user I've only ever encountered that when using some rich text editors, but primarily word processors where your literal text isn't what you want (WYSIWYG editors). When dealing with plain text, even in browser comment blocks, it's always left my ' and " characters alone.

I think I've run across that in the wordpress editor as well.

Right, and, again, it's likely what you want 99% of the time (unless you're copy/pasting more code examples than prose). I'd be surprised if this isn't typical behavior of many blogging platforms.

After reading some of the comments below the article, I think that takeaway is that C is less portable than you may believe.

No not really, you just have to program it in a portable way hence knowing this stuff is important. Big difference.

Well, this is what I told, but I agree, this wording is more understandable.

Reply on news.ycombinator.com