Hacker News Reader

Comment by AnthonyMouse | Hacker News Reader

> In C you could have made some 'safe buffer' struct together with some safe set of functions. But people didn't do that.

Exactly.

  struct buffer
  {
	void* data;
	size_t len;
  };

  void smemcpy(buffer* dst, buffer* src, size_t n)
  {
	if(dst->len < n || src->len < n)
		abort();
	memcpy(dst->data, src->data, n);
  }

I mean how hard is that, honestly?

Buffer bugs in C don't come from the language, they come from the tendency of C programmers to value performance highly. "I just checked that the buffer is big enough, I don't need memcpy to waste cycles checking it again. I don't need that length variable sucking up memory when I know how big the buffer is." Except when you do.

View on HN

It's hard to define semantics. Here, you have potentially copied less than dst->len bytes into dst->data. Is the remaining data (up to dst->len) still valid, is it garbage, should you have adjusted dst->len, etc?

When does it even make sense to partially mix byte-level representations of some data?

The buffer abstraction is necessary, but far from sufficient to build security-critical code. To do so, you also need to code in semantics somewhere, and buffers know zilch about the meaning of the data.

Check again. Hint: "abort()" is not "return" -- that code doesn't prevent you from having to know how large the buffer is, it just shoots your program in the head if you don't. Which is basically the same semantics Java et al have by throwing an out of bounds exception when you don't catch it.

Incidentally, this is another reason that the language can't save you from yourself. Program termination is sometimes totally unreasonable, e.g. in real-time settings. And it's unreasonable regardless of the language. When an airplane falls out of the sky, you don't get to declare victory just because the reason it did was an uncaught out of bounds exception rather than a buffer overrun. In that context the programmer must explicitly handle the condition that the buffer is too small, regardless of the language, because "abnormal program termination" is not an acceptable outcome.

Your code checks whether dst->len < n, but not whether dst->len > n; i.e., after copying 5 bytes to dst, there may be another 10 bytes of garbage in dst. This 'garbage' might as well be a password from a string that was just free'd. A fix would be to set dst->len to n after this operation.

OK, I see what you're getting at. There is a difference between the end of the buffer and the end of the data. But copying some bytes into a buffer doesn't imply that you intend to discard the rest of the buffer. It's quite common to want to replace a header or some other subsection of a buffer and have the rest of the buffer remain unmodified.

Moreover, the purpose of the above is to achieve the level of bounds checking that you get with the likes of Java. If you want to go beyond that then you need something more complicated -- maybe add separate variables to the buffer struct for buffer_len and data_len and then have different memcpy functions based on whether you want existing data in the buffer to be truncated. But there comes a point at which further complexity produces more confusion than safety.

Ideally, you would have a parsing and validation routine which fills in a struct. Afterwards, you manipulate only the struct instead of raw byte representation.

> But there comes a point at which further complexity produces more confusion than safety.

Buffer is simply not the right abstraction for complex protocols, regardless of how "complex" operations you define.

> Ideally, you would have a parsing and validation routine which fills in a struct.

The parsing and validation routine is where you need the buffer abstraction. If you make a mistake there then a copy function that won't let you read or write past the end of the buffer will mitigate the damage.

    if(obj != null) {
        obj.stuff = ...
    }

I mean how hard is that exactly? yet Java applications have null pointer exceptions all time

Nothing wrong with null pointer exceptions - they are specific checks to see if the pointer is valid and are a good thing. If you see one, it means the programmer made a mistake and the runtime caught it for him.

There are also tools to help with it in the form of @NotNull, @Nullable, etc.

It's a completely different issue compared to actually accessing memory in an undefined way as you can in C. The bounds check you are replying to is manually creating something like that exception through the use of abort() while in java it is done always at the cost of some performance in exchange for safety.

Consider the following: some languages don't have null. When you need nullable what you really want is an Option/Maybe type. No null pointer exceptions, ever! It just won't compile until you fix the bug.

Or, more generally, I think languages should have union and intersection types - "(IFoo && IBar) || (IBaz)", etc. (I also think languages should have arbitrary Boolean expressions as types, where the compiler emits a warning and a runtime check if the compiler cannot prove either if the condition will be always met or it won't be always met, but that is another matter.)

Null is just a built-in final/sealed type(enum?) with a single value and no operations defined, like boolean is a type(enum?) with two values.

That, combined with proper type inference, would alleviate a large chunk of the boilerplate associated with statically typed languages in general, and Java in particular. (For how much Java is pushed as moving errors to compile time, there are an awful lot of things that fail only at run time that could be checked at compile time...)

Same thing with checked exceptions. A checked exception is really just an option - return foo or failure.

Even when Java fails at runtime, because it forces garbage collection for memory management, buffer overflows and dangling pointers can't really happen - yes, if you're not careful, the process can still have unexpected behavior, or it might crash. Actually, the Heartbleed bug wouldn't have been such an issue if OpenSSL wouldn't use its own wrapper around malloc/free (at the very least, the malloc/free underneath would be a variable in that equation), e.g. http://article.gmane.org/gmane.os.openbsd.misc/211963

So while I love strong and static type systems, the issue at play here is memory safety - the very reason for why people still use C/C++ is the same reason for why Heartbleed happened.

You're assuming that the JVM is secure. But yes, good point.

Although I'd argue that a truly strong type system should also catch problems with pointers. The compiler should try to prove that any pointer access/arithmetic can or cannot cause undefined behavior - if it could cause undefined behavior, emit a compiler error, if it cannot prove that it won't cause undefined behavior, emit a warning and insert a runtime check.

(Still have the underlying "unsafe" functions. But group them into part of the standard library as an "unsafe" package.)

That's what Rust does. Its type system prevents all memory and concurrency errors. But to be able to have a self-hosting compiler (and to be able to write an OS in it) there's also `unsafe` blocks.

If you use @NotNull etc in Java, then it also won't compile until you fix the bug.

They come from the C language allowing it.

All Turing complete languages allow you to do anything all Turing complete languages allow you to do. You can allocate a huge block of memory in Java or C# and then write your own malloc to allocate buffers from subsets of it. In some cases that may have better performance. It also allows you to have buffer overruns. You can run right off the end of one buffer and into the next and the language won't stop you because it only knows that you haven't reached the end of the array it allocated.

Do Turing machines have a concept of "undefined behavior" in the same way that C does? AFAICT all behavior of a Turing machine is defined -- sure you might not calculate what you want, but that doesn't admit arbitrary behvaior outside of the semantics of the machine as in the C language.

"Undefined behavior" means "undefined by the language specification", but it is well defined by the compiler implementation. In other words, you never use the C language, always an implementation.

You're thinking of implementation-defined behaviour. Undefined behaviour is not well-defined; the compiler does not have to implement it consistently or sanely, and certainly gcc won't.

> Do Turing machines have a concept of "undefined behavior" in the same way that C does?

Sure they do. For example, nothing in the Turing machine specification tells you how fast they are. Whether a given program will run on a Turing machine in a given amount of wall clock time is undefined.

Here's a real example with real machines. Suppose you have a program with two threads. The first thread computes data and puts it on a queue for the second thread, the second thread writes it to the filesystem. When the compute thread has computed all of the data, it sleeps for 10 seconds and then terminates the program without checking whether the write thread has actually completed writing all the data to the filesystem.

That results in undefined behavior in every real language I've ever heard of. Whether the data is all written depends on how fast the CPU is, how fast the filesystem is, what other programs are scheduled on the machine that compete for CPU and filesystem access at what priority, etc. Languages don't define any of those things so you have undefined behavior.

That's a different abstraction than "Turing Machine". Turing machines do not have '"undefined behavior" in the same way that C does' - it would look like a "in state 3, if you read a 0, move to an arbitrary state, write arbitrary symbols to arbitrary locations on the tape, move head to an arbitrary position on the tape, and possibly restructure the state machine in arbitrary ways."

Turing machines have undefined behavior in exactly the same way that C does. What do you suppose happens if a Turing machine reads from a location on the tape that has never been written? The value the machine observes in that circumstance is not required to be well-defined in order for the machine to be able to do all computations required of a Turing machine. Moreover, the more you add features of actual machines like parallelism and timers, the more opportunities for undefined behavior you introduce. What do you suppose a multiprocessor Turing machine does if you write concurrently to the tape without locking?

"What do you suppose happens if a Turing machine reads from a location on the tape that has never been written?"

It reads the default symbol (see e.g. the description of a TM on Wikipedia). It's semantics are well-defined.

The semantics of TMs are well-defined. The semantics of C's "undefined behavior" is not well-defined.

I should have said "blank symbol" not "default symbol".

Clarification: all Turing complete languages allow you to do any _computation_ all Turing complete languages allow you to do.

"All Turing complete languages allow you to do anything all Turing complete languages allow you to do."

This is false. All Turing Complete languages allow you to compute anything other Turing Complete languages allow you to compute, but that says nothing about encoding, or about interaction with the world. For an obvious encoding example, imagine you have a turning machine that can write only the symbols 0 and 2 to its tape. This is obviously Turing complete - it's isomorphic to a TM that writes 0 and 1, the TM doesn't care how the symbols are labelled. Now, if we ask it to encode 7 it has no trouble doing that in some encoding, but that can't ever be "111" on the tape.

The Python language allows constructs such as socket.write(my_private_key). Do you think that just because Python makes it possible to write your private key to a socket that it must necessarily happen?

> Buffer bugs in C don't come from the language, they come from the tendency of C programmers to value performance highly.

You have to use other people's code / language constructs which are underspecified and/or you don't have time to fully understand. Throw in versioning to the equation and now the problem is exponentially worse. This is why C code will always be full of memory vulns. Heartbleed is a very oversimplified example of the problems in C.

Reply on news.ycombinator.com