Hacker News Reader

Comment by epx | Hacker News Reader

One thing that C should have is an automatic string type (more like a byte[] than a String) that knows its own size. A huge amount of programming bugs arise from the lack of strings, and I can't see the reason why they could not be supported even in small systems. Delphi/Object Pascal had something like 3 or 5 function pointers that support the string type (create, free, resize, things like that).

View on HN

The funny thing is that the memory allocator typically knows the size of the string.

You call free(foo) not free(foo, 12) or free(foo, strlen(foo));

But there's no standard way to ask for this size. Why doesn't strcpy just check with the allocator about how much memory is available and refuse to write past that.

Not every string is allocated from the heap. There is no way to know if a string passed to a function, for example, came from the heap or not.

That's true, although there's no reason the stack allocator couldn't record this information somewhere as well.

And if the memory allocator doesn't know about the string then you could just revert to the existing behaviour so you're no worse off.

There's more ways to allocate memory than the heap and stack.

What are you referring to? sbrk? mmap? I typically use either the heap or the stack myself.

Static memory is at the bottom of the address space, below the heap. But that would really only be relevant in the case of string literals unless I'm mistaken.

You can have a static buffer:

    static char buf[128];

and use strcpy() at any point to copy into it, so I don't think statically allocated strings must be literals.

There is a distinction between const and non-const static memory, though, in that the former can be stored in Read-Only Memory. Still, static memory is generally used for many fewer purposes than the stack or the heap, and if I understand correctly, sbrk is still working with the heap, just more manually.

That's right. Allocators use the sbrk and mmap to manage the heap. IIRC high performance allocators tend to use mmap exclusively.

Something like memory pools or manual memory management. Yes the memory pool will come from the heap or stack but the heap/stack allocator will have no idea what is going on in there.

And all of them know how much memory they've allocated.

Yes, but unless you expose at the language level an interface for all types of allocators and enforce it then a whole lot of allocation techniques like memory pools couldn't be used with strcpy or would have to use char* type of strings again.

The point is not that allocators don't know how much memory has been allocated, but that C has no idea about the underlying allocator, so it cannot ask it about the length of the string as it was proposed.

Okay but you don't have to implement a 100% solution to make 90% of existing code much safer.

How much C code is out there that only uses malloc/free? All of those programs could be made safer if the default string methods were able to prevent writing past the end of the array.

Clang offers a memory sanitizer that does a lot of extra checking. People don't ship their software with the sanitizer activated because the performance hit is nontrivial.

You're free to do as you please here. You don't have to use malloc()/free(), just implement your own LIBRARY_Alloc()/LIBRARY_Free()/LIBRARY_GetMemorySizeBytes() :)

However that doesn't mean that you shouldn't be able to either

Would be nice to be able to tell if something is allocated on the heap or stack in a portable way.

As in being able to ask if and object is thread local or not. The language and run time could support that but studiously does not.

Just like the language could allow you to directly monkey with the ABI but does not.

That something could be allocated by custom allocator like new or g_slice_alloc. You can monkey with the ABI as numerous FFI libraries do, but it is not portable by definition, unless you’re using, well, portable FFI library. Personally I can’t see why string, abi, etc libraries must be included. Include GLib++/GObject++ then and use its actually large ecosystem — what problem are you trying to solve by blaming low-level? That low-level is too low? Do you need high-level low-level that everyone will obey next morning? Check Java?

> portable FFI library

I looked into that, not available on many many platforms.

strcpy has no access to the "allocator". It doesn't know about malloc, realloc, or free.

Yes -- and that has turned out to be a bad design decision. It's somewhat convenient to use methods like strcat() to build up strings, but it also leads to very insecure code.

I think the inclusion of strdup() shows that it was a mistake to make C strings completely allocator-agnostic.

Because the programmer can do pointer arithmetic, treat non-char[] memory as char[] memory and probably other weird stuff that makes this more complicated than it sounds, I guess.

just make a type where doing pointer math and other such use cases are a compiler error?

Just use a language where types automatically know their size? I thought the point was to make existing code safer by exploiting information that's already there (e.g. the allocator knowing the size of things on the heap).

The point is that C could be made much safer to use with a few "simple" (at least seemingly) tweaks. C has great tooling support on all platforms, including embedded, so it would be a compelling advancement if the language standard were to include such a thing. The important thing would be to coerce such a string type to character arrays so it remains compatible with all libraries without casting.

Strings don’t have to fill the char array they live in. Overallocating a buffer is a common technique to avoid frequent reallocation.

We have tried to do this by adding a "null terminated string" type. Start in section 2.4 of the specification (https://github.com/Microsoft/checkedc/releases/download/v0.7...).

I've used something similar --- a null-terminated string, with a length prefix at a negative offset. Resizeable ones have an "allocated size" field before that one. This makes for strings that are both easy to manipulate and compatible with existing functions that only read a regular C string.

Basically this: https://github.com/antirez/sds

Good to see I'm not the only one who thought of that idea, although my implementation has a different order of the fields (and IMHO better, since it allows for statically allocated strings with only a length field --- a sort of subclassing.)

So in a sense implementing Pascal strings in C?

Yes. But also "abusing" C's permissiveness with pointers to get pascal strings that interop with standard functions. It get's passed around as a pointer to a char array but the length function would return the value at that pointer - sizeof(size_T). In most languages this wouldn't be possible.

So, BSTRs? They’re common in old-timey COM apis

BSTR size field is at positive offset, IIRC.

You do not RC :)

The point is to keep C small and flexible. It supports whatever kind of string implementation needed by the present environment, platform and algorithm.

That said, several “high-level” string libraries for C exist that make it safer, more efficient, etc. Some of them are even pretty clever, allocating extra space right before the C pointer to a null terminated C string that it gives you, so it can do its own bookkeeping.

The main issue is that there's already a lot of standard library support for classical strings--things that are just a pointer to a chunk of bytes, and no more; rewriting those functions would be destructive. The C standards group could add to the standard library, I suppose, but generally they are a bit reticent to do so.

It is not the case that C has a benevolent dictator with a particular vision and drive for the language. It's more like a small group of people who don't want to mess things up. And so not much changes. This is neither good nor bad to me--it simply is; and anyway, these days, you have quite a few options that do offer automatic strings that can also compile into machine code (Go, Rust I suppose?).

> rewriting those functions would be destructive

Getting rid of standard string functions would be the best thing to happen to the c language in the last 25 years.

There are two types of c programs. Those that scrupulously avoid standard string functions and brittle programs shot through with security holes.

At the point when you have to convert between string types and can no longer easily use much of libc without conversions, why not just use a different language that's already gone much farther in the name of safety, usability or other features?

In other words, if your pseudo-c candidate already suffers from most the interop problems that Rust, D, Nim and Go do, why not just use one of those and at least reap the other benefits they provide?

Anything to help legacy C code base is welcome. Hard part is knowing whether touching decades-old, tested code is worth the risk.

I mean, I guess it's the best thing if you want to wreak havoc and chaos? Which, perhaps, you do!

Your suggested dichotomy is, of course, a little bit false. But I'm sure you knew that when you wrote it down.

It's entirely possible to write secure programs in C, even with standard functions. Writing your own code does not somehow confer a level of security-consciousness that you lacked when sticking to strings.h. (It does give you a wonderful opportunity to write your own security holes that no one has discovered yet!)

I mentioned this somewhere else, but we're in a pretty good place right now with languages; we finally have really solid alternatives to C that can compile to machine code, in both Go and Rust.

> It's entirely possible to write secure programs in C, even with standard functions.

You realize that most standard string functions are outright banned by organizations that care about security. As in you're not allowed to use them not even if you pinky swear to be 'careful'

https://msdn.microsoft.com/en-us/library/bb288454.aspx

I think Fortran uses a number followed by a byte array to represent strings. This is the big argument with C which used null terminators. In retrospect, maybe Fortran was right after all.

Edit: Actually seems I confused Fortran and Pascal. Also, Fortran uses an internal length variable.

I think the failure of C to support a built in array type is exactly like Steve Jobs refusing to give up one button mice.

The reasons aren't technical or based on merit but on stubborn ideology.

Arrays are actually pointers to the first element... They get translated to pointer arithmetic: adding an offset to the first element. You can easily create a struct with one field for the size and another with the pointer to the first character

Semantically, arrays are different from pointers. They do, however, degrade into pointers in many situations, which causes that misunderstanding.

A language with an array type, e.g. Java, distinguishes between a 'null' array reference and an array of zero length. which could substitute in languages that don't have an option type; SOME([]) vs NONE

But a plain *char in C makes no such distinction.

There is a difference. An empty string in C is a non-null char* pointer pointing to a zero byte. Passing a null pointer to strlen will crash.

A zero-length array can't ever be a valid cstring at all since they have to be null terminated. I don't think that is actually relevant to the claim you are replying to, which is about arrays and pointers and not strings.

That type of thinking led to C++, a language whose contributors do not know how to stop adding features.

Reply on news.ycombinator.com