Adding strlcpy() to glibc | Hacker News Reader

Adding strlcpy() to glibc

lwn.net

67 pointscorbet51 comments

Deep in the comments, 'slibc' is mentioned. I hadn't known about it, but this library provides str..._s() implementations of all the standard str...() functions as defined in Annex K of the C11 standard (which I also hadn't known about).

https://code.google.com/p/slibc/

At a glance, this seems like a better solution than any of alternatives: require the buffer length to always be specified, and call a user controlled handler instead of allowing an overflow. This handler defaults to abort().

While there are fine arguments to be made for using a real string library like bstring, are there downsides to the str_s approach compared to using the str, strp, or strl family of functions?

One issue with the str..._s() functions is that the invalid-parameter handler is a global setting - all code that calls the slibc functions and hits an invalid parameter problem will call the handler. This could cause anything from your application code to some random library you never heard of to make your program abort().

I personally don't think this is a big problem, as the kinds of issues that would cause the invalid parameter handler to be called usually are not recoverable no matter where they originate from.

But I can definitely see why the loss of control is technically troubling - those random libraries could be calling set_constraint_handler_s() and thus causing your code to keep running and not abort(), forcing you to rely on your second defense of checking the return values from all the functions.

(There's also a potential political issue with str..._s(), but I won't go into that one.)

Annex K is so useless that last time I checked not even Microsoft, the sponsor, implemented it completely.

Also, if you dig into the Annex K API there are quite a few problems with it, both technical and practical. It was half-baked. Some parts are salvageable, but really the API as a whole should be trashed.

strlcpy is useful and prevalent. Let's not let perfect be the enemy of good.

Trying to solve string management problems in C by getting everybody to agree on a "safe", comprehensive API is misguided and a waste of time.

I usually use strncpy and enforce null at end, that virtually is the same that does a strlcpy.

But if you know how long is the source and destiny buffers (and if you are using str[nl]cpy, probably you know it), you could use memcpy and get a much more faster copy.

strncpy() zero-fills the array. Which seems overly wasteful.

    char buffer[1 << 24}]
    strncpy(buffer, "a", sizeof(buffer));

will set all of the array to zero, instead of just copying two bytes.

And as you noted, it won't NUL terminate the string. Since strncpy() doesn't return C strings, it's not really a "C string" function. It's a horrible, inefficient bastardization which should be used by no one.

Meanwhile, strlcpy() returns the length of the source. So if you just want to copy a part of the source, or you're not sure if the source is terminated past the length you want to copy, it's either wasteful or dangerous.

Those are completely made-up objections. strlcpy also fails to fix me coffee in the morning, ergo it should be avoided.

    size_t len = (src_size < dst_size) ? src_size : dst_size;
    memcpy (dst, src, len); // or memmove
    dst[len-1] = 0;

So not fills unecesary 0's and it's much more faster that any str?cpy

You rarely want to use strncpy. Keep in mind:

1) char buf[1024]; strncpy(buf, "hello", sizeof buf);

now fills in 5 characters, and 1019 0 bytes in the buffer. (i.e. you're writing 1018 nul bytes that's usually not needed)

2) char buf[5]; strncpy(buf, "hello", sizeof buf);

results in buf NOT being a string. (it isn't nul terminated). So with most use of strncpy, you'll have to also do

    buf[sizeof buf - 1] = 0;

Safe string handling in C can be done, but not with char*.

PHP's Zend Engine has safely-handled strings, for example, but we do that by reference-counting them and having an explicit length.

My understanding is that strlcpy() exists to update existing programs that use NULL-terminated char arrays as strings. New programs should probably use some other more sophisticated implementation of strings, such as Git's strbuf:

https://github.com/git/git/blob/master/strbuf.c

https://www.kernel.org/pub/software/scm/git/docs/technical/a...

SDS[0] from the creator of Redis is also useful. It doesn't do reference counting, but is helpful for many other string manipulation issues.

[0] https://github.com/antirez/sds

The clever implementation allowing it to function like any other char* and that makes indexing work is what really sets it apart from the alternatives.

Hopefully it will be added to the standard C library so it will be everywhere - eg Microsoft, Apple, etc.

That would occur in something C11+, but Microsoft has yet to implement even C99.

They haven been very clear about this, from Herb Sutter's blog, Connect message postings and presentations at BUILD.

Somehow, the message seems to keep getting lost.

Visual C++ is a C++ compiler, that is the language that is relevant to keep supporting for systems programming in Windows. Time to move forward.

As of Windows 8 it is now possible to use C++ in DDK and kernel team is making the code compile C++ compatible.

C89 support is good enough for backwards compatibility and what has been added from C99 was what is required from the C++11 standard.

If only C++ were actually "forward" from C, it would be reasonable to tell people to move forward.

I have been on the C++ camp since 1993, after having to deal with C for one year, and a couple of Turbo Pascal years behind me.

C was already feeling primitive in 1993 when compared with languages like Turbo Pascal and C++ offered so much more in terms of type safety and modular programming.

So yes, it has been forward from C since a few decades already.

Actually, about 6 months ago Microsoft did an about face. They're now approaching near complete C99 support, adding features deliberately rejected by C++, such as designated initializers and compound literals.

In fact, compound literal syntax is probably incompatible with C++, because similar syntax in C++ requires entirely different object lifetimes--expression scope in C++, block scope in C99.

Actually, I think the target may be C11, as I doubt they'll add VLAs.

Is this really true? Last I saw, they were adding some the of C99 features (because the C++11/14 spec actually required some of it), but not going all the way. And it was unclear if they were only focusing on the language or the standard library or both.

(I'm willing to skip C99 for C11 because am sick of dancing around Visual Studio for compatibility. Clang for Visual Studio is looking really appealing to me.)

Why they have the only world's C++ compiler that can't handle C baffles me. What did they screw up so badly in their implementation that every other C/C++ compiler got right?

> Why they have the only world's C++ compiler that can't handle C baffles me.

Why? C++ was born at AT&T, the same place C was born.

Given its design to be as compatible with C as possible, it was quite natural for C compiler vendors to bundle a C++ compiler with their C compiler.

There isn't any requirement in ANSI/ISO C++ that a C++ compiler vendor is obliged to support C.

Microsoft's decision is not technical, but political.

From their point of view, if you want to write C style code, there isn't nothing in C++ that prevents it.

On the other hand, C cannot provide at the language level the mechanisms available in C++ to write safe code.

Not to mention that Windows APIs are mostly COM based since the Windows XP, anyway.

If you want C on Windows, or to have fun using COM or WinRT from C, there are plenty of other compilers available.

Both Symbian and BeOS were C++ based OS. I for one, enjoy seeing Windows going into the same direction.

See my post down thread. Or just read these pages

http://msdn.microsoft.com/en-us/library/hh409293.aspx

http://blogs.msdn.com/b/vcblog/archive/2013/07/19/c99-librar...

http://blogs.msdn.com/b/vcblog/archive/2014/06/03/visual-stu...

> They're now approaching near complete C99 support, adding features deliberately rejected by C++, such as designated initializers and compound literals.

No they did not.

Herb Sutter was very explicit about this.

- C99 language features as required by the C++11/14 standards

- compound literal syntax just to allow certain well known open source project to be ported to Windows, mainly game related projects

Nothing else is planned as shown on his slides.

http://channel9.msdn.com/Events/Visual-Studio/Launch-2013/VS...

Edit: Added link to Visual Studio 2013 launch presentation from Herb Sutter.

I know what he said. But I think he's understating the changes.

Visual Studio 2013 supports both compound literals and designated initializers. That required core language changes, and are arguably the two biggest language changes in C99 from a syntax standpoint. Both were rejected by C++ and will never appear in C++, so they're entirely unnecessary.

See http://msdn.microsoft.com/en-us/library/hh409293.aspx

And Visual Studio 2014 added most of the missing pieces, including a standards compliant snprintf. See http://blogs.msdn.com/b/vcblog/archive/2014/06/03/visual-stu...

Heck, they even met C99 half-way on the restrict qualifier by adding a __restrict keyword. C++ also rejected restrict, FWIW.

Those are all comprehensive, non-trivial changes, and most are irrelevant to C++. Microsoft can talk it down however they want, but the new features speak for themselves.

I don't even use Visual Studio. Or Windows for that matter. But I value standards compliance and portability, and I was happy to see Microsoft change their mind on this. They did it rather quietly ("No, we aren't targeting C anymore, but here are almost all the new C features which we said we'd never support in a million years."), perhaps because they so adamantly refused to support it before.

Yes, the reason they finally capitulated is because there was simply too much open source C99 software in the wild, and it was becoming a pain point for Windows developers not being able to use that software with their preferred development environment--Visual Studio. But at this point they've implemented almost all the features that actually matter.

Features like VLAs and _Complex are rarely used, so I doubt Visual Studio will get that. But both are optional in C11 anyhow. With little effort--one of the biggest changes being _Generic--Visual Studio could become C11 compliant, at least in practice.

Thanks for pointing this out, it seems I missed some posts and need revise my statements.

Still, I was a supporter of their decision, given the added safety and expressiveness that C++ has over C.

System programming languages only get replaced when an OS vendor decides to push them aside from their SDKs.

Perhaps the message keeps getting lost because it is a bad message. They have made the wrong decision, to such a degree that people have a hard time understanding just how wrong their decision is, and keep naively assuming that this is some kind of schedule/resource failure and not a decision to do the wrong thing.

Except many of us on the C++ side think of it as a good decision.

But as someone pointed out to me, things seem to have changed a bit more than I thought.

OS X inherited it from FreeBSD. VC++ is a c++ compiler and any c support is incidental. I suspect this won't be included in c++17.

C11 added a different set of safe string functions, although as an optional standard annex (Annex K). They have _s suffixes, e.g. strcpy_s(), and somewhat different behavior. It seems unlikely that ISO would also add strlcpy() at this point, given that one set of "safe" functions is already (sort of) in the standard. But maybe POSIX could standardize it.

Which came directly from MS.

Except that it was a collaborative effort that involved far more than just Microsoft, such as CERT, and the WG14 committee as a whole...

Too bad Microsoft Visual Studio refuses to get with the times and update their antiquated compiler from C89. They are now 2 standards and 25 years behind.

VS has a C++ compiler, not a C compiler. There is no getting with the times. That's like advocating that the Java compiler should get with the times and start compiling C++11. C and C++ are quite different languages, despite looking superficially the same. Think of it as a DVCS history that evolved into two very different branches at one point (C89) with the occasional feature branch of C merged into C++.

Visual Studio is the only major C++ compiler that is inept at handling C. This is not unreasonable to expect proper C support and there is something terribly broken with Visual Studio in that they can't. Even Stroustrup has called for better compatibility with C.

I don't see any particular reason you'd expect a C++ compiler to handle C, except the very specific part of C (largely C89) that is a proper subset of the C++ standards. They're very different languages at this point.

gcc's C++ compiler doesn't handle C past C89 either; you can't use C99 features when compiling with g++. There is a separate C compiler that's part of the GNU Compiler Collection, but then there's also a Fortran and a Go compiler: gcc includes a lot of languages. MSVC++ doesn't; it's just a C++ compiler, not a Fortran or Java or C compiler. Linux norms are a bit more polyglot, so gcc is more polyglot than MSVC++ is.

I personally like C, so it'd be nice if Microsoft shipped a C compiler in addition to a C++ compiler, but they've chosen not to. As a result, the "C" you can write is precisely the C that is also valid C++, and you can't use any features that aren't valid C++. If you really want there to be a language "C/C++" again that all C++ compilers can be expected to handle, you'd have to convince the C++ standards committee to add C99 or C11 features to the next C++1x.

strcpy_s() doesn't appear to be in glibc. At least "man strcpy_s" give nothing on a Linux box.

Yes, glibc has so far chosen not to implement Annex K.

Microsoft has its own (ugly) safe string APIs like StringCchCatEx():

http://msdn.microsoft.com/en-us/library/ms647466.aspx

http://msdn.microsoft.com/en-us/library/td1esda9.aspx

I agree with drepper on this; it's a solution in search of a problem. You should either know WTF you can accept or use a higher level construct that can resize. Silent truncation seems bad - truncation attacks are a real attack vector that SSL, for example, tries to prevent.

strlcpy is designed to make it easy to detect truncation. You get back the buffer size you need to store the result. If this size is >= the size of the buffer, you truncated.

You can just keep going, building up your string using strlcpy/strlcat as you go, and check for truncation (whether from this call, or a previous one) using the return value whenever it's convenient. Depending on what sort of programs you write, truncation might even not be a problem anywhere from some to virtually all of the time, so in those cases you can just let it happen and the code ends up even simpler.

(When I first discovered strlcpy/strlcat I went and changed a bunch of string-fiddling code to use them and it was really amazing how much simpler everything became. Virtually all of my bounds checks could go away, leaving just the string stuff. Much nicer? Well, I wouldn't go that far. But certainly fewer ways for it to go wrong.)

But they don't silently truncate, so your entire argument is bogus.

I am curious why strlcpy() was not modified to check the length going in and out to check for truncation? I am not a C/C++ guy but that is one question I had when he said that was one of the gripes for the function.

Because its return value already tells you if truncation occurred.

> The primary complaint about strlcpy() is that it gives no indication of whether the copied string was truncated or not.

So how hard would it be to add a return value indicating this?

If you did, it wouldn't be strlcpy() anymore. The point is compatibility with existing users and expectations.

Can't do that -- strlcpy already has a defined return value. To change it would break vast amounts of existing code.

The author of the article got this wrong. The return value is the number of characters it tried to write so it's easy enough to compare with the buffer size to see if truncation occurred.

The correct approach is to use the strxxx_s family of functions. This is also in the most recent C standard.

the strxxx_s family blow up your program if they encounter something bad.

What most people want is some simple functions that that can tell you if something would have gone bad. (e.g. truncate the copied string and tell you if it was truncated)

Good. Attempting to copy or concatenate a too long string is indicative of a logic bug. If it really bothers you, use _set_invalid_parameter_handler to change the behavior to something you can recover from.

Reply on news.ycombinator.com