I'm not a fan of strlcpy(3)
nrk.neocities.org
nrk.neocities.org
This holds even if the total length of the data in an interaction is not known ahead of time. E.g. an audio stream can be of indeterminate length, not known when the first byte is sent over the network, but each UDP packet has a well-determined length given in the header.
The length field can be made variable-size in a rather fool-proof way [1], allowing to economically represent both tiny and huge sizes.
(Zip files, WAD files, etc have that info at the very end, but this is because a file has a well-defined end before you start appending to it; fseek(fp, 0, SEEK_END) can't miss.)
[1]: http://personal.kent.edu/~sbirch/Music_Production/MP-II/MIDI...
My current favorite string implementations are the various compact string crates for Rust. Generally, you want a string to be able to do at least three things:
- Pointer, length, capacity tuple, for heap-allocated strings, 24B on x64.
- String inlined into the 24B buffer.
- Pointer, length tuple where pointer points to rodata.
You can do any of that and the discriminator in 24B, given the healthy assumption that all strings are shorter than 2^63.
Sadly, switching costs are massive and every programming language is pretty much struck with the string they started with. Hopefully, whatever comes next can crib from smartstring or compact_str or the like.
Yet...since nobody reads the literature, we have all continued to suffer.
Haskell is _almost_ flexible enough to be able to use a different string than the one it started with.
The language itself actually is flexible enough, but many of the libraries are not.
The main thing making Haskell flexible enough is that a literal like "foo" can be statically determined to be the right string type that you want to use. (And that happens at compile time, it's not a runtime conversion.)
Imagine you are writing performance sensitive code. You want to get a substring from a string, one that is not going to live outside your hot loop. In standard C you can just reference a part of a string with a pointer offset. All standard functions will continue working and you didn’t have to make any calls outside of your loop, not to the allocator, not to memcopy, nothing.
With strings being objects that are prefixed by a header cannot do this. At a minimum you need to allocate a new header, if not the whole string. Yes that’s the safer route but also a lot less performant.
Most crucially, you can build the header string implementation on top of C strings. You cannot do the opposite.
Realistically though C strings (aka null terminated strings) are just not a great thing because of the null termination. For my money, I would prefer to just use untermianted arrays and a separate size variable, as well as wide character strings for actual display stuff. This way all the interop must include string lengths (or some other way to determine length), and all internal stuff may be just ASCII but must not leave your internal logic and never be shown to the user.
If you want your substring to terminate in the same place as the original, at a null terminator. But that sadly is almost never the case, and as many C practitioners know, references like this are often unsafe and so APIs that substring tend to copy. That's just what they have to do to pass address sanitizer and static analysis checks.
If you want arbitrary views on a null terminated string, well, it's no longer null terminated and that's just the start of your problems in C.
In languages like Rust and Go, taking a view of a string or array is safe and doesn't copy the underlying data or require an allocation. So if you are writing performance sensitive code where substrings are a major contributor to CPU cycles, best go with those language (or C++) rather than C.
Any function that does not need to modify a String takes a &str. Any function that does modify a String typically takes a String, which means they consume their input. (Because of utf-8, in-place modification is generally a pipedream.)
Also, the headers are typically allocated on stack. Rust is a lot less shy about types that are larger than a pointer living inline whereever they are used, and this is something that seems to work a lot better than the alternative.
Of course, if your string type is a struct containing a size and a pointer, you can easily have multiple substrings pointing into the same byte array.
Zig uses slices for this (and everything else except interop): a pointer and a length, with some nicely ergonomic syntax for making another one, like `slice[start..][0..length]`.
When you're building strings then you have an ArrayList, which keeps a capacity around, and maybe a pointer to an Allocator depending on what you want. It's trivial to get the slice out of this when you need it.
Doing anything useful with a string requires knowing where it is (the pointer) and how much of it you have (the length) so keeping them together in one fat pointer is just good sense. It's remarkable how much easier this is to work with than C strings.
The rare case should be possible, just not the default.
In Rust, you would make custom string handling unsafe for the bottleneck.
I think that's a pretty bad case of premature optimization, especially because the first CPUs with 57 bit support are now hitting mainstream. Just use 3 words, it's not that much extra space.
Two 64 bit values can be returned in registers on the systemv x64 abi, three get passed as a pointer to stack memory. It's an optimisation but I think it's a valid one.
57 bit address space has been coming any year now for maybe a decade, I'll worry about that when it happens.
Of course, once you're done processing and are sending it along (as in serialization, that you mention), it's not an issue.
If there are bugs with truncation in the resulting buffer, those are the program's bugs, and they existing before strlcpy(3) came into the picture.
It's not a drop-in replacement, though. Not even if you ignore the different return type.
strncpy guarantees that the buffer will be completely overwritten (filling with null chars at the end), while strlcpy will happily leave remnants of whatever was there before.
Just dropping in strlcpy wherever strncpy appears can lead to data leaks or inconsistent hashes, for example, depending on how the buffer's contents are used.
"No true Scotsman..."
A truncation bug is a hell of a lot easier to debug than memory corruption.
I think a lot of the confusion in the C string discourse comes from people thinking they should rely on the NULL termination byte for string length. You really shouldn't, and if you have to do it, you need to be extra careful to check all your assertions that it will be properly terminated. Just carry around the length, and bundle it with the pointer in a struct to pass it around when it makes sense. Not the most ergonomic, but it's C, what can ya do.
Nowadays, things like C++ string_view's and Rust str slices handle this for you automatically, but those came around much later and require more sophistication at compile time.
LOL. It does not force anything - you can mishandle source or destination buffer lengths very easily and compiler won't say anything.
I sometimes wonder what kind of disaster will have to happen to make C programmers agree on a standard buffer (i.e. pointer+size) type with mandatory runtime bounds enforcement ....
Nitpick: it’s not quite a drop-in. Prototypes of these functions are
char * strncpy(char *dst, const char *src, size_t num);
size_t strlcpy(char *dst, const char *src, size_t num);
strlcpy(dst, src, num) returns the total length of the string it tried to create (https://www.unix.com/man-page/posix/3/strlcpy/). Callers can use that to detect that the string didn’t fit the buffer and reallocate a buffer that’s long enough.
The reason for the existence of strlcpy isn’t that it is perfect, it’s that it’s the best option with good UX for integration into an existing C code base.
strlcpy is a stopgap, whack-a-mole solution for buffer overflows. It is rationalized by the reasoning that it does not make the program less wrong, while (probably) making it more secure.
When truncation matters and you have a fixed size buffer, that buffer should be large enough in order for it to be justifiable to say that someone is misusing the application. Perhaps a tester trying to break it.
Nobody’s surname needs 128+ bytes. No reasonable URL for a firmware update download needs 4096 bytes.
If truncation matters, no, it does not always make sense to accept a gig of data and be ready for more. You can impose a limit. A violation of the limit is an error, treated like a case of bad input.
E.g., its length in its native alphabet, or its length as a UTF-8 string?
$ swift
Welcome to Apple Swift version 6.0 (swiftlang-6.0.0.5.15 clang-1600.0.22.6).
Type :help for assistance.
1> "झा".count
$R0: Int = 1
2> "झा".utf8.count
$R1: Int = 6https://en.wikipedia.org/wiki/Hubert_Blaine_Wolfeschlegelste.... would beg to differ.
I'd considered base64-encoding my ripped DVD collection, and using them to store another backup copy for me.
Is there a picture of the ID page of that man's passport? Or of a driver's license or similar?
Whatever is on that is his actual name.
Adolph Blaine Charles David Earl Frederick Gerald Hubert Irvin John Kenneth Lloyd Martin Nero Oliver Paul Quincy Randolph Sherman Thomas Uncas Victor William Xerxes Yancy Zeus Wolfeschlegelsteinhausenbergerdorffwelchevoralternwarengewissenhaftschaferswessenschafewarenwohlgepflegeundsorgfaltigkeitbeschutzenvorangreifendurchihrraubgierigfeindewelchevoralternzwolfhunderttausendjahresvorandieerscheinenvonderersteerdemenschderraumschiffgenachtmittungsteinundsiebeniridiumelektrischmotorsgebrauchlichtalsseinursprungvonkraftgestartseinlangefahrthinzwischensternartigraumaufdersuchennachbarschaftdersternwelchegehabtbewohnbarplanetenkreisedrehensichundwohinderneuerassevonverstandigmenschlichkeitkonntefortpflanzenundsicherfreuenanlebenslanglichfreudeundruhemitnichteinfurchtvorangreifenvorandererintelligentgeschopfsvonhinzwischensternartigraum Sr.
tl;dr Make you bed; lie in it.
For a more extreme example, consider conventions of Japanese. Middle names do not exist in Japanese. In fact, middle names are impossible to input into the 戸籍 (family registry). Forms in Japan are designed around the assumption that each person has exactly two names. Many Europeans would be unable to input their full name in such system. In this example, it'd be unreasonable to suggest most Europeans are acting "outside of the bounds of reasonable human societal expectations".
In general, the most effective solution I've seen for handling names is to have a single name field and treat it opaquely. If you need an inflection, ask for that separately.
And he's not even the guy with the longest name, and his parents did not make up his name to spite some length restrictions.
https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-...
Also note that while the counter examples might sound extreme, in some languages each character might need 3 bytes in UTF-8, and 128/3 ~= 43 characters doesn't seem to be that outrageous.
Oh we're doing that "false things that programmers believe about the world" again! Fun! Let's consider cultures where people can have more than one surname. Ever heard about Pablo Diego José Francisco de Paula Juan Nepomuceno María de los Remedios Cipriano de la Santísima Trinidad Ruiz y Picasso? Yes, that's his full name. Of course, he didn't frequently use it, but if you make a system dealing with people's names, you'd eventually end up having to support something like that.
But that brings us to the assumption that people have just one family name with is just one word, which is very much not the case in many cultures around the world.
I.e. the assumption that the family names come last is not necessarily correct even in Europe. Let alone if one deals with the various Chinese languages, Japanese, or Korean. And probably others.
Same with the dates. I always ask my colleagues to please write the years in full, otherwise it can be very difficult to know if it is DMY or YMD (thankfully we don't have to deal with MDY).
According to Wikipedia some joker did have a 666-character surname, officially, in the USA. Perhaps the best thing to do would be to truncate the field at some reasonable limit to prevent people from crashing your system with ridiculous values but make sure your system works properly with the truncated names so, for example, it doesn't panic because the truncated name isn't equal to the original name.
With spaces, punctuation and diacritics, comparing even short names for equality is a bit dangerous and probably best avoided. If you expect two text fields to match and they don't, even after normalisation, you could consider flagging the case for human review later but continuing without an error for the time being.
Like, yes, nobody is forced to accept their name unless they're running a government service, but using it as an excuse is just that, an excuse.
A non-lazy programmer will determine an appropriate limit, document it, continuously test that the entire system can handle that length correctly, and continuously test that helpful errors are returned when too-long names are input.
If you look at the 666-character name, it's no more or less ridiculous than 500 trillion characters.
If you're working in a stack that nicely handles arbitrary lengths, it takes extra consideration and effort to put in limits.
I dunno, would you expect that the government should be allowed to dictate how long a person's name can officially be? If yes, then problem solved, nobody may have names longer than X, and all services will accept X. If no, then there has to be a practical limit on name sizes that government services can accept, and people will be unhappy because it doesn't accept their "official" name.
There's always them pesky foreigners.
> Because programmers are too lazy to properly handle long names? That's a stretch for denying someone service and you know it.
I don't think someone should be denied service if they happen to have a long name, but I genuinely don't think it's a stretch not to try to handle people going out of their way to subvert expected norms. In this case, the argument is more philosophical than technical because there isn't an obvious heuristic for determining whether a name is intentionally made long or not, but there are places where I do think it's worth it for programmers to consider.
As an aside, I'd argue there's more nuance than "properly handling long names" or "being lazy. There's already an inherent limit on how large a name can fit into memory, and that limit might even fluctuate depending on the number of users being processed in a server at a given time and how much memory a given server has. Is a 1 GB name too long to be worth handling, or is not handling it "laziness"? If you're arguing that any name that the government accepts should be accepted by any software, how do you know what the limit is that the government will accept? If you have international customers, is the limit larger? If there's no documented limit, do you just need to hope your software is at least as robust as the government's? My point isn't that these situations are equivalent to a name that's 666 characters long, but that arguing that not handling 666 characters is lazy already is a blend of implicit technical assumptions (servers have enough memory that handling names with 666 characters isn't an issue) and social assumptions (it's possible for someone to actually have a name that long), and I don't think that "pretend all names can fit into memory fine and just crash or time out or something if there are too many names that are too long according to the parameters of the runtime and the hardware" is the obvious best choice from a fairness perspective.
You don't get to just stipulate new conditions for paying your bill. If it's not in the service contract that your giant name has to be spelled completely on the bill for it to be payable, then that condition doesn't exist.
Or maybe Wikipedia is wrong, or the source was bad. You have to pay to read the 1955 article and I can't be bothered right now. Citation 16 below if you're interested.
https://en.wikipedia.org/wiki/Hubert_Blaine_Wolfeschlegelste...
This attitude ensures that the US software industry will never conquer the world.
The US only gives you 21 characters on the DS-11 for a surname.
I once heard that "decision" comes from a Latin root word meaning "to cut off (the other options; to pay opportunity cost)". I will decide to optimize for my use case.
We'll just both need to learn to live with disappointment.
I believe that's #6 falsehoods programmers believe about names (with examples)
https://shinesolutions.com/2018/01/08/falsehoods-programmers...
Ingoring multi byte characters there are still plenty of long names https://www.ancestry.com/c/ancestry-blog/discovering-the-his...
If you're going to try for a "reasonable" max name length it would probably need to be at least 4kb.
If you must, record fields such as:
Full Legal Name - Freeform input no string input limitation. If you feel like this is an attack vector, send the data out for human review.
Full Mailing Address - Don't try to break this down, allow multi-line, free form input. This is something you might want to validate with your shipping carrier and/or a human.
A short 'nickname' used as such.
Also, in almost any situation where you need a legal name, you actually want to follow a lot of rules. This idea that people's original names are somehow sacrosanct is a misunderstanding. If you're doing business in a European country for example, you have to write your name in Latin/Cyrillic letters, perhaps with a few symbols like ' or - allowed as well, and typically with a few accents/diacritics specific to each country. You certainly can't register as 田中 in any context that requires a legal name in France, you'd have to write that as Tanaka.
And this is natural because legal records are meant for authorities in some specific country to read, and compare with other legal docs - so they need to at least be readable to those authorities.
What size is for sure enough? Well I'm not so sure. What if someone has a lot of titles. What if society decides that someone's Legal Name requires a post quantum cryptography key that happens to be 20MB (binary) long?
Also, FYI, at least PostgreSQL doesn't give you a free lunch for any variable length string; length requirements are a column constraint that DEGRADES performance because it has to check.
An external name validator of some sort could check things. Commonly allowed cases could pass by computer check, while actual humans could review edge cases. Someone trying to abuse the name field like that probably needs human review elsewhere anyway.
And the point of adding limitations on length is that you shouldn't even accept the HTTP request if it passes some size, as it will severely degrade performance if you allow someone to upload a 10GB string, even if you separate it into a human review area.
Finally, if the legal requirements change and legal names can legitimately contain cryptographic material, than your system has to change. There is no point in designing a system that tries to work for any possible use case.
128 bytes would only be 42 characters if each character uses 3 bytes, as would be the case in some languages. Which isn't an unreasonable length, especially if the name has a lot of combining characters.
That is only 32 astral characters... Seems kind of close for comfort. Not to mention combining characters
Why not? Confusion if function call with section number as argument?
- I see the entire computing world with Unix blinders on my eyes.
- I can't imagine a strlcpy function being used on a system that isn't Unix and that doesn't have a man page for it in section 3.
- I don't care that C has been internationally standardized since 1989 with a printf(3) function; printf(3) it's just another Unix function in section 3 of my man pages.
- If I don't affix (2) or (3), how will people know I'm not talking about something other than a C library function? I don't understand this "context" stuff in writing and speaking.
But I forgot and now do not.
So printf(1) is the man page for the /usr/bin/printf command, while printf(3) is the man page for the libc printf() function.
Alternatively, readdir(2) is the man page for the readdir syscall, while readdir(3) is the man page for the libc wrapper, which no longer actually calls readdir(2). See also syslog(2) and syslog(3).
Or, time(1) is the man page for /usr/bin/time to time how long commands take. time(2) is the syscall to return the number of seconds since the epoch. And time(7) gives you an overview of time and timers on the system.
You have to quote or escape the parentheses because they're shell metacharacters, which IMHO makes that syntax more trouble than it's worth.
Other commands that work are "man 3 printf", "man -s 3 printf", and "man printf.3". I think "man 3 printf" is the oldest version.
At least one other version of the man command (NetBSD) doesn't accept "man printf.3" or "man 'printf(3)'".
...and surely the "seconds" field of a timestamp is always between 0 and 59 inclusive, addresses will include a state and a building number, phone numbers contain only digits and maybe a leading + sign, etc.
Wrong assumptions like this are one of the root causes of (in the best case) bad UI or (worst case) annyoing bugs.
128 bytes for a surname is only about 60 unicode characters, less if you include RTL markings and characters outside the BMP.
A URL can contain SHA hashes (think: reproducible builds) and can thus be very long (okay, 4k characters is pushing it quite a bit but I wouldn't rule it out like you did...)
This is not a matter of "wrong assumptions". At the end of the day, all you can do is set a limit such that you're comfortable with the risk that someone will be outside the limit you have set. And risk tolerance, as always, is a matter of opinion and not fact.
Firstly, "how many characters can someone have in their legal name" is decidedly not infinite, because it has to be sufficiently short that some governmental entity was willing to record it.
Secondly, as a reply to a comment (quite reasonably) pointing out that 60 unicode characters may not be enough for a surprisingly large number of people, this makes even less sense. Memory and storage space are not infinite, but 128 bytes per name is still unreasonably low. One could buy a single 12TB hard drive and store the names of every single living human, allocating 1.5KB per person.
How short is that, exactly?
Assumptions, or features? I'm all for inclusive behavior, however I'm also for well tailored solutions. Having support for 8k characters when you are going to usually use maybe 20 isn't smart or correct either. That's why we have utf-8, not utf-32, you can grow the bytes when you need to, but only then.
> A URL can contain SHA hashes
It can, or it can not - again, perhaps a feature, not a bug. The hash can also live in a file named by convention, and downloaded/checked separately. Maybe there are other scenarios where you might need a really long url, but domain + release path + name + major.minor.patch should get you 99% of the way.
What's "reasonable" is relative, always designing for the edge case is good in some cases, but its also OK (and perhaps better) to optimize on occasion.
If you allow users to type emojis in their name you will definitely run out as the color/gender selectors take up an additional code point.
Even if Rust had chosen to make &str literally just mean &[u8] rather than promising it is UTF-8 text, the fact &str existed in Rust 1.0 was a huge win. Every API that doesn't care who owns the textual data can ask you for this non-owning slice type, where in C++ it had to either insist on caring about ownership (std::string) or resort to 1970s hacks and take char * with zero terminated strings.
And then in modern C++ std::string cares about and preserves the stupid C-style zero termination anyway, so you're paying for it even if you never use it.
I don't think this in itself is a real problem. You pay for the zero at the end, which is not much. The real cost of zero termination is having to scan the whole string to find out the size, which with std::string is only needed when using it with C-style APIs.
This isn't as much a fault of a string_view type of mechanism, but rather API's wanting NULL terminated strings. Which are kind of hard to avoid on mainstream systems today, even at the syscall interface. Oh well..
Is this required now? I've seen a system where this was only null terminated after calling .c_str()
Only saw it trying to debug a heap issue and I was surprised because I thought surely it's a null terminated string already right? They also checked the heap allocation size, so it would only reallocate if the length of string data % 8 was zero.
The bug will be crazy rare, but of course there are a lot of Facebook users, so if one transaction out of a billion goes haywire, and you have 100M users doing 100 transactions on average, the bug happens ten times. Good luck.
You never need to walk the string to find the \0 byte. e.g. for strlen.
For short strings no heap memory needs to be allocated or deallocated.
It seems C is going around in circles while everybody else has moved on
No, speed and "efficiency" are not a be-all, for-all.
Safety is more important than that except in very specific cases. And even in these cases there are better ways and libraries to deal with this issue without requiring the use of "official" newer C functions that seem to still be half broken
There's so much fiction regarding memory issues and limited memory issues and what to do if we hit limited memory issues when in practice terminate and give up is the (often enforced by the OS) norm.
At some point we are just outlining in general terms what we want from a language. If C was a toolbox it is a limited number of essential tools, other languages add so many things Alton Brown would faint from the unitasking nature of them.
C programmers love this. I know I do.
I could do pretty much whatever I wanted in (DOS) user space with Pascal.
I can do pretty much whatever I want in a modern OS user space with whatever lang I prefer. "Oh but you might need C bindings" because the OS was build that way! (And with Windows/COM you might prefer a C++ bindings - just saying ;) )
> If C was a toolbox it is a limited number of essential tools
C is an old toolbox where 1/3 of them is a rusty finger-remover, 1/3 is a clunky barely do nothing metal crap and 1/3 of them kinda works
I'm all for a simplified set of essential tools, but not one where it's sharper on the user handle than it is on the business end
bUt C iS jUsT hIgH lEvEl aSsEmBlY no it is not
They're counted, zero terminated, ASCII or Unicode, and magic as far as I'm concerned.
Oh... And a string copy is an O(1) operation as it only breaks the copy on modification.
Edit: correct to O(1), thanks mort96
But as with _all_ string solutions, it's the POSIX interface, standard library, and other libraries that screw you. If you're programming in C today, it's because you're integrating with a ton of C code, and thus it's impossible to make progress since the scope of improvement is so small.
It's always struck me as weird that Rust treats strings the way it does - the capacity value is not useful for many cases of strings, and it would have cost them one bit to special case the handling constant strings without the cap measure, which would be better. Most strings are _short_ which makes the overhead worse, proportionally.
Antirezs sds is just one example https://github.com/antirez/sds
The kind of person who calls a single pass through the string a "horribly inefficient solution" will faint at the idea of burdening every string with 16 more bytes of data.
I'm not sure about that. There are plenty of 64-bit systems with less than 4GB of RAM
I would additionally do the exact opposite: lock Python & Ruby developers in a room with only C for a few months.
C is a great language to learn programming, but Python or Ruby are, nowadays, in the most cases, better languages to program with. For example, C's sharpness is a notoriously famous source of bugs; yet it forces to develop rigor, discipline.
https://bstring.sourceforge.net/
The string keeps track of the buffer size and how much has been used, allowing allocations to be somewhat amortized. The string buffer itself is zero-terminated for easy interop with code that expects standard C strings.
struct tagbstring {
int mlen;
int slen;
unsigned char * data;
};
To be honest, every time I need to deal with strings in C I feel like I'm banging rocks together, regardless of approach. I try to avoid it at all costs.
What I do remember is that virtually all string problems can be solved with snprintf/asprintf.
my rule of thumb is that if its name begins with str then it shouldn't be used.
There is a long history of "this time, the strcpy replacement will be safe" followed by cve after cve after cve. At some point I feel that the response really should be to give up on trying to make c-style strings safe.
You really want
int str_try_copy(str_buffer *dest, str_slice *src)It might also be that in some programs with different access patterns that doesn't happen and it makes sense to optimize for the slow case, sure, but the author should acknowledge that variability instead of being adamant on what's better, even to the point of calling "schizo" the solution it doesn't understand. In my experience the pattern of optimizing the fast path makes a lot of sense.
BTW, the strlcpy/"schizo" variant could stand some improvement: realloc() already copies the part of the string within the original size of the buffer, so you can start copying at that point. Also, once you know that the destination is big enough to receive a full copy of the source you can use good old strcpy(). Cargo cult and random "linters"/"static checkers" will tell you shouldn't, but you know that it's a perfectly fine function to call once you've ensured that its prerequisites are satisfied.
Any attempts to add more letters to the strx functions is just polishing the turd.
The portable assembly myth is a recipe for getting Undefined Behavior. C is built on an abstract machine described in the C spec, not any particular machine code or assembly.
Buffers (objects) in C already have an identity and a semantically important length. C just lacks features to keep track of this explicitly and enforce error handling.
Languages exist to provide a more useful abstraction on top of the machine, not to naively mirror everything even where it is unhelpful and dangerous. For example, BCPL did not have pointer types, only integers, because they were the same thing for the CPU. That was a mess that C (mostly) fixed by creating "fictional" types that didn't exist at the assembly level.
I would say C precisely likes to mirror everything even where it is unhelpful and dangerous. The spirit is captured in the Hole Hawg article: http://www.team.net/mjb/hawg.html
It is the same sort of fun one has with self modifying code (JIT compilers) or setting 1 to have a value of 2 in Python.
ed: https://en.cppreference.com/w/c/language/object is what is being referred to. I'm still pretty sure in the 80s and 90s people thought of and used C as a portable low-level language, which is why things like Python and Linux were written in C.
Do you want to copy and truncate, or just copy?
Within that, do you want to manage your own allocation, or do you want that abstracted?
There's too many decision points and tradeoffs to just neatly hide behind a single "one true function" for copying C strings.
As well as reaching for the %ms format for scanf for reading input.
For buffers, I use memcpy and length tracking. Any other approach seems like unnecessary headache. Or maybe modern hardware has spoiled me?
so yeah, good things need time to adopt, no wonder it's not up-to-date tech, lol.
and because of NIH-syndrome we now got lot's of strXcpy functions to choose from.
Also not a huge fan of locale controls and wchar APIs :)
So refreshing to see a common-sense take in a world of shrill low-level programming alarmists.
https://github.com/uecker/noplate
(attention: this is experimental and incomplete for trying ideas and is subject to change.)
You no longer need to worry about running out of stack space, even in a deep call-stack with each function having its own little set of buffers for strings.
#define strlcpy strncpy
mempcpy() , then add the terminator yourself to be sure. memcpy has the slightly annoying flaw of returning a copy of the dest argument (already known data) rather than the more useful new information of a pointer to the end of the set memory.
#IFNDEF _GNU_SOURCE
#DEFINE mempcpy(X, Y, Z) (void *)((memcpy(X, Y, Z) - 1) + Z)
#ENDIF // Not quite the same but close enough mempcpy, fails for Z = 0
#DEFINE mempcpy(X, Y, Z) (void *)(memcpy(X, Y, Z) + Z)
#DEFINE wmempcpy(X, Y, Z) (void *)(memcpy(X, Y, Z * sizeof(wchar_t)) + Z * sizeof(wchar_t))CUT TO:
"I'm not a fan of strlcpy(3)"
It's also pretty telling that every article that tries to explain how to safely copy or concat strings in C, like this one, only ever works with ASCII, no attempt whatsoever to handle UTF-8 and keep code points together, let alone grapheme clusters. No wonder almost all C software has problems with non-English strings...
There are many strings that are not user facing, which you expect to be of a certain nature, e.g. ASCII based protocols, and therefore you know what to do with them.
So the multi byte situation and strcpy, or std::string, or any other "standard" string function isn't really relevant as it's some other libraries duty.
the task of truncation and otherwise formatting UI strings is the preserve of the rendering layer(s).
Note that the Unicode grapheme cluster, word, sentence, and line break algorithms are locale specific. Now consider how often programmers casually truncate strings, even in high-level languages, without accounting for the locale.
C is okay — not fan-worthy, but okay — for one specific kind of program: a mostly-portable implementation of a real language in which to write every other kind of program. that’s not nothing: implementing a backend for every CPU-OS pair one wants to support is a pain, and a different skillset from writing, say, a web browser or text editor.
I wonder how much C has held back the development of computing.
It’s the C hackers who insist on using it for inappropriate applications who create the problems.
Anything string-heavy and internationalized is a terrible place for C, even if you have spent 30 years dancing through the C minefield and are sure you’ll get it right this time.
Newsflash, a few decades ago the majority of software was the minimalist C that they claim sucks less. There are many reasons we moved on from that world, security firmly among them.
Lisp is older than C. It didn't just hold computing back, it actively walked backwards.
And this is precisely what I want, in some cases. I use C when I need a low-level byte wrangling code that tightly interfaces with the operating system, or when I can't afford to allocate memory at will (like all effortless string-handling languages do under the hood), or when I don't want any runtime and need my code to behave like a shellcode, basically, or...
I guess C is still used because people find it useful. No need to be a fan of it.
This was my mantra for the last 30 years. But now, even the tiniest thing has 1GB, linux and virtual memory. Except for very edge cases, this excuse is rapidly dying
Coreutils should continue to do their job, and not need to be recompiled against whatever encoding is in fashion.
Then again, I was surprised to find support for literal encodings in C23. So perhaps my knowledge was even more outdated than I anticipated.
Can you expand on this? Why does it matter to keep code points and grapheme clusters together in the case of truncation? If you’re already truncating the string, then you can just copy as many bytes as possible. Then, later when you interpret that string, you’ll hit a malformed codepoint and ignore it. I guess what you might be getting at is that if you have a codepoint sequence, then you shouldn’t copy the bytes if they can’t all fit in the truncated string?
I feel like this is an edge case and not “the reason almost all C software has problems with non-English strings”. 99% of the time, copying up until the null byte is fine, whether or not the string is UTF-8 or ASCII. The reason most C software doesn’t work with non-English strings is because the developer never added support. The bytes are still there, they just need to be interpreted correctly in the UI portions of the code.
Truncating mid-grapheme-cluster can change semantics in unexpected ways. For example, the "family" emoji(s) can be encoded as a set of conjoined codepoints for each family member: https://stackoverflow.com/questions/49958287/printing-family.... Depending on where you truncate, you might just get the father, which probably won't cause you any serious issues but it might cause confusion depending on context.
(IMHO, if you're truncating a way where this would matter, you should probably be truncating in screen-space, i.e., way higher up the stack)
As far as this being the reason most C code doesn’t handle UTF-8 though, I’m still skeptical.
So many C developer seem to think they're "smart enough" to handle strings correctly using only the standard library, not needing "training wheels" like the plebs - but then where do the countless string related memory safety errors and all the broken Unicode handling come from that plague every C program, no matter how clever the programmers, no matter how tough the review process, no matter how much static analysis is used?
The issue is that many C developers simply can't seem to admit that C string handling, as it was invented many decades ago, was simply fundamentally flawed. So easy to misuse with catastrophic results, even for plain 1-byte-strings. It has nothing to do with how "smart" you are or how careful you are.
I think it's more a case that programming is English skewed and localization concerns are rare for the programmers these day who still need to work in the realm of C (so, mostly for older legacy software or the embedded realm).
The result is that the same mistakes with string handling get repeated again and again, often with catastrophic results.
I mean if your viewpoint is to handle heavy UIs then yes maybe C is not your go-to language. However, there are other solutions to the character space problem besides UTF-8. Rather than complaining that your screwdriver can't bang nails, perhaps try using a screw?
No, you won't be able to support a language like chinese within a 256 bit charset, but perhaps it's not important to in all situations. The strength of ascii is that its small, simple. If that's too anglo-centric for you, maybe there's a better symbolic tiny charset that is more applicable. I'd support something like https://lojban.io/ becoming more prevalent, but there could be advantages to having a simple(r) symbolic transmission language not burdened with anglo-centric concepts.
Unlikely to occur? Do you think that will put the naysayers at ease?
> , and from your comment I suspect you don’t actually know what that problem is (because you seem to be pointing at a different issue which is actually irrelevant). Maybe you or someone else can describe what the real concern is?
Amazing that people have the patience for this.
Especially because concatenation is a very common operation so those sliced codepoints will be everywhere, including in the middle of text.
Yes, many languages allow strings (UTF-8 or otherwise) to contain null bytes, and C's str*() functions obviously do not, but null-termination vs not is an orthogonal issue to ASCII vs UTF-8.
i.e. Yes it's (depending on context) an issue that C str*() cannot handle strings with embedded null bytes, but that's not a UTF-8-specific issue.
Which is?
[1] https://manpages.debian.org/testing/linux-manual-4.8/strscpy...
Sure screens were smaller before and you didn’t have autocomplete, but reaching out for documentation was also harder, and there was less of it. I guess “real programmers” memorize libc down to every detail and never use dependencies they didn’t write themselves.
And to this day I strive to minimize the dependencies of my software - any dependency you have is an essentially unbounded cost on the maintenance side. I used to sign up for the security mailing list for all the dependencies, and would expect to have to follow all the drama and politics for the depenencies in order to have context on when to upgrade and when not too. With Go I don't do that, but with Python I sometimes still do. And I've been reading the main R support email list for ever. I still think that a periodic reading of lwn.net is an essential part of being a responsible programmer.
I will also say, reading the article reminds me of the terror of strings in C without a good library and set of conventions, and why Go (and presumably all other modern languages) is so much more relaxing to use. The place I worked have everything length prefixed and by the end of the 90s all the network parsing was code generated from protocol description files, to remove the need for manually writing string copying code.
It's just a bit frustrating to deal with so many names that are hard to understand and remember. C-style naming forces you to refer to the docs more often, and the docs are usually more sparse and less accessible than in other ecosystems. Man pages are relatively robust and they were a delight back in the day, but they have not been the gold standard for decades, and the documentation conventions for third-party libraries tend to be quite weak.
However the implementations being much more open source now means a lot of bad documentation can be overcome with code reading or, if needed, stepping thru the code with a debugger. Wrong documentation is still expensive. I have a bitter taste in my mouth from integrating with OpenTelemetry Go libraries. It seems to be sorted now in 1.27and q 28 but 1.24 and for a few versions the docs were wrong, the examples were not transferable, and it took 5x the time it should have.
Would you expound on this please? I've seen people reserve the first element as the size of the buffer but then, say a char buffer, has a size limit based on the size of char itself. What advice would you give yourself if you were just becoming a senior developer today? I do embedded myself
It's a holdover from 1970s-era linkers, many of which required each exported symbol to be unique within the first six or so characters.
I just wish that C style used fewer single char names and hard-to-decipher acronyms. Smart programmers can't read minds either...
One of the common features of "modern" languages that I can't comprehend is the love of single character sigils that completely change the meaning of a line of code or perhaps the entire enclosing function.
> I guess “real programmers” memorize libc down to every detail
Or, we just use 'man' and the massive 'section 3' of that system. A feature that no other language has managed to correctly replicate.
Software developers should never rely on guessing. Always, always read the specification.
man (strxcopy)
It is available in user programs only if you define it yourself, e.g. by using memccpy.
Many words are used with different and derived meanings, usually disambiguated by context. For example, an offensive player is not offensive in the sense you used the word in.