GNU C Library 2.30 | Hacker News Reader

GNU C Library 2.30

sourceware.org

95 pointsjrepinc35 comments

> * Memory allocation functions malloc, calloc, realloc, reallocarray, valloc, pvalloc, memalign, and posix_memalign fail now with total object size larger than PTRDIFF_MAX. This is to avoid potential undefined behavior with pointer subtraction within the allocated object, where results might overflow the ptrdiff_t type.

I did not think they would take this decision so soon, but it is, in my opinion, the right decision to take. There will be complaints from users of memory-heavy programs running on 32-bit platforms though.

For context, this blog post shows how things break when allocation functions are allowed to create blocks of more than PTRDIFF_MAX: https://trust-in-soft.com/objects-larger-than-ptrdiff_max-by...

Absolutely, this is the right thing to do.

If one needs to allocate more than 2 GB in a single allocation, time to move to 64-bit.

> If one needs to allocate more than 2 GB in a single allocation, time to move to 64-bit.

time to move to (anonymus) mmap - it still has size_t as type for the size argument

That first clang listing was hilarious.

> There will be complaints from users of memory-heavy programs running on 32-bit platforms though.

I would think this would be a small one of many complaints such users would have on a daily basis.

> There will be complaints from users of memory-heavy programs running on 32-bit platforms though

In all of recorded history, has a malloc() call for more than 2GB ever actually succeeded anywhere? Most OSes on such platforms never supported any more than that amount of addressible memory in a user process at all.

This is fine. Honestly it's seems like mostly pedantry on modern systems, but it's clearly correct.

I've done about 64GB in Go but I think it uses mmap.

On a 64 bit platform, sure, that's inside of PTRDIFF_MAX, so it's legal and supported.

> Most OSes on such platforms never supported any more than that amount of addressible memory in a user process at all.

And even when they do, it's going to be very rare to find 2GB of contiguous unused address space. With the default executable mapping low in the first GB and the stack right below 3GB, a single library mapped around the 2GB mark can fragment the address space enough to make a 2GB allocation impossible.

> In all of recorded history, has a malloc() call for more than 2GB ever actually succeeded anywhere?

Yes, on OS X 10.5, and on 32-bit Linux with Glibc until two days ago.

The article I linked, written before Glibc 2.30 was released, is from a period when every Unix had been allowing “malloc(0x80000001);” in 32-bit processes until recently; only OS X had had the courage to make that allocation fail. Sorry if the article doesn't make it clear enough that this is the context it is written in, but in its defense, you only needed to try it (and still need today to try it if you didn't upgrade Glibc) to see that it succeeds. Or do you think that the Glibc developers wrote a Changelog entry to explain that they changed something that didn't actually change?

Linux's default limit on 32-bit has been 3GiB for a while, i think: https://stackoverflow.com/a/5080778/139746

Windows's limit is 2GiB by default, but this is only a default and 32-bit processes can be allowed access to more memory, up to IIRC nearly all of the theoretical maximum 4GiB for 32-bit processes running on 64-bit Windows.

The (sarcastic) point was about the fact that no real world code actually relied on a malloc() of half the address space.

I'm sure it "worked" in some sense, though I'd be really surprised if you could make that happen with a default-linked C program on any distro that ever shipped. The holes just aren't big enough. You'd need to link the app with special care, and potentially write your own dynamic loader to keep the region you wanted free. And if you do that... you might as well just mmap() the thing.

The point was that doing this with the system heap on a 32 bit system was never a serious thing. There are apps that would do management of memory spaces that large, but they didn't do it with malloc.

> * The twalk_r function has been added. It is similar to the existing twalk function, but it passes an additional caller-supplied argument to the callback function.

I thought this was standard practice for designing C APIs taking callbacks.

> * The Linux-specific <sys/sysctl.h> header and the sysctl function have been deprecated and will be removed from a future version of glibc. Application should directly access /proc instead. For obtaining random bits, the getentropy function can be used.

That's gonna break the world, a lot of code includes that header and uses the sysctl function.

Well on the bright side it's perfectly reasonable for the function to exist in a perpetual state of deprecation. Let's hope they don't do anything rash.

If they do, Linus will probably scream at them for breaking compatibility. (Even if this isn't the kernel)

This is a glibc wrapper for the sysctl system call, which has been deprecated since forever in the kernel, is compiled in only if an option is specified (major distros don't enable it), and is likely to be removed completely at some point. Currently trying to use it, even if enabled, generates a warning in the kernel log.

http://man7.org/linux/man-pages/man2/sysctl.2.html

> I thought this was standard practice for designing C APIs taking callbacks.

This is to supplant non-reentrant shared state for node depth. Same as strtok() -> strtok_r(). Doing this with the generic closure arg is just a bonus since the callback is already present.

Whoa, a gettid wrapper? What changed the maintainers' minds on making that available?

Static linking works now? Or I have to use Musl to have this feature working?

static linking works also with glibc, AFAIK (using it)

IIRC NSS (/etc/nsswitch.conf etc.) needs dynamic linking for anything beyond the basic files backend. But, again IIRC, musl has never supported NSS anyway so that's kind of a moot point.

> IIRC NSS (/etc/nsswitch.conf etc.) needs dynamic linking for anything beyond the basic files backend. But, again IIRC, musl has never supported NSS anyway so that's kind of a moot point.

I frankly have never ever ever seen anyone actually configure NSS outside of the defaults.

You have never been in an organization that has used ldap or other user backends then.

However even the defaults on Debian and Centos are affected here, as it means that the dynamic user/host stuff in systemd also won't get picked up when something doesn't read nsswitch

It's needed for Bonjour/avahi/mdns.

It's also needed for systemd dynamic users.

If you have either of those, you will have an /etc/nsswitch.conf that requires dynamic linking :)

Anyone able to expand on:

"The dynamic linker accepts the --preload argument to preload shared objects, in addition to the LD_PRELOAD environment variable."?

Does one ever invoke the dynamic linker directly? Why? How?

yes, you can. the linker loader is the default executable for all ELF binaries

    $ /lib64/ld-linux-x86-64.so.2 /bin/true --version
    true (GNU coreutils) 8.28
    [...]

If you run it without arguments it will tell you usage.

Ah, yes of course. I actually do this often to look for missing runtime dependencies. I hadn't thought about preload in that context - or ldd as a way to run executables "by hand".

IIRC it lets you get around the noexec flag on a mount.

A lot of this sounds like great work, and the GNU project is doing great work. However, I assume I'll have to prepare for more software breaking? When 2.28 rolled around, Electron and a bunch of GNU software (which relied on glibc specific stuff which changed) broke.

Software compiled against glibc links to versioned symbols, which are backwards compatible in ABI and behavior. I'm unsure of the reason you experienced breaking software when upgrading your glibc version.

The GNU software which broke just didn't compile (with no upstream fix available for a long time, which I found incredible; I had to go find arch linux' repos' patch and apply that whenever I wanted to compile GNU build tools).

The electron thing was apparently an LLVM linker thing according to your sibling comment.

EDIT: the m4 patch in question: https://git.archlinux.org/svntogit/packages.git/tree/trunk/m... - apparently they still use it.

> FIXME: Do not rely on glibc internals.

Seems that's less of a glibc breaking compatibility, and more developers relying on something outside of the guaranteed API.

I mean, it's GNU M4. It's at the core of GNU's build system. It's GNU developers depending on glibc internals. I'd be with you if it was just some random project, but it's pretty bad of an update to glibc to break the GNU toolchain.

glibc goes to some length to ensure software linked against an older release will always run unmodified when a newer release is loaded so your comment surprised me a bit. Turns out it was actually an LLVM linker bug [0], fixed in Electron when they moved to building with a newer version of LLVM [1].

[0] https://github.com/electron/electron/issues/13972#issuecomme... [1] https://github.com/electron/electron/pull/13988

How is the support for the 68k in this release?

Reply on news.ycombinator.com