Apple's libc shells out to Perl to implement wordexp | Hacker News Reader

Apple's libc shells out to Perl to implement wordexp

github.com

103 pointsdevbug47 comments

This is source code from 2011. Checking http://opensource.apple.com will show that it's not longer implemented this way in 10.10.

Here's the current implementation: http://opensource.apple.com/source/Libc/Libc-1044.1.2/gen/Fr...

Can you explain why wordexp.c on the referenced site has a completely different (and older) copyright? Did they merge in another implementation?

I'm guessing that the linked implementation probably hadn't been touched since 2008. My comment that this was source code from 2011 was referring to the fact that this release of libc (Libc-763.11) is from 2011.

The current one is a derivative of NetBSD's implementation (where it shells out to use a sh builtin)

Judging by both the source code and its placement in the repository (gen/FreeBSD/wordexp.c), I'm thinking it's a derivative of the FreeBSD implementation (which may itself be a derivative of the NetBSD implementation, or perhaps the other way around).

Yes, the OSX code seems to come from FreeBSD. Here is the FreeBSD version of the source:

https://github.com/freebsd/freebsd/blob/master/lib/libc/gen/...

It looks very similar to the current OSX file posted in an earlier comment. Calls /bin/sh too.

License-wise:

- this piece of code is 2BSD-licensed

- it calls out to /bin/sh

- on OS X, /bin/sh is hardlinked to /bin/bash

- OS X's bash is GPLv2

Forking to shell is the only way to reuse bash's code. More often than not I really wish sh were not bash.

"Forking to shell is the only way to reuse bash's code."

well, the other obvius way is to license this piece of code as GPLv2.

AFAIK this is THE reason for GPL. So that you either release your code and contribute to open source or suffer from slower access methods.

The BSDs doesn't ship with bash and their wordexp shelling out to /bin/sh doesn't suffer, so I don't see how this is a reason for the GPL.

Probably I've written it not clearly enough: "Reason for GPL not allowing you to use code in your own programs, but allowing use through exec: you either release your code and contribute to open source or suffer from slower access methods."

ahh, ok; that's clearer

Which would require libc to be GPL, something not even glibc does.

As it's already written in the manual...

http://linux.die.net/man/3/wordexp

wordexp, wordfree - perform word expansion like a posix-shell

So, the implementer decided to take the short route, and just spawn a shell which, essentially, gets passed the input data to wordexp(), to do the work. But, of course, having something that starts with such a comment...

/* XXX this is _not_ designed to be fast (...) wordexp is also rife with security "challenges", unless you pass it WRDE_NOCMD it must support subshell expansion, and even if you don't beause it has to support so much of the standard shell (all the odd little variable expansion options for example) it is hard to do without a subshell). It is probably just plan a Bad Idea to call in anything setuid, or executing remotely. */

...in your standard C library wasn't such a smart idea to start with. Scroll down, there are many more gems in the comments!

Sometimes it's better to just implement it as

        void wordexp() {
                fprintf(stderr,"wordexp() is a security nightmare. Not implemented.\n");
                assert(0);
        }

or decide to deliberately only implement a safe subset of the full functionality specified (e.g. only ~user-homedir expansion and $VARIABLES), to at least cover the common use-cases without creating a security nightmare.

(EDIT: typos)

"Like a POSIX shell" includes expanding command substitution. wordexp("`echo hi`") should return {"hi", NULL}. So it isn't totally ridiculous to reach for a shell; tracking multiple subprocesses, and nested command substitution (like "$(echo $(echo $(echo foo) $(echo bar)))") is a pain to reimplement correctly.

In an ideal world, we'd have a libsh that exposed all of the steps of what /bin/sh does in a nice fashion, this function would call one of the libsh functions, and /bin/sh would be a 10-line while (true) { libsh_this(); libsh_that(); }.

In a slightly less ideal world, the shell would have a way to separate things with null characters, without bothering Perl.

We really don't live in an ideal world.

I understand that spawning a shell to do the expansion work probably is a good idea when your goal is to "expand a string like the shell".

But my guess is that most users will call wordexp() just to expand a few variables or tilde-homedirs.

> In an ideal world, we'd have a libsh that exposed all of the steps of what /bin/sh does in a nice fashion, this function would call one of the libsh functions, and /bin/sh would be a 10-line while (true) { libsh_this(); libsh_that(); }.

THIS, GODDAMNIT, THIS.

Scroll down, there are many more gems in the comments!

...and what a spelling errors:

"This kludge is needed because /bin/sh seems to set IFS to the defualt even if you have set it; We also can't just ignore it because it is hard/unplesent to code around or even a potential security problem because the test suiete explicitly checks to make sure setting IFS 'works'"

ESR writes about sloppy spelling: "Write in clear, grammatical, correctly-spelled language. We've found by experience that people who are careless and sloppy writers are usually also careless and sloppy at thinking and coding (often enough to bet on, anyway)."

"...and what a spelling errors...ESR writes about sloppy spelling..."

:-\

ESR hasn't been a role model for anything related to programming since, like, 1998.

The standard itself at http://pubs.opengroup.org/onlinepubs/9699919799/functions/wo... says "While wordexp() could be implemented entirely as a library routine, it is expected that most implementations run a shell in a subprocess to do the expansion" and gives guidance on how to implement it that way.

Is wordexp() a requiement for being POSIX compatible?

I searched through the FreeBSD repo on Gitbub and the only references to wordexp is basically the code itself. Searching the whole Github does not show many things either.

How common is the function in practice and how important is it?

requirement? yes. http://pubs.opengroup.org/onlinepubs/9699919799/functions/wo...

common in practice? openbsd doesn't support it. afaik few programs use it, and those that do are usually easily patched not to (can use glob instead or disable whatever feature entirely).

Here is a vulnerability in the current version [1] for your amusement:

    #include <wordexp.h>
    #include <stdio.h>
    int main() {
        wordexp_t we;
        char *s = "$(($(sleep 10)))";
        printf("->%d\n", wordexp(s, &we, WRDE_NOCMD));
    }

(Since the manual page strongly recommends not trusting wordexp with untrusted input even with WRDE_NOCMD, and based on a code search the function is rarely used in the first place, I don't think it's really sensitive.)

http://opensource.apple.com/source/Libc/Libc-1044.1.2/gen/Fr...

Well, you are warned: /* XXX this is _not_ designed to be fast */

Performance is the least of my worries here. I'd be more worried about security, and plain dependency management.

I mean, let's say this libc thing is supposed to be installed in quite a lot of systems, are we really sure that perl thing is installed in all of those systems? And what if, by some crazy coincidence, perl happens to depend on libc?

In Debian, at least, there's actually a number of circular dependencies within the (small) set of packages marked as "essential"; they all sort of lean on one-another like a house of cards. Perl is in that set, if you're wondering.

Almost everything depends on perl. Unless you're running a very small embedded setup, you'll have extremely hard time living without perl.

Circular dependencies are perfectly normal.

I'm aware that a lot of projects use perl for the build. But on runtime? I doubt it.

It's Apple's libc. They ship always ship Perl on OSX (and presumably iOS too?), so what do they care whether other platforms have to deal with an extra dependency? Their platform has Perl anyway, so the dependency is essentially free.

Are you on a Debian based OS? Try

  aptitude search '~i~Dperl'

Apple libc in particular is supposed to be installed on exactly two base systems: OS X and iOS. The former has perl installed by default, and the latter marks wordexp as unavailable.

> I mean, let's say this libc thing is supposed to be installed in quite a lot of systems

Apple's libc is not an independent thing to them - it's a part of OSX and iOS. You're not supposed to be able to take it out and use it for other things.

Fair enough.

Sorry, I don't have a Mac to check, but does the OSX build of Perl depend on libc?

Also, that code seems to be from 2008 (or at least that's the latest year in the copyright header) despite the commit being from 2012. Does anyone know if this has been updated? Inserting a NUL byte between each word, and at the end doesn't sound like it requires Perl...

> otool -L /usr/bin/perl

/usr/bin/perl:

/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1151.14.0)

/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)

Thanks, much appreciated.

I thought there may be some strange cyclic dependency going on, but it seems not.

It would only be a problem if that specific perl command (perl -e 'print join(chr(0), @ARGV), chr(0)' -- args....) or the shell invoking it depended on wordexp(). Recursion is fine as long as it terminates.

> Inserting a NUL byte between each word, and at the end doesn't sound like it requires Perl...

No, it's more clever than that. The shell will actually execute something like this:

/usr/bin/perl -e 'print join(chr(0), @ARGV), chr(0)' -- your input string goes here

So, what happens is that the shell (in this case, bash) performs argument expansion on your input string, then calls perl -e '...' with the expanded words as its arguments. What perl does is join all of those arguments with a NUL byte and spit them back out for the calling process to read, which makes figuring out where each expanded "word" begins and ends really simple.

Could you do this in shell? Probably. Would a correct implementation be as short and easy to understand? Probably not. Therefore, perl.

They appear to be using a "wordexp-helper" in Yosemite. I found this in /usr/lib/system/libsystem_c.dylib (and no references to perl).

    [ $# -gt 0 ] && export IFS="$1";/usr/lib/system/wordexp-helper

Source to wordexp-helper here: http://opensource.apple.com/source/system_cmds/system_cmds-6...

Also the source to the Yosemite implementation is available at: http://opensource.apple.com/source/Libc/Libc-1044.1.2/gen/Fr...

sometimes you have to get shit done

> sometimes you have to get shit done

And Apple have always been masters in selling polished, nicely packages shit as the most advanced technology ever conceived.

With respect, is there evidence to support your claim?

My direct experience is that I have received value far in excess of what I have had to pay for the Apple products and services I have purchased over the years.

For example, I'm typing this reply on my PowerBook, which I use all day, every day in my work. I also have my iPhone with me constantly, and use it heavily.

Finally, the Apple brand was said to be worth about $104.7B vs. $62.8B for Microsoft in 2014. [1]

So, I am not saying you are wrong, only that I think some evidence would be helpful in making your case.

[1] http://bgr.com/2014/03/19/apple-vs-google-vs-microsoft-brand...

OT, but you're still using a Powerbook? That's pretty impressive. Which generation\year was it? I'm guessing you've had to swap out batteries and upgrade hard drive - have you had to modify anything else?

Sorry, my mistake. It's a mid-2012 MacBook Pro. 16GB DDR and 512GB SSD.

http://support.apple.com/kb/SP694

He meant MacBook Pro.

That said, I still have an iBook G4 from 2004 in perfect operation for mail/web/music (though one button is missing).

Reply on news.ycombinator.com