How does `cd` work?
blog.safia.rocks
blog.safia.rocks
You see, at that point Unix had no fork system call. There were multiple processes, but they were created statically at startup rather than on-demand. Running a command in the shell would cause the command to replace the shell in the address space of the process, and the process quitting would put the shell back in there.
This worked perfectly with cd being a normal command. Then they implemented fork() and were for a while very confused trying to debug how in the world fork() could have broken the chdir() system call :)
> As a historical curiosity, in the very first versions of Unix chdir was a normal command rather than a shell builtin.
> You see, at that point Unix had no fork system call. There were multiple processes, but they were created statically at startup rather than on-demand. Running a command in the shell would cause the command to replace the shell in the address space of the process, and the process quitting would put the shell back in there.
> This worked perfectly with cd being a normal command. Then they implemented fork() and were for a while very confused trying to debug how in the world fork() could have broken the chdir() system call :)
The pattern of development:
- Make A
- Make B better
- A now broken because it had dependencies on B
- (maybe) Realize a fundamental pattern that unifies A and B.
- (maybe) Unify principles so that A and B become lemmas.
This is a very scientific way of working. The truth emerges as the byproduct of keeping all the plates spinning in your model. As long as you remain aware of all the plates, you'll eventually realize something deep. It's also the strongest argument I can think of in favour of rewriting a codebase plan-9 style. Once you have the truth in mind, bake your knowledge for big wins. Baked knowledge can form the foundation upon which you can climb higher.But many developers don't keep all the plates spinning. If you don't have everything (or some subset) making sense in some unified theory, don't rewrite. Without a unified grand theory, then you can't make something better. This is why I write unit tests — doing so keeps your plates spinning. It's not about proving your language and libraries are doing what is expected (although that's valuable too in some languages without unified libraries) — it's about showing that everything is working together the way your grand theory intends. Unit tests should be written to reject some part of the null hypothesis.
Would the title "How does 'cd' work in MacOS?" be better for readers wishing to learn, or worse? I wonder.
The blog posts from her friend at jvns.ca have been similarly Linux-centric but never use titles that inform the reader as such, i.e., "How does X work?" versus "How does X work in Linux?"
It makes me ask, how important is it for students to be aware of the Holy Grail of UNIX: portability.
How long should one remain blissfully ignorant of incompatibilities between UNIX-like OS that work against those "still seeking the Holy Grail"?
As a user, the UNIX programmers I admire the most are the ones who understand the value of portability, set it as a goal and through broad knowledge and utmost care can get very close to achieving it, despite the mind-numbing work or the tradeoffs this might entail.
An article much more worthy of the title "How does `cd` work?" would maybe, you know, actually go through the code that makes `cd` work.
I am a little confused. What is the parent process and why does that matter here?
What you want cd to do is change the caller's (or parent's) working directory, or in this case the shell's working directory.
There are hacks to do it that you can find elsewhere in this thread (process injection and debug APIs to name two) but frankly the shell changing its own working directory via a builtin is a much simpler and more reliable solution that is also cross-platform.
If we have a shell and we use it to start up a separate chdir program, we get two processes: shell (parent) and chdir (child). The kernel isolates them from each other, so they both have completely distinct working directories. The chdir executes the relevant system call, changes its own working directory and exits. The shell is not affected by the system calls performed by its children, so the chdir process effectively does nothing.
By implementing chdir as a built-in function, the shell is able to change the state of its own process.
$ type cd
cd is a shell builtin $ alias cd=ls
$ cd foo
<lists contents of foo>
$ command cd foo
<changes to directory foo> % which which
which: shell built-in command> The Bash shell invokes the chdir function.
Nope! Not quite.
Bash:
~$ ls -ld foo
lrwxrwxrwx 1 kaz kaz 4 Mar 1 6:50 foo -> /etc
~$ cd foo
~/foo$ pwd
/home/kaz/foo
~/foo$ cd ..
~$ pwd
/home/kaz
Raw chdir and getcwd syscalls: $ pwd
/home/kaz
$ txr
This is the TXR Lisp interactive listener of TXR 190.
Quit with :quit or Ctrl-D on empty line. Ctrl-X ? for cheatsheet.
1> (chdir "foo")
t
2> (pwd)
"/etc"
3> (chdir "..")
t
4> (pwd)
"/"
See the difference? While of course the shell will invoke chdir, it has its own idea of a current working directory, and translates the argument of cd to something else. For instance cd .. doesn't translate to (chdir "..").So the article does point out that CD isn't a direct alias for chdir
-- Can we nominate Julia Evans to the most positive influence in tech prize or something?
To control the shell process.
It works, but I'm not entirely sure it should work.
https://www.amazon.com/Advanced-Programming-UNIX-Environment...
https://www.amazon.com/Design-Implementation-MTX-Operating-S...
no, you came across the definition of bash's 'cd' builtin there..
On FreeBSD with a source checkout, the definition of /bin/sh's (the 'ash' shell, https://en.wikipedia.org/wiki/Almquist_shell) cd builtin is in:
/usr/src/bin/sh/cd.{c,h}
or, on the web:
https://svnweb.freebsd.org/base/head/bin/sh/cd.c?revision=32...
similarly, most anything in base is available in:
/usr/src/{path with '.' instead of slash}/{command name}
which is by the way, very handy, and in combination with having all your self built packages in /usr/ports/distfiles/<package>.tar.<whatever> a big plus of running a BSD system from source..
I don't think it's particularly controversial to say that when you want to explore how the internals of unix work, you're almost always better off reading the BSD source code first. Even if you don't run a BSD. (musl libc is great, too, but limited to libc.)
I do most of my systems programming for Linux platforms, but when I have a question about semantics my first stop is POSIX to learn how it should behave. My second stop is the BSD source so I can quickly grok the mechanics. Lastly would be the Linux or GNU code, to confirm exact behavior. Setting aside the fact that Linux or GNU code usually feels like it was written inside-out and upside-down, you can't well understand why and how something works without having a more general understanding of the problem and solution space. This applies to everything in life, but in the context of systems software programming I've developed a very concrete process.
Having a copy of POSIX locally (greppable, but also the local HTML frames version is super easy to navigate), as well as easy access to BSD code in /usr/src, can make this a very fast and efficient process. Much faster than Googling, wading through Stack Overflow, and other haphazard habits.
cd.. (no space between "cd" and "..") works on Windows
You can simulate that behavior on bash with an alias such as this one:
alias cd..="cd .." cd\foo\x
The cmd parser in Windows is the most bizarre piece of software in common use. I don't believe there is a single person that actually understands how it works completely.I know it's a low-level system call in most on-disk operating systems to change directories. But, for instance, how does it translate physical address to the human-readable name? Do modern (WinXP+) implementations actually take the time to translate folder names from 8.3 to the extended name field? How?
Heck, I'll be dumb enough to ask - why, specifically at the call level, does 'cd\' stick to one level up/down whereas 'cd ' can just pull from just about anywhere? And why, dare I ask, can 'cd' not display like 'tree'?
These are questions about how 'cd' works, to me. Not just, oh, in (foo)Nix it's a system call.
* https://superuser.com/a/380231/38062
* https://unix.stackexchange.com/a/251215/5132
... and so on.
https://github.com/freebsd/freebsd/commit/0bc1bed704cc7b7292...
Unfortunately, I learned that there's is no way to invoke cd programmatically from a program, but I didn't get an explanation I could understand why this is not possible.
Can someone explain why you can't invoke cd from a program?
As far as I know, you will need to provide the user with an alias or a function or something that can put in their .bashrc or something, and there is no other way to do it. (Anything that could would constitute a violation of the UNIX process model.)
You can't change another process's internal state without sending a message that the other process can choose to interpret (or by hacking the memory that the process is using)
# eg. save mc
# cd mc # no '$' is necessary
if [ ! -f ~/.dirs ]; then # if doesn't exist, create it
touch ~/.dirs
fi
alias show='cat ~/.dirs'
save (){
command sed "/!$/d" ~/.dirs > ~/.dirs1; \mv ~/.dirs1 ~/.dirs; echo "$@"=\"`pwd`\" >> ~/.dirs; source ~/.dirs ;
source ~/.dirs # Initialization for the above 'save' facility: source the .sdirs file
}
source ~/.dirs # Initialization for the above 'save' facility: source the .sdirs file
shopt -s cdable_vars # set the bash option so that no '$' is required when using the above facility
What this does is, whenever you're in a directory that you'd like to "bookmark" as you'd call it. Just type `save whatevername`. Then, when you navigate somewhere else you can type `cd whatevername` and it'll change you back there. It's simply adding to this ~/.dirs file and so overwriting is taken care of by just saving the same name in a new (or the same) directory. It just appends a line.Also, you can just type `show` and any point and it'll tell you what you've saved and where.
The nicest thing is that this persists with new logins (the only drawback is that other shells that are running don't get the update automatically).
----
$ which cd
/usr/bin/cd
$ cat /usr/bin/cd
#!/bin/sh
# $FreeBSD: src/usr.bin/alias/generic.sh,v 1.2 2005/10/24 22:32:19 cperciva Exp $
# This file is in the public domain.
builtin `echo ${0##*/} | tr \[:upper:] \[:lower:]` ${1+"$@"}
Oh, bother! Reading shell scripts can be such a hassle sometimes. I know the tr command is used to translate characters. In this particular case, the second half of the command, the part after the pipe symbol, basically converts the command cd dev to CD dev. I have no idea why this is. In any case, this modified command is passed to the builtin command which is handled by the shell (Bourne shell) that we are using.----
1. The first mistake is typing `which cd`. `which` is a separate program that looks things up in $PATH, which may not actually be what happens when you run the command. You should have used `type cd`:
$ type cd
cd is a shell builtin
As you discover later in the article, `cd` must be a shell builtin. Which makes it a little mysterious (and interesting!) why the file /usr/bin/cd exists; it won't really do anything, try it: $ pwd
/home/lukeshu
$ /usr/bin/cd /usr
$ pwd
/home/lukeshu
$ # but it will print error messages
$ /usr/bin/cd /bogus
/usr/bin/cd: line 4: cd: /bogus: No such file or directory
So, why does /usr/bin/cd exist? The comment with the CVS ID gives us a hint: It's a common "src/usr.bin/alias/generic.sh" that is copied (hard-linked) in to /usr/bin for several shell builtins ( https://github.com/freebsd/freebsd/blob/0bc1bed704cc7b7292be... ). For other builtins that don't need to be builtins, it makes sense; let other programs call them with exec. For `cd` it doesn't make much sense though, and I'm not sure why it exists. Is it just for consistency with other builtins, or does it serve a real purpose? IDK.(edit: the short answer is "POSIX says so" https://github.com/freebsd/freebsd/commit/55d0b8395514ae4055... , but why does POSIX say so? See my child comment for further citation.)
2. The second mistake is about what `tr` is doing. You claimed it's converting lowercase to uppercase; but that's backward, it's converting uppercase to lowercase.
So, why does it convert to lowercase? Recall that we learned that it's the same script being used for all builtins. If it weren't literally the same file (at the cost of a few more bytes disk space), it could have just done a search/replace within a template, having each be `builtin BUILTIN_NAME ${1+"$@"}`. But they wanted to save a few bytes, and instead the script must detect the appropriate builtin name by translating its program path to a builtin name. If you execvp("cd", ...), it will invoke the script with $0 set to "/usr/bin/cd". If /usr/bin is on a case-insensitive filesystem, and you execvp("CD", ...), that will also call the script, with $0 set to "/usr/bin/CD". How is it going to translate from "/usr/bin/CD" to "cd"? The ##*/ bit trims the leading directories, then the tr bit converts the remainder to lower case.
(as an aside: the `${1+"$@"}` is a little interesting too; why not just write `"$@"`? "$@" will expand to the full list of arguments (after argv[0]). The ${1+...} bit says to only do that expansion if the first argument exists (i.e., there are >= 1 arguments). But that should basically be happening anyway; if there are no arguments, "$@" should expand to a zero-length list. IDK, perhaps a weird historical shell?)
POSIX actually mandates this behavior for any builtin not on a specific list, though most shells (even dash, which is typically obsessive about complying with POSIX) do not implement this behavior.
The scripts we're discussing were written in 2002 [1], so the time-appropriate version is POSIX-2001 (Issue 6).
[1]: https://github.com/freebsd/freebsd/commit/55d0b8395514ae4055...
But you don't have to take my word for it: I've paste-bin'ed the entirety of what POSIX-2001 had to say about shell built-ins (in general, I didn't include man-pages for individual built-ins): https://lukeshu.com/dump/posix-2001-builtins.txt
As for whether that's still true today, looking at POSIX-2008 (Issue 7), 2013 edition (I don't have a copy of the 2016 edition handy), none of that has changed.
That seems like fairly rare edge case, considering that UNIX typically had case-sensitive file-systems, or rather handled filenames as opaque blobs. I wonder what actual system caused the need for case folding? Maybe HFS?
https://github.com/freebsd/freebsd/blob/master/usr.bin/alias...
One of the most famous shell-portability issues is related to "$@". When there are no positional arguments, Posix says that "$@" is supposed to be equivalent to nothing, but the original Unix version 7 Bourne shell treated it as equivalent to "" instead, and this behavior survives in later implementations like Digital Unix 5.0.
The traditional way to work around this portability problem is to use ${1+"$@"}.
[1]: https://github.com/freebsd/freebsd/commit/55d0b8395514ae4055...
That's not bad, but I'd also suggest checking man pages for syscalls. `man 2 chdir` will give the some documentation on both Mac and Linux OSes and call out the specs that are relevant to the call. (Why `man 2`? That searches section 2 of the man pages which is dedicated to syscalls. How on earth would someone know that? `man man` of course. :-) )
On linux, an interesting thing is that you can inspect a process's current working directory (IE the last thing they chdir()ed to) by looking in /proc/<pid>/cwd/. It presents itself as a symlink to the actual current directory as stored in the kernel.
In this case, the author got tangled in the shell script and code, and totally missed the whole subtlety and complexity of the Unix architecture. I expected to see something about the processes, file system, and directory entries. Even after the author added key information that was emailed to him by a reader, he didn't really follow up or try to understand more. I understand that not all coders have or need a degree in computer science, but it really surprises me what this author doesn't know (and doesn't know he doesn't know).
Another example from his next blog about 'ls': "I’ll admit, scrolling through all this C-code can be a little tiresome. Oh, how I miss the days when all I had to do was read JavaScript source! Because C gives you so little out of the box, a lot of the code that you end up reading is not that interesting. It’s largely the kind of stuff that higher level languages implement in their standard library." [https://blog.safia.rocks/post/171381157060/looking-into-ls]
Here's a book about Unix, among many (I got this one as a CS graduation present): https://www.amazon.com/Magic-Garden-Explained-Internals-Rele...
I am quite surprised/curious how it got to top page here.
Others come to that knowledge by reading a 650 page book about System V after graduating from university. Maybe they already learned it before getting the book, in some practical circumstance, like the author. Maybe they have always known it. But for me, an article by someone in the midst of their learning is a great help, an opportunity for others to share what they know, and exemplary of the hacker spirit.
Coming to it now, with all the layers that have built up -- the side-track for the script that uses tr, and the idiom with ${1+"$@"}, and all the cruft build up within sh(1) -- it must be pretty hard to separate what is incidental from what is fundamental.
I will say I did learn about some shell along the way and the bit about processes was interesting and starting to get at the core of 'cd'. It just seems like there are many other pieces of the puzzle, and I encourage the author to keep researching and keep writing about them.
* https://unix.stackexchange.com/a/413225/5132
* https://unix.stackexchange.com/a/251215/5132
* https://superuser.com/a/380231/38062
* https://unix.stackexchange.com/a/34386/5132
... and so on.
>It’s largely the kind of stuff that higher level languages implement in their standard library
I think it's extremely interesting. It's in these libraries that the hidden fun stuff happens. For example, memory allocation and related terms like the heap seem like magic but it becomes clearer once one learns about how it works.
Here's one: https://github.com/bminor/bash