Quoting command line arguments the wrong way (2011)
blogs.msdn.microsoft.com
blogs.msdn.microsoft.com
Not this. Frankly, few developers will even know about the need for careful coding such as this, and even fewer will actually do it because it will muck up each and every program with dozens of lines of extra stuff to work around a deficient part of the PLATFORM.
Then, for new OSes, write a compatibility layer to implement the old functions on top of the new ones, and for old OSes, write a compatibility library to implement the new functions by quoting and passing to the old ones.
Mark the old functions as "deprecated, do not use in new code", and point to the new ones.
1. Fix old solution
2. Implement alternative that is feature complete and document
3. Write comparability layers
If people followed this life as a software developer would be easier on bloodpressure. I remember fondly working with many APIs thst were deprecated but had no alternative and the devs admitted it.
> These functions appear to be precisely what we need: they take an arbitrary number of distinct command line arguments and promise to launch a subprocess. Unfortunately and counter-intuitively, these functions do not quote or process these arguments: instead, they’re all concatenated into a single string, with arguments separated spaces
But that's the opposite of "does exist"?
Any API or C program main() running on Windows that suggests anything else is fiction - the C runtime parsing what the kernel gave it to a string array on one side, or concatenating into a single string on the other.
NT's native process creation functionality is powerful, but baroque: see [1]. There's a ton of stuff that processes can be passed in addition to the command-line. One trick that's not well-known is that CreateProcess allows parent processes to pass an opaque binary blob to subprocesses via the lpReserved2 member of the STARTUPINFO structure. Cygwin uses this blob to pass information about file descriptors, ttys, and other POSIX context; this information block bootstraps Cygwin's fork implementation. The Microsoft C runtime uses it for a vaguely similar purpose: it's how file descriptor inheritance works when neither NT nor Win32 know anything about file descriptors (which are private to libc).
[1] http://www.rohitab.com/discuss/topic/40191-ntcreateuserproce...
[2] https://msdn.microsoft.com/en-us/library/windows/desktop/ms6...
For example, suppose the console app thinks that apostrophes should be treated as quotes. I pass "'x", "x'" into the new API, and that gets serialized to "'x x'" (two strings to most applications), but this particular app interprets that as one string that says "x x". The OS can't even escape the apostrophes to avoid this, because it doesn't know what language the console application speaks.
PowerShell had to deal with this problem because native command lines need to be rehydrated from its AST before they can be passed to CreateProcess, and (IIRC) they ultimately had to add an operator that means "everything after this point in the command line should be passed to the command verbatim" to cover all of the corner cases from this.
Microsoft have just chosen not to make it work like that because that would involve admitting they were wrong.
At present, they're both "everything in one big string"; if an alternative "array of strings" API were added at both ends, you'd need to come up with shims for when the caller passes a string & the callee expects an array, and vice-versa. It's not immediately obvious how you'd do that in a way that works reliably in all cases, especially considering the callee currently has total freedom to parse its command-line however it likes.
[1] https://msdn.microsoft.com/en-us/library/windows/desktop/bb7...
Even when finally it was aknowledged that the command shell needs some love too, the answer was PowerShell, which yet again defined an object interface between cmdlets[4].
[0] https://en.wikipedia.org/wiki/Dynamic_Data_Exchange [1] https://en.wikipedia.org/wiki/Object_Linking_and_Embedding [2] https://en.wikipedia.org/wiki/Component_Object_Model [3] https://en.wikipedia.org/wiki/Local_Procedure_Call [4] https://en.wikipedia.org/wiki/PowerShell#Pipeline
They create much more closed, difficult and obscure technology which needs, training, certification and lot of care (services provided by Microsoft).
Sure, Windows command-line argument passing is bad, but have you ever tried using wait/waitpid/wait4/waitid/etc.? That's a nightmare in the POSIX world; Windows has nice, clean process handles, not the garbage /proc stuff that makes it fundamentally impossible to write a safe pkill(1).
If you're writing a brand-new system, for the love of God, do a good job of designing the APIs. You will not have a chance to go back and fix the APIs later.
To list quickly from memory (and my game machine experiences) - all from windows 10 pro.
* new install with ms office, I run autoruns.exe and see that I have 70+ things being run on startup. Yikes. Default install.
* registry: multiple things, biggest for me: you can't easily move part (or all) of it between machines, due to being tied to specific machine: imagine in unix you can't move whole /etc between to a newly installed separate machine.
* still lack of decent package management (check chocolatey (BTW they try to do some work here): among dozens of problems they have, they still (and in fact probably never) can't tell you simple 'list all files of given package'. Don't event want to start laughing about appx microsoft newest invention: it does not return even HALF of default installed apps of new laptop.
* logging. During update 1607, process is stuck (6hrs, and still 'please wait'): no simple place (log) to analyze what's happening (and no, eventlog is not such place). All daemons and windows do not have sane logging with enough information constantly being written to logfiles to analyze problem (this is big and deliberate).
* file system mess: e.g. system drivers running with kernel permission installed in 'program files' (new dell xps from 2016) and also usual common day programs being installed in c:\windows - while windows happily allows it - again, clean new install :)
* naming of services/technologies and their (microsoft) general approach to architecture design (boundaries and namespaces): this is mess, one example among hundreds: check what is short name of background transfer service (the bits one) you need to restart if windows update (sic!) stops working - no, it's not bits or bts :)
This is only written on the phone high level things, There is on the net comprehensive listo of 500+ things could have been fixed.
So???
Encode the program name together with all arguments as one big string, discarding all type safety, spawn the process of your favorite shell, let that shell process the string in order to spawn one or multiple processes, in each process have the standard library process the arguments into an array called argv, and have your average process call out to yet another library to parse these string argument strings into flags and parameters, and prints a not standarized, potentially localized string if an error occurs, and if the error is fatal exits with a semi-standarized return code. The calling program tries to make sense of the output of the called program, often by fuzzy matching against known output.
That's the standard way how it's done since the beginning of Unix. Some APIs skip the shell, but that's a minor detail in all this. If this sounds simple and like the obviously best solution, then congratulations. The windows developers disagree and tried (and continue to try) to find a better way. They mostly failed so far, but I think we should thank them for at least trying to innovate.
1) serialize data structure to semi-typed array of strings 2) pass array of strings directly to target program 3) have target program parse the array of strings to flags & parameters
With no shell or C library touching the command line arguments at all.
The only way this could be more direct would be if you used JSON instead of arrays of strings, web-API style.
You know, in all these years I've never bothered to find out how DDE worked, having used COM instead, and now I find: https://msdn.microsoft.com/en-us/library/ms648774.aspx and it's a terrifying abomination built on wparam/lparam.
> In this world, the command line, the pipe philosophy and the 'less is more' mindset were not only not welcome, they were the adversary.
I agree that this is Microsoft's greatest heresy and also an effective tool against interop.
Umm... Windows 3.1, I think? It could use DDE to pass the path to the file opened in the File Manager to the program specified in the Registry.
Unix got this right by forcing the shell to provide the kernel a pre-parsed list of strings, so the only insanity the tool integrator needs to understand is the shell's quoting syntax. Which is still insane. But it's only insane in one particular way.
However, at minimum it is still useful to shell-escape arguments when displaying them to the user (for ease of copy+paste), so it's unfortunate that many languages don't have any standard library function to do this - including C on POSIX, and Python before version 3.
So your rule won't work. You can't single-quote a string that itself contains single quotes, which makes for some fun when you have arbitrary strings (file names are the big frustration) that need to be substituted into a parseable command line.
But like I said above: Bourne syntax[1] is only one kind of insanity, which is still much better than the DOS/Windows world of a separate parser for every app.
[1] We shall not speak of the C shell here.
It's a bit ugly but it works.
The first ' closes the current single quotes string. The \' adds a single quote character outside a quoted string (and outside single quotes you can use backslash escapes). Finally, the last ' reopens the single quoted string that we closed before.
(Well, they are also in a system DLL called MSVCRT.DLL. That is an internal library which is undocumented and considered by Microsoft to be off-limits to applications.)
This is for example why you should avoid giving your files such cute names as '-rf'.
Still, it would be right and proper if Unix programs a little type-safety in their arguments, for example by requiring that ALL arguments be flags, as in this hypothetical smart_rm: "/bin/smart_rm -rf --pattern foo/bar"
It would remove the problem of programs all being required to implement --.
The kernel should ban these names. I'm a big fan of dwheeler's proposal for fixing filenames: see http://www.dwheeler.com/essays/fixing-unix-linux-filenames.h...
These is no god damn reason why a filename should be able to contain, say, LF, DEL, or BEL. None whatsoever.
It's like suggesting we don't allow sql to store quotes so we can use quotes to enclose data.
Your stance is like being against ASLR because developers just shouldn't have buffer overflow vulnerabilities in their code.
Ambiguity would remain in how a given shell parses input to determine what are options and what are arguments: but this would at least be out of the control of individual programs. Notably, the shell would be the tool which parses the -- convention. Programs wouldn't see the -- delimiter which separates options from non-options, so it would be impossible for a program to neglect to implement support for --.
OK you want ASCII 0x07 to be disallowed. Should a filename be allowed to contain "㜇"? (U+3707)
Firstly, its IEEE standard (1003.1 or "POSIX") specifies the -- convention for separating option arguments from non-option arguments. The tiny handful of utilities like "echo" which do not implement it are also documented that way.
Secondly, Unix provides the POSIX standard getopt C library function, and getopts command. Programs and scripts which use these standard functions for processing options will implicitly support the -- convention.
Developers of new command line programs can ignore the documentation and standard functions, of course, developing their own non-conforming parsing from scratch. But at least users have something to point to if they report that as a problem: look, your program isn't supporting --, meaning that you ignored both the POSIX standard convention and the library function which enforces it.
Only if the designer is using command-based functions like the ISO C system or POSIX popen; not when forking and exec'ing programs.
The problem is also YAGNI and validation-thru-testing, instead of up-front design. The "ad-hoc" and "text" parts aren't what's important, it's the whole approach of not doing any more than the bare minimum of up-front work. Which seems to historically give overall better results, even if it does come with interesting bugs that need fixing later.
And while it's true that text isn't a necessary part of the general problem, in my experience text-based formats seem especially prone to it. How many times have you seen people attempt to use regular expressions to parse HTML/validate e-mail addresses/whatever?
rename *.txt *.bak
In the Unix environment, the shell expands the asterisk to all files that matches that pattern, and then passes these files to the "rename" command, who never sees the asterisk. Therefore it's trivial to create a new "rename' utility because it doesn't need to parse wildcards. However, renaming all .txt to .bak is awkward in a UNIX system.
find --name *.txt --exec mv {} {}.bak\;
If this use case were important, it's simple enough to write a purpose-built program that generated command-line argument pairs according to a spec.
find -name '*.txt' -exec bash -c 'mv -nv "$0" "${0/.txt/.bak}"' {} \;
zmv -W '*.txt' '*.markdown'
Of course, avoiding the shell is the best way to avoid the problem. Sometimes, you can't avoid the shell.
My preferred shell quoting method, for unix, is to wrap each parameter in single quotes. Then only single quotes inside a parameter are a problem. They can be replaced with '"'"'
Probably a lot of things use double quotes and perhaps try to escape $ and ' and " but miss details to do with \ and perhaps other characters that some shells treat specially.
Another way is to pass the filename in the environment: system("rm -rf \"$DIR\"")
What circumstances have you got in mind?
>> What circumstances have you got in mind?
> My favorite awkward environment to cite is running a command remotely over ssh.
Dude, if you run remote commands by calling them through SSH, you didn't just got things backward: you fucked things up heavily. SSH was never ever designed as a batch, unsupervised tool, despite many people using it as such.
Remote code that is parametrized should be run exactly as that: as a remote procedure call, a technique known for over thirty years now. One of the reasons is quoting (because for non-interactive SSH call the command needs to be quoted exactly twice if in shell, and exactly once when run from exec()), but there are problems with distributing keys, maintaining usable home directories, and disabling unnecessary things that are enabled by default (port forwarding, among the others), and that doesn't exhaust the list of issues.
Proper RPC protocol, like XML-RPC (which was released twenty years ago and is still usable while being quite simple), covers quoting -- or actually, serializing data -- without programmers worrying if they got their list of metacharacters right and did enough passes for things to work correctly.
On the other hand, I'm not surprised that people do this through SSH (and a variant of this stupidity: adding apache user to sudoers, so a web panel can add firewall rules). After all, I've never seen an easy to use RPC server that has all the procedures passed in its configuration. I needed to write such thing myself (once in Perl, as xmlrpcd, and recently in Python, with custom protocol that can do a little more, as harpd of HarpCaller project).
Suggesting that RPC be used instead of SSH while ignoring security is terrible advice.
Security-wise, your advice to use something that makes building correct system virtually impossible (because quoting issues, unnecessary features enabled by default, and others) is simply stupid and dangerous.
And yet, sometimes you're working in an environment that already has a good ssh configuration for other reasons, and you're very low on engineering time that you can invest into something, and ssh is good enough for a first pass implementation. Alternately, you may be working on some kind of ad-hoc data collection or maintenance task that's not going to become part of any long-term infrastructure (or will be replaced by something better), and you don't yet have any better systems in place to run ad-hoc programs across the cluster.
I completely agree with you that good RPC is a much better foundation to build reliable systems on.
[edited to add]: HarpCaller looks like a pretty interesting project, and similar to several things I've considered building in the past. Nice work.
I would agree personally before I wrote xmlrpcd. After I wrote it, I, its author, have no excuses for using SSH as an RPC protocol. Though I'm not good on the marketing side, so I understand that people just don't know about such tools.
> Alternately, you may be working on some kind of ad-hoc data collection or maintenance task that's not going to become part of any long-term infrastructure (or will be replaced by something better), and you don't yet have any better systems in place to run ad-hoc programs across the cluster.
Honestly, this is yet another matter.
To properly manage a set of servers, one needs three different services[&], each for different thing. One service is for running predefined procedures (that can possibly be parametrized) -- this is what HarpCaller and earlier xmlrpcd are for. Another service is for managing configuration and scheduled jobs -- this is a place for CFEngine and Puppet. Then there is what you just said: a tool for running commands defined in an ad-hoc manner and collecting their output synchronously. From the three, the first and second don't match how SSH works and is used, but for the last one it actually makes sense.
[&] It doesn't have to be three services, but we don't have one that would cover all three in an uniform way.
> [edited to add]: HarpCaller looks like a pretty interesting project, and similar to several things I've considered building in the past. Nice work.
Thank you. I'm quite proud of how it turned out, and the middle part of it was an excellent pretext for me to write something for production use in Erlang.
SSH + unix + existing cli already have excellent answers to those. Plus more, like better response streaming.
That being said, SSH has plethora of problems as an RPC mechanism. Host key distribution sucks heavily if you have more than a handful of servers. User key distribution is even worse, unless you incorporate external mechanisms. You need to maintain a usable home directory for a service, which otherwise woudn't need such a thing. SSH has plenty of obscure functions, like port forwarding in three flavours, X11 forwarding, VPN baked in, and others. And to use SSH-as-RPC for a service you need to disable them. Are you sure you have covered every single one of them? And then there is also mixing a debugging channel with regular operations. Break just one and you cannot recover (and it's easy to break a debugging channel, as you want it reconfigured and limited to only allowed accounts and whatnot). Those two should be separated.
The only thing that SSH has better than XML-RPC is streaming a response. But first, it's a rarely used function for a setting where you need to execute a remote operation, and second, because it sometimes is actually useful, my HarpCaller (RPC daemon) needed a custom protocol.
SSH is very, very far from being an excellent protocol for running predefined remote operations, even if it were only for issues with quoting, which are difficult on their own.
Right. But then again, it's the not fully predefined operations that have the most need if escaping.
However, I agree that quoting for ad-hoc synchronous commands to be run through SSH is very troublesome and tiring, especially when one doesn't understand how the commands are executed through non-interactive SSH (and most people don't).
https://tools.ietf.org/html/rfc4254#section-6.5
And I guess it can't really do any quoting/escaping, because there's no guarantees in the protocol as to how such things will be interpreted server-side - the command line could be interpreted by cmd.exe on the server for all `ssh` knows ;)
With that in mind, I've made it a habit to always quote the command-and-parameters part of `ssh` lines into a single argument - I figure, that's what it's doing anyway, so it's best to be explicit about it. And when I know I'm working with bash on both ends, my pattern of choice is
ssh host "$(printf '%q ' cmd arg 'arg with spaces' "$var" ...)"The difference occurs when a process A created process B and process B spawns one or more processes.
— It is possible to “see” a command line (in "ps" output, etc.) in ways that you won’t see environment settings. This can be useful when passing information to a sub-process that you need to keep private.
— It may be that you are configuring your sub-process in a place that is “far away” from the point that actually executes the command. Rather than have to thread an extra command-line argument through your code to make sure it is part of the final command invocation, it can be quite convenient to just set a variable. I have often used this to enable debugging features or test experimental features, or even to disable entire features when unexpected problems arise.
— In a similar way, your program may use multiple languages or otherwise be difficult to manage in any common way without environment variables.
— In a cross-platform scenario, environment variable names might be far easier to keep constant across UNIX, Windows, etc. than command-line syntax.
I’m sure there are other reasons.
One argument for having command-line arguments is that it can be less typing, as argument names can be implied. Compare:
>cp foo bar
>from=foo to=bar cp
There's a longer debate about this topic on Stackoverflow titled "Argument passing strategy - environment variables vs. command line". [1]
[1]: http://stackoverflow.com/questions/7443366/argument-passing-...
echo *.txt | read_stdin_and_get_filenames
export SPOOLDIR=$HOME/myspooldir;
empty_spool_directory
echo *.txt >list_of_filenames
mycomand list_of_filenames~~Also, the paired asterisks ate some of your comment by treating it as an italics marker.~~ fixed now, thanks.
(that should read star dot txt in the line above, not sure how to disable the italics meaning of star in posts)
I don't think it is the case in Windows, and this seems not to have changed since DOS days, when some programs would be abled to handle wildcards (internally) while others could not, because it was done by the individual programs, not the shell (COMMAND.COM or nowadays CMD.EXE).
A quick test:
$ python -V Python 3.5.2
$ type test_arg_list.py
import sys print(sys.argv)
$ python test_arg_list.py a t b ['test_arg_list.py', 'a', 't', 'b']
(that should read t star (not just t) in both the lines above)
So wildcards are not expanded. I'm sure there are Windows calls to expand them (there were from the DOS days, like FindFirst and FindNext (awkward approach, IMO), but your program has to actually use them for the expansion to work.
In fact, that is what I did, via the Python glob module, in this recent post:
Simple directory lister with multiple wildcard arguments:
https://jugad2.blogspot.in/2016/12/simple-directory-lister-w...
Whereas, in Unix, the shell (at least sh / bash) does it automatically for all arguments for all command-line programs, before the program even sees the arguments. This is one of the (many) key benefits of the shell. In fact, all metacharacters are interpreted by the shell and/or the kernel, acting together. This includes redirections, piping, the many special symbols that start with $, backquotes, and many others.
Within that explanation, we all know that global/shared variables are a code-smell for the most part. Say you want to call the same command with different logic, or multiple times even.
result1 = func("hello", "world"); result2 = func("hello", "another world");
vs
greeting = "hello"; greetee = "world";
result1 = func(); //How do we even know that func uses greeting and greetee variables?!
greetee = "another world"; result2= func(); //Did func change my greeting variable? I don't know.
So let's assume that greeting and greetee are actual important variables. You are essentially then sharing your "state" with the func in order to alter its behavior. I think in some shells, the functions themselves can alter global environment variables, so it would be a giant mess making sure that functions are idempotent and don't have artifacts.
Environment variables also made shell scripts reusable by other users. The file $HOME/special would refer to the "special" file in the user's home directory.
Command line arguments are only passed to the one single child process. And if that process wants to launch a new process, it must create it's own command line arguments.
I think I may have the answer (at least for Unix). (IIRC had read about this somewhere a while ago, also had thought about this issue myself earlier, so it's a combo of reading the reason somewhere and (maybe) figuring it out. Anyway, here it is:
It is because it allows 3 different ways of setting options for commands: rc files, environment variables (env. vars from now) and command line arguments, with each subsequent one able to override the previous one. The logic being that they go, in order, from more permanent to less permanent (as settings). rc files (rc stands for run command, a term I think I read Unix inherited from some previous OS) are config files for commands, like .exrc and .vimrc for vi/vim, .bashrc for bash, .netrc and many more. Any command can create or require users to create its own rc file, and can use it if present to read settings. The setting in a file is less easy to change on the fly than an env. var (not really difficult, of course, just that you have to go edit that file in an editor - or use sed etc.), and an env. var in turn is (a bit) less easy to change than a command line option, when we are talking about multiple different invocations of a command, in which you want the values for that option to be different in some of the invocations.
Let's take the example of a setting for a port (for a network server or client):
First, put the most common and permanent setting for the option, say, PORT=8080, in the rc file, say .foorc (for command foo - whether foo is built-in or written by you).
Second, for times when you want to change it for say today's work, set (i.e. change) it via an env. var, like:
export PORT=8181
foo args ...
# this setting will remain in effect until you change the var or you logout/reboot, and as long as it is present, will override any PORT value in .foorc each time you run foo.
It can also be shortened to:
PORT=8181 foo args ...
# but this is now a one-time setting of the env. var, so will override any PORT value in .foorc for this run only.
# In both the above variants, the args will not include PORT, since the foo command will be written to check for an env. var called PORT internally (and similarly checks for a PORT setting in .foorc before checking for an env. var called PORT, with the latter overriding the former if both are present).
And third, for the time(s) when you want to change the PORT setting on the fly, maybe just once for today, do:
foo --port 8282 args
which will override the settings for port (if any) in both the rc file and the env. var.
So the order is: command line option overrides env. var. and env. var overrides rc file setting.
This is what I read/figured out. It gives a lot of flexibility. Many Unix commands work that way. If you want your own to work that way, you have to write the code for it, like checking for presence of the rc file and for the setting in it, checking for the env. var with getenv() and finally checking for the command line option.
Edit: for typos.
Also, I've never understood where this exit(-1) idiom comes from. It's nonsense.
Also dynamic scoping can be very powerful when stitching together pieces separately designed. To this day emacs lisp is still dynamically scoped by default and arguably it derives some of its power from it.
I still hate environment variables.
PROPS="-Dprop1=foo bar -Dprop2=bar foo"
java $PROPS SomeClass.class
You see this often rear its ugly head with daemon scripts.
I left an example of an argument with a space in there (the last one).
PROPS=("-Dprop1=foo" "bar" "-Dprop2=bar foo")
java "${PROPS[@]}" SomeClass.class
(edit: Please read the replies to my post, I didn't think about the fact that this syntax is bash specific. Thanks to those who pointed it out)
I tried to find a failsafe solution once while rewriting a daemon script and just gave up.
PROPS="'-Dprop1=foo bar' '-Dprop2=bar foo'"It's a sensible rule, when you think about it - otherwise, for example, an expansion that introduced mismatched quotes would cause total chaos.
Example
#!/bin/bash
Q1="'"
Q2='"'
echo "$Q1$Q2"
'"
#!/bin/bash
Q1="'"'"'
echo "$Q1"If it's an issue for you on Linux, you can always run a python shell and spawn your process there. On Windows you can't.
If you want spaces to be unparsed in parameters, you "quote" the parameter.
A="spaces in the argument"
touch "$A"
ls -l
total 0
-rw-rw-r-- 1 me me 0 Dec 27 15:40 a file with spaceshttps://github.com/PowerShell/PowerShell/pull/2182
It is extremely common to get this wrong. Apache Portable Runtime even gets this wrong :/. (I haven't submitted a patch for this yet, but I intend to: I ran into it a couple months ago and then got distracted after working around it in my program by predicting what incorrect escaping might be performed by APR and compensating by adding quotes and escape characters to my input to their open process function... my build is statically linked so I don't feel bad about this temporary hack ;P.)
There is no general correct way to quote command line arguments in Windows, because every application receives just a character string which it parses however it wants.
There is no single specification for the syntax by which arguments are delimited within the command string.
How do you avoid it if it's the only API available for spawning processes on Windows?
Process.Start(executable, args);
So while it seems to be pretty easily fixable, it hasn't been.
But yes, if MS ever fixes this it's more likely that they'll go into the route you described. I can't wait to see how many bug reports with crazy descriptions it will create, and how many more overlays will be written to fix the bugs preserving backward compatibility.
Edit: and the DOS / Windows exec family of functions was likely derived from Unix's exec().
It does seem rather misleading to say "everyone" does it wrong when it's specifically a problem with Windows APIs (though not terribly surprising from the Windows team).