/usr/bin/time: not the command you think you know | Hacker News Reader

time: not the command you think you know

hackernoon.com

253 pointsactivatedgeek76 comments

The reason 'time' is a builtin is because it can time a complete shell pipeline, not just a single command. If you type

    time foo |bar

Then the result is the total time taken by foo and bar together. This requires it to have special-case syntax. Whereas

    /usr/bin/time foo |bar

Would run foo and give its time statistics as input to bar.

Yes, as a consequence, it also lets you do this:

    time {
      sleep 0.1
      sleep 0.2
    }
    # result is 0.3 seconds

which you obviously can't do with an external command. I wrote about that here:

https://lobste.rs/s/hwgt2h/usr_bin_time_not_command_you_thin...

The `time` syntax in combination with for loops or just shell functions is pretty useful as well.

It looks like

  /usr/bin/time foo |bar

would actually give the output of foo as input to bar, since the time statistics go to stderr.

Since it is not an error, what we really need is a stdmeta file descriptor:

http://unix.stackexchange.com/questions/197809/propose-addit...

Just think of it as std dbg, not err

I've been thinking a bit about this before and I think that not just one but a whole lot of additional streams would be useful. By default all commands would still use stdout and stderr only but you would be able to tell them to use other streams as well for their various data.

Furthermore, each stream should be possible to pipe to a different process if you wish independently of the other streams.

To a small degree this might be possible already with named pipes.

I just started playing around with IPC ideas with processes tellin each other which flags they would like: https://github.com/mrquincle/bipipes

I think socket pairs would be great for it.

Nothing implemented yet!

Honestly this is one area where PowerShell kind of nailed it.

I still don't use it enough, but when it comes to scripting PS's objects beat the hell out of these

It is possible already - you can do `command 2>&4` to separate the output from one command in a pipeline into its own stream, then do `command2 <&4` to pull from the further down the pipeline.

I would like more pipes for each process.

This would allow many things, for example searching for multiple terms across all files in the whole file system in one go without opening each file more than once, and then grouping the different results together in accordance with the term that they matched.

Run these commands for a better demonstration:

    time echo | sleep 1

vs.

    /usr/bin/time echo | sleep 1

The former times the entire pipeline whereas the later only times the first command in the pipe.

This also gives the time of the entire pipeline (since the commands all begin together and only finish when the last command has completed):

echo | /usr/bin/time sleep 1

I'm pretty sure this is wrong.

As an example :

sleep 2 | /usr/bin/time sleep 1

This gives an output of "1 second" even though the entire pipeline took 2 seconds.

Ok you can wangle your way out of it. But if you're doing something like `grep "$something" | /usr/bin/time some_job` then generally you're fine.

Wouldn't the system/user time be wrong though?

The sleep sub-process of time is not part of the pipeline, however. It will exit after the requested time has passed.

That is merely an optimization though. You can simply

    time sh -c 'foo | bar'

But now you are timing the startup of `sh` as well.

...which (surprisingly) is a measurable amount. About 1 ms for `dash` and a whooping 9.5 ms for `bash` on a relatively slow CPU:

  pi@pi:~ $ time (for x in {1..1000}; do time echo | sleep 0.01; done 2>/dev/null)
  real    0m24.173s

  pi@pi:~ $ time (for x in {1..1000}; do time sh -c 'echo | sleep 0.01'; done 2>/dev/null)
  real    0m25.170s

  pi@pi:~ $ time (for x in {1..1000}; do time bash -c 'echo | sleep 0.01'; done 2>/dev/null)
  real    0m33.643s

1ms is on the order of process creation time, so not very relevant: The measured program itself requires process creation. Most programs run considerably longer than that (otherwise timing would not make much sense).

It's really not more than that: an optimization.

`/usr/bin/time -o foo_time.txt foo | bar`

No, it wouldn't require special-case syntax. It could just pass it's argument to '/bin/sh -c'. You'd have to shell-escape the pipeline, though.

Yeah it does require special-case syntax. When using bash's built-in time, the command is executed exactly the same way as if you weren't using time, which means the command can set variables or define functions in the current shell. For example, you can say

  time foo=bar
  echo $foo

and it will correctly print out "bar". It can't do this if it executes the command in a subshell. Or similarly you can say `time cd /foo` and it will actually change the current working directory.

Wow — that is extremely surprising. I did not expect that behavior at all. Who the hell thought it was a good idea to let a builtin command absorb the pipe symbol?!

On bash you can do this to have it not use an alias or built-in command:

  $ \time echo

  0.00user 0.00system 0:00.00elapsed 100%CPU (0avgtext+0avgdata 1764maxresident)k
  0inputs+0outputs (0major+66minor)pagefaults 0swaps

The built-in version:

  $ time echo


  real    0m0.000s
  user    0m0.000s
  sys     0m0.000s

Is this behaviour documented anywhere? The closest I can find is https://www.gnu.org/software/bash/manual/bashref.html#Aliase... which remarks,

> The first word of each simple command, if unquoted, is checked to see if it has an alias.

I observe that t\ime works just as well as \time.

Just quoting it normally also works, apparently, as your excerpt from the man page suggests:

  $ help|head -1
  GNU bash, version 4.3.30(1)-release (x86_64-pc-linux-gnu)
  $ "time" --version
  GNU time 1.7
  $ time --version
  bash: --version: command not found
  
  real	0m0.001s
  user	0m0.000s
  sys	0m0.000s

t\ime is rather surprising. Is there any documentation or explanation on why it does what it does?

https://www.gnu.org/software/bash/manual/bashref.html#Escape...

I think it's because using the backslash to escape counts as quoting. So the following are all equivalent:

  $ \time foo
  $ t\ime foo
  $ 'time' foo

The conventional way of forcing bash not to use a built-in is by using the full path (/usr/bin/time in this case).

The more obvious way of doing this is

  command time echo

The usage of the `command` keyword makes it ignore shell functions (including built-in shell keywords like time).

Since the article doesn't clarify on shell command priority, commands are search in order:

1. Alias expansion.

2. Defined shell functions.

3. Built-in shell functions.

4. Command path.

If you provide an unqualified command that matches more than one of these elements, the first match wins.

Prepending a backstroke: "\command", will inhibit alias expansion. E.g.:

    alias date="echo no date"
    date
    \date

Should return "no date", and your current system date, respectively.

To invoke a system command directly, call the full path. If you don't feel like running the fish shell (not that there's anything wrong with that).

Some simple shells (e.g., dash, and IIRC the original Bourne shell, though that is not what you'll find as /bin/sh on most modern systems) don't include a time builtin, and can invoke the system time command directly.

You can also use the 'command' built-in to skip aliases and shell functions.

$ command time --version GNU time 1.7

Perhaps, but time is a syntactic form, like `if` or `case`.

https://www.gnu.org/software/bash/manual/html_node/Pipelines...

I'm sorry, that's not clear to me. What's the significance exactly?

A command takes a series of arguments, stdin/out/err pipes and returns an error code on completion.

time prefixes a pipeline ( the most basic case being a single command without a pipe into another ), a command block { ... } a subshell block ( ... ), a for statement, an if statement, just whatever really.

This "time" would output how long "a" took to run:

    /usr/bin/time a b c | { d e ; f g ; } | h i ;

This "time" outputs how long the pipeline "a", "d", "f", and "h" took to run:

    time a b c | { d e ; f g ; } | h i ;

This is a syntax error:

    /usr/bin/time { d e ; f g ; }

This returns how long the command group takes to run:

    time { d e ; f g ; }

Further examples:

    $ time sleep 10 | sleep 1 ;
    
    real	0m10.073s
    user	0m0.000s
    sys	0m0.000s
    $ time sleep 1 | sleep 10 ;
    
    real	0m10.003s
    user	0m0.000s
    sys	0m0.000s
    $ /usr/bin/time sleep 10 | sleep 1 ;
    0.00user 0.00system 0:10.00elapsed 0%CPU (0avgtext+0avgdata 1796maxresident)k
    0inputs+0outputs (0major+80minor)pagefaults 0swaps
    $ 
    $ /usr/bin/time sleep 1 | sleep 10 ;
    0.00user 0.00system 0:01.00elapsed 0%CPU (0avgtext+0avgdata 1808maxresident)k
    0inputs+0outputs (0major+82minor)pagefaults 0swaps

Note the two /usr/bin/time's output their timing information as soon as the first command is done, but the pipeline doesn't return until both commands have exited.

> hope this rant helps in some way

Thanks, that does raise some interesting points.

What of:

    /usr/bin/time ( sleep 1 | sleep 10 )

Explicitly invvoking a subshell. Which I understand the builtin to be doing.

A more interesting command is /bin/[.

Yep, it's the same as /bin/test, but basically requires the last argument to be a ]. Typically /bin/test and bin/[ are hard links, that is, they are physically the same file and share the same inode(s).

You know, so you can type

  if [ $foo = $bar ]; then
    yes
  else
    exit
  fi

This actually runs the command /bin/[, except most of the time it doesn't, because it's built-in to the shell.

As as side-note, an article like this should at least mention which OS and shell the author is using.

> As as side-note, an article like this should at least mention which OS and shell the author is using.

Very much agree! I just tried this on my 2.6.32 RHEL system, and it's never heard of "-l". It outputs very similar-looking information as in the article, though, when given "-v" .

In both bash and zsh, you can force the shell to use $PATH for lookup (bypassing functions and shell builtins) by calling a builtin name with 'command' ('command time -l ls'). You can equivalently force a builtin with 'builtin', but that does not work with reserved words (and 'time' is a shell reserved word).

TIL that a shell reserved word is different from a shell builtin.

http://unix.stackexchange.com/questions/267761/differences-b...

Me, too. I was surprised because I was trying to use `builtin time`. :)

Yeah. Shell semantics can be pretty unintuitive sometimes. I often find it helpful to translate these ideas to standard programming language terms.

* Commands are like functions * Commands in /bin etc. are like library functions * Builtins are like a language's primitive functions * Keywords are keywords

You can also just prefix the command with a backslash. "\time" will run the version in the path.

    $ which time
    time: shell reserved word
    $ ls /usr/bin/time /bin/time
    ls: cannot access '/usr/bin/time': No such file or directory
    ls: cannot access '/bin/time': No such file or directory

Looks like something specific to the author's distribution.

He is on MacOS.

It's time -l on BSD derivatives and time -v on GNU userlands.

It's a BSD thing.

There's a GNU time as well, but it appears not to always be installed by default.

Yep.

GNU source: http://git.savannah.gnu.org/cgit/time.git/tree/

BSD source: https://github.com/freebsd/freebsd/blob/f5d95e1f8d32db4ccccf...

Busybox source: https://github.com/mirror/busybox/blob/f88e3bfa8ded4f1c7e7d2...

Toybox source: https://github.com/landley/toybox/blob/master/toys/posix/tim...

This would have been a lot easier if I'd known GitHub has a `filename:` search

Exists on a debian(/testing?) system I've:

$ time ls / bin boot dev etc home lib lib64 lost+found media mnt opt proc root run sbin srv sys tmp usr var

real 0m0.004s

user 0m0.000s

sys 0m0.000s

$ /usr/bin/time ls /

bin boot dev etc home lib lib64 lost+found media mnt opt proc root run sbin srv sys tmp usr var

0.00user 0.00system 0:00.00elapsed 0%CPU (0avgtext+0avgdata 2304maxresident)k

0inputs+0outputs (0major+109minor)pagefaults 0swaps

Edit: formatting

Sometimes. It depends whether you have the time[1] package installed.

1. https://packages.debian.org/sid/time

On my Android system with Termux installed, the system 'time' is the Busybox implementation, so yes, YMMV.

Which gives new meaning to the relativity and vagueries of time....

I had to use "--verbose" instead of "-l" in my RHEL7

  \time --verbose  echo

same on my ubuntu

Same on Debian Stretch

I bet you're using GNU time rather than BSD time.

It did seem odd to me that the author didn't bother to mention which OS he is using, though from the hostname I have a pretty good guess.

No, /usr/bin/time is indeed what I know, and its extended stats is why I suggested using it in my last perf book (time -v).

"/usr/bin/time: not the command you think you know" -> "/usr/bin/time: may not be the command you think you know"

There, I fixed the title.

/r/iamverysmart

Or even "may not be the command I think you know".

/usr/bin/time doesn't seem to be a thing on my Arch Linux install. ¯\_(ツ)_/¯

Same here. I had to fetch it via pacman. Also, the -l option doesn't exist on the non-bsd version.

`-v` (--verbose) Seems to give the same (although differently formatted) output on GNU's implementation.

I love arch. Sudo doesn't exist until you install it (directly or indirectly).

That's the same on Debian.

Yeah, I don't think it's all that rare for sudo to be an optional package not installed by default in distributions. At least that used to be the case back when I used try various ones a lot more.

  $ pacman -Fo /usr/bin/time
  usr/bin/time is owned by extra/time 1.7-8

If that command doesn't work for you, try adding "-y". `man pacman` for details.

So also are /usr/bin/{cd,[,echo,pwd,fg,..etc} a lot of them having subtle differences with their corresponding shell builtins. Most of the time the differences are not worth the hassle to remember, unless you somehow end up on a system with a broken filesystem (for example where /lib or /usr/lib is destroyed) and need to rescue stuff.

On Fedora Linux, '/usr/bin/time -v <command>' rather than -l.

So it's just like /usr/bin/cd–a builtin that also has a binary.

How would that work? A forked/execed subprocess somehow forcing chdir() in it's parent?

Guessing it isn't terribly useful.

I too don't see how it is useful but it's certainly a thing. The Solaris implementation looks like this:

    #!/bin/ksh -p
    # ...
    cmd=`basename $0`
    $cmd "$@"

I just noticed that the what(1)-string (I haven't seen on of those for a long time) references "alias.sh", perhaps this is a clue?

    #ident  "@(#)alias.sh   1.2     00/02/15 SMI"

Were builtins actually aliases in an early shell? I still don't understand how this works though.

That's a funny implementation. It would end up being an infinite loop for any command that wasn't a builtin, or didn't have an identically named thing higher up the PATH. Like if you renamed it from /usr/bin/cd to /usr/bin/mycd.

On my Mac, it's

  #!/bin/sh
  # $FreeBSD: src/usr.bin/alias/generic.sh,v 1.2 2005/10/24 22:32:19 cperciva Exp $
  # This file is in the public domain.
  builtin `echo ${0##*/} | tr \[:upper:] \[:lower:]` ${1+"$@"}

Guessing it isn't terribly useful.

You guess correctly. The issue is that POSIX states that

    ... all of the standard utilities [...] shall be implemented in a manner
    so that they can be accessed via the exec family of functions as defined
    in the System Interfaces volume of POSIX.1-2008 and can be invoked
    directly by those standard utilities that require it (env, find, nice,
    nohup, time, xargs).

Note however that /usr/bin/cd isn't completely useless: It has the same diagnostics (error message and exit code) as the command, so `env cd $FOO` is a way to check if you can change to that directory without actually doing it.

Reply on news.ycombinator.com