Unix Command Lines and Relations
merrigrove.blogspot.com
merrigrove.blogspot.com
If only I didn't have exams! (Been working on it 50% of the time though.)
Like when people shed a "new" light on seemingly simple tools.
Please, people, if you're using Blogspot, stop. Or at the very least, avoid the craptacular "dynamic" themes like the pox they are.
cat file | grep 'expression'
select * from (select * from table) as q where column like '%expression%'
It's also more expensive:
# wc -l /var/log/secure
97845 /var/log/secure
# time cat /var/log/secure | grep root > /dev/null
real 0m1.600s
user 0m1.517s
sys 0m0.294s
# time grep root /var/log/secure > /dev/null
real 0m1.275s
user 0m1.237s
sys 0m0.036s
As a simple example: cat somefiles.txt | pipeline_script > stored_output.txt
Other times I'm reading gzipped files or remote log files, I don't want that data mixed in with the pipeline logic. If I want to move the output files to a different directory on one set of servers, that may not impact my generalized pipeline.
I work with trillions of lines of log files and there are many ways to scale up pipelines. I wouldn't start optimizing the difference between 1.6s and 1.275s unless it made an economic impact on the problem. How often is it run? Can you process more lines in an economic unit of time with the faster version? If this is a job that runs once an hour or even once a minute how many lines are collected during that time?
Intermixing the grep or wc -l logic into the pipeline logic can have adverse support and maintenance costs and many times saves machine time while spending programmer time. Which one dominates in this scenario?
It's like the old saying...there is more than one way to skin a "cat". :)
You're assuming a constant factor here. That might be the case. Or it could be one-time start-up overhead as the linker finds and opens library files.
Unless you profile the process over a range of input sizes, you don't know which you're observing. And you know what they say about premature optimization.
No. And if you actually go to the link to the original question, you'll see that cat doesn't exist.
Last, I would disagree that `cat file` is equivalent to `select * from table`, his argument makes the comparison that `cat file` is equivalent to `table` itself, or `load data into table` which needs to happen before any relation can be performed against it.
You could condense the first three stages: cat /var/log/nginx-access.log | grep "GET" | awk -F'"' '{print $6}'
down into: awk -F'"' '/GET/{print $6}' /var/log/nginx-access.log as well.
Or with the fourth stage (cut -d" " -f1): awk -F'"' '/GET/{split($6,a," ");print a[0]}' /var/log/nginx-access.log
Add the fifth stage (grep -E "^[[:alnum:]]) by doing:
awk -F'"' '/GET/{split($6,a," ");if(a[0]~/^[[:alnum:]]/){print a[0]}}' /var/log/nginx-access.log
And the 6th and 7th (sort | uniq -c):
awk -F'"' '/GET/{split($6,a," ");if(a[0]~/^[[:alnum:]]/){b[a[0]]++}}END{for(i in b){print b[i} " " i}}' /var/log/nginx-access.log | sort -rn
Which is still just about a one liner and a lot faster then the original. Actually you could make it shorter, even faster, and more awky by tinkering with the field-separator to get rid of the split() and if:
awk -F'[[:space:]]|"' '/GET/ && $17~/^[[:alnum:]]/{a[$17]++}END{for(i in a){print a[i] " " i}}' | sort -rn
And then replace the last sort with awk's builtin asort() function but thats left as an exercise for the student ;)
But why learn the basics when you can be do Big Data and be buzz word compliant instead.