Comment by saidajigumi | Hacker News Reader

Comment by saidajigumi | Hacker News Reader

Parent Full thread

> Dreadful for the long term.

Here comes a bubble-bursting: I've lead a team that built data processing tools exactly like this, and the performance and ease of manipulating vast amounts of text using classic shell tools is hard to beat. We had no problems with any of: operational supportability, restart recovery, or maintainability. Highly testable, even. No, it's not just cowboy-coded crappy shell scripts and pipelines. Sure, there's a discipline to building pipelined tooling well, just as with any other kind of software. Your problems seem to stem from a lack of disciplined software engineering rather than the tools, or maybe just an environment that encouraged high technical debt.

The kicker? We were using pipeline-based tooling ... running on a Hadoop cluster. Honestly, I'm a bit surprised to see such an apparent mindshare split (judging by some recent HN posts) between performant single-system approaches and techniques used in-cluster. The point that "be sure your data is really, truly big data" is obviously well made, and still bears repetition. Yet the logical follow-on is that these technique are even more applicable to cluster usage. Why would anyone throw away multiple orders of magnitude performance going to a cluster-based approach?

Unix/POSIX backgrounds are pretty common among the Hacker News crowd. Not so in "Enterprise" development. (Beam me up Scottie, there's no intelligent life here, only risk avoidance)

Enterprise development is predominated by 2 or 3 trusted operating systems: Windows (/ .NET), and the JVM. POSIX systems are only useful in-so-far-as they are a cheaper (or sometimes more reliable) place to host Java virtual machines. Enterprise dev groups generally have very limited exposure to, and a lot of fear of, things like Borne shell, AWK, Perl, Python. These languages don't have Visual Studio or Eclipse to hold your hand while you make far reaching refactorings like renaming a variable.

Sure, you and I would crawl log/data files trivially with a few piped commands, but that's a rare skill in most shops, at least since the turn of the century.

Ugh, that sounds cliche, but it's hard not to feel that way after being drowned in "Java or nothing" for so long at work.

http://tvtropes.org/pmwiki/pmwiki.php/Main/ElegantWeaponForA...

I agree with @roboprog. Most software shops employ engineers who don't have exposure into UNIX tools. Only few hardcore engineers have exposure or interest learning UNIX tools. For majority of engineers it is just a job. They simply use the same tool for everything. And they tend to use the tools that seem to get them into well paying jobs. If hadoop can get them good paying jobs, they would like us "hadoop" for something in their current job, even if that job can performed by a set of CLI utils. I have seen 100s of resume builder projects in the my past experience.

Windows has PowerShell, which can be even more powerful than Unix shell depending on the data and processing required.

Does anybody without a unix/POSIX background even bother tinkering with PowerShell? Yes, it's a cool idea, as well, especially if your source data is in MS-Office, but I've not seen it put to much use.

The problem with shell scripting is that nearly nobody is very, very good at it. The Steam bug doing an rm -rf / is an example, but it's very common for shell scripts to have horrible error handling and checks for important things. The shell is just not suitable for extremely robust programs. I would bet that 80%+ of people who think they're good at shell scripting... aren't.

Yeah but 80% of the people writing Java and think they're good aren't as well. And plenty of companies support Java.

The answer isn't "don't use it", it's "train your programmers in the languages they use".

The sorts of bugs people experience with Java mostly result in a crashed/stalled/hung process. Bash bugs erase your entire file system. The thing about Bash is that it is trivially easy to make these sorts of mistakes- the language just isn't suitable to general purpose scripting.

> Bash bugs erase your entire file system.

    EMPTY=""
    rm -rf $EMPTY/

Is this the kind of bug you're referring to?

> The thing about Bash is that it is trivially easy to make these sorts of mistakes

I fail to see how any other scripting language would have a different effect when you told it to do:

    system("rm -rf "+""+"/")

> the language just isn't suitable to general purpose scripting.

Yes, it is. Bash is deeper than it looks, but not by much. Learn how to handle errors and you'll be fine.

It shouldn't be able to erase your filesystem unless you are running as root or doing something equally stupid. That's pretty much common sense stuff for anyone that isn't a beginner.

Yeah the "common sense stuff for anyone that isn't a beginner" argument is repeated ad nauseam, and even the largest companies make this mistake in their largest products. Take Valve - they should know how to write good code, right? And yet, last week an article was on top of HN, outlining how they put:

"rm -rf '$STEAMROOT'/*" in their code, used to remove the library. But hey, no one checked if $STEAMROOT is not empty, so when it was for one user, Steam deleted all of his personal files, whole /home and /media directories next time it started.

I'm not saying that command line tools shouldn't be used,but sometimes they are just too powerful for some users,and stupid mistakes like this happen.

You're right to an extent, but this isn't relevant to the Java vs Bash discussion. The largest companies make this kind of mistake in whatever language they happen to use.

People delete data and screw things up in MapReduce jobs for Hadoop. A lot.

If you're worried about that, don't give the script permissions to access your entire filesystem. Easily handled with separate users, cgroups, assorted containerisation, and more.

> The problem with shell scripting is that nearly nobody is very, very good at it. The Steam bug doing an rm -rf / is an example

The steam bug is an example of of utter incompetence; not of someone not being very, very good at it. Whoever is happy with shipping `rm -rf $VAR/` without extreme checking around it should get their computer driving license revoked.

> The shell is just not suitable for extremely robust programs.

Incorrect. "The shell" can go as robust as you can handle. In bash, `set -e` will kill your script if any of the sub-commands fail (although ideally you'll be testing $? (exit code of prev. op) at the critical junctions), `set -u` will error on usage of undefined variables, etc.

A huge part of the "glue" that holds your favourite linux distro together is bash files.

> I would bet that 80%+ of people who think they're good at shell scripting... aren't.

The same probably goes for driving[1], this doesn't make cars any less robust.

[1] http://en.wikipedia.org/wiki/Illusory_superiority

> rm -rf $VAR/

> /

facepaw.jpg

Without the trailing slash, null or undefined $VAR would cause an error instead of a request to delete all the things.

The Steam line was more like ` rm -rf $VAR/* `, eg, they didn't want to delete the $VAR directory. Still, ` rm -rf /* ` is no fun.

Part of the journey into Linuxdom is learning a healthy dose of fear for that command. I always pause over the enter key for a few seconds, even when I'm sure I haven't typo'ed.

> The same probably goes for driving[1], this doesn't make cars any less robust.

I don't think I can imagine anything less robust than cars, in terms of the frequency and severity of operational failure. They're pretty much the deadliest thing we've ever invented that wasn't actually designed to kill people.

It's actually a good example of the point developer1 was making: cars and shell scripts are perfectly safe if operated by highly competent people, and only become (extremely) dangerous when operated by incompetents, but in practice most operators are incompetent, in denial, and refuse to learn from others' mistakes.

> I don't think I can imagine anything less robust than cars, in terms of the frequency and severity of operational failure.

Maybe US cars :P

> They're pretty much the deadliest thing we've ever invented that wasn't actually designed to kill people.

It's a box weighing 1-2 tons that travels at 100kmh+. Millions (billions?) of km are driven every year. There will be accidents for both good drivers and bad. This won't change.

> cars and shell scripts are perfectly safe if operated by highly competent people, and only become (extremely) dangerous when operated by incompetents, but in practice most operators are incompetent, in denial, and refuse to learn from others' mistakes.

That's simply untrue - both points. Highly competent drivers will have accidents. I highly doubt you feel extreme danger when you get behind the wheel/in a car. The way you phrase it, one expects millions of fatalities daily.

> The shell is just not suitable for extremely robust programs.

Absolute statements like this are usually wrong. This one does not escape the rule. When Linux distros init is mostly bash scripting, there is very little need to further prove that robust systems can be written in bash scripting without the language fighting the developer.

Wait, is it really a good argument for shell-based approach when all major distros are switching to the systemd due to the configuration/maintainability/boilerplate issues with bash init scripting?

I'm not going into the systemd VS sysvinit discussion. For my argumentation, it is enough to recognize bash based sysvinit has been with us for circa 20 years with no stability problems.

I think most of was was written also applies to any normal programming language. You could write this in Python, Ruby, Javascript, Java or C# without any problems. The code would probably be easier to read also. The only special thing is the web page scraping that could be done by a library but the same thinking about scalability and the use of a single computer instead of a hadoop cluster still holds even if you're reading from file systems or databases.

Yeah I learned this trying to massage some data with Elasticsearch. curl -XDELETE host/index/type/$id ... Except $id didn't exist.

Something to keep in mind is that while a single app might be best served on a single machine piping data, multiple apps working the same data set probably wouldn't scale. Hadoop for all its faults does provide a nice, relativily simple programing platform to support multiple data processes.

Edited for phone swipe mistakes.

This "orders of magnitude" less performance going to a Hadoop cluster approach is nonsense.

There are plenty of options for Hadoop that make it dramatically faster than the naive example in the article. Spark ? MR-Redis ? Storm ?

Reply on news.ycombinator.com