Hacker News Reader

Comment by ninjamayo | Hacker News Reader

why?

There was a post earlier on using command line tools instead of Hadoop for quick data processing. This shows a non-trivial example of how you could implement a complex data pipeline and an overview of some of the commands you could learn if you’re interested.

Yeah, this is a pretty cool post I never considered doing something like this in the shell. It seems silly to me that I forget how powerful basic tooling in the native shell can be.. sometimes I have a jackhammer and I forget that a sledgehammer will do the job just fine.

I've been there. 100s of lines into a ruby program then "oh yeah, cut sort grep"

Just for anyone else, you could probably end up in a similar corner with perl - but in both cases it's likely a case of "holding it wrong" - ruby borrows heavily from perl which borrowed heavily from shell with sed, awk, grep, cut and friends.

So this kind of thing should be quite doable in a short ruby script - or a few short scripts - albeit written in "shell" style, with eg '-n or -p (wrap code in "while gets...end",-p with "puts _"), probably along with -a (automatically split lines).

Its in some senses an entirely different dialect of ruby, though.

Some examples here:

https://github.com/learnbyexample/Command-line-text-processi...

Because there's a big trend towards overengineering. I checked the Azure search solution, and my god it's complex. I ended up doing it in sqlite, and it perfect for a small fts document search engine. The article is similar: try to keep it simple; there are barely any cases where you'll use the real benefits of something lkke Azure search.

> I ended up doing it in sqlite, and it perfect for a small fts document search engine.

It even has it as a feature!

    CREATE VIRTUAL TABLE something USING fts5(x, y, z)

Reply on news.ycombinator.com