Show HN: Nip – Use JavaScript instead of awk, sed, or grep | Hacker News Reader

Show HN: Nip – Use JavaScript instead of awk, sed, or grep

github.com

42 pointskolodny53 comments

I think we should all make an effort to try harder with our comments.

The author writes "this is for people who aren't 'devops' and who can't crank out a fancy piped shell script using awk and sed."

Whatever we may feel about people who have inexplicably failed to learn to use awk or sed, or perl, or python, or ruby, criticism on the basis of these methods being "better" (if only one learns how to use them) is a waste of everyone's time.

If you really want to make a point about how less performant javascript is than perl, that's a different matter. If you want to critique the goals, concept, implimentation, or whatever, that's a different matter.

But it's simply too easy to write "why not do 'X'", especially if you're going to ignore the fact that the constraints on the problem have already ruled out 'X' in the first place. That takes less than no thought. It takes an active deficit of thought.

Without putting too fine a point on it, I think people should make an effort to try harder with their tooling. "This is for people who aren't 'devops'" can be just as easily read as "this is for people too aggressively incurious to step outside their sinecure for even the teensiest of seconds and maybe learn something."

Reinventing every wheel "because $my_pet_language" is fractious, generally worse than the alternative, and creates fairly unproductive noise--I think it's reasonable that people should catch at least a little flak for doing it. I mean, if criticizing anti-cooperative behavior is a waste of time, what was engaging in it? The idea that bash and GNU coreutils are "devops" tools, instead of the tools of basically competent Unix-based software developers, is so ludicrous as to be worth a little roasting. (And it has been a little. Nobody here, not even me in this post, is being more than mild.)

HN doesn't go for gratuitous negativity, and that's pretty cool, but if this thread's any measuring stick, negativity at this sort of behavior is a long way from gratuitous.

In the end, nobody cares how I renamed all the images from lolimag_{i}.jpg to 2015_05_20_img_{i}.jpg, they just care that I did it.

Yeah sure if you mastered perl, grep/awk/sed and can whip up what you need in a few seconds, more power to you, but I'm not going to waste an hour of my time googling and reading man pages when I can take 3 seconds to write a quick function in JavaScript to do what I need for that one task.

Learning to use sed for that kind of trivial task would take, what, a minute? Maybe two? If you know regex, you know enough of sed to do anything, and if you're not using regex in your JavaScript transform you're quite inarguably (and I don't use that word lightly, but regular expressions really are required knowledge) screwing up anyway...so what's the point?

Polluting the global namespace with yet another tool that's worse at its job than the existing ones because of a particular breed of aggressive incuriosity and epistemic closure is real weird. (This isn't a failing unique to the JavaScript community. Go people love doing this, too, creating bad solutions to solved problems. You used to see it a lot in Ruby, but the community seems to have grown up a little.)

> Go people love doing this, too, creating bad solutions to solved problems.

My anecdote: most "Go people" doing this are actually "Go beginners" that are still learning the language and attempting to write something familiar to cement their understanding. I imagine this is often the case for other languages.

You'll also see this (that is, go replacements for common tools) as a direct result of a Suckless challenge to do exactly that. The idea being to get rid of the complexity of the current bash tools (let's be honest, when `cp` has options you would expect from `rsync`, it's gotten rather complex), with the side benefit of being statically compiled.

I'll admit, I've written a few of these myself to help nail down both my knowledge of Go and the basic toolset I use every day. It was quite useful as a learning tool, even though they'll never replace the originals.

Heck, I'd even recommend re-creating `dc` to anybody who is interested in learning a language, or writing a parser/interpreter for a language.

> You'll also see this (that is, go replacements for common tools) as a direct result of a Suckless challenge to do exactly that.

This is what I was referring to, yeah, and it is to my mind a sign of a certain amount of cultural insularity and maybe a little hubris. Reimplementing something like `dc` to learn a language is great, it's when you start movements to start replacing things--or even trying to be taken credibly as an alternative--that I think you had best come very correct or be ready to defend your contention that the status quo is for some reason unsuitable. "I don't already know sed" is not, to my mind, a good defense, which touched off this subthread.

basic regexp search and replace with sed is pretty simple, but it gets hairy VERY fast beyond that in my opinion. awk I love, I find it very simple and intuitive.

Also gnu awk, sed, and grep are not the same as BSD variants. The gnu versions have (some would say suffer from) a lot more features.

For me, anyone who uses unix and doesn't have working knowledge of how to use find, xargs, sed, awk, and grep is missing out on a lot and very likely wasting time reinventing wheels.

I'll use awk for that stuff when it occurs to me, sure. Most of what I do is pretty straightforward transforms, and chaining together a couple sed commands often comes to mind first.

I use the GNU tools for consistency, as it's easier to install them on a BSD system than the reverse, but your point is well-made.

No, nobody will care. However, you'll care when ten years down the road you will still working around your lack of knowledge about your operating system, when you could be spending that time working on something interesting.

Let's look at it briefly from the flip side. I'm in Devops myself, and I am quite confident chaining together seds, awks, and greps. Those didn't help me when I had to troubleshoot a Node application, though. What did? I also learned enough about Javascript to be able to not only navigate code, but debug what was happening.

Artifically limiting yourself by not taking these opportunities to learn about an OS you seem to spend a fair bit of time in is only hurting you.

sed and awk have very little role in modern operating systems. It's easy to be a Linux whiz and still have to look at the manpage every time you see an awk script pulling out columns.

Heck, awk is two generations removed. Even Perl one-liners are fading into history.

What do you see as a replacement for Awk, then? And what does your Linux whiz use instead of sed and Awk for getting things done?

Well, and in your comment here, you're writing something more substantive than "why not use...", which was literally the gist of several of the top level comments.

I'm not sure I would characterize it as a "because my pet language" tool; rather, it allow one's main production language to use shell pipes, which can be quite handy for a variety of uses.

Yeah, that's fair, I got a little carried away with that characterization. A thing that does shell piping is great. Where I get salty is the suggested use case of using it to replace bog-standard tools because they're not JavaScript. Monolinguists are bad hires in virtually any shop; at prior gigs I've dealt with loads of developers who honestly thought that anything that isn't the programming language they argue about and reinvent ways to concatenate and minify is "devops". People who push these mindsets are toxic and should be burned out of functional engineering cultures.

I like this a lot.

...

As a developer who, while (I like to think) competent, hasn't learned to use sed/awk very well .. I find it aggravating and at times arrogant that long-time users scoff at the idea of replacing them with more general and accessible tools. Why should I have to learn an esoteric syntax with loads of historical baggage and obtuse documentation(1), just because you put up with it?

(1) man pages suck, sorry. Good tool documentation starts with basic examples. If I just need to delete a line, search a file, or uncompress a tarball, I should be able to find the simple command in O(1) lines read.

...

Case in point, there's a comment here:

"Wait: nip 'function(l) { return /^var/.test(l); }' is supposed to be better than egrep '^var' ?"

Yes, it absolutely is better (except when performance really matters), because it's very easy to see how to extend that to more complex things, whereas I don't know how to extend egrep without doing lots of reading - if it's even possible without switching tools. And because I can trivially wrap the former in a shell script to produce the latter, more or less.

...

I feel this way about a lot of things. I think some people who 'grew up with' certain older technologies end up saying everyone else should use them (which may be true because they're omnipresent) and then extend that logic to say that they're actually good (which is not necessarily and often not true).

One of the big reasons to learn the old fashioned tools is that they're always there. Period. Yes, you have node on your server, and your development machine, but what about when you need to do some work over on the database server? Those antique shell utilities have saved my ass when my more preferred methods weren't available to me.

Though the example in question, you're putting in a huge amount of boilerplate code that you're going to have to write a wrapper for with each new script. Most of the basic tools -- sed, grep, awk, even perl -- tend to have a quick and dirty form that does what most people need, with the more powerful portions at the ready. The "new and improved" tools tend to end up a lot like your examples, and get swiftly ignored because it becomes a pain in the ass to type the same thing over and over again, or they grow shortcuts and secrets as time goes on until they too are full of history and shortcuts that only the old timers know. Someone's going to eventually want a patch for a command line flag that wraps all that boilerplate in, and you're going to end up with perl's -n flag, and the horribly ugly code of

    perl -ne '/^var/ and print'

which will confuse beginners, and be a godsend to veterans.

> what about when you need to do some work over on the database server?

You get the angry DBA coming over with a bat to smack sense into you because You Do Not Work On Servers.

> Good tool documentation starts with basic examples

No. Guides and tutorials do that. Documentations teaches how to use a tool, so you can understand it and fully utilize it.

For snippets, bit of working scripts, etc, you have how-to guides, and alike on the internet.

The documentation is targeted towards someone wanting to learn about the tool; not somebody who just wants to use it once.

I disagree. I don't mind if man pages continue to contain all the content they currently have. Just add a section at the top with basic examples and you halve my effort to start using them. That makes them better. Now, if we're picking about the definition of the word 'documentation' - well, I don't care. I just want utility.

It's especially better you lower the barrier to entry into the world of command-line-fu. There's a huge class of people in the world who will never, ever touch a command line tool - it's the "non-DevOps" people in the article. Or you can just call them 'non-engineers'.

I don't blame these people for avoiding them, either, because ours tools are actively hostile to new users. (in my opinion) The linux community frequently takes a stance of being actively hostile to new users, by intentionally making simple tasks difficult and refusing to provide the most basic of hand-holding.

And, of course you can look these things up on the internet, but by that argument we should do away with local man pages entirely. Nah, I'd rather just make the local documentation better.

> Just add a section at the top with basic examples and you halve my effort to start using them. That makes them better.

Why is moving examples from the bottom to the top better?

> There's a huge class of people in the world who will never, ever touch a command line tool

So that's your argument in defence of a new command line tool? Oh, the irony.

Also, how many of those non-cli users actually know enough programming so as to write raw JavaScript on the cli?

Examples at the top is better because experienced users who already know what the tool can do will find what they need regardless of where it's placed, but beginners need to have the barrier to entry lowered. It's not uncommon to open the document, not see what you need, and close it and go looking elsewhere. It's the same reason you put a useful description of your product on its landing page instead of nestled inside subdirectories.

I guess I shouldn't have mentioned the people who never touch command-line tools. It's the people who almost-never do who are more convertible. Anyway, there's a marked difference between an easy way to run code on the command line, and a special-purpose tool with special-purpose syntax. Those non-advanced users are the ones who would benefit from having something with easily composable parts. (this is the same argument for building lots of small CLI tools and then composing them together, taken further?)

I wager lots of amateur web developers can manage their way around basic JavaScript. It used to be people started coding with shell hacking so they were forced to learn the bizarre tool suite. Now they probably start with web development [citation needed].

Why use this instead of perl/sed/awk?

Because you program in JS all day long and don't want to look up man pages every time you want to do some simple text processing, that's why.

Why? (Yes, I read that section on the readme.)

If you learn sed and awk you can slice and dice text damn near everywhere. Nip is only going to be useful where you have enough control to install it.

If you already know sed/awk and you don't know JavaScript then as you say, it would make sense to just use the former tools.

But for those of us for whom it's the other way round, it makes far more sense to use the scripting language we already know for the occasional text processing task than to spend time learning another language for the purpose.

Unless you're really a JS-only developer, it's hard for me to see why this is better than using Perl (or just sed/awk/grep themselves) for text-munging.

I don't know perl or sed/awk/grep that well, but I know JavaScript pretty damn well.. So what do I do?

Look up man pages or Google Stack Overflow answers for 15 minutes to do what I need?

Or I can just use nip and write a quick snippet that does what I know and move onto what's most important, the task I was trying to accomplish.

In the end, nobody cares how I renamed all the images from lolimag_{i}.jpg to 2015_05_20_img_{i}.jpg, they just care that I did it.

I would argue that you could spend 1 hour of your time to learn the bare minimum of sed and grep and grow as developer/*nix user.

Learn Perl and increase your value instead of being a one-trick-pony !

OK, I am an intermediate to advanced BSD sed user. I can do things with only basic regex that I have seen others struggle to do with extended and Perl-compatible regex.

But I am as green as it gets when it comes to Javascript.

I am sometimes forced to use a Windows workstation.

So I installed "super sed" (ssed.exe), a Windows sed, from sed.sourceforge.net.

It is painfully slow. Ridiculously slow.

I'd like to try using Javascript to do some sed-style editing on text that comes through the browser.

Let's say I open up the Javascript console in a browser, e.g., Chrome.

I know little of Javascript but I do know about window.location() and XMLHttpRequest().

Assuming I can get the text I want to edit into the browser window... what do I do next?

If you have Chrome dev tools open, you can go to the console and define a variable that holds your text using an ES6 string template, like so:

   var text = `Hello World:
   Here is a multiline
   piece of text`

I would then follow this guide on JavaScript regex:

https://developer.mozilla.org/en/docs/Web/JavaScript/Guide/R...

Here's a short example/sample, based on the text above: (You should be able to copy-paste this directly into the console as well, following the declaration of `text` above)

    // Logs the lines to console that match a certain regex.
    // This is a very poor approximation of grep.
    var re = /(Hello|text)/g

    text.split("\n").forEach(function(line) {
      if (line.match(re)) {
        console.log(line);
      }
    });

Using the JavaScript console of your browser is a great way to learn the language, prototype ideas and sometimes to debug into your webapp. (Sometimes it feels like hotswap when using the JVM :))

I remember seeing someone who did this (for Python) with magic for auto-importing modules. I can't seem to find it for the life of me, but that feature was really nice.

I'll give a shoutout to a similar project I did with Python (but that doesn't auto-import modules) https://github.com/samzhang111/pit

Wait:

    nip 'function(l) { return /^var/.test(l); }'

is supposed to be better than

    egrep '^var'

?

I think he was demonstrating function syntax. This shorter version also works:

nip 'return /^var/.test(line)'

An implicit return, added if the input doesn't match /[function|return]/ would be nice, along with aliasing line as l, then you'd have:

nip '/^var/.test(l)'

I wrote a similar tool in Perl, called "flt" - short for filter. It executes the command for each line in the input file and gives me a bunch of extra functions to work with. I use it dozens of times every day, about as often as I use cat, grep and sort. But when the one-liner becomes 3-4 lines long, it's time to write a proper Perl script with its own source file.

I would appreciate some more common use-case side by side comparisons. I've been hesitant to learn awk or sed myself, instead using Ruby, but my intuition tells me that javascript will require comfort with at least some basic utilities library, and lacks a lot of built-in niceness that Ruby has.

A Python version of this is called 'The Pyed Piper', yes it was named before the show :p

https://code.google.com/p/pyp/

Most *nix machines will have Ruby already installed, it's much better suited, right out of the box, for this kind of work than Javascript is, having descended from Perl and all.

I wrote a similar CLI in Ruby (https://github.com/jpb/rb) inspired by the Python equivalent (https://github.com/Russell91/pythonpy). I've never really used rb though - these tools are very limited by their dependencies (in comparison to the common unix utilities you find on almost every system).

Or just use Perl! Perl is like sed/awk on steroids, anyway.

Perl has a quite steep learning curve. Not really worth picking up for anyone who would consider using OP's solution. Not when you can use Ruby, anyway. I started to learn Perl several years ago, switched to Ruby pretty quickly.

If a task is difficult and a tool has specific capabilities to deal with it, the only way is to learn that to solve the problem. Using a different/easy-to-learn tool doesn't reduce the complexity at hand. You will ultimately end up reinventing all the features in the easier tool, what was available out of the box in the original one.

This is precisely the problem with languages that make things easier for novices, once you are beyond that stage you will see yourself doing a lot of work to accomplish simple tasks. This is the reason I switched to Perl, from Python. Python was good for a month, once I started running into more complicated tasks, Perl starting emerging as the clear winner.

Never liked Python much, can understand why you'd switch. Still think you'd have been better off with Ruby. Ruby handles complexity like an old kung-fu master.

In my experience, Perl is preinstalled on a lot more systems than Ruby, but I fully admit that Ruby is a more pleasant/approachable language than Perl. I'm not that experienced with Ruby, but how is it for running one-liners from the shell?

I could see all the 'end's being kind of a pain. Perl's ugly terseness is arguably an advantage for that use case.

Isn't python usually considered great for string processing?

I don't know perl or Ruby, but I know JavaScript, what now?

Identify the value that learning more than just one language would offer you, and use the opportunity to broaden your horizons.

Javascript is not the be-all-end-all programming language. Neither is Ruby, Awk, Haskell, or Prolog. However knowing more than one will help you grow in your chosen profession.

I'm honestly curious why all three of your posts in this thread espouse not learning things.

This is awesome, simple and powerful. awk and sed are especially annoying to work with -- now I can use one of my favourite languages to do the same tasks.

I will shed a single tear for grep though.

Hmmm, how performant is it?

There's really not that many possible way to beat GNU grep at least in the regexp matching case. It's as close to the optimal solution as you'll find (especially with `--mmap`) and really, as close as is possible.

https://github.com/ggreer/the_silver_searcher

Okay, that does sound like it might be faster. If Ag unrolls the main loop and ignores line breaks unless there's a match, the JIT regex might well give you the decisive speed edge. (I'm ignoring the parallel part of Ag, obviously that's a winner in the ack use case.)

I'm going to seriously consider switching to Ag from ack, especially since my work source tree has recently switched to an SSD.

That 33% gain on typing speed is the real gain ;D

Reply on news.ycombinator.com