Hacker News Reader

Comment by mrb | Hacker News Reader

Heather, I agree that people are unnecessarily rude to you. Don't sob. Your code shows you are already a more capable programmer than most of the ones I interview. No one has given you a clear answer as to why your code is reinventing the wheel, so allow me to do it politely. I took every single example from your README, and show you below how everything can be reimplemented with sed -r (nice extended regex syntax, mostly like js regexs), and find/xargs:

  replace 'var' 'let' *
  sed -ri 's,var,let,g' *
 
  replace 'var' 'let' . -r
  find . -type f -print0 | xargs -0 sed -ri 's,var,let,g'
 
  replace 'var' 'let' test/file1.js test/file2.js
  sed -ri 's,var,let,g' test/file1.js test/file2.js
 
  replace '(\w+)_(\w+)' '$1-$2' *
  sed -ri 's,(\w+)_(\w+),\1-\2,g' *
 
  replace 'var' 'let' . -r --include="*.js"
  find . -type f -name '*.js' -print0 | xargs -0 sed -ri 's,var,let,g'
 
  replace 'var' 'let' . -r --exclude="*.min.js,*.py"
  find . -type f ! -name '*.min.js' ! -name '*.py' -print0 | xargs -0 sed -ri 's,var,let,g'
 
  replace 'var' 'let' . -r --preview
  find . -type f -exec sh -c "echo == {}; sed -r 's,var,let,g' {}" \;
  (not sure if --preview does exactly that, I echo the filename followed by the modified content)

There is only one case that justifies reimplementing things: if your tool has the requirement of supporting the exact js regex syntax, then yes you did the right thing to reimplement this in js. I have run into similar situations myself. One time I had to support Perl regexs, and I started by simply using Python's standard regex module, thinking that it would work because both Perl and Python use PCRE. Well as it turned out the regexs I encountered used some advanced features (such as negative/postive look-behind/look-ahead, etc) that Python's regex module plainly did not suppport. So I ended up rewriting part of my implementation in Perl.

Edit: I spoke too fast. As a commenter pointed out, other legitimate cases for reimplementing this in js would be when you can't afford to or don't want to fork a process to run find/xargs/sed. It sounds like you were running the tool from the command line, so I didn't think that would be your situation.

Edit 2: Yes, the exercise of reinventing the wheel is also useful for learning... I am not going to argue that.

Edit 3: If you care about simplicity, I would personally rather write a small shell script wrapping find/xargs/sed and hiding their arcane options, as opposed to writing 173 lines of js.

View on HN

Honestly, I think you ended up really making the case for replace. The nix CLI examples are convoluted at best.

I've used CLI tools for years, and it would probably take me 5-10 minutes of scanning the man pages to reproduce all but the simplest of those snippets. Granted, this boils down to the sporadic use-case, but that's the entire point.

The replace examples are all clean. It would mean I would not be juggling little peculiarities of arcane syntax, such as which sed flags to invoke, or whether to use exec or xargs.

Then take the 5-10 minutes to scan the man pages. Write a simple shell script wrapper exposing a syntax as simple as "replace". And be done with it in 15 minutes top. It seems to me that writing all this js code would take a lot longer than 15 min.

I think the point everyone is trying to make here is. Should every person spend 5-10 minutes doing this?

If the problem occurs too frequently, it makes more sense to make the solution re usable. Which is exactly what 'replace' is doing here.

> I think the point everyone is trying to make here is. Should every person spend 5-10 minutes doing this?

Exactly. People tell me "Nobody needs ack, you just write wrappers around find+grep" and I say "You could, but I published it and now people don't have to."

This reminds me of the first of Larry Wall's three virtues:

"[Laziness] makes you write labor-saving programs that other people will find useful, and document what you wrote so you don't have to answer so many questions about it. Hence, the first great virtue of a programmer."

I use ack, but that's because it searches for the right file types, ignores source control directories automatically, and has pretty colors, not because it's find+grep.

Other than "pretty colors", your statement contradicts itself. Doing the things you mentioned is being find+grep. That's the entire point of ack, to be find+grep.

I didn't say that find couldn't ignore source control directories or find only specific file types. However, ack does that far better as it has a better semantic awareness of both concepts.

I don't have to worry about updating my find command lines to account for new types of source code or source control folders in unfamiliar programs, I need only ensure that ack is kept up-to-date.

I'm not sure what you are trying to say. Your post contradicted itself. You literally said "I don't use ack because it is find+grep, I use it because it is find+grep". I am simply pointing this out.

Except by that logic, you could say that any user of a high-level language was really claiming (in the end) to be using assembly.

C++ once compiled to C, but even then people wouldn't claim they were using C, they'd claim they were using C++. "I don't use C++ because it's C with 'classes', I use C++ because it sets up my vtables and overloaded functions"

That analogy makes absolutely no sense. You are claiming an intended use. And then contradicting yourself by saying you don't use it for that intended use. The analogy would be "I don't use C++ to write programs, I use C++ to write words that have a particular syntax that a C++ compiler can understand and translate into executable machine code". And that statement is just as contradictory as the one you made.

Then write it once and share the wrapper script on github. No?

C'mon we are long past the days where tools had to written in $somelanguage. These are days where even things like compilers and databases are not being written in C.

If its of any help I think its time for the Unix community as a whole to sit down and self reflect. If people are beginning to think something like 'sed' as too level to use, then better job needs to be done to make them more accessible and usable. There are enough examples on this thread where people are saying that they look at sed as an API on top which a program needs to be built up on. And not a complete program in itself. Also shell is not a full fledged programming language.

I respectfully disagree with you here. The beauty and the wonder of these small command line tools (for me, at least) is that they are a) small and b) fast and c) ubiquitious. I do a lot of data manipulation, and while I'll do anything complex in a real programming language (with a REPL) sed, awk, grep and find are really really handy for quick data finding and manipulation.

Any new API on top of them would only be present on a fraction of machines, and thus would reduce the major advantage of them in that they are available everywhere.

Wow, you actually stated something that was wrong enough to force me to comment. The beauty of these tools are that they are small, fast, ubiquitous, AND easy to replace with other tools if they fit your problem better. Unix was built exactly for swapping in these small one-off programs and to deny that is to deny the beauty of it's design and to basically ignore the fact that the system is built to string arbitrary programs together to solve complex problems in simple ways.

One advantage of her Javascript replace is that she can drop it on any machine with a Javascript runtime and it will work, unlike sed.

And the real truth is, she wrote something that solves a problem she has, which is the measure of her success, not someones arbitrary comments.

Why do you have such a strong objection to someone spending their own free time to simplify and / or mimic [insert unix tool functionality here].

It clearly serves a purpose and it's apparently useful. Most importantly, it exists, so there's no need to argue over how you think it should have been implemented.

I'm sure if you were to release your own [script / wrapper], someone would find use for it, in the same manner that some have apparently used 'replace'. I honestly fail to see the big deal here.

I know it someone's right to do this, of course. I object because I come from an environment (software industry) where I have spent hundreds of hours undoing people's poor jobs where they wasted time and money reimplementing (incorrectly) library functions, system commands, etc. Because of this personal experience, I reacted the same way many reacted when seeing her script. So, I decided to explain to her and attempt to convince her that she should have spent her time and skills on something else (especially because she seems a rather good programmer!) Does it make sense?

Fair. I can see where you're coming from.

I object because I come from an environment [...] where I have spent hundreds of hours undoing people's poor jobs where they wasted time and money reimplementing [...] system commands

What you fail to take into account is that there's definitely a niche for Node.js-based development tools such as this script: The GNU userland is not ubiquitous.

Node.js did not come with a system command to search and replace across files - now it does.

It's a one off tool, unlikely to be used in any project you are writing software for. She provided the source code freely, and the code isn't bad at all.

I fail to see why you will ever need to rewrite this?

You have better things to do than tell people to invent time machines to go back and not do things that are already done.

The point everyone seems to be making is that there is no reason to do this in javascript if it's for command line use. Implementing it in js like this is just silly.

JavaScript has a widely used, native regular expression engine. That's enough technical defense on its own for that particular design decision, besides the usual, "Reimplementing a unix command-line tool in your favorite language is always a useful exercise."

> It seems to me that writing all this js code would take a lot longer than 15 min.

Probably, but I can't really speak to that.

Either way, we were originally talking about the syntax & implementation. It doesn't make a lot of sense to object to a hobby script on the basis of development time.

Add another 30 minutes or so for learning how to write a shell script. I mean how to pass variables into a shell script and stuff like that. Actually no idea if 30 minutes are sufficient for learning how to write shell scripts (it can get pretty confusing with environment variables that are suddenly not available and what not).

The issue is that it's 5-10 minutes to scan each man page every single time I want to use sed, find, xargs, etc.

I quickly reach a point where I just think "Fuck it, I'm writing a Python script for this."

A simple tool that only takes a couple of minutes to learn, then a five second look at the man page to refresh the syntax is very valuable, particularly if it focuses on doing the one job that people want to do most often.

You completely skipped over reading his post, and thus your response makes no sense. Read the docs once, and write your weird syntax version as a shell script. Then you don't need to look it up ever again, because you are using the weird syntax you invented for yourself. It is just much simpler and easier to do it the reasonable way than to write it in javascript.

I didn't misread it at all. I just completely disagree that it's 10 minutes to work out how they all work. And that's every time you want to use them, because few people use them often enough to learn the tools properly.

You said:

> The issue is that it's 5-10 minutes to scan each man page every single time I want to use sed, find, xargs, etc."

Which means you could put in a one time investment, write a wrapper shell script, and be done.

And if you insist on disagreeing still, then that's fine, but the this whole point was brought about in response to:

> I've used CLI tools for years, and it would probably take me 5-10 minutes of scanning the man pages to reproduce all but the simplest of those snippets.

You keep saying that you have to research how they work every time. But we tell you: no, you only have to write the wrapper shell script once. It is a one-time effort. Just like harthur put a one-time effort in writing "replace". You wouldn't have to remember find/sed every time, just like harthur doesn't have to re-write "replace" every time.

You are the one who claimed 10 minutes. If you didn't misread, then you are deliberately ignoring what people are saying just to be obstinate.

Well, the whole point is that writing sed-replacement in the language of your choice really is simpler/quicker than learning to write decent "simple shell script wrappers exposing a syntax".

Such wrappers aren't lingua franca anymore - nowadays they are as arcane and as COBOL, since [for most X] it's easier to write a perl|ruby|JS script to do X than a shell script to do X.

Starting from scratch (say, from a Win background) it might be easier to learn a language like Haskell or Scala than to learn the shellscript language with the whole pack of sed/awk/grep/etc flags and tricks.

Who cares how long it took the author? The API is what matters. And I personally think her API beats the snot out of obscure sed, awk,and grep compositions.

I agree. I've used Linux for 18 years, and I'm comfortable with all the find and xargs use. But sed is too obnoxious. Beyond the extremely trivial, I'd do stuff in Ruby instead rather than bother with scouring the sed documentation to figure out what I need.

I might have an issue with doing a replacement in JS for node, though. But for those who are already using node, why not.

I've been coding since 1993 and I have never made room in my brain for stuff like -print0.

Moreover, there are times when you'll want to do this stuff from within another program. What are you going to do, shell out to a find(1) pipeline? Then people like me will be giving you shit.

There's nothing wrong with rewriting sed in Javascript.

Please, tell me you're jocking: on your platform you cannot afford a pipe but you can launch node.js?

by the way: "-print0 | xars -0" is really idiomatic, you know? Lucky you didn't have a single filename with spaces to handle...

I don't know what this even means. I'd already be running v8 runtimes. They're unavoidable. But there's no need to fork and exec (many times over) just to sweep a filesystem tree for files matching patterns.

It's the shell, by the way, that cares about "spaces in filenames". Not dicking around with shell metacharacters is just one of the many wins you get by not shelling out to accomplish basic programming tasks.

I get what you're saying for the "already in a REPL" thing, but as far as fork/exec overhead, xargs does a single fork/exec for every thousand or so files you search; the overhead is tiny compared to everything else.

Not even that, but xargs can also be made to run N processes in parallel ('xargs -P N'), making a find|xargs sed pipe perform much better than anything you could easily invent on your own.

Note that 'xargs -P' behaves very similar to 'make -j' and is often useful as a simple work-unit scheduler.

"find -print0 | xargs -0" handles spaces in filenames perfectly correctly.

That's his point.

" I've been coding since 1993 and I have never made room in my brain for stuff like -print0."

Seriously? find/xargs is nearly impossible to use (safely) without it.

You never need things until you begin to need them. If a way exists to solve your problems without learning arcane syntax people will take that way.

I use find|xargs pretty much exclusively for /usr/include and in my source code directories. Anything more complicated than that and I tend to be in an irb.

So you know who you're dealing with here, I used Emacs for something like 10 years before I figured out how to do regexp search/replace.

I'm pretty comfortable with my abilities as a software developer. If I was Steve, I'd be about as embarrassed about the idea that someone's "level of understanding" of find|sed|xargs was worth commenting about as I would about the rudeness. Of course, if I was Steve, I'd have apologized privately and not fed into the shitstorm, too. :)

Also - it's much easier to debug the equivalent dynamic language's code than to try to figure out what went wrong in your xargs pipe.

> There's nothing wrong with rewriting sed in [X]

Indeed, especially since there's sed and sed.

The examples given use -i, but no luck with BSD sed, where you will use -i~ at best. Also, lack of -e before the expression may make things blow up at times. And the sed regex does not support stuff like +, so many times I end up replacing sed -i~ -e with perl -pi -e or ruby. Besides, sometimes your looping logic is not implementable with a find easily or at all, especially when its context is readily available in the original language, so maybe you use the CLI at times, but this CLI might just be a frontend to a lib part saving you from forking a thousand times over.

> And the sed regex does not support stuff like +

Are you talking about BSD sed here, because GNU sed does support + with the -r flag.

The problem is that it's hard to use sed portably, because GNU sed's "-r" is "-E" in BSD sed. (There's also the differing behaviour of "-i" as mentioned earlier in the thread.)

Not it's not. GNU Sed compiles pretty much everywhere. Just use that.

You just made his point.

And nobody seems to have mentioned the fact that to run this script you'll need to install node and npm (although you can argue that node comes with npm, but anyway…). You might as well just install GNU sed in BSD if that's the problem. Also, I believe GNU sed is much faster than any javascript script.

Moreover, if you want to use this as part of your node app, then it's a good reason to reimplement some of sed's functionality.

I still don't see why you'd laugh or ridicule someone else for doing something like this, and sharing it (if anything at all, it serves as a good tutorial on file manipulation with javascript/node IO.

"You might as well just install GNU sed in BSD if that's the problem"

Often not possible depending on what authorizations and hosting you have.

What are you talking about? Yes, GNU bloatware is massive and hideous, but it is still far easier to install than javascript + framework + package manager + package. If you don't have the ability to run binaries in ~/bin then you can't use this sed alternative either.

I took it that I already had node.js installed because that's what we were using. Asking for a replacement for an already existing sed is probably not going to happen.

You don't need to ask unless /home is mounted noexec. It seems rather absurd to assume a seldom used and rather massive ecosystem of language, libs, framework and package manager are installed, but that it is impossible to upload a copy of gsed to ~/bin. The number of shared hosting environments where you can't run your own code, and you already have node installed is miniscule.

And although its getting a bit off-topic at this point, doing replacements directly in the file isn't even a good idea. The standard "move $file to $file.backup, run sed against $file.backup and pipe it into $file" method is not just recommended because it is portable, it also saves your ass when you messed up your search+replace and broke all your files.

"You don't need to ask unless /home is mounted noexec."

You'd be surprised what is restricted in a lot of corps beyond the "approved" software list. It is amazing what big things get approved and what little things don't.

No I wouldn't, I've dealt with them extensively. I have no idea how that is relevant though. If Enterprositous Corp X needs a way to do replacements, they install one. They would have to install node and a ton of other crap anyways, why do you pretend installing a single trivial binary is beyond hope of possibility, but installing a huge language+framework ecosystem is totally cool?

Or by escaping: \+ I didn't even know there is a -r option, I always escaped the plus sign (and parentheses)

Sure, if this were intended for doing it from within a JS program.

But it's not. It's a command-line tool. To replace the best, most powerful, most flexible, most stable tool (and additional tools) made for the command-line for this exact purpose. This is the definition of reinventing a square wheel. It's just a bad idea.

(p.s. by using xargs (with or without -print0) you can use one option to parallelize the function, vs writing it all in your own code. but i guess getting mad over a fork/exec call is a better idea)

I'm not going to downvote you, but you've not really put your case. Using replace simplifies the workload for a lot of people, the commands (which incidentally, I actually like using) is still somewhat arcane even to the most experienced Unix guru.

If you can replace a long line of pipes with a simple command line, then why not?

Perl was created as a replacement for sed, awk, grep, and other tools of the day. It can be used as one command to perform anything you could ever want to do, with a minimal amount of typing and several command-line arguments to perform useful tasks with ease.

When admins use it, they often simply shell out to the identical console tools. Why? It's easier. They just do everything you need, they're universal, and they work well together.

Go ahead and write a new Perl in JS, and nothing will change; the console tools will still be there, better than anything else.

I agree that it simplifies the workload. Unfortunately, it does not reduce the workload.

Yes, find options are very strange, and it's certainly flexible, still, the learning curve is steep going beyond the basics

Still, someone probably rewrote something similar to sed in JS

I looked up what print0 does as I use find and xargs a lot.

Here's what print0 does - it separates the filenames with a null byte so that names containing spaces and newlines can be interpreted correctly.

Hopefully someone else will find it useful.

Wish I knew that a year ago. Really.

On the bright side, normalizing file names before batch-processing them was a good idea too.

The TL;DR of -print0 is just to make sure find works with all kind of weird filenames.

IMO, if you deal exclusively in shell on UNIX-alikes, you'll likely only need -print0 once in blue moon because people just don't tend to create filenames with spaces or other amusements in them. The instant you add Windows to the mix, suddenly it's the best feature ever and becomes the default for any 'find -print0 | xargs -0 ...' pipeline.

I also think that binary sized string should be the default for shells and command pipelines; plain text was great in the 60s, but it's now the 21st century and I use too many shell flavours on too many systems to remember their arcane quoting rules, locale rules, unicode bugs, etc.

Regardless, it's a shame that there's not a standard separation of UI from functionality in all these utilities - more than once I've wanted to just be able to dynamically link libsort.so from another program because writing files and exec'ing ... urgh.

You might be surprised at how often a filenames I make is OSX becomes a PIA (because of spaces) to handle in the terminal.

One of the subtler issues I've had with piping find in a scriptis that you end up calling a program N times instead of just opening N files in the program.

So if the program has overhead when loading, using a -r option instead an outer find loop can make a magnitude of difference to program operation.

Look at using find's --exec with the + ending instead of \;.

Yeah, when I first saw this project I didn't get the big deal with why it was so bad. Something I might of done if I could have found the time. It probably made doing something easier for a time. I've also never liked using sed much and pretty much have to re-read the man-page and google around a bit almost every time I have to use it again, because I just can't be assed to remember how it works.

If you use GNU Parallel instead of xargs, then you probably never need to make room in your brain for -print0. GNU Parallel separates on \n and deals correctly with files like: My brother's 12" records.ods

To my (naive) eyes, your examples act as a compelling advertisement for her library.

The sed/xargs/etc. examples look increasingly like impossible-to-memorize line noise to me, while the --include, --exclude etc. options in `replace` are comprehensible and seem to follow standard command line argument practices.

Again, this is from a very entry-level unix tools perspective, but there it is.

It also seems to me that reimplementing things with the stated goal to make them easier to use has to be one of the best justifications around, no?

I hope I sound as respectful here as you did in your comment -- I certainly appreciated seeing a clear response to her library, so thanks for that.

Impossible to memorize? I wrote all these examples from memory. I tested them in a shell and had a single error (sed -ir needed to be changed to sed -ri because -i takes a glued "suffix" argument). I really don't think it is that hard to memorize the basic find/xargs options. You only have to memorize them once in your lifetime. They are a basic cornerstone of a Unix skillset. By comparison, I often forget the custom syntax of my custom scripts I wrote years ago.

I can write complex vector math in C from memory - doesn't mean everyone needs to be able to do it.

That replace library does what it's supposed to do - it replaces shit easily.

Don't like it - don't use it.

If people want to do more - then they'll figure it out for themselves. For everyone who already knows more - eat some humble cake.

>Don't like it - don't use it.

This is a false dilemma and completely misses the point. The replace library was written by a person who either never heard of or never learned how to use SED. Just because they created their own hack as a workaround doesn't mean it's the right thing to do.

Right is arbitrary.

Just because it does something that you like to do a different way, doesn't mean it is hurting anyone for someone to write it.

By this logic, there is no way for beginners to write much of anything since there are existing implementations of almost any idea, and it should be disallowed ever to make a new thing which could replace any old thing.

Why would you assume that, because they wrote an alternative to an existing tool, they don't know how to use it? Perhaps she just doesn't like sed, or would like to use the replace functionality in an environment where it's not clear ahead of time whether GNU or BSD sed are available. Or any of a number of reasons. Your assumption that she must be ignorant to have created her own solution is just as boorish as the original tweets, imo. Was tmux written by a person who had "either never heard of or never learned how to use" GNU screen? What about GNU itself? Why bother when UNIX already exists? People have lots of reasons for writing software, and deciding for them whether it's "the right thing to do" because some other software already exists is absurd.

If you take a look at the readme, the first commit back in 2011 says this:

"It's similar to sed but there are a few differences:" and then goes along to list out the differences.

I would expect this person to 1) have heard of and 2) have learned to use sed in order to write a tool that is similar but has slight differences in syntax to sed.

There is no wrong thing to do when writing tools for personal productivity, as OP clearly stated. If it works, it works. If they decide to open source it, that's great. Maybe someone has use for it. The end.

If a program is nontrivial to learn and use (as the sed examples shown in mrb's post illustrate) then a good solution is to make a better, more easily usable tool tan sed.

As for editors - there is a niche for emacs and there is a niche for notepad. If a program is weaker on some aspects but clearly superior in another aspect, then it has a valid niche and making it really is the 'right thing to do'. Usability/discoverability is a very important aspect of software, and sed is a bit lacking there.

> Just because they created their own hack as a workaround doesn't mean it's the right thing to do.

Are you honestly at HackerNews posting that someone hacking together some code to scratch their own itch is not "the right thing to do"?

Please don't follow up a false dilemma with circular reasoning.

>Impossible to memorize? I wrote all these examples from memory.

Yes. And people can site Pi to the 10.000 digit. Doesn't make it any less useless.

The arguments to common UNIX commandline tools like sed or xargs are without doubt going to be longer-lived than the arguments to your helper script, likely even longer than the language your helper script was written in, and maybe even longer than the idioms and philosophies you had in mind when you were building your tool(s).

Calling the basic operation of these cornerstones "useless" speaks more about your disposition than the tools, I think.

>The arguments to common UNIX commandline tools like sed or xargs are without doubt going to be longer-lived than the arguments to your helper script, likely even longer than the language your helper script was written in, and maybe even longer than the idioms and philosophies you had in mind when you were building your tool(s).

Yes. Sad isn't it?

It's not as if they were perfect when the were designed.

It would appear they're the basic cornerstone of YOUR unix skill set. I spend all day in *nix, I didn't understand everything you wrote there.

Guess you've never heard the KISS principle. So what if you can remember arcane commands? If there was a "replace" program, I'd be sure to use it. I understand all of these as well, trying to pipe them together or to use a "replace" command - I'd choose replace every time.

There's another reason to reinvent the wheel:

    $ replace "Foo" "Bar"

is common sense and easy to understand.

Not to mention that sed -i is a GNU extension.

I haven't used replace but it seems like a pretty innocent piece of helpful code for someone who likes JS.

I agree and also I took a peek at the code and my eyes did not bleed... it's a nice little one-off that doesn't try to do too much and is written in a decent ad-hoc manner... what's wrong with that?

Heather, you've created a nice and useful tool with a modern tool-set, did a very decent job coding it and took the time to open-source it. In my eyes at least it puts you in the top 0.01% of coders in the world.

I actually had something similar myself, a shell function called findRE, the entire purpose of which was to automate searching a given path recursively for a given regex.

However I used it so infrequently that whenever the need would arise, I would invariably screw up the order of the 2 (only 2!) arguments. I could have added support for --help, --usage or similar, but that seemed like overkill.

I eventually just learned how to "properly" use find and grep (both of which I was already using anyways, which helped me remember the command line for each).

With that said I do agree that I'd never use shell script or Unix userland on purpose when writing scripts that are intended to be portable across different Unices (or even Windows), that's the kind of thing I'd reserve for personal use on my own userland. Hell, even on one machine you never know with some distributions what kind of shell you'll login to by default...

Am I the only one who finds replace's syntax much nicer?

I haven't spent much time learning unix utilities, so that might explain it. But I can't help but shudder at find . -type f ! -name '.min.js' ! -name '.py' -print0 | xargs -0 sed -ri 's,var,let,g'

She was able to simplify her command line interface in order to support only the features she needed, so of course her syntax is nicer. Whereas sed/find/xargs can do a lot more things, so of course they come with a more evolved syntax.

It is the usual tradeoff between simplicity and features.

Right... she had different requirements, so she wrote a different tool. Which is why I find your near-rant above somewhat disturbing. It seems predicated on a (seemingly) very arrogant belief on your part, that you know more about what harthur needs, then she does. Personally, I am very sceptical on that point.

There is only one case that justifies reimplementing things

Or, um, how about the one that goes "I felt like it"? That's sufficient justification in my book.

Yeah, who needs TextMate when you have vim, right? Or Ruby when you've got Perl?

I beg to differ on your first argument. Learning Vim was one of the best decisions I've ever made in terms of how I write code. Other editors feel like toys in comparison.

I'm siding with mrb on this one. Learning flexible tools gives you immense power in comparison to low-learning-curve ones. The time invested is considerably smaller than the time spent figuring out how to solve edge cases with simple tools. This is something which appears to be impossible to communicate to people with words. Learn some of these tools and you will see for yourself.

By 'learn' I do not mean, for example, becoming familiar with hjkl for navigation in Vim. I mean mastering the tools to the point that using them requires no effort - where just as you experience the impulse "I want a glass of water" and without any effort go to fetch one, you experience the impulse "I want to create a new class in file X, navigate to file Y then modify method Z", your fingers start moving and suddenly you're done. I get frustrated watching my IDE using co-workers write code - it takes them a very long time to do very simple things. Their tools are narrow and do not facilitate expression - they constrain it.

Pre-emptive strike on the argument that code completion is better - Eclim (Eclipse with a Vim frontend) solves this problem very well. Nonetheless, I have it installed and configured but can barely find a use for it.

My only point is that what's right for you isn't necessarily what's right for everyone.

Exactly, and its something happening all the time and many people seem to like it. If not, everyone would be using Linux Desktops vs OSX/Windows, vim/emacs vs Sublime/Textmate/an IDE, plain ruby/python/etc vs Rails/Django/etc, OpenGL/DirectX vs Unity/any Game Engine etc.

Even in the consumer world, this is Apples strongest selling point for example. Make products simple and easy to use, but still do 90% of what anyone would want to do with it. If that isnt enough, there are other things to use.

I think the same can be said over replace. I find your skills with sed/grep impressive and its certainly very useful in some cases (mostly devOps) but in case i need it myself i would just look it up and in the meantime use simpler tools and get the job done.

>If you care about simplicity, I would personally rather write a small shell script wrapping find/xargs/sed and hiding their arcane options, as opposed to writing 173 lines of js.

I would rather use my language of choice to implement the above (mine wouldn't be JavaScript, but it clearly was Heather's) than to research all of those arcane command lines.

Rather obviously YOUR language of choice for this kind of problem IS sed. So OF COURSE you would use a shell script wrapping sed etc. to implement these functions.

But guess what? To handle all of those options at the command line would still take a pretty large fraction of 173 lines of code, if you added in an equivalent level of parameter checking and option handling.

And if you want to make MY eyes bleed, hand me a file with ~150 lines of logic in shell script that spawns cryptic command lines. Despite not REALLY knowing JavaScript, I would much prefer to maintain the JavaScript than a shell script with the above commands.

I believe that your complexity estimate in writing the wrapper indicates you are perhaps not a proficient shell scripter. However I sincerely believe it is possible to write such a wrapper in less than 15-20 lines. The trick is to do as little argument parsing as possible but enough to be safe (for example pass the ! -name "*.min.js" as is to find).

PS: I will do it in 15-20 lines, if only to prove a point, if someone sends 1 BTC to 1F43fceWvwznv4rH7s1B3du2w5VznFJY58 :) (yes, my time is valuable - I have other things to do)

No interest in seeing a proof of concept. And I make no claims about being a "proficient" shell scripter: You've nailed me there. I can write a shell script if I absolutely have to, but I avoid it when possible since Bash is such a terrible "language."

When I DO see non-trivial shell scripts, they are just this side of arcane gibberish. The command lines above are completely opaque to anyone but an expert in EACH of those tools (I know some of the tools, but not all well enough to know EXACTLY when I need to backslash a paren, or how many backslashes to use, or when I need quotes or not...).

When I see the JavaScript that spawned this conversation, despite not knowing JavaScript well at all (probably about the same level I "know" shell scripting) I was able to follow everything that was happening to the point where tweaking it to add features would have been easy.

The objective shouldn't be to write code in the fewest possible symbols. Everyone would use APL (or a more recent evolution of that language) if that were the case.

The fewest possible readable and modifiable lines of code is my objective. Shell scripting tends to have lots of obscure and unpleasant syntax to accomplish the most basic flow control.

I'd present the argument that, once you learn sed, it has the possibility to replace more then just this 'replace' command. If you tend to live a lot on the commandline, you'll naturally start using it where appropriate, and become more familiar. After learning how regular expressions work, it really amazed me how many things I used to think "I'd have to write a script for this", when now I just write a quick line of sed.

I'd ague that her version is at the very least syntactically much easier to use.

Hear hear!

I've had a sed book on my shelf for years and I've used it more than once, but 99% of the time I end up giving up on sed and using a GUI-based tool instead of trying to concoct the cryptic lines given above as examples.

I absolutely would rather use the clean interface of replace, written in JavaScript or anything else, than to memorize that steaming pile of cruft.

Two reasons why it's better to memorize the cruft:

1. You can pipe find/sed/grep/cut/awk/whatever to other utils. Today you might be replacing, tomorrow you might be analyzing text logs, and the more core utils you learn the more things you can do with them.

2. When you're ssh'd into another box that doesn't have that spiffy replace utility, now you don't know how to do an inline replace.

Your argument that it wouldn't be available on other boxes can be applied against any new command, no matter how fantastic, so it doesn't seem very constructive to me.

I think it is a good argument to learn basis before reinventing the wheel. When using unix, there are already far too many commands to learn. Not having to learn useless commands is a bonus.

And yet, it's not useless to so many.

1. Perhaps learn that when you actually need to analyze text logs?

2. Perhaps you don't need to do this. Or perhaps you'll work it out when you ssh into another box.

> 1. You can pipe find/sed/grep/cut/awk/whatever to other utils. Today you might be replacing, tomorrow you might be analyzing text logs, and the more core utils you learn the more things you can do with them.

Fair.

This boils down to an individual's needs. I think most people in a position to be making this choice are fairly well aware of what's possible with standard nix CLI tools, and can decide whether or not to invest the time to learn them.

In my experience the ROI for nix wizardry is fairly high at first, but begins tapering off. There are a half dozen or so command chains I use daily without which I could hardly see myself working, and the rest I may not touch for weeks or months on end. YMMV.

> 2. When you're ssh'd into another box that doesn't have that spiffy replace utility, now you don't know how to do an inline replace.

It doesn't make sense to optimize a workflow around an edge case. You don't often hear this brought up as an argument against dotfiles, key bindings, vim plugins or what have you.

You actually do hear this brought up as a case against dotfiles and vim plugins, even the use of vim as opposed to vi. There is just this school of thought which says you should never use anything which isn't in a default CentOS install.

As a sysadmin, I can totally relate to those thoughts.

If you run nonstandard, you basically have to learn vim/vi as a default install for an OS and another with all your add-ins and customization. Sure the key bindings are similar, but I don't have my macros and why is my tabbing funny, there's no line numbers, and damn it, the backspace key doesn't work. I forgot I set that in my .vimrc.

Sure, I can remember most of the latter with set tab and set no, but when you're popping around systems, switching to root (with it's on .vimrc), keeping applications as stock as possible is very helpful.

I personally have a slightly custom .vimrc for my local use, but I don't rsync my dotfiles across the globe for my other home directories. I basically try to keep it as code development (with .vimrc) and sysadmin (may not have niceties, but I'm not editing code).

I know the core basics of find/sed/grep and I've even used awk on occasion (though I typically have to Google for that one), and occasionally I use them when I'm ssh'ed into a box.

But as soon as I want to do something more sophisticated, I again reach for the tools I know better.

No I can't use my GUI tool on a box I'm ssh'ed into, but I can push my changes to a git repository and pull them to be local, where I can do complex things really quickly and without having to memorize tons of obscure options and corner cases.

In a case where that's not as feasible (hundreds of megabytes of log files, say), and what I need is beyond my find/sed/grep skills, I usually drop into my language-of-choice and produce a tool that is much easier to use for my purposes. Honestly I haven't touched awk in a long time because the pain is so much greater in awk to accomplish what I can in my own more familiar toolset.

I get that those tools work fine for any old-school Unix hacker, and the combination is quite powerful in its own right. I don't have the desire to BE an old-school Unix hacker, though, and my tool-set is quite powerful enough to accomplish what I need without going through that particular hazing ritual, thank you.

My need for tools like this happens rarely enough that even if it would be faster once I'd learned the tools to type out the appropriate command line, the amortized time to GET to that skill level in those tools would be far more than the time it takes me to just use my "less optimal" tools.

i've downvoted you while i wanted to upvote you.. wtf.. i agree with you

Thank you for taking the time to write these translations. In fact, I'm going to bookmark this thread and use it as a reference in the future.

I feel these examples serve more as a selling point for why one would use something like `replace` over a cacophony of flags and pipes. The interface for `replace` is simple and elegant. It does one thing, and it does it well. I think that's commendable.

P.S. The `sed -r` flag does not exist on POSIX systems like OSX. You'll have to use something like perl -pie, or write a convoluted buffering loop.

Or install gnu sed

I don't even get what the issue is. People have a problem that she even wrote this? I wrote something similar because there IS no standard replace tool along the lines of grep. Sed is awful (it's essentially impossible) for multiline expressions for example. Could this really be penis measuring because people think they are so cool that they know unix commands exist? Congratulations, you have turn invocations of a simple, convenient tool into a shitpile trainwreck of find and sed. Golf claps for infinity.

You have a valid point but its incomplete, allow me to explain you why we need projects like replace.

You are right in telling that sewing a few utilities using pipes can emulate 'replace'. But remember you are talking about learning those 'few' utilities first. Trust me unless you have practiced your way out of these thing with hundreds of hours with experience over years its humanly impossible to remember and use a tool like sed to its full capacity. The more utilities you add to your pipe chain the number of ways in which these utilities can be used more or less follows a exponential curve.

Now if some one comes along and gives you a version 1 of a tool that can simplify these things and give you simple way of doing things in a command instead of doing it with a ten pipe chain commands, he/she is ideally solving a problem of a great importance.

As some one who loves the Unix CLI, I have often wondered why people spend a whole day writing a java program what could have easily done with sed and awk in a line or two. But these days I understand why. Learning and using sed/awk to their full potential is a time consuming task. You not only have to learn them factually through a manual. You also need to study the problem cases for which they are appropriate. Then you need to go and put their solution patterns in your brain. Then you need to map all problem patterns with solution patterns. Then you need to learn all the arcane special cases. Generally this comes to a person over years practicing those utilities.

As a matter of fact Perl was invented because using C, with all these utilities was getting very difficult.

Now if somebody can give things like these. Which do many things out of the box, without you having to go through the pain of learning things and practicing over the years. They would have indeed given you a tool of great value.

In my experience even if some one fails working on something like this, by third or fourth version of the tool they generally come up with a thing of great value.

Think of this like Vim Vs Sublime Text 2. In case of vim there are ways of solving problems. In ST2 many of those solutions are automated and sometimes given to you out of the box.

I have just publish a report on monday where I explain that the command line should be used instead of some small scripts that encapsulate for loops.

The argument is that basic shell knowledge should be trained in order to not lose skill (or to improve). At my work, these skills are useful when facing unexpected urgent needs.

Having a short script is a very poor benefit (simpler syntax and reduced key typing) for a big loss: - loss of knowledge - loss of understanding of what the script is doing - need to learn the name of the script which is not a reusable knowledge.

Learning the basic usage of sed, awk, shell is an investment that is quickly rewarded.

Learning all the intricacy of "replace" needs to know many javascript details and how they interact with shell escape sequences and your LOCALE configuration. I am sure it is less documented than the sed command.

All I can say is I'm glad I don't work somewhere where someone writes a report on how I should get my work done at the most basic level.

It is a report on How and why I have done my work the way I did, What were the advantages and disadvantages, What could be done to improve the process.

This kind of reports are useful for people doing similar activities.

This is like saying you should learn C before using [some-lang] when all you want to do is cobble together a web page. Yes, it might be beneficial to some, but it's not absolutely required.

Some people just don't live in the shell. We're taught to build upon useful building blocks to make useful things - why would you not abstract away the monotonous tasks?

> There is only one case that justifies reimplementing things

Who are you to decide how she may spend her time? Why in the world do you think that she has to justify to you what projects she creates?

The clear answer is probably appreciated however the implication that reimplementing something needs to be "justified" is absurd, even setting aside the fact that the syntax you use is representative of the arbitrary, inconsistent, and generally horrible CLI of unix commands, and does not represent a good case against implementing a better interface. Open source is packed to the brim with "reimplementations," some out of ignorance, some for learning, some to improve, some to port to different languages, some for ego, some for principal, and some for many other reasons.

Frankly the code is hardly deserving of derision. It looks perfectly capable to me, and the syntax is far superior to arcane unix commands. Just because you'd rather write a sed wrapper than write this code doesn't mean it makes sense to deride others for trying to move the human race forward.

Some of my examples can be further simplified (no -print0, no xargs), without a perf issue ("{} +" passes multiple filenames as arguments), with GNU find which is standard on Linux:

  replace 'var' 'let' . -r
  find . -type f -exec sed -ri 's,var,let,g' {} +

  replace 'var' 'let' . -r --include="*.js"
  find . -type f -name '*.js' -exec sed -ri 's,var,let,g' {} +

  replace 'var' 'let' . -r --exclude="*.min.js,*.py"
  find . -type f ! -name '*.min.js' ! -name '*.py' -exec sed -ri 's,var,let,g' {} +

To all the Unix shell critics: don't try to argue that writing a 173-line js script is "simpler" than learning 3 find options (-type -name -exec), the bang(!) 'not' operator in find, and 2 sed options (-r -i). This is just wrong, seriously.

You are still wrong, she is still right. If we wouldn't have to care about either 1) the resource use or 2) the easiness of use, then most of the programming would be unnecessary. Her solution is better than yours on 2) always, and on 1) as soon as you need that functionality in node.js contexts.

You are wrong about #1: her design goal was use from the command line (according to her readme), not from node.js.

You are wrong about #2 too: I have said this 5 times already: you can write a very short (much shorter than 173 lines of js) a wrapper shell script exposing an interface as easy as "replace".

Please read what I wrote!

Even if she designed it to be used from the command line, and this is still not the only possible use of the JavaScript code which she open sourced, she had one more reason: sed doesn't have JavaScript regexps. Anybody who really wants to use JavaScript regexps doesn't have other choice but to use her tool. Now you can say that you don't need that, but you still has to admit that others may. So you're still wrong.

With all due respect, you add nothing to the discussion, I already acknowledged this in my OP:

"There is only one case that justifies reimplementing things: if your tool has the requirement of supporting the exact js regex syntax..."

(Please read what I wrote, really!)

a) Can you agree that the main reason of open sourcing a JavaScript code is for others to use the same code in their projects, not to "run the command line"?

b) Regarding those that do want to "run the command line," can you agree that there are users who really want and need to use JavaScript regexps from the command line and not some other regexps with different semantics?

c) Can you accept that these are the valid use cases that your "use sed" arguments just don't cover?

I've read your arguments and I've never seen clear acceptance of a) b) and c). I've also seen that you had to carefully think and even improve your "use sed" implementations, giving also the good argument in favor of her approach. Having the tools that demand less from the user is a good thing. Can you accept that (d) too?

Can you see why we discuss, when you just quote yourself as "There is only one case"? That's why I claim you're wrong. There's not only one case.

>To all the Unix shell critics: don't try to argue that writing a 173-line js script is "simpler" than learning 3 find options (-type -name -exec), the bang(!) 'not' operator in find, and 2 sed options (-r -i).

In the context of discussing the open-sourcing of the script, the issue isn't really whether or not writing the JS is simpler than learning "3 find options", it's whether using the JS is simpler than learning "3 find options".

You don't need to write a 173 line js script, in the same way you don't need to write the thousands of lines of C code behind find or sed.

You just use it.

Let's face it. Most people will never bother learning shell commands in and out. I've been a *nix user for 20 years and I still have to be checking man pages, and I often stumble upon basic shell command options running differently in different shells.

Other than for sysadmins, shell commands are going the way of the dodo. This isn't new, it's been happening for many years already. Increasingly capable computer people from the newer generations will be focusing their skills and time elsewhere. Arguing against this reeks of grumpy old coder bitterness. I know my shell stuff and it serves me well. I use vi/vim daily. I don't impose my usage patterns or expect that something "existing" means it's appropriate for everybody.

Replace looks a LOT cleaner to read and write though and more intuitive though for smaller/simpler things such as these examples.

Learning how find, grep and a bit of sed works is a better investment of time, they have a broader use than this replace utility.

I can think of no better advertisement for replace than this post.

both Perl and Python use PCRE

No, Perl doesn't.

PCRE was written because a lot of people liked some of the ideas that were first introduced in Perl's regular expression engine. But PCRE was a reimplementation, and never was all that compatible with what Perl was doing.

The funny thing is, that Python doesn't use PCRE either and never had.

True enough. It's not the first time I hear about PCRE and Python, so maybe it's a common misconception.

Other than "re", there's "re2" and "regex". None are PCRE but "regex" is close enough most of the time.

I'd argue that wrapping crappy syntax is decent syntax is 90% of what tools do.

I often write wrapper scripts to make complex syntax simple. Just for me.

Don't get me wrong. i actually _know_ the f* syntax because I'm and old crazy fart like that by now. But when it comes to use it, it takes a while to type. Sometimes I make typos. Sometimes i forget one specific option or character and I gotta look in man again.

The simplified syntax wrappers make it f' easy.

The other day I made a csv column grepper. It just calls awk and assign numbers to column names, so i can just grep by column names no matter their order.

So yeah, a huge line of awk can do it. An even less understandable line of grep can actually do that too. Some other tools as well. Mine is just ./supertool colname colname colname ... | awk "$2 != blah" |column -t

(For the record, there were similar wrappers in python and perl, each of which were 5 to 35x slower (200 to 1000 lines script) than my 20 lines bash script, which is another complain I have. Reimplementing that stuff in high level languages is generally a bad idea, specially when you don't understand what happens underneath.

sed -i the way you use it is not portable. BSD (and therefore Mac OS X) requires a mandatory argument to -i (the suffix to use for the backup files). In Linux sed the suffix is optional.

In his(her?) own words: http://harthur.wordpress.com/2011/06/06/replace/

Ironic that the post just before that on her blog is about dealing with the assholes you will likely encounter in Open Source.

I would rewrite these functions for fun, even with an understanding of them. It doesn't matter if the end result is useful or not, we're programmers, we enjoy programming, not using. Of course that's to a point; time spent making something useful that's fun to make is more rewarding to most.

I'm just making the point that it's not just about learning, it's about writing the code. If you're the type of programmer that is only looking for a means to an end, this isn't for you, but if you're like me, the fun is in figuring it out yourself without the help of libraries.

Why are people so opposed to reinventing the wheel?

In a professional context, I understand[1], but this is a personal project someone wrote to scratch their own itch. Why is it unjustifiable? I can think of plenty of reasons to reimplement something but I think the burden is really on you to say why it shouldn't be done. If you think replace is a waste of time just don't use it - simple as that.

[1] I've spent the last two years maintaining an application written in a homegrown web framework - don't get me started.

I've been working in software companies for a few years and I could say that something like this "replace" application is pretty much useful:

- Node.js can be installed on Windows and sed/find are not available on this platform.

- Many of my coworkers don't understand sed/find syntax, and even if they could learn it, they don't have plenty of time.

Btw, it's terrible that a few guys make rude comments about a free and open source software. If you don't like, don't use it or if you think you can make it better, fork it or shut up.

Why are people even arguing about this? Is there some religion that forces people to only believe in and use GNU tools that they can't appreciate a competing tool?

A tool's a tool. If it's useful to you, use it. If you don't need it, then don't. Why does this have to turn into a debate about why someone shouldn't have invented another tool because one already exists?

Look around the world, there's competition and choice in literally everything. Why can't they be choice in what CLI tools to use?

I think it can be valid to do stuff with a scripting language you know, rather than shell commands (isn't that the good old Perl philosophy?). I've actually even looked up sed once and did stuff with it, but it doesn't come up often enough to remember the gory details. And learning sed just to replace stuff takes too much time.

That said, the javascript solution does look a bit long winded.

Also, please don't take it as a negative comment on your comment, I actually appreciate the solutions and it makes me consider sed again.

You just made a point why this 'replace' may actually be useful to do things instead of bothering with using something as needlessly complicated as you have illustrated sed.

It would be far quicker for me to write an one-use program in a number of languages than to find out how to do this in the sed oneliner style that you show - ergo, the sed approach has drawbacks in some use cases, and alternatives should be tried.

Replace looks better to me. I can figure out the command line options from just looking at them, and it does what I expect. I like it.

Very nicely done. It was cool that you outlined this in a lesson format and pointed to how things could have been done differently and attempted to show the perspective of the commenter. Talk about constructive input.

The good news is that in every single example, her command-line equivalent is fewer keystrokes (sometimes many fewer).

Depending on how much you need to this sort of workflow, this could be extremely valuable.

Hers is concise and elegant. sed and find are just defaults.

I guess I'll point out that if you want to play "golf", the replace command was shorter in every single one of your examples.

ack-grep does quite a lot of this as well.

>I took every single example from your README, and show you below how everything can be reimplemented with sed -r (nice extended regex syntax, mostly like js regexs), and find/xargs:

And why assume a POSIX command line?

Reply on news.ycombinator.com