Comment by segmondy | Hacker News Reader

Comment by segmondy | Hacker News Reader

Parent Full thread

I can't tell, I know many developers today who are on the new shiny js thing, golang, elixir, scalar, rust train, but they never got really good in their first language be it Java, python, or php.

They use linux or OSX. But they don't know awk/sed/bash, give em a very simple problem, they would write 1000 line of OOP, united test code when a few line of scripts would solve it.

I know many programmers, who graduated with a CS degree, they go out of their way to implement ridiculous rule engines and overly complicated systems, when they can just use prolog and glue their app to prolog.

I've witnessed many so called fullstack developers who break out in cold sweat at the sight or thought of writing SQL. They want an ORM.

This is what I call chasing, new things, when the old things are rock solid. I see this today, where people cry about PHP, but yet don't even use it's good features. Go use NoSQL solutions but never even had a clue about their SQL DB.

/rant

> They use linux or OSX. But they don't know awk/sed/bash

Am I in the minority of developers then? I don't know awk and sed and I'm honestly not very interested in learning them. But bash I definitely agree with. I still use a cheatsheet to get the syntax right because it's weird as hell imo.

Worth reading "Area Number Three" on this blog post from Steve Yegge: https://sites.google.com/site/steveyegge2/five-essential-pho...

Isn't that just a `grep -R [regex pattern]`? Not sure where something like awk or sed fits in to solve such a problem.

The primary point was one of mindset, but the second example in the cited section is a tailor-made sed example:

Let's say you're on my team, and I've decided I'm a real stickler for code formatting. But I've got peculiar tastes, and one day I decide I want to have all parentheses stand out very clearly in your code.

So let's say you've got a set of source files in C, C++, or Java. Your choice. And I want you to modify them so that in each source file, every open- and close-paren has exactly one space character before and after

Whoops, didn't notice that one. Thanks for linking that post by the way. Great read.

Most Yegge blog rants are well worthwhile, IMO. I wish he wrote more of them.

Y'know - after thinking about it a little, I'd argue that this is a question that could serve a second purpose. The obvious out-of-the box implementations of this in sed/perl/etc. would all fail to properly handle open and close parens in strings and comments. Particularly multi-line comments and strings with escaped quotation marks inside of them.

I like sed. I like perl one-liners. I would consider it a mistake to try to do this on a real codebase in a batch manner with just a regexp. You'd be far better off googling to find a source code formatter that includes a real parser, like clang-format.

The question specification is such that you don't need special handling of comments, strings, etc. "every paren" isn't qualified as "that is part of the code".

The point of the question is more to gauge your understanding of/familiarity with available tools and ability to think laterally rather than immediately jump to "I've got to write some real code to solve this"

I'd probably reach for ANTLR to write a quick parser since working with such a stickler is likely to require more code formatting changes in the future.

As an aside, the stuff I used sed/awk for in the past have pretty much been replaced by Python for me. These days it's pretty rare to not have a default Python install on any Unixy box one spins up and in the rare cases I use Windows I tend to install Anaconda so it's also slightly more cross-platform for my use cases.

awk and sed are fantastic for command line processing of text. You really should take a look at them. All that said, if you don't find yourself trying to do something in bash and thinking "there must be an easier way" then you're problem sets may just not need them.

edit: comment was about perl, I didn't notice discussion below

"I don't know awk and sed and I'm honestly not very interested in learning them. But bash I definitely agree with."

Awk/Sed are optimised for text/data processing.[0] While Bash might work, will the script work on all systems (and version)? [1] awk & predecessor sed are good for one liner programs and suitable for problems that need fast, reliable solutions.

reference

[0] Opening, closing files; reading, breaking and counting record fields.

[1] OS use Bash or Bourne? https://en.wikipedia.org/wiki/Bash_(Unix_shell)#Portability

Then again, sed works differently on different platforms, and awk is not the same as gawk or nawk, both of which have many more features than standard awk. If you're used to the GNU implementation, you might be surprised to find your script doesn't work on a system without the gawk implementation.

On Linux, I can use this sed command: sed -i ‘s/^”//’ example.txt

That deletes any quotation marks that are at the beginning of a line. It does it in-place, and while you have the option to rename the old file and replace it with a new file with the original name (using "sed -i.bak ‘s/^”//’ example.txt" would move the existing example.txt file to example.txt.bak and write a new one in-place), if you leave out the .bak, it just writes it in-place with no backup file.

On OS X, though, that command won't work. You have to specify a backup file, even if it's blank. "sed -i “” ‘s/^”//’ example.txt" would replace the text in-place without a backup. sed -i “.bak” ‘s/^”//’ example.txt would replace the file and move the original to "example.txt.bak".

Even with sed and awk, you still have to stop and wonder if it will work cross-platform.

This is why I have gripes about the OS X command line. I can't confidently use the scripts I do on Ubuntu/Mint/Debian. Like, get with the program, Apple. It infuriates me and I'm moving entirely off their platform as soon as my Mac kicks the bucket.

OS X is not Linux. It's a BSD-derivative, and doesn't use the GNU toolchain. It isn't that it's "not with the program." It's that it's not the operating system, or toolchain, you're familiar with. Install homebrew and GNU tools[1] if you want them.

If you used an operating system that shipped with BusyBox, you'd be similarly frustrated. But operating systems are tradeoffs with pros/cons. Using a different set of tools makes it distinct, not "out of date".

1. https://www.topbug.net/blog/2013/04/14/install-and-use-gnu-c...

"OS X is not Linux. It's a BSD-derivative, and doesn't use the GNU toolchain."

True, but I was thinking awk/sed was a good multi OS tool with minimal code effect.

Oh, my mistake. I was under the impression that sed/awk/grep/ack/etc etc were part of the POSIX standard toolchain, rather than the GNU toolchain.

I have a script which installs and aliases all the equivalent GNU commands on OSX. You can fix a lot of the annoyances.

"Then again, sed works differently on different platforms"

Well that kills my idea why to use awk/sed. What would you use @freehunter?

Perl, Python, Ruby

True, defeats the idea of a one liner in bash/awk/sed.

Well the beauty of one-liners is that they're quick and dirty. They don't need to be portable, you don't usually maintain a one-liner for years. If you need something portable, you should use a real cross-platform language like Perl, Python, etc. Something that's independent of the OS you're running.

"Well the beauty of one-liners is that they're quick and dirty. They don't need to be portable, you don't usually maintain a one-liner for years."

Given that description, I'd agree. Now I understand the crux of the problem, problem solved using disposable code in terse language or script.

I don't think you are in the minority, assuming you have a reasonable way to quickly parse text. I think most developers just Google for one-liners or throw something together in Python to solve text-parsing tasks at hand, rather then trying to remember which tool uses which type of regex with which argument.

There's no reason to use awk or sed when perl exists (a2p and s2p have existed for decades)

There is no reason to use perl in situation when awk kicks its butt with clearer, shorter code free of distracting sigils and other line noise

Awk is also POSIX standard; a good remaining reason to use it (if it is a feasible alternative for a task) is that a script has to be portable outside of GNU/Linux.

I like Perl a lot better than awk, buy I do agree with the general sentiment of using the language that makes for the clearest code for the task at hand, especially the smaller ones. Which makes me a frequent awk proponent.

So I'm surprised at the following part of your comment:

> There is no reason to use perl in situation when awk kicks its butt with clearer, shorter code free of distracting author and other line noise

In my experience, the shorter ask code tends to be the one with sigils.

But in a lot of ways awk seems way more modern than Perl (Perl 5 at least). You actually have functions with parameters rather than having a weird @_ array that you need to shift values from. Honestly, if I didn't know otherwise, I'd think awk came after Perl.

Functions with parameters in a () is already in Perl 5. Search for subroutine signatures.

http://perldoc.perl.org/perlsub.html#Signatures

If you're getting along fine without awk then by all means, don't learn it. But just today I had to quickly get a raw list of all US area codes (and only area codes). After a short Google search, I realized the fastest way of downloading and formatting it without having to do a secondary step was this:

  curl https://raw.githubusercontent.com/kedarmhaswade/cities/master/area-codes.csv | \
  awk -F, '{print $1}' \
  > ~/area_codes.txt

Now, could you use grep? Yeah, sure. But for really quick examples like this - things that lend themselves to it (e.g. a raw CSV with mostly crap you don't need but something you do), this is super fast. Anything more complicated and I'd break out a regex, though.

I think that's fair enough, but I'm not sure I equate one to the other. Your examples are complaining about a lack of exposure to different kinds of tools that already exist rather than lack of depth in tools that the developers use.

Although grossly unfair, it's sometimes instructive to split people in two categories in order to consider a point. With that in mind, I notice that there are programmers who want to learn as little as possible in order to get the job done. They want tools to make things easy for them so they don't have to understand the underlying concepts. How many times over the decades have I had to unwind bad commits in source code repositories because some programmer refuses to learn how to use the software directly and leaves their IDE to do it for them? Same again for build systems. How many times have I worked with programmers who refuse to understand what a pointer is, because now that we have GC apparently we don't have to worry about leaking resources? How many programmers never bother to learn the syntax of the language, preferring to generate snippets and cut and paste from StackOverflow (or back in the old days, you would have entire applications built primarily from cutting and pasting examples from MSDN)?

Yeah, this is a problem (and has been a problem for as long as I've been a professional programmer -- i.e., a long time ;-) ). But the people who are chasing after the new shiny don't really exhibit that attitude for me. To the extent that everybody who wants to work tomorrow has to keep learning, no good developer is exempt from this. But I find the people who really chase new stuff without getting a proper grounding in the old stuff have a few overwhelming things in common. They are young (to the industry). They are excited. They want to learn.

My experience has been that if you show them the very, very cool things you can do with awk/sed/bash/prolog (Really? You use prolog!? I'm seriously impressed) they will get similarly excited by that.

Really, you use Prolog (in the shell)? What for - have you blogged about it anywhere?

I break out in a sweat contemplating how to get an ORM to efficiently do something that would be simple in straight SQL...

Use SQLAlchemy, ORM gets easy :)

Interviewer: sort this file of 1,000 names. Here's a Linux laptop.

Candidate 1: writes some Python code

Candidate 2: writes some C++

Candidate 3: runs "sort names.txt"

I'm pretty new to the industry, being young, but as soon as I got a machine to run a Linux VM on, I spent hours reading docs and man pages and books on Bash and general *nix commands. Definitely helped me become a better programmer. It's amazing what you can do with simple POSIX commands. I didn't even know about sort until now, but it'll help me a bunch now.

Run `info coreutils` and get a general sense of the tools available. There are things you'll use all the time like sort, uniq and wc. Things you'll use occasionally (`foo | column -s $'\t' -t` reads all the lines of standard input, splits each on a tab and then lines everything up in nice columns) and things you might rarely need, but when you need them they're amazing: tsort, for example, is really useful for managing dependencies in certain circumstances.

Candidates ask no questions?

Although, it'd be fairly simple to do this in python:

python3 -c 'import sys; list(print(line.strip()) for line in sorted(sys.stdin, key=str.lower))'

Agreed for the most part, although we should acknowledge that those "new and shiny JS things" make for some extraordinary RAD times and generally don't take too long to learn (at least to become productive). They may have serious limitations, but for side-projects developers can get to an MVP with haste.

"for side-projects developers can get to an MVP with haste."

That's the caveat you want to hear. Also Facebook with php and HHVM [0] not wanting to break things too much.

[0] https://en.wikipedia.org/wiki/HipHop_Virtual_Machine

Reply on news.ycombinator.com