PAWK - A Python line processor (like AWK) | Hacker News Reader

PAWK - A Python line processor (like AWK)

github.com

63 pointsrachbelaid24 comments

I'm going to go on a small tangent here and warn strongly against parsing the output of `ls` (cf. the examples given in the README).

Parsing `ls` is inherently fragile[1] and it will break at some point. Please don't do it.

[1]: http://mywiki.wooledge.org/ParsingLs

I really find this amazing.

I've been watching Larry Wall's Perl 6 talks off late, where he talks of designing Perl programming languages(1,2,3,4,5 and 6). He goes back to 1987 and talks about problems in 1987 about difficulties in sanely doing text processing then. He mentions the problems really were that if you wanted to do simple text processing work there was always awk/sed and other little unix utilities in form of a command line program using pipes. But any thing more than that and you really had to write program combining them in a language like C(The dominant language of the day;circa 1987).

I see the situation hasn't changed at all.

Almost anybody who doesn't use Perl, continues to bend <insert their favorite language here> in a way that badly implements what Perl was invented to do in 1987. Which is to really solve problems like these, in the most appropriately designed programming language for it.

Perl was designed to fit the niche between [c/python/java/<insert your language>] and [shell/unix text processing utils].

When you bend your language to fit that niche, like this tool does(This tool makes a good attempt at turning Python into a Perl like language). You make it largely look Perlish. This is dilemma languages trying to replace Perl must face, everytime you try to invent a language to replace Perl, you end up inventing a tool/language that largely looks and works like Perl(Read: Ruby/Perl 6). People who have learned Ruby after learning Perl know this fact, it really feels like you are just learning some more extra syntax above Perl.

Giving a powerful command line language is very difficult. Because by very definition you have to design a powerful syntax in the most terse ways ever. Where does that leave us? This in a C based syntax and what you get is Perl.

It will take a little while to understand that languages like Python and Ruby are not replacing Perl. That is for the same reasons why Perl even in CGI days never came to replace Php. Its just these languages and tools like Perl/awk/sed are designed to rule a very different niche.

When you begin to understand the design and niche they occupy you will suddenly discover that your 100 custom scripts and jar files you spend days writing do badly what was already solved in one constant design paradigm some 2-3 decades ago.

This is the reason why the grey beard unix guy smiles at you every time you show him your world changing script.

Have you ever maintained some else's poorly written Perl code? I have. Rational or not, I hate Perl.

And yes you can write crappy code in any language, but Perl is just asking for a bad programmer to write bad code.

It's one of those never ending tradeoffs. The premise of the parent comment strikes a chord with me.

We necessarily can't do terse and readable. To be terse enough we need to encode. Encode is just another way to say obfuscate, in this context anyway.

I wouldn't necessarily agree Perl asks for a programmer to write bad code (saying this as someone who left Perl because he was fed up with write-only code). I'd phrase it as Perl readily facilitates bad code.

I have maintained Perl code.

I don't what you mean by the term 'poorly written Perl code'. I have maintained more horrible code in languages like Java than in Perl.

>>but Perl is just asking for a bad programmer to write bad code.

And what is that wand that other programming languages have, that when cast with a spell magically transform a bad programmer into a Dennis Ritchie?

It's just that Perl's power/expressiveness and TMTOWTDI mindset tend toward harder to _read_ code. You can write awful hard-to-understand code in any language, but in those other languages the reading part is usually easier. Except for maybe C++. :-)

> if you wanted to do simple text processing work there was always awk/sed and other little unix utilities in form of a command line program using pipes. But any thing more than that and you really had to write program

In the current landscape of scripting languages, what makes Perl superior to Python for text processing?

Asking as I have not touched Perl in over 10 years, after it was supplanted by Python for my needs. I use awk for many of the tasks I could call "text processing", and Python for more complex stuff.

>>I use awk for many of the tasks I could call "text processing", and Python for more complex stuff.

I use Perl for both.

That was the whole point of my post. Perl was invented because you shouldn't have the need to use (awk/sed etc) + (some other language) for text processing and what you call as 'complex stuff'.

What I found happening (as someone who started with Perl 4, then Perl 5, then largely moved to Python) is that my scripts often went through this evolution: shell (+ sed, + awk, ...), then Perl, then Python. That is, as the program grew more complicated, the best tool for the job (in its entirety) often shifted.

The Perl to Python transition was usually due to Python including something in the standard library that Perl didn't. e.g. the example usage of pawk outputs json, which Python includes in the standard library. Doing that in Perl would mean installing a CPAN module.

Anyway, at some point I started going from shell to Python and skipping the Perl step. These days I mostly reserve Perl for one liners and 90% of the time that's perl -[pn] -i -e '...'.

I have a pattern of the opposite. At one point almost everything was in Python, even very simple things. I "discovered" awk, and found I could do a lot of things in a few piped commands.

Python is still my go-to for a tasks > 1 line of code, but it is still surprising what can be accomplished in just one line of bash.

To be clear, I still write a ton of stuff in shell. I try to think ahead to whether what I want to do can be accomplished in shell + standard unix utilities more easily than in Python. I use Python if it's not easily possible in shell + unix utilities, or the script is likely to be more than a few hundred lines.

Here's one I probably should've written in Python - https://gist.github.com/jaysoffian/f39fa2da8e7f340550f9

Then use Python for both. Everyone's happy.

Python's nice syntax also makes pawk nicer to read than awk or perl, to my eyes.

This is all just your opinion and IMO it's totally wrong. I programmed Perl for over a decade and I don't reach for it for anything anymore. The main thing it was good at was regex and regexes are just a poor way to do most things. Even perl6 was moving to grammars.

Yet another option: pyliner - A Python line processor (like Perl): https://gist.github.com/dgulino/4750088

This looks like a really useful tool, although the automatic importing of anything that smells like a module makes me a little uncomfortable:

    modules = re.findall(r'([\w.]+)+(?=\.\w+)\b', all_text)

then later:

    for m in modules:
      try:
          key = m.split('.')[0]
          self[key] = __import__(m)
      except:
          pass

I love awk but personally cannot remind the syntax to do complex stuff without googling or dig in the man. I didn't test the performance but I don't always need performance (or even rarely) ... I'm adding pawk in my toolbelt and I glad somebody built it

You may find this useful, also in python:

http://code.activestate.com/recipes/437932-pyline-a-grep-lik...

And if you find that useful, here's an extension of that same idea: https://github.com/gvalkov/python-oneliner (I haven't worked on it in months though)

Hmm I feel uncomfortable if there's an eval going on in there; what if the commandline is tainted etc? Its something to keep in mind.

Surely if your command line is tainted then you have bigger problems?

people often have websites that take some parameters and then pass them to grep or whatever, in the exact same way that they pass parameters that came from the user to an SQL engine.

As long as its data, no problem...

Injection attacks are injection attacks.

neat. not sure i'll use it, but neat nonetheless. thanks for sharing. i've been a big awk user for well over a decade (and a python user for nearly as long), so this was neat to see.

mawk is fast like nobody's business. I mainly use awk for speedy filetype conversions for large data sets. I am not sure if I would reap any benefits from switching my awk interpreter into a JIT language, but this is a really cool project nonetheless.

Reply on news.ycombinator.com