Is it a must for every programmer to learn regular expressions? | Hacker News Reader

Is it a must for every programmer to learn regular expressions?

programmers.stackexchange.com

31 pointsriyadparvez58 comments

Yes, it is, without a doubt. It's one of the most universal tricks of the trade that you'll literally never regret learning, mainly because just about any environment you'll ever work in will by necessity support regexes, and in many, it will be the primary way you interact with text.

But it's also a must that you realize that despite the fact that they're exceptionally useful and widely supported, regexes are a disgusting abomination, one that we should be absolutely mortified to be associated with. It's one of the worst syntaxes to ever be invented, and every one of us should feel the cold stink of UX failure wash over us every time we write a regex. If we ever catch ourselves writing a DSL that in any way, shape, or form resembles regular expressions, we should stop immediately and ask what the fuck is wrong with us and why we're being so opaque and random. Regular expressions are quite literally one of the worst syntaxes to ever be introduced in our field.

I worry a lot when someone doesn't know regular expressions at all. But I worry far more when someone thinks they're beautiful. That person has far too high a tolerance for unintuitive syntax and code, and will cause vastly more damage to my codebase than even the rank amateur that still uses "goto" on a regular basis.

Which is not to say that we shouldn't lean on regexes heavily anyways when they're appropriate - as programmers our primary job description is that we're paid good money to work with shitty interfaces in order to express simple ideas and algorithms.

>It's one of the worst syntaxes to ever be invented, and every one of us should feel the cold stink of UX failure wash over us every time we write a regex.

Okay, so they're ugly and hard to read at first glance. Considering the purpose of a regex, I can't think of another way to implement them that doesn't involve typing more characters needlessly (therefore even making it more hard to comprehend).

How would you design a regular expression syntax more intuitively? Personally, I find beauty and simplicity in regular expressions. Sure, they can grow to hideous atrocities, but you can achieve such disastrous feats with any language/syntax. Maybe you could back up your claim of regexes being a disgusting abomination with, at the very least, anecdotal evidence.

A typical regex looks like this:

  \b[A-Z0-9._%-]+@[A-Z0-9.-]+\.[A-Z]{2,4}\b

Which is also what happens when a cat walks across the keyboard.

Call me weird, but I find that very readable.

I find that perfectly readable, except for the \b which I hadn't seen before. It's matching an all-uppercase email address.

There's plenty of upper case e-mail addresses that won't match that expression.

There's characters missing[1] and the tld is too short[2]. And even if that's fixed, we still don't match internationalized addresses or actually validate that the e-mail address exists. You're probably better off with something like...

   if "@" in email and "." in email.split("@")[1]:
       send_verification(email)

...but you should probably also check for common misspellings like "gmial.com" etc.

[1] http://en.wikipedia.org/wiki/Email_address#Syntax [2] http://en.wikipedia.org/wiki/List_of_Internet_top-level_doma...

Well, if you want to be fully compliant you can go for this 6kb monster

http://ex-parrot.com/~pdw/Mail-RFC822-Address.html

There's nothing wrong with regex syntax. But there _is_ something wrong with the formatting of your example: it's not readable. Perhaps you _should_ write (assuming Java-syntax):

  (?x:              #standard token is an uppercase letter, digit, dot, or hyphen
    \b
    [A-Z0-9._%-]+   #1 or more of standard token, underscore, or percent sign
    @               #at-sign
    [A-Z0-9.-]+     #1 or more standard tokens
    \.              #dot
    [A-Z]{2,4}      #2 to 4 uppercase letters
    \b
  )

We can easily optimize for readability with regex syntax.

You can say that about pretty much anything if you aren't familiar with the syntax. That looks pretty readable to me - certainly far more portable and readable than equivalent code.

[It's also perfectly obvious that if this is an attempt to match email addresses that it's not a very good one - but I don't know the context where it's supposed to be used, it might be good enough for whatever the author intended].

IMO, the main problem is that the syntax is too terse. The syntax is taken wholesale from the algebraic notation used in mathematics and dropped in unmodified. Essentially, regex code ('cause regexes really are code) is not skimmable.

The greatest syntactical atrocity in regexes is that they don't have the `x` modifier (in Perl parlance) on by default. This means that you can't use whitespace to chunk code into meaningful bits, nor can you comment it to easily document what does what or explain a particularly hairy section to handle some weird edge case. This means that regexes degenerate a lot faster than ordinary code in terms of readability.

Edit: Misplaced close paren.

"Easy to write, hard to read". Perl's influence on Regex shows. Which is fine in most cases.

I always wonder what regexes would look like if they were derived from Python instead.

Actually, regex syntax isn't primarily a Perl invention. Regexes as software tools go back to early Unix text processing tools (ed and grep, according to Wikipedia), and Perl took the syntax from those tools.

It is true that Perl reformed the syntax in important ways (to the better, if you ask me), and later on extended it a lot, but it's certainly not a Perl invention.

Hmm - you know I actually think it's not correct to look at regex as an interface - even though we use it as one. It's really more accurate to look at it as a grammar (type 3 if I remember correctly).

Anything that comes out of the whole chomskian hierarchy stuff isn't going to look intuitive. But the point is that it is a particular, very rigorously defined system of representation. And various systems of representation are always more or less intuitively accessible - and come with a whole set of trade offs around what they can represent vs their ease of use etc...

These things just are - written into the laws of the world. They are discovered - not invented. We pick them up and use them as we would rocks left lying around. We find better ones when we can and fashion better ones when we can too...

> Anything that comes out of the whole chomskian hierarchy stuff isn't going to look intuitive.

Why not? Type-0 (recursively enumerable) languages are equivalent to Turing machines, and we've managed to invent some pretty good syntaxes for that. The main problem with regexes really is the syntax. Regular languages are a lot easier to understand (IMO) if you look at the left/right-linear grammars that define the same language as the regex.

Regex syntax as a representation is very close to the FSA used for matching, and that's not necessarily the representation best suited for human consumption.

> These things just are - written into the laws of the world. They are discovered - not invented.

This applies only to the theoretical computer science regexes. The practical regexes are very different in this aspect.

Practical regexes are neither discovered nor invented - they are constructed. The theoretical regexes are just the basis but on top of that there's a lot of features added. Some of them are just syntactic sugar for theoretical regexes but others actually make the language non-regular.

Groups that match what previous named groups have matched are definitely in this category.

Please name 2 concrete ways to improve the "interface" of a standard regex.

I think that regular expressions have a good syntax for most typical (small) uses of regular expressions. Maybe the choice of special characters isn't ideal, and it might be nice to have 'English' versions of more special characters (similar to e.g. [[:digit:]] in POSIX), but for small regexes the (mostly) one-to-one mapping between characters in the pattern and characters in the string is a very nice and intuitive syntax.

I think the real problem is that we lack (or don't learn) good tools to bridge the gap between regular expressions and 'custom parser'. We're reluctant to refactor from '1 line of just-starting-to-be-horrible regex' to tens or hundreds of lines (depending on language and libraries) to do it 'properly', and so we end up stretching regular expressions beyond the point where they make life easier.

Perl has Parse::RecDescent (and probably several others), which is pretty close to the right thing, and clearly it's very doable in a lot of languages - anyone got any suggestions in other languages?

I could imagine a Jquery-like DSL where something like /^[A-Z]+[0-9]{2}$/ could be expressed like Match(str).BeginsWith().BigCaseAlpha().AtLeastOne().Numeric().FixedLength(2).EndsWith()

Of course, you would also have to be able to nest these for more advanced matching...

Is there something like this already in existence? :)

How about pyparsing? http://pyparsing.wikispaces.com/

I really liked eykanal's answer [1].

    > Regular expressions are a tool. It happens to be a
    > very useful tool, so many people choose to learn how
    > to use it. However, there's no "requirement" for 
    > you to learn how to use this particular tool, any
    > more than there is a "requirement" for you to learn
    > anything else.

Nails it. I do think most programmers will eventually run across a problem to which the solution is 'Use Regex', but it's not an absolute "must" like boolean logic.

[1] http://programmers.stackexchange.com/questions/133968/is-it-...

No requirement to learn anything else? Reading and writing are pretty good skills no matter what your situation.

good point indeed. I spent my first years programming DSP algorythms on embedded systems. Hardly any strings used, let alone there was any need for regular expressions. Learning their ins and outs back then would have been a waste of time, and would have been like a bricklayer having a fork in his toolbag.

Probably worth revisiting this bit of commentary from Rob Pike:

"Regular expressions are hard to write, hard to write well, and can be expensive relative to other technologies... Standard lexing and parsing techniques are so easy to write, so general, and so adaptable there's no reason to use regular expressions.

"Another way to look at it is that lexers and parsing are matching statically-defined patterns, but regular expressions' strength is that they provide a way to express patterns dynamically. They're great in text editors and search tools, but when you know at compile time whatall the things are you're looking for, regular expressions bring far more generality and flexibility than you need.

"Encouraging regular expressions as a panacea for all text processing problems is not only lazy and poor engineering, it also reinforces their use by people who shouldn't be using them at all."

http://commandcenter.blogspot.com/2011/08/regular-expression...

http://news.ycombinator.com/item?id=2915137

Personally, I think they're pretty darn useful and too powerful to not learn, but Pike's comment makes me think that maybe they're also a crutch that I've relied on too much rather than learning enough about lexing/parsing.

Yes. Next question?

From time to time I re-read Larry Wall's "Apocalypse 5" - http://www.perl.com/pub/2002/06/04/apo5.html?page=6 It is still in interesting look into why regexs are the way they are.

Also if you are familiar with this quote:

  "Some people, when confronted with a problem, think 
  'I know, I'll use regular expressions.'   Now they have two problems."

Read Jeffrey Friedl's research into the quote - http://regex.info/blog/2006-09-15/247

While knowledge how to use regular expression is invaluable, I also recommend learning how they actually work under the hood. It really gives a good lesson when regular expressions are applicable, and when they're not. From my experience, while many programmers are apt in tools like regexps or grammar->parser generators, they very rarely know how it actually works, which results in people trying to parse HTML with regexps or similar things. It is also a good starting point to some very interesting theoretical stuff like the theory of computations.

Also, you see people doing simplistic string comparisons using regexes. Which is ok sometimes but is an easy target if your system has performance issues.

I find one the hardest aspects of using a regex is all the subtle differences between languages/environments. I use them about four times a year but I feel like I have go read a mini tutorial each time. I know the concepts but I can't remember the special char for whitespace or digits in the particular language I'm working in. Also the java implementation is so bad. Having to encode it as a string with its own escaping rules is not easy. Then the 3 line api usage is a real hassle.

The ability to parse text quickly is invaluable when writing code.

I would say it's a must to take your coding skills to the next level.

There's no "a must." But I don't consider a programmer to be a good one if he doesn't know regexps.

If you ask "do you personally really need regexps" I'll tell you: don't learn them. As you're asking that question at all, I understand that you're not interested to learn and that you are looking for an excuse not to learn, so do something that interests you.

I regularly forget regex intricacies, so now have an app called "regexbuddy" for Windows. Very very useful, has a killer help file, has the ability to adjust and test regexes for many languages.

Surprisingly, here: http://www.regexbuddy.com/

The help file alone will make you want to buy it.

I think they're a valuable as a simple tool e.g. using :s/^#// in vim.

However, I try to avoid using them in my code unless they improve readability. Using re.VERBOSE can help in Python.

If you find a regex online, you should definitely reference it in your code, to help provide background understanding, such as validating a UK Post Code.

I taught my friend how to use regular expressions for search and replace. He's found them invaluable and he's NOT a developer. He uses then to clean up and filter lists of keywords from Adwords.

I reach for refex frequently but almost never as something I add to the code I'm writing. They're just amazing for filtering and bulk editing text.

I was thinking about this question and it occurred to me. The programmers who probably don't need to know regexes[1] almost certainly already learned them. So I guess that's a yes.

[1] Thinking of embedded systems developers creating code for, say, automotive entertainment systems, or control code for scientific or medical hardware.

[1] Thinking of embedded systems developers creating code for, say, automotive entertainment systems, or control code for scientific or medical hardware.

Don't they have log files to read?

That's what grep is for ;-)

But the point is I can't imagine someone getting to that point without previously having learned it.

Err, grep uses regular expressions.

You don't need to know regular expressions to use grep for reviewing log files though.

I think every programmer should have an understanding of what Regular Expressions are, how they can be used, and where to find a cheat sheet or reference (like www.regular-expressions.info).

It helps to know the basic syntax by memory, but you can just as easily look it up if you understand how they work.

I think every programmer should know them, but should try to avoid using them unless they really are the best solution to your problem. Too many people reach for them too soon, leaving hard to understand and hard to maintain code that could have been better expressed some other way.

Yes! They're not that hard and there's only so much syntax to them (vs. an actual programming language). I essentially learned from this site by just entering in random strings and trying to match them. http://rubular.com/

Regex is a good example for a bad interface. When usually half of the input needs to be escaped something must have gone fundamentally wrong. I regularly forget regular expressions and look them up again when I really need them.

I think it's essential to know of their existence, but not necessarily to know their syntax inside-out. Basically, enough enough that you're not writing looping and parsing code yourself, using a regex where applicable.

The problem is that regexp is often misused as things like html parsers and e-mail validators. Learning regexp syntax without learning when and how to use it, makes you a worse programmer not a better one.

At least is good to know what can be solved with regular expressions, then if you need to solve something you can always look it up in a book or use some software for it (I use RegexBuddy).

Only if you're doing a lot of string parsing. Just like learning about algorithms on graphs is a must only if you're working with graphs of a non-trivial size.

I don't agree. Regular expressions come in handy as well during your day to day work: you can save a lot of time if you know how to use them in your editor "search & replace" function, or just when a simple search in a directory gives you too many results and you need to switch to power tools (read: egrep).

Learn the 20% that you will use 80% of the time. If your not willing to do that your probably not serious about programming in the first place.

Amen to that. For reasons I can only guess at, I'm the "regex guy" at work. When people have a regex question they come to me. And the interesting thing is that the answers are not hard, well within the 20% area. I've found that most people don't care to learn how to use character classes, or even what they're for. For example, to match the single letter 'i', I've seen people write:

[i]

Many don't know the difference nor implications of .* ? vs .*

And on and on. It amazes me that these people actually code for a living.

Learning about the nongreedy modifier is IMHO a must when it comes to regexps. It makes them much more convenient to write.

If you wanna do it for serious. At the very least you really ought to work how grammars work (which are strictly more powerful than regexes).

They are useful but non-intuitive, so I imagine most people relearn them each time they meet - unless you are stuck doing it all the time.

I wonder if Perl programmers are resented for our regex knowledge ;-)

I regularly tell non programmers that it is the most useful concept that they can learn (if they deal with data)

You can get by as a programmer without knowing regexes. You can also get by without knowing about pointers. Heck, maybe you can even get by without knowing SQL or OOP.

But you will be limited. You will run into a lot of situations where your ignorance will create friction and limitations. Something as simple as editing an nginx configuration file, for example.

You should learn regexes, you should be able to use a regex as necessary and be able to understand regexes you come across, you should understand when they are most useful, and when they are not. You should learn their limits and your limits in using them as well. When used appropriately they can be a potent addition to your technical knowledge. When they should be used but are avoided the result is typically a massive inflation of the effort to solve a simple problem. And when used incorrectly they can result in an inflation of intractable complexity (much like any technology).

Reply on news.ycombinator.com