Hacker News Reader

Comment by Ocerge | Hacker News Reader

Regex is one of those topics I just can't seem to commit to memory. The concept is easy enough, and every time I need one I can usually work out what to do after a short amount of research, but I feel like I forget immediately afterwards. It may have to do with how regexes of any complexity look like machine code to me.

View on HN

In addition to the 'practice' stated elsewhere, the other thing someone needs to really get to grips with something like regex is a 'reason'.

I never really got the point of regexes until I discovered capture groups. Pretty much everything I use them for is based on some problem like:

"Change date format from 'DD-MM-YY' to 'YYYY-MM-DD'"

"Keep only lines in the log file with a date in them"

"Swap the first name and last name of every contact in this list which has a comma in it"

"Rename all the files in this folder to strip out all the gubbins and put the useful stuff in a standard format"

And so on. You need a problem where doing it any other way than regex would be lengthy and painful and the regex itself is light and breezy. You build up little snippets or "phrases" that you can re-use easily to solve new problems.

Regexone (https://regexone.com/) is one of the best online tutorials I’ve come across for anything. I got a decent grasp of Regex after going through the series 1-2 times.

Reminds me of websites like https://www.sqlteaching.com/ It's obviously not for regexes but after that I understood basic SQL so much better.

(also I love comments that point out amazing learning resources with full conviction)

The creator of regexone.com also created sqlbolt.com for SQL, which I also highly recommend.

Strongly agree with this recommendation. I ran through regexone.com once, and even though I already knew regexes, it helped me fill in some stuff I had gotten rusty on.

I highly recommend the lessons. If you do them, I also highly recommend donating the recommended $4 via Paypal (or more), although it's technically free to use. It's well worth it.

I haven't seen it mentioned here yet, but wanted to throw out (https://regexr.com/) as a decent resource into writing regular expressions. I think I haven't needed to use a tutorial because of how easy it makes writing them.

I'll second this, it's immensely useful when I can't wrap my head around a regular expression.

regex101.com is my go-to, because it supports regex flags and Python regexes.

Going to add regexper.com which is really good for explaining RegEx patterns.

Practice, my friend. You might have developed a systematic methodology for quickly learning and internalising (many) other things. (Kudos.) But sometimes it just comes down to wrote practice and repetition; time and effort might be key here for your mastery of this subject.

The problem I have with learning regex is similar to the above. It's easy enough to (re)learn when you need it and it doesn't come up often enough for me to consider actually learning it as being useful.

To add to that, regex mojo has just kind of built up in my head over years. It's not something a topic I was able to study, learn, and retain (well). Every time I stumble onto a regex use case I need to understand & don't, I level up at that time, and that's always been totally fine.

A few months ago I needed to add a regex for validating IPv4 addresses that were in a non-standard format. I had to do a little research, then I wrote & tested it, and promptly forgot 90% of what I had just learned.

>time and effort might be key here for your mastery of this subject.

It doesn't have to be this way. If most programming languages were designed like regular expressions they would be unusable. I can spend a year w/o writing a single line of Pascal or C++ and then it would come back immediately. I occasionally found regular expressions useful in my work, but only a couple of times could justify spending time on learning a little bit of it only to forget the next day. In a few other cases I simply googled a solution and used w/o understanding it. Most often I just write code instead, it is verbose but at least it's clear what it does.

I suppose what I'm trying to say is that from personal and anecdotal experience I got 97% in a first year calculus course. I went in with a marginally good algebra result, but I resolved to learn the shit out of everything that came my way.

I systemised as much as I could, but there was a lot of write practice when it came to learning trigonomic idendities and transidnetal functions. And not just the normal ones; I mean all of them ... both the inverserse and hyperbolic identities.

I did all of the problems and got extra ones.

In short, I knew the material backwards and forwards; I finished hour long exams in 20 minutes, and was usually the first or second person to leave.

What I am trying to say is that REGEX is similar to calculas and in my view requires significant focus and practice to master. Or you can muddle your way through it when you have to.

This is a new and excellent way to actually remember how to write regex https://www.executeprogram.com/

Spaced repetition works!

I did a static analysis code tool based on regex and only after writing hundreds of them, some of thousands in length, regex clicked.

That was how I was as well until I got tasked with making feature rich text inputs (decorate hashtags, user tags, replace names with emails, vice versa). At a certain point it gets too difficult to respond to user input as it is just too variable (user highlights 40 chars and presses ‘A’? user pastes 100 chars?). A bulletproof method is running find and replace when needed and the ideal way to do that is regex.

Forcing myself to get tough regex working is what really made me grasp some of the more advanced concepts that unlock regex. It’s mainly just that we never need regex, and if you use it for a simple task you’re a jerk for introducing unnecessary complexity. Why would people learn it?

It's because it's terse with quite a low amount of mnemonics (a meaningful feature is only 1-2 characters, there's no padding for a spacial understanding of hierarchies, etc.., compared to your host language), and even though it comes in very handy, most developers aren't spending a large amount of time writing them in your average project.

I learned first about Regular Expressions i.e. the finite automata that do a simpler subset of what Regex can do. Learning about this and drawing out the graphs etc, may help commit to memory.

Also start using them to help you find and replace stuff in a text editor. They are very handy for converting for example a csv into code.

> It may have to do with how regexes of any complexity look like machine code to me.

Regexes get a bad rap because programmers who write otherwise maintainable code throw code hygiene out the window when writing long regexes. Your host language is much more complicated than the regex language, but writing shitty unreadable regex is, for whatever reason, acceptable. Some unfortunate and unnecessary limitations of typical regex libraries make the problem worse.

1. Regexes are typically written as one-line strings instead of as structu/red code. When writing C/Java, programmers understand that they should put each element of a sequence on a separate line and use indentation to visually signal branching and loops. But for some reason when writing character-processing programs that also have sequences, branching (|), and loops, programmers almost always golf it and put the whole complex program all on one line.

If you're writing a regex longer than 5 or 10 atoms, place each sequence on a separate line and use newlines+indentation to visually offset disjunctive choices (|) and loops (star).

Rule of thumb: you should be able to roughly sketch the rough shape of the finite automata by crossing your eyes and eyeballing the shape of the regex, just like you should be able to roughly sketch the control flow of a program by crossing your eyes and eyeballing the shape of the code.

2. Character group names are too terse (e.g., \s instead of "\whitespace" and \d instead of \digit), probably because of #1. Also, very few people give names for long subexpressions (e.g., factoring code out into functions with well-chosen names).

3. Gotos/try...catch (i.e., backtracking) are not used judiciously and aren't well-documented/tested. I often see backtracking or confusing mixes of lazy and greedy matching instead of just writing out a slightly longer disjunction.

4. Regexes are often used for languages that aren't even almost regular. A bit of backtracking is OK (gotos are sometimes OK), but if there's a lot of backtracking then you need to use a different class of languages/machines.

5. There's no way to embed non-regular matchers into a regex.

Due to the combination of 1-5, matching an email address with an optional recipient name (so something like "asdf@asdf.com" or something like "John Smith <asdf@asdf.com>") you need an insane regex that many people implement with lots of backtracking and so on all stuffed into a single line.

But something like this would work just fine and is much more readable:

    \emailAddress := ...
    # todo: need to support dashes in names.
    \name := (
        [a-zA-Z]*
        \whitespace?
        [a-zA-Z]*
    )
    # matches asdf@asdf.com or John Smith <asdf@asdf.com>
    (
        \emailAddress
    )
    OR
    (
        \name
        \whitespace? 
        <
        \emailAddress
        >
    )

where e.g., the implementation of \emailAddress could be written as a stand-alone parser in the host language. But even without digging into email address you can already see how this is way more readable than:

    \emailAddress|[a-zA-Z]*\s?[a-zA-Z]*\s?<\emailAddress>

Writing readable regular expressions shouldn't be difficult -- just treat the regex like any other piece of code and allow inter-op with the host language. But few people/libraries put in the effort, and for whatever reason golfing regexes in production code is considered acceptable even in orgs where you'd be fired for code golfing in the host programming language.

Lua's Lpeg module (http://www.inf.puc-rio.br/~roberto/lpeg/) is probably what you are after:

     lower = lpeg.R("az")
     upper = lpeg.R("AZ")
     letter = lower + upper

Personally I prefer though the terseness of regular expressions.

Python has another somewhat reasonable solution. Either of those solutions can be combined with good programming to constitute a reasonable solution.

> Personally I prefer though the terseness of regular expressions.

I think there are legitimate use-cases for both.

If you're quickly hacking out a small good-enough parser for something regular or "almost regular", terseness can be great.

However, if you're parsing a large regular language, terseness isn't really a benefit. Perl is on its bed for a reason, and overly terse regular expressions should die for a similar reason. Overly-clever write-only coding culture sucks.

But the terseness of regular expressions is basically terrible beyond maybe a few hundred characters. E.g., the following has no place in production code -- you might as well just include a binary:

    (?:(?:\r\n)?[ \t])*(?:(?:(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t]
 )+|\Z|(?=[\["()<>@,;:\\".\[\]]))|"(?:[^\"\r\\]|\\.|(?:(?:\r\n)?[ \t]))*"(?:(?:
 \r\n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(
 ?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|"(?:[^\"\r\\]|\\.|(?:(?:\r\n)?[
 \t]))*"(?:(?:\r\n)?[ \t])*))*@(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \000-\0
 31]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)*\
 ](?:(?:\r\n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \000-\031]+
 (?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:
 (?:\r\n)?[ \t])*))*|(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z
 |(?=[\["()<>@,;:\\".\[\]]))|"(?:[^\"\r\\]|\\.|(?:(?:\r\n)?[ \t]))*"(?:(?:\r\n)
 ?[ \t])*)*\<(?:(?:\r\n)?[ \t])*(?:@(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\
 r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[
 \t])*)(?:\.(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)
 ?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[ \t]
 )*))*(?:,@(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[
 \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[ \t])*
 )(?:\.(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t]
 )+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[ \t])*))*)
 *:(?:(?:\r\n)?[ \t])*)?(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+
 |\Z|(?=[\["()<>@,;:\\".\[\]]))|"(?:[^\"\r\\]|\\.|(?:(?:\r\n)?[ \t]))*"(?:(?:\r
 \n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:
 \r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|"(?:[^\"\r\\]|\\.|(?:(?:\r\n)?[ \t
 ]))*"(?:(?:\r\n)?[ \t])*))*@(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \000-\031
 ]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)*\](
 ?:(?:\r\n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \000-\031]+(?
 :(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?
 :\r\n)?[ \t])*))*\>(?:(?:\r\n)?[ \t])*)|(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?
 :(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|"(?:[^\"\r\\]|\\.|(?:(?:\r\n)?
 [ \t]))*"(?:(?:\r\n)?[ \t])*)*:(?:(?:\r\n)?[ \t])*(?:(?:(?:[^()<>@,;:\\".\[\]
 \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|"(?:[^\"\r\\]|
 \\.|(?:(?:\r\n)?[ \t]))*"(?:(?:\r\n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \t])*(?:[^()<>
 @,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|"
 (?:[^\"\r\\]|\\.|(?:(?:\r\n)?[ \t]))*"(?:(?:\r\n)?[ \t])*))*@(?:(?:\r\n)?[ \t]
 )*(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\
 ".\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \t])*(?
 :[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[
 \]]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[ \t])*))*|(?:[^()<>@,;:\\".\[\] \000-
 \031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|"(?:[^\"\r\\]|\\.|(
 ?:(?:\r\n)?[ \t]))*"(?:(?:\r\n)?[ \t])*)*\<(?:(?:\r\n)?[ \t])*(?:@(?:[^()<>@,;
 :\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([
 ^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\"
 .\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\
 ]\r\\]|\\.)*\](?:(?:\r\n)?[ \t])*))*(?:,@(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\
 [\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\
 r\\]|\\.)*\](?:(?:\r\n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\]
 \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]
 |\\.)*\](?:(?:\r\n)?[ \t])*))*)*:(?:(?:\r\n)?[ \t])*)?(?:[^()<>@,;:\\".\[\] \0
 00-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|"(?:[^\"\r\\]|\\
 .|(?:(?:\r\n)?[ \t]))*"(?:(?:\r\n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \t])*(?:[^()<>@,
 ;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|"(?
 :[^\"\r\\]|\\.|(?:(?:\r\n)?[ \t]))*"(?:(?:\r\n)?[ \t])*))*@(?:(?:\r\n)?[ \t])*
 (?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".
 \[\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \t])*(?:[
 ^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]
 ]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[ \t])*))*\>(?:(?:\r\n)?[ \t])*)(?:,\s*(
 ?:(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\
 ".\[\]]))|"(?:[^\"\r\\]|\\.|(?:(?:\r\n)?[ \t]))*"(?:(?:\r\n)?[ \t])*)(?:\.(?:(
 ?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[
 \["()<>@,;:\\".\[\]]))|"(?:[^\"\r\\]|\\.|(?:(?:\r\n)?[ \t]))*"(?:(?:\r\n)?[ \t
 ])*))*@(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t
 ])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[ \t])*)(?
 :\.(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|
 \Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[ \t])*))*|(?:
 [^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\
 ]]))|"(?:[^\"\r\\]|\\.|(?:(?:\r\n)?[ \t]))*"(?:(?:\r\n)?[ \t])*)*\<(?:(?:\r\n)
 ?[ \t])*(?:@(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["
 ()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[ \t])*)(?:\.(?:(?:\r\n)
 ?[ \t])*(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>
 @,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[ \t])*))*(?:,@(?:(?:\r\n)?[
 \t])*(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,
 ;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \t]
 )*(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\
 ".\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[ \t])*))*)*:(?:(?:\r\n)?[ \t])*)?
 (?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".
 \[\]]))|"(?:[^\"\r\\]|\\.|(?:(?:\r\n)?[ \t]))*"(?:(?:\r\n)?[ \t])*)(?:\.(?:(?:
 \r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\[
 "()<>@,;:\\".\[\]]))|"(?:[^\"\r\\]|\\.|(?:(?:\r\n)?[ \t]))*"(?:(?:\r\n)?[ \t])
 *))*@(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])
 +|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[ \t])*)(?:\
 .(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z
 |(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[ \t])*))*\>(?:(
 ?:\r\n)?[ \t])*))*)?;\s*)

Thanks for your post, I learned a lot and I feel excited when I come across an experience to use a regex again to put this into practice.

Reply on news.ycombinator.com