Hacker News Reader

Comment by kibwen | Hacker News Reader

Aside, but I'd love for someone to come up with an alternative syntax for regexes that sacrifices speed-of-typing for ease-of-reading. Regexes are obviously still fantastically useful for things like command-line scripting and jumping around in text editors, but the fact that people still reach for regexes within programs that are meant to receive long-term maintenance tells me that we're missing a tool in our toolbox.

View on HN

People have a number of good suggestions, but let me suggest unit tests as one excellent way to make regexes maintainable.

The maintenance problem for me with regular expressions is that when I write them I have carefully studied a variety of inputs and then make something that matches. Typically I will also have tested it as I go, trying out various known strings. But then a year later I have forgotten all the cases, and I have to somehow decompress them from the regex.

If I save my experiments in well-named unit tests, though, it's much easier for me to figure out my original intent, and to see whether the new case I'm thinking about is covered.

Combinators make for a very nice Regex API.

In OCaml, most people use the re[1] library for regular expressions (example[2]). I wrote a library called tyre[3] for typed extraction that follows a similar API. Of course these APIs are much more verbose, but they are also very regular(hah!): Regular operators are normal functions of the language, typechecking and completion works, etc.

[1]: https://github.com/ocaml/ocaml-re

[2]: https://github.com/ocaml/ocaml-re/blob/master/benchmarks/ben...

[3]: https://github.com/Drup/tyre

This was one of the places CoffeeScript really innovated, IMO (though of course Perl did it first). The "Block Regular Expressions"[0] allow whitespace and comments:

    NUMBER     = ///
      ^ 0b[01]+    |              # binary
      ^ 0o[0-7]+   |              # octal
      ^ 0x[\da-f]+ |              # hex
      ^ \d*\.?\d+ (?:e[+-]?\d+)?  # decimal
    ///i

[0]: https://coffeescript.org/#regexes

Rust allows this too:

    Regex::new(r"(?x)
      (?P<y>\d{4}) # the year
      -
      (?P<m>\d{2}) # the month
      -
      (?P<d>\d{2}) # the day
    ")

Another way to write more readable Rust regexen: https://crates.io/crates/pidgin

(I'm the author.)

Yup! rust/regex has had this for a while. More recently, its error messages have also gotten a lot better, which I'm proud of!

    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    regex parse error:
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    1: (?x)
    2:       (?P<y>\d{4}) # the year
    3:       -
    4:       (?P<m>*\d{2} # the month
                   ^
    5:       -
    6:       (?P<d>\d{2}) # the day
    7:     
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    error: repetition operator missing expression
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Playground link: https://play.rust-lang.org/?version=stable&mode=debug&editio...

CoffeeScript sourced a lot of its design from Python. This is one of those.

https://docs.python.org/3/library/re.html#re.X

Python version: https://docs.python.org/3/library/re.html#re.VERBOSE

CL-PPCRE turns a PCRE string into a parse tree, as a native data structure, and the programmer has full access to this interface. For example, the sample regex:

    /(?<year>\d{4})-(?<month>\d{2})-(?<day>\d{2})/u

could be specified in your program as something like:

    ((:named-register "year" (:greedy-repetition 4 4 :digit-class))
     "-"
     (:named-register "month" (:greedy-repetition 2 2 :digit-class))
     "-"
     (:named-register "day" (:greedy-repetition 2 2 :digit-class)))

(Whether this is easier to read depends on your relative familiarity with CL and PCRE. It's a lot easier to generate or manipulate with native CL functions, though, if you ever need to do that.)

Most HLLs today wrap an existing C regular expression library, and I don't know any C regular expression library that provides a public interface to its parse tree, so it's unlikely that other languages will be able to do something similar without a lot of work.

Of course, regular-expressions-as-strings are still strings, so if you only need to write them, you can get most of the benefit by using your language's native string facilities: https://news.ycombinator.com/item?id=241373

> Most HLLs today wrap an existing C regular expression library, and I don't know any C regular expression library that provides a public interface to its parse tree, so it's unlikely that other languages will be able to do something similar without a lot of work.

I don't think I know of one either. But Go's regexp library provides access to the syntax[1], and so does rust/regex[2]. In the case of [2], it provides both an AST and a high level IR for regexes. It's not as convenience to build expressions as in your Lisp example, though, there's nothing stopping someone from building such a convenience. :-)

[1] https://golang.org/pkg/regexp/syntax/

[2] https://docs.rs/regex-syntax

Yup! There's performance reasons not to write a regular expression engine in a language like Python or Ruby. I'm not surprised that other natively compiled languages like Go and Rust are the ones that come closest.

Perl has always let you split regexs across multiple lines, with freeform formatting and even comments. Absolute godsend for those (rare?!) times your regex gets a bit too complex for its own good...

Absolutely. The rare times I had to have a regex that wasn't obvious, it was great to be able to break it down with comments.

Emacs Lisp has rx: http://francismurillo.github.io/2017-03-30-Exploring-Emacs-r...

As someone who never found regex difficult to understand, and observed how some others are the same while others seem to view any moderately long regex with a look of shock and horror, it appears to largely be a matter of attitude and perception --- the ones in the former group have, for lack of a better term, an "APL mindset" to learning. That is, they're not afraid of the fact that something looks completely foreign and incomprehensible at first, and realise that it's just another language to be learned.

The main issue I think is that they get convoluted...fast. Combined with varying dialects, the issue that they’re usually write-once, update rarely, and all regexes are imperfect and will fail in unexpected fashions, and that they usually don’t appear that many times in any given codebase, means that they’re never really worth the effort of properly memorizing. Understanding is fine, its not that complex and each operator is relatively simple conceptually, but I have to re-lookup the regex syntax anytime I’m writing/reading any regex that goes beyond pure regex (*+.|)

Character classes beyond \w, d and b always require a doc-lookup and I always misremember the look{ahead,behind} operators.

The issue is that its too small to remember fully, makes no effort to be self-documenting, the docs are always annoying (I can never get a ^F search to not return 20 items), and theres always an uncovered edge case somehow

Rosie (https://gitlab.com/rosie-pattern-language/rosie) is an interesting approach to this problem, and I think PEGs in general would be better for reading. My memory is a bit foggy on the details, but I do recall that PEGs have some problems parsing certain inputs performance wise, though I could be mistaken.

Wolfram Language / Mathematica has “string patterns” which extend the built in pattern language to character-based matching. Within the context of Mathematica is a quite readable and familiar syntax.

https://reference.wolfram.com/language/guide/StringPatterns....

This JavaScript library popped up recently here: https://areknawo.github.io/Rex

Perl6 attempts this: https://docs.perl6.org/language/regexes

I'm not sure so much innovation in syntax is going to help adoption, but the aim was to make the syntax less code-golf-y and more readable.

The terseness of regex isn't about speed-of-typing but ease-of-reading. Once you get used to the syntax, you can pattern match in your head pretty easily (for any non-insane use of regex).

Pattern matching with a verbose syntax that is "easy to ready" has been done hundreds of times but it rarely translates into "easy to read" for anyone familiar with regular expressions.

Rebol's parse dialect is basically easily readable regexes (actually even more, but that's not relevant here).

For some examples see: http://blog.hostilefork.com/why-rebol-red-parse-cool/

Most regexp libraries let you use named groups and comments which let you write more maintainable expressions. The fact javascript does not may explain why so many people don't know how to use them.

https://www.regular-expressions.info/freespacing.html https://www.regular-expressions.info/named.html

You can have some of this in JS too by using libraries such as this one (which is mine): https://github.com/jawj/js-xre

It does now! (named capture groups)

ES2018 introduces named capture groups.

They're available today in Chrome, unsure about other browsers.

Reply on news.ycombinator.com