A human way to define regular expressions in Ruby | Hacker News Reader

A human way to define regular expressions in Ruby

krainboltgreene.github.io

68 pointsvbv68 comments

Am I the only person who thinks that things like this are totally unnecessary? Is learning/reading regular expressions really that difficult for most people?

Here's the subset of regular expressions that has gotten me through nearly all of the regular expressions I've ever needed to write. As a plus, it has no dependencies!

* - zero or more of the preceding character/group

+ - one or more of the preceding character/group

? - zero or one of the preceding character/group

$ - end of line

^ - beginning of line

. - one of any one character

\ - escape the following character (for a literal '$' or '.', for example)

[<some characters>] - one of the given characters

[a-zA-Z0-9] - letters and numbers inside a group can have ranges!

(<something>) - capturing group (anything that matches inside it will be accessible in the match object)

<thing1>|<thing2> - either the first thing, or the second thing (or the third, or the fourth...)

This isn't a complete, or even precise, definition, but knowing those things will get you to the point where you can read and write expressions like this:

^(-|+)?[0-9]*\.[0-9]+$

which matches things like -.2, 0.123, +0.1, etc. (floating point numbers, basically). This likely has bugs, since I haven't tested it ;)

You know what? I agree with you that regexes are not that hard, but the main problem from my POV is the lack of a standard. Each programming language implements those in a similar but always so slightly different syntax.

If we could come up with some sort of high-level API for regexes we might end up in a better situation, given that this would make them effectively language agnostic.

And you could always revert to your platform's native option if there are some missing features on the API (Given that those type of things generally end up with a least-common-denominator approach)

Not true. Regexes have been around for a long time and there are a few standards -- POSIX basic, POSIX extended, and PCRE. Any language/tool worth its salt these days will follow one of these three implementations (usually it's PCRE as it's the most complete). It's usually older tools like awk, vim, find, etc. that have their own quirks -- they were created before these standards existed and so they generally follow extended regex but then introduce their own syntax for certain concepts (in vim, for instance, \<...\> does the same thing as \b...\b in PCRE).

Didn't you just make the previous commenters point, that regex implementations vary? Within those three there are variations too with the flags and how they implement word boundaries (either exclusion of word characters or inclusion of certain spacing chars)

I think the case that there are too many implementations is overstated. Almost everyone can get away with knowing only PCRE these days. Most programmers I've know don't even know that they're using PCRE, they just know they're using "regexes"

I haven't had the incentive to learn regexes well enough to understand why I have to escape random things in different languages and environments. And occasionally something will fail to work all my cases and I have to fall back on a non-regex way of matching.

I've generally found that my practice is basically "try regex; if it doesn't work after ten minutes of effort, abandon regex completely".

For me, the biggest advantage of this sort of "library" approach is that instead of only being able to spedify regexes using literals you can treat them like other values in the language: you can name subregexes, you can create functions that combine or modify regexes, etc.

I'm not sure the "builder pattern" in the OP does this well though...

Thoughts on doing it better? The DSL is just a wrapper for the objects.

I may as well make one suggestion along these lines ( implemented in a similar JS library I whipped up not too long ago: https://github.com/wyantb/js-regex ).

Macros! https://github.com/wyantb/js-regex#macros

Given some named macro, whenever the user wants to use it, they use addMacro (or some analogous method).

Well, it's part of Ruby's philosophy: provide clean, readable and understandable code.

I'm a huge fan of regular expressions and I love to discover new ones, try to improve my own, etc. But, it does not mean there isn't place for a human readable regex syntax, and I will definitely give it a try.

totally unnecessary? The fact that solving problems with a regex is a common joke answers this question handily enough ("Some people, when confronted with a problem, think "I know, I'll use regular expressions." Now they have two problems.")

I only use regex occasionally and, as such, typically need to refresh each time I go back to it.

Obviously you should choose the level of abstraction that fits your needs but I would love something like this if I did any ruby.

It's just an alternate syntax, trading verbosity for readability. Debating whenever it's necessary or not is bikeshedding.

For one thing I agree it's totally unnecessary. I also think that people who're not familiar with regex should stay away from them as far as possible. Or at least get some understanding on how regex is implemented, learning about finite automata is very helpful to the understanding of regex.

> which matches things like -.2, 0.123, +0.1, etc. (floating point numbers, basically). This likely has bugs, since I haven't tested it ;)

The fact you can't come up with a regex and be sure it's correct answers your first two questions.

Regexes are incredibly useful, but the syntax is just a bunch of noise. This library turns it into self-docummenting code, which is incredibly valuable. Novices might prefer showing their regex-fu, but after you've spent a good chunk of your life reading and writing code, you value clear code more.

Under the hood, what's being generated is still a regex though. How can you be sure ANY code is correct, aside from testing it?

True, but your argument can apply to any sort of abstraction (e.g. using C instead of assembly or even assembly over 1's and 0's).

He's arguing that the nicer syntax is a valuable abstraction over regex's overly-complicated syntax, not that the syntax will be 100% the same as regex 100% of the time.

> How can you be sure ANY code is correct, aside from testing it?

Apparently, we can't. So, you really want to improve your odds of writing correct code at the first attempt by making the language work with you rather than against.

Replacing a bunch punctuation with code that reads like prose is one way.

> This library turns it into self-docummenting code, which is incredibly valuable.

The problem is that for anyone who knows how regexps work, this library provides an obfuscation layer. I personally prefer the free-spacing regexp support[1] found in a number of popular parsers. It lets you write a complicated regexp in small and/or indented chunks across multiple lines, including comments. This provides as much or more documentation than this "self-documenting" form and allows the full power of the regexp engine (which big regexps often seem to need). I've occasionally had to work out some pretty gnarly regexps, but when written in two-column code/comments form they end up being pretty understandable and maintainable.

[1] Example from the Ruby docs. IIRC, Python and a few other parsers have similar support: http://www.ruby-doc.org/core-1.9.3/Regexp.html#label-Free-Sp...

I was prepared to think this was totally unneccesary, but ended up thinking it's actually pretty nice.

> Am I the only person who thinks that things like this are totally unnecessary?

Not unnecessary, “useless”, “kidding” are more appropriate words IMHO.

Where is named captures? Oh, you cannot. Well, ordinary $1 $2 $3? Oh, you cannot capture at all! How do you use back refs? Oh, you cannot? Wait, why the heck then you even need regexps?

There are problems with complex regexes, yes. But this silly toy is not the solution. For solutions look at Perl6's “regexps”, Perl6's Grammars (example down the thread), Perl5's Regexp::Grammars, look how Haskell do it with parser combinators. They are all light years ahead.

Ouch, thanks for the harsh words.

But yes, 1.2 will likely have named captures and back refs. It's a regex, so $1 etc come with turning it into a regex and doing matching.

OK then. But look at state of the art solutions I mentioned ;-)

Perl5's R::Grammars is doing it the same way you do — it generates one-big-scary regex. But this lib is nowhere close to what it can do.

For idea: it will provide fully parsed struct like {proto: 'http', domain: { all: 'www.exampe.com', subdomains: ['www','example'], tld: 'ord', ... }

You can do this with regex captures.

It's not $1="this", $2="that", it's a tree.

So, no, you cannot. And there're many more cool things you cannot

Library owner here!

> Am I the only person who thinks that things like this are totally unnecessary?

No, it's been made clear to me that my library is totally unnecessary by quite a few developers. I like it though.

> ^(-|+)?[0-9]*\.[0-9]+$

exp.float, and you only have to define it once.

Not to mention that if you "learn" regular expressions with this library, you'll become an expert at this library and useless with "regular expressions", which is just one more example of things that can result in your inability to learn and use other languages effectively.

Regular expressions are a pain, but I strongly believe that you shouldn't use abstractions unless you have at least a functional understanding of what you're abstracting.

Am I the only person who thinks that things like this are totally unnecessary? Is learning/reading regular expressions really that difficult for most people?

No, you're not the only one. I have a feeling that things like that massive regular expression that is meant to match RFC822 email addresses is part of the reason for the focus on this kind of "easy to use, human" regular expression interfaces. The problem is that that email matching regular expression is an extreme case that matches a whole bunch of things that MTAs/MUAs need to know about, but that most people, who are accepting email addresses in web forms, don't need to know about, like "email comments", the real name fields, bang paths, etc. 99.999% of email addresses are going to match something simple like ^\S+@[\w-.]+\.[\w-]+$, and you're going to pass that off to the MTA to parse and validate and bounce anyway. For example, most people should be more concerned with having decent bounce processing support rather than trying to make email address matching regular expressions.

I really don't understand why some people are so afraid of regex. Sure, it's not perfect, but these symbol-less solutions feel to me like one step forward and two steps back. A more interesting development would be something along the lines of what Perl 6 is trying to do (if only someone could implement it).[1]

We should start treating regex like a computer language in its own right, rather than some second-hand citizen that we stuff into a single line without any delimiting whitespace. Use the /x switch and comment your code as you would with any other programming language, and you'll find that regex really isn't that scary.

Take the example pattern. The regex written by a human might look something like so:

    m{^
        https? ://        # Protocol
        (?: \w+ \. )?     # Subdomain
        ([\w\-]+) \.      # Domain
        (?: com | org )   # TLD
        /?
    $}x

It's certainly easier to parse. Debugging it is also a lot easier. We can see that this won't match anything with more than one subdomain. We can also see that it won't match subdomains that have hyphens in them. It also looks at a glance much more like a URL and less like some arbitrary Ruby code.

[1]: http://www.perl6.org/archive/doc/design/apo/A05.html

> something along the lines of what Perl 6 is trying to do (if only someone could implement it)

What, like, perl6?

    grammar URL {
	rule TOP { <proto> '://' <domain> '/' }

	token proto { 'http' 's'? }
	token subdom { \w+ }
	rule domain { <.subdom> [ '.' <.domain> ]? }
    }

    say URL.parse('http://lelf.lu/');



    Betty:hacks lelf$ perl6 gr.p6
    ｢http://lelf.lu/｣
     proto => ｢http｣
     domain => ｢lelf.lu｣

And for those interested a full RFC 3986 (URI) in Perl6 grammar can be found here - https://github.com/ihrd/uri

Here is direct link to grammar module - https://github.com/ihrd/uri/blob/master/lib/IETF/RFC_Grammar...

i have no trouble writing regexes, even quite complicated ones. I probably end up writing a really complicated one once a month or so; but basic ones weekly or daily.

I have no trouble writing them. What I have trouble doing, when it comes to very complicated regexes, is _reading_ what I've already written a couple months ago (myself! I pity those who aren't me even more). And even more so, when it comes to complicated regexen, _modifying_ what I've already written to fix a bug or change it slightly for new requirements.

A complicated regex meets the requirements for the old joke about 'write-only languages.' Somewhat ameliorated if you use the Perl-originated conventions for insignificant whitespace in a regex, which I _think_ ruby does too now, but I am not sure, cause I never use it, cause I can never remember how it works and never bother to look it up (have you tried looking up docs on ruby regexes on the web? good luck! I have no idea _where_ the official docs, if any English exist, are.)

I have no trouble writing regexes. I think every programmer should be comfortable doing so. I think using this tool without understanding regex basics first is a mistake. And I _still_ think this is a kinda neat and potentially useful tool.

Library owner here, just want to point out that

    m{^
        https? ://        # Protocol
        (?: \w+ \. )?     # Subdomain
        ([\w\-]+) \.      # Domain
        (?: com | org )   # TLD
        /?
    $}x

Isn't really any different from

    hex {starts
       protocol
       subdomain
       domain
       tld
       maybe("?")
    ends}

I think you mean 'maybe("/")'?

I did!

I tend to write such code without comments (using some pseudo-language):

  Protocol  = 'https?://'
  Subdomain = '\w+'
  Domain    = '[-\w]+'
  TLD       = 'com|org'

  Re = WholeTextMatches(
         Protocol + Optional( Subdomain + '.')
         + Capture(Domain) + TLD)

Typically, I wouldn't go that far, but storing separate parts into well-named strings before assembling the entire regex, IMO, makes things much more readable.

CoffeeScript's block regex (delimited by /// instead of /) does exactly this. It is a vast improvement over normal regular expressions, and I even prefer it to the syntax described in this post (which is unreasonably verbose).

This is interesting. I find that regexes are one of the few places where a comment explaining what a block of code does is generally necessary. Since the "ruby way" is to use aggressively replace comments with method and variable names, a tool like this is a good way to achieve that goal.

That said, this also makes a huge tradeoff against conciseness. Personally, I'd prefer

    # Is foo a valid klingon email address?
    foo =~ /gibberish/

Over

    foo = Hexpress.new.
        start('g').
        maybe('i').
        many('b').
        find('b').
        ...

In another rant about how my library isn't needed someone helpfully noted you can actually do comments with the %r{} syntax:

    %r{
      ^
      # foo
      bar
      $
    }

Please yes!

I did a little research and this derives from Verbal Expressions, as explained [1] and implemented [2]. In any case, I'm emphatically in favor. I'm the sort of programmer that needs to write maybe one regex per month, and I'm tired as hell of relearning the human-meaningless syntax for regexs, let alone all the little variations between languages. Not to mention, if it can vastly reduce the number of symbols I have to escape when matching special symbols, so much the better.

[1] http://thechangelog.com/stop-writing-regular-expressions-exp... [2] http://verbalexpressions.github.io/

I really need to link those in the readme, thanks! I also discovered on releasing this a really old ruby library that predates either of those and I'm adding that too.

Wow. My mind is blown.

It almost feels like what high level programming languages are to assembly code.

Sure, we can (and should) learn Regex constructs, but it is undeniable that this library provides an unmatched level of clarity.

Kudos to Krain for coming up with such an elegant solution to the issue of Regex building/reading.

Personally, I would not use this. One of the powers of regex is the similarity (depending on complexity and structure) between the regex pattern and what a matching string looks like. By doing it this way, a lot of noise is introduced that can make parsing more difficult.

I don't find:

    find { matching { [word, "-"] }.multiple }

Clearer than:

    ([\w\-]+)

It might be an alternative way introduce regexes though.

I would also like to see a more complex regex written this way.

In that example I wanted to show off a the matching and multiple syntax.

That's really too much praise, and I'm definitely not the first person to come up with this. See: Verbal Expressions and TextualRegex before that.

Have a look at Rebol parse dialect for an alternative mind blowing way of doing things!

Here's is OP example ported to parse:

  ; build some sub rules
  r: context [
    domain: charset [#"a" - #"z" #"-"]
    proto:  ["http" | "https"]
    tld:    ["com" | "org"]
  ]
  
  ; simple URI rule
  rule: [
    copy proto r/proto
    "://"
    some [copy d some r/domain #"." (append domain d)]
    copy tld r/tld
    ["/" | end]
  ]
  
  ; capture
  proto: tld: d: none
  domain: []
  
  ; usage example
  parse "http://www.example.com/" rule

  ; Above returns true with following variables set to...
  ;
  ; proto  = "http"
  ; domain = ["www" "example"]
  ; tld    = "com"

Here's a nice tutorial on parse (though it's probably Rebol 2 centric) - http://www.codeconscious.com/rebol/parse-tutorial.html

Regular expressions were very invented to avoid this. Because this works fine only as long your regular expressions are small and few.

Once that fact changes you will find yourself writing and staring at walls of text.

You will do this enough number of times, then only hope that you have a more succinct and powerful way of expressing such a idiom, you will do all that only to realize using regular expressions are the only way to solving a range problems which it was designed to solve.

The easiest analogy I can give you math. Prior to manipulation of symbols, math was pretty much text. The whole of math looked paragraphs of puzzles and word play. Worked fine when you want to do small things like postulates, axioms and a few things derived from that. To move a higher level of abstraction and interplay of concepts we had to get into symbols.

This is something similar.

Python's re.VERBOSE let you write [1]

    a = re.compile(r"""\d +  # the integral part
                       \.    # the decimal point
                       \d *  # some fractional digits""", re.X)

Instead of

    a = re.compile(r"\d+\.\d*")

But that is as far as it goes for me. From that point, masking REGEX with additional layer of 10's of functions is not a wise move.

[1] http://docs.python.org/2/library/re.html#re.VERBOSE

Ruby & Perl use /x to ignore whitespace & comments like this.

eg.

    /\d+ # some comment
     \w  # another comment
    /x

It's not meant for simple regex like that. It's meant for complex composed regular expressions.

Seems that most languages with built-in regexes lack any operation on them (like `/foo/ + /bar/` or `/foo/ | /bar/`). I'd blame the standard library, not praise a third-party one.

This completely illustrates what I hate most about ruby: the conflation of 'understandable code' with some sort of insane directive to turn everything into an English sentence without any regard for what the code actually does. At its worst it's almost an obsession with enforcing technical ignorance.

That said, regexes can be tricky to parse and making them clearer to the average person is a worthy goal.

I don't understand why so much hate for the library. For me, this is a very ruby way to tackle regular expression problem that newbie like me have.

Not that I'm saying that newbie shouldn't learn regex. But any level of abstraction would be great. Rails abstract the complexity of web development. Hexpress abstract the complexity of regex. Both are win in my book.

Like VerbalExpressions, this a good idea carried out without full acknowledgement of the nature of regular expressions. There's no shortcut - concatenation, union, and kleene star. A URL operator doesn't replace any of them.

If there isn't isomorphism between the traditional symbols and the new names, then the expressions will be limited in expressiveness.

Here the same[1]. This library squats vocabulary in a bad way. e.g.:

#word is (\w+). This works in english, but breaks very early. Still, it might be correct in some time. "word" is extremely context-sensitive.

[1] https://news.ycombinator.com/item?id=6319584

Good point, and more importantly I should be making use of the special Ruby matchers.

It seems like the argument against regexes is that they can get really complex and unreadable. For something as simple as that url, isn't it easier to just use a regex?

Seems like in the 'unreadable' regex case, you'll have a boatload of verbose function calls to construct it.

Library owner here. A huge draw for me to make this library was composability, not avoiding regex syntax.

I want to share patterns without having to do much.

So as computer scientists, we're in the business of tradeoffs right. I think the conciseness tradeoff for clarity is good. Abstracting away from pure regular expression syntax is a good thing, I think, because it offers better AT A GLANCE reading.

Why would that be better? I recently saw Bret Victor's Inventing on Principle talk on Vimeo, and one of the things he mentions that I really agree with is that most of us must 'think like computers' to understand what our code will do. The less we have to do this, the better, because we're terrible computers

Now am I likely to use something like this? No, mostly because I don't program in ruby TOO often, and I am pretty well aware of how to use regular expressions.

Here's a link to the VerbalExpressions organization that contains implementations of a similar DSL for various languages: https://github.com/verbalexpressions

Very cool. I like the standalone "start_of_line" more than the start("some string").

We actually match (and beat) the VerbalExpressions standard if you `require "hexpress/verbal_expression"`

Cute, but I prefer just using Rubular to test out regex's

Or instead of learning start maybe with words multiple has either ending you learn ^?\w+(|)$.

I'd imagine that wrappers such as this one do not enable users to create any more complex patterns, than ones they could make with a most basic understanding of regular expressions. Whilst providing neither a base for more complex regex use nor a community to further an understanding of regex.

Considering the composability I've designed here I'm not sure how you think it wont "enable users to create more complex patterns"?

Now we need a human way to write Ruby in Python.

The reason that this library is useful is because I don't have to maintain an AST in my own feeble mind about what each special character in the regex means. I can look at the code, at first glance, and immediately know what the hell is going on.

Also, this.

https://xkcd.com/208/

Library owner here, I've added some replies to points made but I just want to note that the readme on Github's website was a bit out of date and provides a little more details now.

Seems easier to just use a regex. Every programmer should know them already, so this is just another API to remember. Still kinda cool though, but prob not so useful.

Reply on news.ycombinator.com