Pomsky – A portable, modern regular expression language
pomsky-lang.org
pomsky-lang.org
https://news.ycombinator.com/item?id=31690878 (242 points | 6 months ago | 189 comments)
Well, maybe. What I REALLY like about this one, is that it fully reverses the quoting/escaping assumptions. The default assumption of old regexes that symbols should by default match themselves, I think is regexes' million dollar mistake. The literal matches are the least interesting part of regexes. If you reach for regexes, it's because you want something more complex than literal matches, and the syntax should be about taming that complexity. Even the Unix world sort of conceded that, in reversing the quoting assumption for ?, +, (), [] and | in egrep. The mistake was in stopping there.
I will take a good look at this. I hope they provide good justifications for their choices.
Also, learning regex syntax has much better ROI and is a much more transferable skill, than learning a specific parser library’s syntax and bugs.
Just invest 30 minutes once in your life into learning basic regex, for Christ’s sake, it’s not that hard!
Of course you can't always use a 3rd-party library, but IMO it should be your default choice without additional constraints. Much like how a memory-safe garbage-collected language should be your default choice for most things.
It's a quick toggle. Or you can just wrap your literal in quotes.
Still, regex remains a great choice for one-off interactive purposes because it's concise, but for an application something with better composability and a clearer delineation between operators and literals is beneficial. You don't replace all uses of regex with Pomsky, you replace it specifically in "apps" where the pattern is part of the code.
Example: https://github.com/VerbalExpressions/JSVerbalExpressions#tes...
It seems to only exist in log parsing ecosystems but this really helps with getting rid of little bugs and wrong parsing of specific regex patterns.
Instead of doing "^\d+(\.\d+){3}$" for IP checking which is clearly wrong, you'd do "%{IPV4:ip}" which is so much better.
List of known patterns : https://github.com/hpcugent/logstash-patterns/blob/master/fi...
Even for PHP a third party library only has 15 stars.
It's not a bad idea, though you'd want to make it a more formal standard; since it's just used internally in one project, it may be subject to change based on that project's own needs. There could also be more documentation, showing examples of strings that are accepted and not accepted by each pattern, as well as advice for generating compliant strings.
It's for parsing rather than search though, you'd still need regex if e.g. searching unstructured logs for IPs.
e.g. (powershell syntax for easy reproduction) returns parsed, or throw if invalid. [uri]"\\ServerName1\Folder\UserName" [version]'1.2.3.4' [ipaddress]'1.1.1.1'
if (int.TryParse(userInput, out var number))
{
total += number;
}
[1] https://learn.microsoft.com/en-us/dotnet/api/system.net.ipad...
If I came across this in code, I still have similar issues to regex. I have to question the source and what unit testing exists, that the tests cover the particular cases we care about, and if composed it needs tests anyway.
For context, I always split my regexs to a function and add unit tests for them, and ask the same in PRs. It is one of the easiest tests to write and most beneficial: if I don't discover a case I missed while I come up with example inputs, it protects any future modifications by future devs. It also empowers devs that don't know regexs well to be able to modify them (because they know it won't break) and to approve a PR with one.
I don't see how grok patterns avoid the need for tests (especially if composed), and if there are tests anyway, they're just a level of indirection.
In the worst case, when there's inevitably a bug (or app-specific missing situation) in one of these, it's harder to fix than just having a regex directly.
[1] https://github.com/mmkjony/awesome-regex-1 , https://regexlib.com/DisplayPatterns.aspx?cattabindex=5&cate...
In your case, putting the grok patterns in your regex specific function seems to make it more readable.
> A specialized regular expression object that matches natural language text for predefined data patterns.
It's targeted at the kind of information end users are interested in: dates, email addresses, like that.
[0]:https://developer.apple.com/documentation/foundation/nsdatad...
A common example is email address validation. Trying to validate that an email address is well-formed with regex will often miss valid addresses (the example you linked would miss some in the HOSTNAME portion alone). It would be better to simply send a confirmation link to the address and wait for it to be clicked since that would ensure that the address is valid
I didn't think some milliseconds is worth more than correctness.
Grok pattern is like commenting on your regexp, except you'll also get logical validation instead of bunch of characters you may make a mistake one day.
Further, what happens if the grok pattern is wrong? Or if it changes?
'Hello' ' '
Did you count the single quotes correctly? Even with syntax highlighting, people WILL mess this up.
In early development, I took a number of cues from existing work that turned out to be bad ideas, in particular using whitespace for concatenation (which all BNF dialects seem to do).
Switching to '&' for concatenation (reading it as "x and then y") made things a lot clearer, as it would also do for Pomsky:
'Hello' & ' '+ & ('world' | 'pomsky')Also, as a binary format enthusiast (I built a binary "Codable" implementation for Swift), KBNF looks great! I'll be following your progress.
https://github.com/jkrumbiegel/ReadableRegex.jl
It is in Julia, but if you have it installed locally it’s just a few taps away. You can even generate the regex, and use that in Python and just add the ReadableRegex in a comment nearby.
If there's any interest in other takes on "better regexps", the Rosie Pattern Language has some neat ideas:
[0] https://rosie-lang.org/index.html
[1] https://www.youtube.com/watch?v=MkTiYDrb0zg&list=PLcGKfGEEON...
That said, after looking at this syntax, I'm not sure that this is much of an improvement. Maybe I've spent far too much time in Regex land [1], but I know I'd perform much slower in this. It's not particularly beautiful, either. The verbosity doesn't seem clearer.
Variables and comments are great, though. We need to add them in future regexes.
Overall, good idea. I'd like to see more takes on this.
Every text editor, shell environment, programming language or desktop application seems to use regular expressions with a syntax which is slightly different to all the others, but not different enough to call it with a different name.
This means that a newbie learning regular expressions will be thrown into an environment where it can learn the basic principles, but the rules it learns are not generalisable to all applications (e.g. do I match an arbitrary string with '*', or '.*' ? Can I reuse matched patterns with (1) or with {1}? Etc.)
A new readable and easy-to-learn syntax that is nevertheless portable may work like markdown, as a simple-yet-universal new way to apply regexp that newcomers may learn with confidence and apply everywhere, replacing all the previous slightly incompatible versions.
And although I don't know for sure, reusing a matched pattern within the regex string using (1) seems like it would be very strange, because () are metacharacters for grouping.
That's what I meant about every system using a different syntax for what's basically the same functionality.
Interesting idea but the syntax adds nothing to the readability of regex.
I’d be more impressed with this if it targeted multiple different regex targets, since not all implementations of regex are equal. But in its current state it has all the problems of regex plus now the problems of this new language on top.
Looks a fun personal project though. Hope the developers enjoyed building it.
pcre has always had java and scala support.
Would you include Ruby in your comparison given that it inherits its regex implementation largely from Perl despite Onigurama's slight differences?
https://scsh.net/docu/html/man-Z-H-7.html
https://www.ccs.neu.edu/home/shivers/papers/sre.txt
It works nicely with Scheme, including for programmatic generation.
(I've done some regular expressions from heck, without the benefit of this, and needed extensive commenting just to keep a few/several levels of nested groupings straight. With S-expressions, that's trivial.)
Of course this led to a certain amount of UI whack-a-mole: people learn a mini-language really fast when they use it all day at their jobs, and people are creative; whereas the "computer people" really needed the system to not grind to a halt because of a massively inefficient regex.
From this I learned to see regexes everywhere when I look at text; and that we should always consider the "threat model" of our own users being good at their jobs.
If I know RegEx, why would I use pomsky?
Insignificant whitespace is needed to support it, and as an added bonus it would make it easier to break up patterns across multiple lines.
The syntax changes and added verbosity do not seem great, though. They'd trip me up for sure.
In general, I think I'd like to see a language more like "Regex 2.0", ie. an extension that doesn't depart too far from what we're used to.
+1 to the syntax
Way back somewhere in the 2010s when I was still keeping an eye on Perl 6, I was kind of hoping that all these improvements would make their way into some kind of PCRE v3 that all other tools that already use PCRE would switch to. Would have been nice.
The greatest problem with regexp is not its complexity nor learning curve: it's its apparent complexity & learning curve. That tends to put learners off, but anyone I know who's persisted has found the journey faster than expected. After they're over it, everyone I've talked to thinks pretty highly of the syntax: it's simple (small set of compostable components) & expressive, especially if using labelled groups.
The second greatest problem with Regexp is learning query optimisation: that's definitively complex but this new syntax doesn't even attempt solving that.
This might do better there.
Hello *waves*
The only thing regexp has on the plus side is that it's compact. I'd prefer something that's more readable without knowledge. I know how to use it but I always have to google it. I only use it once per year and so there's no big win in mesmerizing it.
. * | + ? ( )
Character classes can often be given as ranges:
[a-z] [A-Z] [0-9] ^ $
Lazy matching finds the shortest matching string instead of the longest one. This is given by the symbols:
*? +?
> I only use it once per year
If you use a text editor, any kind of repetitive find+replace activity can be sped up using regexes. Seems strange you only use it once per year.
Some people tell you that you can't parse XML using regexes. THEY'RE WR- right, but you can often do decent one-off jobs on an XML document using regular expressions, as long as you're aware that it's a one-off job that won't work on any other document.
^(?=.\d)(?=.[a-z])(?=.[A-Z])(?=.[!@#$^&()_-]).{8,18}$
That is hard to quickly* read and understand what it is. Sure a comment would help, but if someone said it isn't working, I have to work all that out and then fix it.
There is a reason Perl lost to Python. It just seems like we can do better.
I've certainly seen regexes that remind me of compost heaps.
(I agree that regex syntax is fine. I just liked the typo.)
You can of course disqualify me in saying I can't possibly have persisted at learning it, if I don't like it. But I am inevitably "the guy who knows regular expressions" in the places I've worked. I do use them, for stuff like fixing tedious one-off data munging jobs without having to write an actual program to do it. But I
* Do it step by step, to avoid writing anything resembling a complex regex, because I've yet to meet anyone who can write a complex regexp without bugs.
* Use "undo" a lot in that process.
Query optimisation is totally unnecessary if you use Rust's or Go's implementation.
I'm a big believer that "premature optimisation" is a bad thing but making a blanket statement that any given implementation is 100% performant for all use-cases is going a bit far.
Say that the length of your string is n, and the length of your regular expression is m. Isn't Regular Expression matching in those languages guaranteed O(m n) time? In plain English, that's only linear time - which you practically can't improve on - even if you tried! It can only do worse if your implementation uses backtracking instead of Finite Automata - which was very common at one time, but is now considered The Wrong Way: https://swtch.com/~rsc/regexp/regexp1.html
There is actually a way to optimise your queries, which is by precompiling them to DFAs, but I don't think that's what you're doing.
> There is actually a way to optimise your queries, which is by precompiling them to DFAs, but I don't think that's what you're doing.
You can do that, but the primary advantage of DFAs in practice is backtracking avoidance, so simply reduction of backtracking in manually constructed queries is more what I was referring to. The missing part of the puzzle here being that not supporting backtracking at all doesn't limit capabilities, so removal rather than reduction is the go/rust approach.
Complex password validation: https://gist.github.com/pmarreck/4c5f1076498da1a86062
Email header parsing: https://gist.github.com/pmarreck/8476538
An attempt at a JSON validator (yeah, I know): https://gist.github.com/pmarreck/2775709
Remember that commenting is not just for others, but also for "future you"!
It is for the next person that maintains the code.
I do admit it can be a write once hopefully never need to read again thing, but I still love it.
The basic idea of rx is to use S-expressions to describe regular expressions, and my elevator pitch would've been to embed rx invocations in shell scripts using $(syntax), the main use case being something like sed invocations.
I still think it's a neat idea, and complex regular expressions tend to be hard to parse for humans.
Not having to navigate to a different page would be a boon for the playground. Very interesting though, I use Regular Expressions for the times when you need to extract information but find and replace functions/methods just aren't enough. Mostly for my scrapers.
[:0-255:] could be an option for example to write a number range.
Also regex / variable interpolation should be added to regexes in languages probably :)
> Also regex / variable interpolation should be added to regexes in languages probably
I've often wondered and desired this myself. It feels like the single most useful change one might make, because it enables (albeit very basic) logical decomposition. It comes in handy when a regex works for your use case, but the regex is somewhat long but with many repeated parts. In my experience, that isn't too rare, and the regex readability would likely be helped quite a bit by some kind of decomposition.
let day = "[0-9]{2}"
let month = "jan|feb|…|dec"
let year = "[0-9]{4}"
let regex = Regex(month + " " + day + ", " + year)
The string concatenations also make things a bit slower, but that almost always needs to be done only once.
Advantage is that this makes testing easier. For example, you can do
let dayRegex = Regex(day)
If you decide to have your regex to treat 30-day and 31 day months separately, things get more complicated. I wouldn’t go that far, though, and certainly not as far as handling leap years correctly. When regexes get complex, it’s time to go for a proper lexer/parser.
Still doesn't change what I said.
My original comment was talking about adding variables to the regex language itself. String interpolation and combinator APIs and enums and type safety are all beside the point. They surely solve a similar problem as regex variables in certain contexts, but they go about it in a very different way and are specific to the context in which the regex is created.
https://github.com/search?l=Rust&p=5&q=%22regex%3A%3Anew%22&...
For the importance of numeric range it was quite easy to find actually a good example:
x0rz3q/advent-of-code-2020
Regex::new(r"^19[2-8][0-9]|199[0-9]|200[0-2]$").unwrap(), ); comparitor.insert("iyr", Regex::new(r"^(201[0-9]|2020)$").unwrap());
There are lots of number parsing.
I would enable both [[:1-12:]] and [[:01-12:]] as options without / with leading zeros.
About the variables:
This file would look much more readable with variables that are reusing other regexes:
https://github.com/spcan/common-regex/blob/3238bc8ee85e0e000...
Feel free to add other projects, including the ones in this thread
The embedded string is often precluded from static analysis. Fieldnames or names of capture groups may drift apart from the outside code and maybe nobody notices.
What we need are more embedded DSLs where we get static syntax and type checks and hopefully a good integration between the surrounding code and the embedded code.
While there is nothing stating programming languages need to be Turing-Complete (and there are in fact examples of non-TC programming languages, if only educational), I think calling regular expressions* a "programming language" may be a bit of a stretch.
* Even when its capacities are as extended outside the mathematical definition as modern RegEx is.
edit: specified I meant "non-TC programming languages"
Sure, SQL is only educational or not programming.
There seems to be a lot of programming going on in the world generating a lot of value which you wouldn't recognize as such. Excel uses a terminating (and therefore not computationally complete) dataflow-based evaluation model but it's in no case programming. Are you sure?
I'd think, telling a computer "if you recognize a pattern described as such, replace it with so and so", just regex with replacement, sounds very much like programming to me. Very weird where you draw the line.
> There seems to be a lot of programming going on in the world generating a lot of value which you wouldn't recognize as such. Excel uses a terminating (and therefore not computationally complete) dataflow-based evaluation model but it's in no case programming. Are you sure?
I think this is not the most charitable reading of your parent. You're taking it as if they definitively rejected SQL and Excel as programming languages and arguing from that, rather than that maybe they didn't think of them. I think it would have been easy to make the case for SQL, Excel, and even (ir)regexes as programming languages without taking a sarcastic tone. For example:
> I'd think, telling a computer "if you recognize a pattern described as such, replace it with so and so", just regex with replacement, sounds very much like programming to me. Very weird where you draw the line.
could easily have asked instead, where do you draw the line?—which would surely be a more interesting discussion.
The commenter made that quite clear. Using a computational model that's not complete can not be called "programming" in any real-world context.
Which is weird. This kinda implies that, whenever I forbid myself to write my Python-Code without While-loops or recursion, I'm not programming.
The traditional definitions of programming are often along the lines of instructing a machine to perform certain tasks.
Last time I checked SQL was proven to be Turing-Complete?
While Excel's formula language wasn't always TC, the program as a whole, I believe, was since the very beginning.
>telling a computer "if you recognize a pattern described as such, replace it with so and so", just regex with replacement, sounds very much like programming to me.
I think by adding "if X then replace" capability we already escaped outside the bounds of Type-3 grammar, i.e. it's not a regular expression anymore (despite being build upon one).
Without recursive CTEs, it isn't. And there's nothing much to prove about that. Finite SQL statements on finite databases always terminate and it's therefore not computationally complete.
A comment further up offered a very promising alternative.
https://github.com/VerbalExpressions/JSVerbalExpressions#tes...
It's a bit verbose, but I don't care anymore, I am too much a veteran to care about my code being sleek, I want it readable and workable.
The first type of thinking is generally not composed in code. It happens staring out the window, or looking at data tables or lists and reading documentation, drawing out the architecture. The second type happens in code composition: programmers know what snippet will accomplish which outcome, and it just comes out when we know it's needed. Boom, we have the thing we imagined during the architecture phase. Maybe we look something up or slow down while writing code, but the composition is fairly reactive/spontaneous. It flows.
So coding goes in this rhythm... thinking openly and patiently and deliberately, followed by a burst of code composition, and then more open/deliberate thought while staring out a window. Again and again. Maybe we reverse some of the quick/spontaneous thinking to rearchitect according to our slow/deliberate thought.
Enter regular expressions. I have never, ever spontaneously composed a regex. I have to be very slow and deliberate. I have to recall the precise definition of every character in the regex as I read or write it. Sometimes I stare at a regex for 2 minutes, look up some operator, stare again for 2 minutes, refactor it. Finally I understand it and can now apply the regex spontaneously to whatever environment I'm working in. It always feels like first principles architecture, never like flow. This is odd, as regex is like a giant shorthand engine.
Maybe I have not yet formed the neurological connections required for it to flow – but more likely this is a function of the language. There must be a syntax that better supports flow-state composition. (I don't think Pomsky accomplishes it.)
That syntax was devised as a math notation, where sub-patterns were abstracted as one letter variables. You didn't have to deal with complex patterns at all, and it was very readable for that use case.
It was then re-purposed as a write-only language for searching at the CLI (in `ed` and its descendants, then grep).
They gradually graduated into what they are today in general programming languages because they were familiar and worse is better... However, composition got lost in the process.
Swift takes a radically different approach, and gives RegExps a parser combinator syntax. So every sub-expression can be assigned to a variable and reused/tested independently.
Here's an example (sorry I don't have it at hand in text form): https://pbs.twimg.com/media/FipgjUdUcAAl1xS?format=jpg&name=...
I wrote a lib that does the same in JS (the lib predates the Swift syntax by years). Here's the same example ported to JS: https://flems.io/#0=N4IgtglgJlA2CmIBcBWAzAOgAwCYA0IAzgMYBOA9...
Here's the lib: https://github.com/compose-regexp/compose-regexp.js
This hasn't been the case because I had to work in other languages where usage of regexes was more idiomatic/commonly understood. Parser combinators are great but most of the time, the problem can easily be solved using a regex and we can't really justify adding yet another library in the dependencies for the rare case it would be better suited.
In addition to all the benefits they can provide in terms or readability, re-usability and abstraction, parser combinators can even compete with regexes in terms of performance. https://pl-rants.net/posts/regexes-and-combinators/
I can do simple regexes at pretty close to the speed of thought. e.g. here's one to recognize the "arthurofbabylon 20 minutes ago" bit at the top of HN comments:
(-?\d+ points? by )?[a-zA-Z0-9_-]* ((just now)|(\d+ (minutes|hours|days|months) ago)|(on (Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec) \d{1,2}, \d{4}))
Not saying it's completely correct (probably someone will reply with a counterexample or problem with it) but I think it's a demonstration that regex fluency is possible. I do this kind of thing on the command line a lot, interactively to figure stuff out from logs and so on. For production code I'm more circumspect.
Trouble comes when one need to update or debug a complex RegExp. The one you posted is not that complex.
The equivalent code to parse this is much longer and arguably to me not really easier to grok. (Unless of course you regularly avoid regex, in which case it will always remain hard.)
I'd say the hardest thing for beginners is the character class and how it matches EXACTLY ONE LETTER without a modifier. Everything matches exactly one letter without a modifier. It seems intuitive to me, but obviously a lot of folks have trouble with it, so I accept that we are outliers.
Reminds me a bit of folks who steadfastly avoid SQL. You have to change the flow in your brain; there's more than one flavor of flow. For regex, it's the watching for single character match with optional modifier. For SQL, it's thinking in terms of sets, not instructions.
If regexes make you feel stupid, lean into it instead of recoiling from it. That's where the learning happens: immersion into the uncertainty until it becomes familiar.
I'm one of those people, SQL just doesn't fit in my brain right :(
Just find the right strategy for you to internalize the patterns. Maybe a game?
Are we transpiling to regex?
1. - Xerox xfst
2. - Xerox lexc
3. - Xerox twolc
4. - Mans Hulden's FOMA
Repo: https://fomafst.github.io/
Paper: https://dingo.sbs.arizona.edu/~mhulden/hulden_foma_2009.pdf
Demo: https://dsacl3-2018.github.io/xfst-demo/
Tutorial: https://foma.sourceforge.net/lrec2010/
5. Helsinki HfstXfst:
Homepage: https://github.com/hfst/hfst/wiki/HfstXfst
These tools go back to research by Lauri Karttunen and others at Xerox Research Center Europe in Grenoble, where an attempt was made to create highly efficient compilers and runtime libraries for finite-state transducers, i.e. to move beyond regular expressions to regular RELATIONS. This not only permits to formalize replacements (regular expressions with an "output tape"), but also creates reversible automata (input and output roles can be swapped) and leads to a domain specific language that describes transducers in very readable ways, including sub-automata naming, so that it can be useful for formal specification or linguistic rules (phonology, morphology, i.e. word or sound grammar). The latter two projects are open source clones of the former effort. Once you have used these for a week, you will never want to get back to ugly "ordinary" regexes again.
Books:
(a) https://www.amazon.co.uk/Finite-State-Processing-Synthesis-L...
(b) https://www.amazon.co.uk/Recognition-Algorithms-Finite-State...
(c) https://www.amazon.co.uk/Finite-State-Techniques-Transducers...
(d) https://www.amazon.co.uk/Finite-state-Language-Processing-Sp...
(e) https://www.amazon.co.uk/Finite-State-Morphology-CSLI-Comput...
‘Hello’ (‘World’ | ‘Pomsky’ | NULL)
In order to avoid ?: stuff.
> If you know RegExp's, the syntax will immediately make sense
Doesn't seem like a feature.
Im not so sure this solves the actual issue ;)!!!! Jokes aside, regex has its uses.
You're better off using an actual parser (which will probably implement a state machine), writing a state machine yourself, or being overly accepting of invalid email addresses and just relying on attempting to deliver an email to the address.
See http://cubicspot.blogspot.com/2012/06/correct-way-to-validat... for more discussion.