Some useful regular expressions for programmers
lemire.me
lemire.me
When I write code for personal projects I don't even bother trying to format it correctly. The only whitespace button I use is the spacebar. tab is mapped to the function that reformats the entire document according to the formatting rules I've configured.
At work, whenever I apply a change in a function I will autoformat that function whenever. I can't just format the entire file, but formatting errors are rife in the codebase I work on.
goto fail; is real.
gg=G in vim is fast enough to type for me, though.
One thing I just can't live without, however, is a code formatter. Formatting code by hand is tedious and repetitive and I have no patience for it.
So i can just write, save the file and everything is nice :)
Huh. you know...
I don't want to be too cynical about the article, since these are absolutely all good things to know due to their myriad uses in other circumstances. But if your intention is to apply these to code you're writing, as the article describes, I don't think any of them are practical if you're already using an IDE/IDE-lite. (It probably makes a lot more sense if the author is using a lighter editor like vim without a lot of plugins, though.)
† “Characters” is a deliberately vague term here. Different regular expression engines match different types of things, typically supporting one or two modes. The three most common things to match are bytes (the traditional default, normally encoding-unaware but potentially known to be ASCII or UTF-8), UTF-16 code units (JavaScript’s default, and UTF-16 was an all-round terrible idea that ruined Unicode for everyone else while achieving nothing useful), and Unicode scalar values (which is typically called “Unicode mode” and often enabled by a /u flag).
Depends of the language. In Python 3, files are expected to be utf8 by default, and you can change that by adding a "# coding: <charset>" header.
In fact, it's one of the reasons it was a breaking release in the first place, and being able to put non-ASCII characters in strings and comments in my source code are a huge plus.
I mentioning this because in another article, the author said being used to code in "C++, C, Go, Java, JavaScript, Python, R, Swift, Rust, C#" so I find it kinda weird.
In shell scripts especially (and sometimes elsewhere), if I want an ANSI escape code, I’ll type the actual bytes I want, because it’s more reliable and easier to type <C-V><Esc> and see ^[ in Vim than it is to worry about whether I’m using \u001b, \u{1b}, \x1b, \033 or something else, and whether I have '', "", $'', $"", printf, echo -e, &c. &c. Speaking generally, if you use the actual control characters, the code will either fail to run (parse or compile error) or work, whereas encoding your control codes with escape sequences is more hit and miss. I’ve seen quite a few others using control characters in this way too.
Truth to tell, I’m quite liberal in using all of Unicode in my source code. About the only places I wouldn’t use the actual characters are when (a) the environment requires me to escape the thing; (b) the numeric values of the characters are more meaningful than the characaters themselves; and (c) when dealing with combining characters and modifiers, because otherwise they’ll do things like combine with the opening ' from the string/character literal.
The code will work, but how will it edit outside of your personal environment? Or print?
If you quote it in a PDF paper, will it be copy-pastable?
Basically I would worry about all that sort of stuff.
If I have echo '^L' to clear the screen, and that program is sent to a line printer, will I get:
echo '
[ paper ejected ]
'
It's interesting that many languages avoid unicode and non-ASCII text, yet they make assumptions about file and directory structures about the underlying system. It's as if interpreting directory and file system structures is "okay", but interpreting file formats is not.
> In fact, it's one of the reasons it was a breaking release in the first place, and being able to put non-ASCII characters in strings and comments in my source code are a huge plus.
Sorry, but as a Python dev that went from 2 to 3, yes native unicode features are nice, but no, it was not worth breaking two decades of existing code.
Up to 2015, my laptop was plagued with Python software (and others) crashing because they didn't handle unicode properly. Reading from my "Vidéos" directory, entering my first name, dealing with the "février" month in date parsing...
For the USA, things just work most of the time. For us, it's a nightmare. It's even worse because most software is produced by English speaking people that have your attitude, and are completely oblivious about the crashing bugs they introduce on a daily basis.
In Asia it's even more terrible.
And I've heard the people saying you can perfectly code unicode aware software in Python 2. Yes, you can, just like you can code memory safe code in C.
In fact, just teaching Python 2 is painful in Europe. The student write their name in a comment ? Crashes if you forget the encoding header. Using raw_input() with unicode to ask a question? Crashes. Reading bytes from an image file ? Get back a string object. Got garbage bytes in string object ? Silently concatenate with a valid string and produce garbage.
I live in Europe and I (mostly) agree that (most) code shouldn't (usually) contain any codepoint greater than 127. It's a simple matter of reducing the surface area of possible problems. Code needs to work across machines and platforms, and it's basically guaranteed that someone, somewhere is going to screw up the encoding. I know it shouldn't happen, but it will happen, and ASCII mostly works around that problem. Another issue is readability. I know ASCII by heart, but I can't memorize Unicode. If identifiers were allowed to contain random characters, it would make my job harder for basically no reason. Furthermore, the entire ASCII charset (or at least the subset of ASCII that's relevant for source code) can easily be typed from a keyboard. Almost everyone I know uses the US layout for programming because European ones frankly just suck, and that means typing accents, diacritics, emoji or what have you is harder, for no real discernible reason.
String handling is a different issue, and I agree that native functions for a modern language should be able to handle utf8 without crashing. The above only applies to the literal content of source file.
Otherwise, chinese coders can't put their name in comments? Or do they need to invent an ascii names for themself, like we forced them to do in the past for immigration ?
And no hard coded value for any string in scripts then? I mean, names, cities, user messages and label, all in in config files even for the smallest quick and dirty script because you can't type "Joyeux Noël" in a string in your file? Can't hardcode my "Téléchargements" folder when in doing some batch work in my shell?
Do I have write all my monetary symbols with escape sequences? Having a table of constant filled with EURO_SIGN=b'\xe2\x82\xac' instead of using "€"? Same for emoji?
I disagree, and I'm certainly glad that not only Python3 allows this, but do it in a way that is actually stable and safe to do. I rarely have encoding issues nowadays, and when I do, it's always because something outside is doing something fishy.
Utf8 in my code file, yes please.
Damn, a perfect example of why it's nice to be able to just do "€" in my file. It's also less bugs.
EURO_SIGN = '\N{EURO SIGN}'
While I don't believe the \N is backwards compatible to Python 2, explicitly declaring the file encoding is, and you can have a UTF-8 encoded source file w/ Python 2 in that way.
I'd also note the box drawing characters are extremely useful for diagrams in comments.
(And… it's 2021. If someone's tooling can't handle UTF-8, it's time for them to get with the program, not for us to waste time catering to them. There's enough real work to be done as it is…)
Scientists writing Julia are laughing on the floor.
It's not COBOL that won. It's qwerty.
For example, to type α², you just type \alpha, hit the tab button to turn it into α, and then type \^2 and hit tab again to turn that into ².
If someone gives you a unicode character that you don't know how to type, you just hit the ? button to enter the repl-help mode and then paste the character in.
help?> α²
"α²" can be typed by \alpha<tab>\^2<tab>
search:
Couldn't find α²
No documentation found.
Binding α² does not exist.It reminds me of GNU TeXMacs, which has a similar interface for equations. That is, typing "\int" followed by tab will render an integral sign.
Works 99.99% of the time, is unreadable/unmaintainable, makes you understand that the only true way to verify an e-mail is to send an e-mail to that address.
[1] - https://emailregex.com/
PS: Gotta love the Perl / Ruby version
this-is-not-a-valid-address@
@this-is-not-valid-either
@/^[^@]+@[^@]+$/
?
“\@“@example.com
The worst regex would be:
/^[a-zA-Z0-9\-_]+@[a-zA-Z0-9\-_]+\.[a-zA-Z0-9\-_]+$/
Many websites use such a regex :(
/^[a-zA-Z0-9.!#$%&'*+\/=?^_`{|}~-]+@[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?(?:\.[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?)*$/
1) \S can be used instead of [^\s]. I haven't come across a regex flavor that supports \s but not \S. GNU grep supports \s and \S but doesn't support using them inside character classes.
2) For the first example, it is a weird mix of formatting for the code snippets. Rest of the examples are properly code formatted and specified without single quotes around the expression.
3) The fourth example uses variable length lookbehind, which isn't supported by Perl, Python (unless you use regex module), Ruby, etc. \K can be used instead if supported. Also, `<` at the start of the regex is a typo
It is supported: https://perldoc.perl.org/perl5300delta#Core-Enhancements
Work-around for all previous versions: http://www.drregex.com/2019/02/variable-length-lookbehinds-a...
$ echo 'asb34 asgfdhgree56 jdf42' | perl -lne 'print join "\n", /(?<=\ba[a-z]*)\d+/g'
Lookbehind longer than 255 not implemented in regex m/(?<=\ba[a-z]*)\d+/ at -e line 1.1. Those expressions fall down on a number of edge cases (eg heredocs / string literals)
2. It is actually perfectly reasonable to have non-ASCII inside source code because these days it's increasingly common for compilers to assume source it UTF8 and, frankly, English isn't the first language for most of the speaking world so it's rude of us to assume all comments and quoted strings should be in English.
3. There's already multiple code formatters for nearly every language out there and they take tokenised versions of your source (ie will take into account stuff like the string literals I mentioned earlier). You're far better off picking whatever code formatter is considered (de facto) standard for your language and using that. They come with the additional benefit that they don't just highlight formatting faux pas but they can also enforce style guidelines.
^( {4})\* {1,3}\S
<[^>]+>The former is obviously painful because poorly formatted code is a constant mental burden. But it’s also true that unrelated formatting changes can present a similar mental burden and make meaningful changes harder to identify.
The latter can detract from more meaningful review. But prioritizing better formatting consistently keeps a codebase more manageable over time.
This is somewhere tooling can help, more than it already does. Automated linting/formatting is great. Depending when it’s applied, and how effective its automation, it can potentially defer the burden til review time. But there’s still a lot of room for improvement there.
Many diff tools present an “ignore whitespace” option. I would like to see this taken two steps further:
1. Better language syntax awareness could go beyond whitespace as formatting-specific “noise”. I know there’s work in this direction and it’s great too. Besides making it more available and complete...
2. “Ignore” is probably not the best option. Ideally tooling can also identify these two sets of changes so they can be both viewed, and more importantly merged separately.
The problem right now is that formatting is noise pretty much everywhere one might interact with a project. Making it a distinct set of changes at the diff and history level would help significantly.
Highlighting trailing spaces are an option in git diffs too, though the devs who submit them typically don't look at their own diffs after they've pushed them (at least in my experience)
I do think formatters work great (much better than format linters) but only if everyone does them on landing from a shared conf. Otherwise you end up playing format tennis and a lot of expression-normalization noise if it's working from AST.
I think the point where it's rewriting braces, adding parens, etc, is the point where I'd typically want to see the reformat split out for review. I haven't seen a code review tool in a long time that didn't let you hide whitespace-only changes, though, so not sure I think pushback would be justified there if the files were being touched anyway.
One thing we're running into at my current company is that there are pre-commit auto-formatters that are employed that will take care of this for us. It's really convenient. The only drawback is that this process was introduced many years after the codebase was created, so there are still occasional lingering files that will pull in formatting changes once someone changes even a single character. The upside is that this isn't very frequent at this point, and we're kind of expecting them to happen on occasion.
A couple of devs have also been planning on running the entire codebase through the formatter all in one go, which will happen during a lull. I've seen another company do this once, and while the result is nice, it always ends up with someone getting frustrated with having to iron out conflicts in some branch they've been working on. (And of course, another drawback is that it complicates the blame process, but that might not be a huge problem depending on the codebase.)
VerbalExpression regex = regex()
.find("a")
.capture().find("b").anything().endCapture().then("cd").build();
/a(b.*)cd/
It’s way more verbose, but not, in any way I can tell, lower cognitive load.
"(?:[^\\"]|\\.)*"- - -
This is my fourth attempt at understanding the intent of this pattern. I’m quite comfortable with regex, so much so I listed it on my resume skills list for years as sort of a glib joke. I think I’ve been overthinking this one. For my benefit I’m gonna describe what I think this matches and ask you to confirm. For others I hope the description helps.
This appears to be intended to match a balanced double quote, where any characters between may be anything besides a backslash or double quote, or any character preceded by a backslash. The content between the bounding quotes isn’t captured, which is to say parentheses usually designate a capturing binding like $1 or \1 depending on language/context, but the ?: tells the regex engine you don’t need/want that binding.
If I’m understanding all of that correctly, it’s probably very close to what you intended but definitely highlights two of the biggest problems with regex:
1. Intent is easily lost on read. This is trite to say, but I’m pointing it out as someone who’s played regex golf for fun and still spent over an hour getting confident enough to describe what this short pattern appears to do.
2. Balanced pairs allowing an escaped terminator are not “regular language”. It’s possible (and I think probable but I’m not at a device capable to confirm) that this pattern is safe because the balanced pairs are single-character and there’s only one possible escape. But the fact that I can’t say so with confidence is something that would make me cautious using it.
All of that said, if I were writing this. I’d try to make it both more clear in intent and a little more predictably resilient.
1. I’ll include slash delimiters to remove ambiguity.
/"(?:[^\\"]|\\.)*"/
/"(?:\\.|[^\\"])*"/
/"(?:\\.|.)*?"/
Final thoughts:
- Almost everywhere you could use regex will not match any kind of line breaks. That might be desirable but depends on what you’re trying to do.
- Even with the abundance of caution I expressed, this is decidedly unsafe for processing user input. For example I’m certain this would be an XSS vulnerability.
In actual code, it is of course attached to some definition like `type: TOK_STRING`.
> 2. Balanced pairs allowing an escaped terminator are not “regular language”
Are you confusing this with that e.g. the language of validly parenthesized expressions is not regular? A single string with the option to escape any character including the escape character and terminator is regular.
/"(?:\\.|.)*?"/
/".*?"/
/"[^"]*"/
This feels like a big leap for something I didn't say was even involved in web development. I see ways that a failure to handle an invalid string (and this will match invalid string in languages that have additional escape forms with stricter requirements e.g. `\u1234`) could be chained with other bugs to create an XSS, but it would be difficult to ascribe the XSS to this part. And extension to handle most such things wouldn't make the language non-regular.
(p.s.
> I’ll include slash delimiters to remove ambiguity.
The two languages I have most often used this in don't use slash delimiters, so I think this has only added ambiguity - now we are considering it embedded in some specific-but-unspecified language's regexp syntax where we might need to consider another level of escaping - compared to letting the regex stand "bare".)
/"(?:\\.|.)*?"/
1. Match a quote.
2. Match the shortest sequence of:
A) any character preceded by a backslash, or
B) any thus far unmatched character, until
The 2.A case should match before laziness kicks in. The only case when it shouldn’t is if the backslash has already been consumed by a previous escape.
"\"
As for the issue of varying between engines: The classic deviation between POSIX and non-POSIX regexp is whether you pick leftmost-longest and tiebreak by alternate order, or leftmost first alternate. (POSIX is the correct answer because there are benefits to commutative alternation.) Non-greedy regexp has a similar issue for whether it should prefer shortest or first alternate. (Proper thinking about regexes doesn't involve temporal relations like "before" or "thus far" or whether something is "consumed" - even in engines that don't implement leftmost-longest this can only confuse.)
Engines that take alternation order into account will also have it match the complete string of:
"\""
tl;dr: Don't use non-greedy operators. Don't think about regular language matching in temporal or procedural terms. Don't try to rewrite regexps without test cases.
Yes, naming helps. I was mostly sympathizing with the people whose brains explode at the sight of a regex. It’s surprising how much time it took me to be sure I understood your simple pattern.
> Are you confusing this with that e.g. the language of validly parenthesized expressions is not regular? A single string with the option to escape any character including the escape character and terminator is regular.
This is a good correction, thank you. I was thinking of cases where balanced pairs allow both escaped delimiters and nested pairs.
> This feels like a big leap for something I didn't say was even involved in web development.
I hope it’s clear I didn’t intend to make that leap, and was just generally reflecting on the ways simple patterns can go awry. That’s why I said “for example” and “would”. If someone were to take this pattern or any of the variants and use it in a context that writes innerHTML, it would be vulnerable to XSS by failing to account for multi-character tokens like " in its escaping.
> The two languages I have most often used this in don't use slash delimiters, so I think this has only added ambiguity - now we are considering it embedded in some specific-but-unspecified language's regexp syntax where we might need to consider another level of escaping - compared to letting the regex stand "bare".)
I understand that many languages don’t use slash delimiters. But it’s quite common to use them for regex where a language context isn’t assumed. My point wasn’t to impose slashes on your example but to make it explicitly knowable, in words, whether the quotes were intended as part of the pattern. Not knowing that was the primary reason I wasn’t sure I understood the pattern’s intent.