My takeaway from this, since the majority of it went over my head: I can continue to parse HTML with regular expressions.
Technically PCRE regexes are powerful and can match anything.
In reality, a complex PCRE regex will almost always be more difficult to maintain than a parser-combinator or hand-rolled parser in a more traditional language.
People saying "Regexes can't match HTML, use an html library" are wrong to say regexes are incapable of it, but they're right to say to use a library meant for the job.
Sure, you can use this regex to match emails (http://www.ex-parrot.com/pdw/Mail-RFC822-Address.html), but using a more normal parsing language than PCRE regex will result in more readable code.
The same is true for almost any regular expression that takes advantage of PCRE features, especially backreferences.
In addition, a regexp will only match html correctly if you write a very complex one. With a naive regexp for an html tag's contents, you'll find that you still might match that text inside a <script> tag even though that is not html.. so you now need to figure out when you're in a script tag and exclude that, or if you're inside an html attribute string, and before you know it you have a 2000 character regexp that no one else will be able to read, all because you didn't want to use an html parsing library where getting a tag's value correctly would be a single xpath expression or css selector away.
https://blog.onyxbits.de/validating-email-addresses-with-a-r...
In the case of perl, where regexps performance has been optimized for significantly, the regexp actually performs better than a more normal parser.
From http://www.ex-parrot.com/pdw/Mail-RFC822-Address.html:
> It provides the same functionality as RFC::RFC822::Address, but uses Perl regular expressions rather that the Parse::RecDescent parser. This means that the module is much faster to load as it does not need to compile the grammar on startup.
Of course, if perl were a statically compiled language, the cost of compiling the grammar could be done at compile time.
No matter what language or programming style you use it's going to be ugly because it's an ugly problem.
Plot twist: the html library is built upon regex's (at least in part).
But they are not built solely of regexes. They always have added control structures that complement regexes on those place they are worst.
Or you can use comments and named groups: https://stackoverflow.com/a/1917982
If you actually just care about retrieving a few specific bits of data within a page, I've found parsing libraries (including ones that allow for CSS selectors) to be just as brittle to changes as regular expression extraction, and not all that much easier to use, given a good grasp of both technologies.
That said, if you need to alter an HTML document in some non-trivial way, parsing is probably the way to go.
Personally, I think to myself before using regexes to parse HTML:
1. Am extracting more than the contents of a single element?
2. Is the input HTML prone to change?
3. Will there be issues if the parsing completely fails?
4. Do I plan to use this code for more than a few months?
If the answer to any of the questions is "Yes", then don't use a regex :)
Some parsers may survive, but it's different enough to break most of them.