Hacker News Reader

Hacker News

Comment by thom | Hacker News Reader

Full thread

thom

Delighted to discover this kind of breakthrough is based on a gnarly regular expression:

https://github.com/Svensson-Lab/pro-hormone-predictor/blob/m...

View on HN

That’s only the tip of the iceberg. Here’s “How Perl saved the human genome project” from 1996:

https://bioperl.org/articles/How_Perl_saved_human_genome.htm...

(And of course all the articles about excel renaming genes, and then people trying to clean up the mess…)

> The study would not have been possible without the use of artificial intelligence to weed through dozens of proteins in a class called prohormones.

I see we’re being fast and loose with the term “artificial intelligence”?

Could be that they relied on AI to generate the gnarly regex ¯\_(ツ)_/¯

Any sufficiently advanced regex is indistinguishable from AI.

Even simple regex ones, since we don't know if they were created by a human or an AI.

I'm trying to think of a funny way to cite Clark's Law, but failing...

Regexes occasionally get called "black magic", and there is is an inverse of Clarke's Third Law: Any sufficiently advanced magic is indistinguishable from technology.

I think you meant “any sufficiently commoditized magic is indistinguishable from technology”, or “any sufficiently analyzed magic is indistinguishable from science.”

With enough if statements, you could create a LLM.

I am still waiting for an LLM trained to focus on effin' regexes and their variants like sed, somebody please do a page with ads for this and you will have a nice little side income and warm fuzzy feeling on top of it.

Natural language -> fully working one, I don't mean some email validators but way more complex stuff. Although, I've recently had a case which was too much even for regexes in any form or spec, then sort of grammar-based parser needed to be done from scratch.

I haven’t had the need for super complicated regexes, but ChatGPT and Claude both worked fine to generate a regex to extract markdown code blocks.

AI is the new, “Algorithm”. It is an overly broad term that basically means, “Technology” to people who use it.

For non-technical people, terms like Machine Learning, Algorithm, Automation, AI, Neural Networks and 'Magical Powers' are synonymous.. Journalists, marketing teams, and executives choose whichever term generates the most hype and clicks.

I like to describe my job as a software engineer as "wizardry" to people. With the way LLMs are writing code, it's only getting closer to writing actual spells.

> "Formerly, when religion was strong and science weak, men mistook magic for medicine; now, when science is strong and religion weak, men mistake medicine for magic."

— Thomas Szasz

"Like a magic crystal mirror, My computer lets me know Of the other world within it Where my body cannot go / The computer is a gateway To a world where magic rules Where the only law is logic Webs of words the only tools" https://mindstalk.net/filk/world.html

My understanding is that the original use of AI was to describe algorithms that solved complicated problems, but only sometimes, and usually through heuristics. As opposed to provably correct algorithms like Dijkstra's.

I think that's how we still use it.

It is a somewhat though provoking question for me where you LLM's fall in automata theory? Are they non-deterministic finite automatons which would make regex's a cousin? Or does it require pushdown automata levels of capability?

Regexes were invented by Kleene in 1951, to describe McCulloch-Pitts neural nets. So, artificial intelligence!

https://www.rand.org/content/dam/rand/pubs/research_memorand...

Jokes aside regexes being AI, lots of protein motifs can be elegantly described by simple regexes. And their classification performance is sometimes perfect.

See the Eukaryotic Linear Motif resource: http://elm.eu.org/elms.

unfortunately, if you do anything computational, when you need to explain it to a journalist/public audience/young person you want to hire/investor/trendchasing early user, you are incentivized to call it "AI".

I got lost at (?R). IIUC, this is a feature of the "regex" module that "tries to match the entire regex recursively". Fun.

Beautiful yet terrifying. I'm glad that regex isn't something I'll ever have to touch.

The good news is it’s read only- if you’re reading for understanding, you’re doing it wrong :-D

It's not that bad. AutoRegex[0] and regex gen [1] make it more accessible than ever.

[0]: https://www.autoregex.xyz/

[1]: https://regex-generator.olafneumann.org

Heh, I just fed autoregex a regex from one of my projects, and it simply times out. It comforts me to know that billion dollar LLMs have to chew on those just as much as I do.

A proper litmus test, what's causing the hang in your case?

Regex to me, is pattern finding and abstraction taken to the extreme. I like the challenge

Good question. The regex I tried is for extracting amounts in EUR and USD:

  /
  (?<=^|[ \t])
  (?<currency_prefix_with_space>
    (?<currency_prefix>
      €|EUR|\$|USD
    )
    [ \t]?
  )?
  (?<number>
    (?<integral>
      -?
      \d{1,3}
      (?:[\.,]\d{3}|\d*)
    )
    [\.,]
    (?<fraction>
      \d{2,3}
    )
  )
  (?<currency_postfix_with_space>
    [ \t]?
    (?<postfix_or_ending>
      (?<currency_postfix>
        €|EUR|\$|USD
      )
    ) | (?<ending>
          [ \t]|$|\n
        )
  )/x

I'd imagine many nested named capturing groups may trip even the best automated system! I do like the solution though.

I would've probably approached it differently, trying to first get the 'inverted' match (i.e. ignore anything that isn't a currency-like pattern) and refine from there. A bit like this one I did a while back, to parse garbled strings that may occur after OCR [0]. I imagine the approach does not translate fully, because it's pattern extraction rather than validation.

[0]: https://observablehq.com/@dleeftink/never-go-nuts

Thanks for sharing! I have to admit I do not have the necessary brain cycles to spare today, but OCR processing is indeed of interest to me, and I will take a more in-depth look in the upcoming days.

The idea of an exclusionary approach sounds interesting as well. I'll have to think about that a bit.

Check out WordNinja in case regex doesn't cut it! [0]

[0]: https://github.com/keredson/wordninja

Will do, thanks again!

I think what's causing the hang is using the site. I gave it [a-f0-9]{8}(?:-[a-f0-9]{4}){3}-[a-f0-9]{12} and it sat thinking about things for a few minutes. So I tried the reverse and asked it "Match a UUID" and it sat and thought about things.

Now I am enlightened.

Aww shucks, and here I was feeling a wee bit proud of myself for managing to break it.

Regex is one of my favorite things to work with. It's surgery. I love having a very specific problem and a very sharp knife. What I hate is chasing down language or library bugs, or undocumented limitations. Of course I would hate it if I had never used it extensively. Its terseness is harder if you only have to pick it up occasionally.

You'll love this one then!

[0]: https://cardoni.net/parsing-proper-nouns-with-regex/

That regex detects things like "eeieiieeeiA __ 1 2 3__" as a proper noun. I'm confused by the decision to accept any number of leading [ei] and any number of trailing (?:\s+[_\d]+).

It's not perfect for sure, but perfectly suitable for a first pass and then a dictionary sweep or another set of regexes. I guess I like the compactness of it!

It only looks for a single leading “e” or “i”, not any number. I’m guessing those tweaks were added to capture specific proper nouns that weren’t captured by simpler “leading capital letter” regexes, like “iPad” or “eBay”.

"*" is "0 or more" and "+" is "1 or more", it looks for any number of "e" or "i" at the beginning and at least one capital letter. The diagram below the regex is wrong.

Instead of

  [ie]*-?[A-Z]+

it looks like they wanted

  [ie]?-?[A-Z]

There was a product called the eeePC or so. It was a "netbook."

The Asus product line, yes (all three Es apparently stand for "easy"). But Asus write it with a capital E on the front. Still, somebody else might not.

I have done something similar before, to help form an index for a large book. This is what I would consider a relatively simple regex.

The harder ones I have dealt with are those looking for malformed syntax where the closing mark that might be missing could be several thousand characters after the opening mark, or the opening mark itself might be missing, across a data set that is several hundred million characters. So you need something very complex to find all the distinctive characteristics of the content that is supposed to be enclosed - while avoiding the many similar structures that give false positives. Sometimes the technically easier solution is too slow to run (look ahead and look behind, etc), so you need to pivot and use other regex features. It can take a day or two to get right.

Remember to get a license for your code

https://regexlicensing.org/

On the list of disasters, they forgot "parsing html":

https://stackoverflow.com/questions/1732348/regex-match-open...

Huh, I tried sharing the Zalgo answer here, but it didn't quite work, never knew HN wouldn't render those kinds of characters

It did, but I kinda went over the top with it once and dang had to tell me off, so it might be they added filtering.

o, there's your problem. Only half is supposed to go over the top. The rest is supposed to go below.

> Licenses: “No regex licenses have been issued at this time.”

Perfect

Just license 007: License to kill.

Yes, once you have a scalpel, everything looks like a very specific problem ;)

That is the temptation! No requirements, no conventions, no best practices, no nagging boss, just me and total root access to all the files :)

I think it comes down to the syntax being used to make regex. It is borderline an assembly language level of making them. Better language tools would make it easier for people to grok. Basically instead of using chars like ?^()$ and the like lets use words people understand.

It’s not just the syntax. Regex isn’t directly composable (unless you just mash together strings). They cannot define nested structures, I.e. no recursion. They can’t maintain context.

We don’t need better language tools. Better parsers can, and already have, been implemented in libraries.

I’d have a look at parser combinators.

(?R) is very much recursion in there.

> They can’t maintain context.

Is true though, or perhaps "state"? I know I had to come up with an algorithm because regexp alone couldn't do what I wanted (not even advanced features like lookahead, lookbehind, etc.)

I played with parser combinators in Elm and found it very useful c.f. regexes - the issues were mainly down to the Elm implementation. Not sure how effective they are in Rust or Go

As @aranchelk said LL and LR grammars are literally that. They allow you to assign names to regexs and compose them.

Libraries like python's lrparsing [0] let you assign regex's (aka tokens) to variables, then build grammars by combining them using python expressions. For example:

    while_statement = Keyword('while') + '(' + expression + ')' + block
    statement = while_statement | ....
    block = Token('{') + Repeat(statement + ';') + Token('}')

These grammars are more complex, with more rules you have to follow, but also more checking is done when they are compiled. They tend to mostly work once they do compile. So they are what you asked for, but there is no free lunch.

On the down side, lrparsing is pure python so it's slower than python's inbuilt regex's.

[0] https://lrparsing.sourceforge.net/

I'm comfortable with it (having previously been paid to write Perl for several years), but what I don't like is how many other people I have worked with aren't, and just blindly paste in (totally incorrect) regexes from Stack Overflow.

Writing one that does exactly what you want is very satisfying. And with Python you can have named groups extracted into a dict and it just feels like magic.

Neat. I always thought Advent of Code 2015's Day 19 (Medicine for Rudolph) was just an exercise in parsing and validation of production rules against a grammar, and any use of regex in that puzzle solution was purely coincidental ;)

Is this AGI?

oh god, can you even imagine?

After using a regex that creates AGI researcher will see the world like Neo sees the matrix.

Ask Deepseek to generate a regex for detecting python import statements.

Is that an anti-pattern to put the comment inside the function vs above

Function documentation goes inside functions in Python, explicitly making them self-documenting. That can then be automatically picked up by any other code (like tooling for automatically generating API documentation) formalized as PEP 257, https://peps.python.org/pep-0257

Python will literally assign the content of your docstring (a triple-quoted string at the start of a function) to a special double underscored ("dunder") property called `__doc__` that any code can access; not just docs generators, but your own code as well (pretty dang useful for generating on-the-fly help output!).

Which also means that if you run into weird behaviour where a function doesn't do what you think it should do, you can just fire up the REPL, import that function, type `print(function_name.__doc__)` and presto, you have the documentation right there, specifically for the exact version you're using. You don't even need to leave your IDE if it comes with an integrated terminal.

interesting TIL too bad we don't have auto doc gen (fastapi what I'm working with atm)

edit: not saying it can't be done but our org isn't using auto docs

Yeah, it's one of those "Once you're used to writing doc strings, you have no reason to ever use anything else" but if you don't use them, it can be prohibitively laborious to move whatever alternative you're using over to "PEP-compliant".

(lots of fun dunder functions in Python that make metaprogramming a lot easier, but someone needs to tell you about)

Fluent Python covers a bunch of this stuff.

TIL __doc__! Pretty nifty.

> type `print(function_name.__doc__)`

Or even easier: `help(function_name)`!

https://news.ycombinator.com/item?id=43266546 just got posted about that =D

How is this not more well known? I thought I was a god for discovering dir() but you’re showing that I was nothing but a peon!!!

There's a bunch of "all dunder properties" pages online, any one of them is well worth reading through at least once. So much hidden goodness.

The actual docs on python.org are quite readable and probably a lot more accurate than random blogspam / LLM glurge pages

Why would you assume it's LLM slop when Python's been around for decades? Lots of folks are enthusiastic enough about "Python magic" to have done excellent writeups of literally ever dunder function under the sun on a single page, at a level of detail and excellence that the python documentation, bless its heart, cannot ever hope to match.

- The python docs are there to give you the information you need.

- Passion pages are there to do a deep dive into all the crazy shit you can do with that information =D

> Python will literally assign the content of your docstring (a triple-quoted string at the start of a function)

Any string. Triple-quoted is just multiline strings and works anywhere.

True, I forget some people think a single line is function documentation; any function that's worth documenting is worth documenting properly, and needs multiline ;)

Annoyingly it pushes the type definitions away from the function body when you have a particularly long doc. It’s a cute trick but docs above would have been better.

You know what's cool: IDEs. They can do block folding. The really good ones just auto-fold function docs because you don't need to see those until you do.

It's not just a good trick: it's the right way to do it, and tooling can be (trivially) made to show you only what you need to see until you need to see more.

If we’re assuming an IDE that can arbitrarily change text layout and rendering I don’t see why it matters where the comment goes

Welcome to the fascinating world of academic code!

Obligatory xkcd:

https://xkcd.com/208/

A very early one at that

Please, if you write regexes like these, comment them...

A great way to do it is to split them up by concatenating them across a bunch of lines, and put a brief explanation at the end of each non-obvious part (to the right, on the same line).

Plus that also lets you indent within nested parentheses, making it that much more understandable.

I'm baffled when I come across a file like this where the code itself is heavily commented, but a gnarly regex is not. Regexes are not strings, they are code -- and with their syntax, they need comments even more.

One thing I always appreciated about CoffeeScript was its [heregex syntax](https://coffeescript-cookbook.github.io/chapters/regular_exp...), which made regexes clear and easy to document.

Many CoffeeScript constructs were adopted into JavaScript and TypeScript, but unfortunately, heregexes weren't among them.

I agree. I'm pretty sure jashkenas got it from Ruby, which got it from Perl, with the `/x` flag (for extended regular expressions).

Later languages have added support for the `x` flag, including C# and Rust. There is also a stage 1 proposal for JavaScript: https://github.com/tc39/proposal-regexp-x-mode

Nice roundup, and thanks for sharing that proposal!

Notepad++ has /x mode as well.

I probably use regexes the most in notepad++

regex101.com is a lifesaver for parsing esoteric regexes after you've forgotten what they're supposed to do.

Great advice that is a bit redundant with LLMs, no? They're pretty great at all forms of translation.

First, I don't want to waste time feeding something into an LLM that should be commented in the first place.

Second, not at all. An LLM can tell you how the regex works (hopefully). It can't tell you what each piece means in terms of the program's logic. Or at least not always and not reliably.

Reply on news.ycombinator.com