Comment by ridiculous_fish | Hacker News Reader

Comment by ridiculous_fish | Hacker News Reader

Parent Full thread

ridiculous_fish

I wrote the JavaScript regex engine in Hermes [1], and also regress in Rust. I want to explain what's sucky about this decision, but first some ES regexp background.

The regexp grammar portion of the ES spec is different, in that it's not sufficient. You can implement JavaScript from the ES spec, but NOT regexp. The grammar doesn't really make sense [2]. The Chakra team found the same thing [3].

I have not confirmed this but I have been told the regexp portion is tightly controlled by Google, more than most parts of the spec. When v8's regexp implementation is found to deviate from the spec, the spec gets changed. This makes sense (why risk breaking websites), but it illustrates how little weight the regexp spec actually has.

With Firefox using irregexp, it further cements this unfortunate reality.

1: https://hermesengine.dev

2: Example: the NonemptyClassRangesNoDash production may include a dash

3: https://docs.microsoft.com/en-us/archive/blogs/ie/chakra-int...

> You can implement JavaScript from the ES spec, but NOT regexp. The grammar doesn't really make sense [2]. The Chakra team found the same thing [3].

I haven't tried to implement the RegExp section, so I can't speak to that, but this is largely why we have Test262[1].

Also that Chakra blogpost is from 10 years ago, talking about ES5. Things have improved considerably since then, though of course we have to live with historical baggage.

> With Firefox using irregexp, it further cements this unfortunate reality.

Is this a specific unfortunate reality? I don't spot any RegExp entry in the last spec Incompatibilities section[2] that seems like a bad change or even having to do with regex grammar at all.

[1] https://github.com/tc39/test262/tree/master/test/built-ins/R...

[2] https://www.ecma-international.org/ecma-262/#sec-additions-a...

Yes, thank goodness for Test262. One implements JS-compliant regexps by doing the natural thing, and then making Test262 pass.

Some things have indeed improved. The 'u' flag introduced in ES6 provides a much cleaner grammar for example. However the process is still busted.

Here is a concrete example. /{1}/ is permitted by the ES6 spec (including Annex B). But in practice, "web browsers" rejected it. Rather than fixing the browsers, the spec was changed in ES8 to reflect "web-reality". https://github.com/tc39/ecma262/pull/303

Browsers could have become ES6 compliant here, with zero compatibility risk. But when there is non-compliance, it is the spec that moves.

That's how specs should work, when there is a feature nobody wants and nobody has implemented.

> I have not confirmed this but I have been told the regexp portion is tightly controlled by Google, more than most parts of the spec. When v8's regexp implementation is found to deviate from the spec, the spec gets changed. This makes sense (why risk breaking websites), but it illustrates how little weight the regexp spec actually has.

A few responses:

1. This was not at all my experience working with the V8 team. For example, when I noticed that Irregexp did not properly implement the Canonicalize algorithm for non-unicode case-insensitive comparisons [1], they were happy to accept my patches [2][3]. Nobody ever suggested changing the spec.

2. Nothing changes in the JS spec without consensus from all the major stakeholders. See, for example, the second sentence in the TC39 process document here [4]: "The committee operates by consensus and has discretion to alter the specification as it sees fit." Mozilla can block (and has blocked) proposals that it feels are bad for the web platform. Google has no special power to bend the committee to its will.

3. Another way of saying "when V8's regexp implementation is found to deviate from the spec, the spec gets changed" is "when the spec is found to deviate from the consensus of implementations, we correct the spec". The Chakra team's post notes: "In practice, all browsers accept regular expressions such as `/]/` and web developers write them." The purpose of the spec is to make the web work better by maximizing compatibility between implementations. In cases where all implementations agree, and real websites depend on that behaviour, the spec should change to match reality.

Don't get me wrong: Chromium monoculture is a real problem. If we thought that Google was going to add a bunch of new regexp features without going through the standards process, or if we had plans to prototype regexp features that we feared Google wouldn't accept upstream, we might have made a different decision. But regexps are not a plausible battleground, and the compatibility benefits of sharing code outweigh the harms.

PS: regress looks really sweet, and I am excited to see how it turns out.

[1] https://tc39.es/ecma262/#sec-runtime-semantics-canonicalize-... [2] https://github.com/v8/v8/commit/3fab9d05cf34a7f0bc0e9405729a... [3] https://github.com/v8/v8/commit/b65fcfe92566415c8aae373d5fb5... [4] https://tc39.es/process-document/

Great reply, thank you for sharing your perspective.

First, I confess to some skepticism of the standardization process. For example, consider ES2018 lookbehinds. At the point of standardization, only one browser implemented them: v8. And in the standardized form (arbitrary width) they are very difficult to retrofit onto existing engines. Moddable had to throw their engine out and start over [1]. JSC's YARR still doesn't support them, neither does SpiderMonkey. Did all stakeholders really agree on this feature and then not implement it? Or was this just v8 in the driver's seat?

Second, regexps actually do have a role in driving monoculture. You can't polyfill regexp syntax. At one point, Steam's website only worked on Chrome because only v8 implemented lookbehinds [2] - again because it's hard to retrofit.

Lastly, it remains a problem that one cannot implement a conforming JS regexp engine from the spec alone. The rest of JavaScript does not have that problem, only regexp. And I disagree with the "real websites depend on that behaviour" qualifier. No website depended on /{1}/ being invalid syntax, yet it was made so in ES8.

Incidentally, it's amusing that v8's Canonicalize was buggy. Non-Unicode case-insensitive char ranges is THE ugliest part of regexp (QuickJS doesn't even try [3]) and I just assumed it was codifying v8's implementation. Guess it was someone else's, hah.

1: https://blog.moddable.com/blog/regexp/

2: https://github.com/webcompat/web-bugs/issues/51385

3: https://github.com/ldarren/QuickJS/blob/mod/libregexp.c#L259

The TC39 minutes are all online [1]. You can see the initial introduction of lookbehind assertions [2], the meeting where they reached stage 3 [3], a status update while waiting for more implementations [4], and then the eventual agreement to move them to stage 4 (full standardization)[5].

They reached that point because: a) everybody agreed that they were a good feature, with precedent in other languages; b) the spec text had been scrutinized by the people who like to spend time scrutinizing spec text; and c) two implementations (Moddable and V8) had successfully implemented them. That's the process, and it was followed to the letter.

Does that put a burden on other engines to keep up? Sure. But that's what we signed up for. The webcompat bug you linked is from April 2020, more than two years after lookbehind assertions were standardized. That's nobody's fault but ours, and we did this project so that we don't end up in a similar situation in the future.

I'm not sure what your source is for JS regexps being impossible to implement from the spec, but if you have specific points of concern, you should open an issue [6]. Writing a spec is hard work, and things get missed, but the people working on this are genuinely trying their best.

(As far as I can tell, Canonicalize is what everybody thought they were implementing. V8 tried to get clever with ICU and ran facefirst into some of the dark corners of Unicode.)

1: https://github.com/tc39/notes/tree/master/meetings

2: https://github.com/tc39/notes/blob/master/meetings/2016-11/n...

3: https://github.com/tc39/notes/blob/master/meetings/2017-03/m...

4: https://github.com/tc39/notes/blob/master/meetings/2017-05/m...

5: https://github.com/tc39/notes/blob/master/meetings/2018-01/j...

6: https://github.com/tc39/ecma262/issues

I want to say four things. First, I think Mozilla probably made the right decision here. Mozilla is an org with limited resources and must direct them to what matters - and as you say, regexps are not a plausible battleground. I wish I had said so in my top-level post.

Second, I cast no aspersions against any of the people involved in the ES spec. I recognize they are engaged in hard and unrewarding work driven by a sincere effort to push JavaScript forward, while everyone's a critic, especially myself. It's awesome that the meeting minutes are available. Thank you for the links.

Third, I honestly believe the meeting minutes support my point. I risk overstepping, because I had no involvement, but from your third link:

> We have two implementations. One in V8 and one in the Dart VM. It's not a JS implementation, but it does implement this feature.

This supports my speculation that it's all about v8. Later:

> I would assume Chakra or V8 would have said something by now if they had issues

SpiderMonkey and JSC are chopped liver? This feature really seems to have been driven by the implementors.

And this one feature (lookbehinds) was so disruptive, so damn hard to implement, that Mozilla abandoned their implementation and now just uses v8's. The linked blog post cites this feature as an impetus to switch.

Was that the goal? Did SpiderMonkey engineers give the nod to lookbehinds with the intention of abandoning their engine in favor of irregexp? I have to believe not; I speculate it was the very human response of conflict avoidance. Easier to say "yes" especially if you do not appreciate how much work you are signing up for. But it means ES2018 regexp advanced the monoculture. One less regexp engine in the wild.

Fourth, I really do know that the ES regexp spec is not useful for implementors, because I am an implementor. Ok, it's not really a question, huh; I should just make a PR.

tl;dr: The spec exists for the web; the web does not exists for the spec.

Or another reading is that Google is yet again abusing their dominant position to alter the spec to suit their needs instead of the need of people using non-dominant browsers.

Reply on news.ycombinator.com