cscottnet

I know, it's the coolest, right?

They should have gone with xwm (pronounced "zoom") for the name, though. Searching reddit for "Parsoid" just turns up "parasitoid" wasps for some reason.

Funny thing: I was hired by OLPC specifically to implement that book: https://blog.printf.net/articles/2011/06/18/narrative-interf...

A slightly longer answer is that we looked into a number of possible reasons and it's not clear there's an easy answer. Lots of differences between the two setups, and every time we come up with an possible answer like "oh, it's the reduced network API latency" we come up with a counter like "but html2wt is also faster and it does barely any network requests". Casual investigation raises more questions than answers. So we've looked into it but don't yet have an answer that we fully believe.

It appears that you were probably describing the lexer pass in your description of docuwiki. Indeed tokenization is a very hard problem for wikitext. We use a pegjs grammar for it, but it contains less of lookahead/special conditions/novel extensions, etc. It's hard. Wikitext is messy precisely because it was intentionally designed to be easy and forgiving to write.

Seems like we've learned many of the same lessons building our parsers. Markup parsers do seem to be a unique thing, not really like parsing either programming languages or natural languages. If we every meet I'm sure we could happily share a beverage of your choice trading stories.

Yes. We're not there yet, but that's the goal!

There are thousands of integration tests. The "correct" output of the parser is well-known for a given input, and those test cases have been accumulating for over a decade. But the internal structure of the parser is much more fluid, and so it wasn't (historically) thought worthwhile to try to write tests against that shifting target.

If you want to dig through the history some: https://github.com/wikimedia/parsoid/blame/6eb00df3e090b20cc... Is a pretty good example of the porting technique. You'll see quite a decent number of lines are still unchanged from the "automatic conversion from JS". https://github.com/wikimedia/parsoid/commit/6eb00df3e090b20c... shows what the initial port process was like. Still quite a bit of work, but you'll see it's almost all "real" work that needs a human to think about things, not just mechanical syntax translation. The syntax translation part was done automatically.

Then https://github.com/wikimedia/parsoid/commits/master/src/Ext/... is a not-too-atypical view of the process after the "intial working port" was done (post Aug 2019). Some nasty bugs fixed (https://github.com/wikimedia/parsoid/commit/34fcb4241aa0f3a0... a GC bug in PHP!), some more subtle bugs (PHP's crazy behavior of '$' at the end of a regexp, unless you use the 'D' flag), etc.

If you look through the history earlier in 2019, you'll even see JS commits like https://github.com/wikimedia/parsoid/commit/2853a90ceda7cdfa... which are to the JS code (in production at the time) preparing the way for the PHP port. In that particular case, our tooling was doing offset conversion between JS UTF-16 and PHP UTF-8 as part of the output-testing-and-comparison QA framework we'd built for the port, and it was getting hugely confused by Gallery since Gallery was using "bogus" offsets into the source text. Since fixing the offsets was rather involved (the patchset for this commit in gerrit went through 56 revisions : https://gerrit.wikimedia.org/r/505319 ) the change was first done on the JS side, thoroughly tested, and deployed to production to ensure it had no inadvertent effects, before that now-better JS code was ported to PHP. It would have been a disaster to try to make this change in the PHP version directly during the port.

I'm on the team. Part 2 of this post series should have lots of interesting technical details for y'all; be patient, I'm still writing it.

But to whet your appetite: we used https://github.com/cscott/js2php to generate a "crappy first draft" of the PHP code for our JS source. Not going for correctness, instead trying to match code style and syntax changes so that we could more easily review git diffs from the crappy first draft to the "working" version, and concentrate attention on the important bits, not the boring syntax-change-y parts.

The original legacy Mediawiki parser used a big pile of regexps and had all sorts of corner cases caused by the particular order in which the regexps were applied, etc.

Parsoid uses a PEG tokenizer, written with pegjs (we wrote a PHP backend to pegjs for this project). There are still a bunch of regexps scattered throughout the code, because they are still very useful for text processing and a valuable feature of both JavaScript and PHP as programming languages, but they are not the primary parsing mechanism. Translating the regexps was actually one of the more difficult parts, because there are some subtle differences between JS and PHP regexps.

We made a deliberate choice to switch from JS-style loose typing to strict typing in the PHP port. Whatever you may consider the long term merits are for maintainability, programming-in-the-large, etc, they were extremely useful for the porting project itself, since they caught a bunch of non-obvious problems where the types of things were slightly different in PHP and JS. JS used anonymous objects all over the place; we used PHP associative arrays for many of these places, but found it very worthwhile to take the time to create proper typed classes during the translation where possible; it really helped clarify the interfaces and, again, catch a lot of subtle impedance mismatches during the port.

We tried to narrow scope by not converting every loose interface or anonymous object to a type -- we actually converted as many things as possible to proper JS classes in the "pregame" before the port, but the important thing was to get the port done and complete as quickly as possible. We'll be continuing to tighten the type system -- as much for code documentation as anything else -- as we address code debt moving forward.

AMA, although I don't check hacker news frequently so I can't promise to reply.

If you use it, we will keep it up.

Sadly, this does seem to be a limitation of our labs infrastructure. :(

Yeah, the domain setting was leaking between sessions before. Oops. Fixed now.

It's actually a 269-line test case for more serious projects at the foundation: the Offline Content Generator and Parsoid. We're allowed to have fun in the service of the greater goal. More technical details at https://meta.wikimedia.org/wiki/Telnet_gateway

Patches welcome!

Fixed!

Accessing Wikimedia content while being Really Really certain that there is no downloadable code being executed, or browser exploits being inserted. Browsing for the paranoid.

Here's a bit more: https://en.wikipedia.org/wiki/Wikipedia:Wikipedia_Signpost/2...

Fixed!

Oh -- and report bugs/feature requests at https://github.com/cscott/wikipedia-telnet/issues please. Patches are even better.

See https://meta.wikimedia.org/wiki/Telnet_gateway for technical details. The technology behind this is used for Wikipedia's Visual Editor, PDF export, and plain text export, so it is very likely to be supported if enough folks find it useful.

re: find_or_insert

http://static.rust-lang.org/doc/0.8/std/hashmap/struct.HashM...

It appears to still have the mandatory copy. (And this is just one example of a pervasive issue with the standard library.)

Please see http://cananian.livejournal.com/68747.html and https://github.com/cscott/rusty-turtle.

Yes, the post is brief since it was written from notes several months after I had been talking to the rust guys. I couldn't justify the time to spend making the examples more concrete (and wasn't exactly encouraged to do so at the time). Sorry, that is as far as I've carried the load, you've got to take it from here.

Sorry, I gave Mozilla about 3 weeks of free work, then travelled to SFO and talked with Rust folk in person. That's a lot of donated time. I was working on another project by the time I wrote this post (from my written notes). Try not to look the gift horse in the mouth.

Boehm is far from "fast in practice".

Oh, look, my rant finally got noticed. ;)

First off -- yes, it is a rant. I actually had some more nuanced in-person conversations, but those aren't nearly as fun to post. So you get the version that was fun for me to write. Sorry.

That said, I'm glad to hear Rust is all wonderful now. It sounds like pcwalton agrees with most of my points and got them fixed. Kudos. I look forward to better GC, fork-join parallelism, and work-stealing.

It doesn't seem like anything significant was done with the type system, though. How does Rust handle "find_and_insert" now?

If folks are interested in actually critiquing code style, feel free to check out the rusty-turtle code at http://cananian.livejournal.com/68747.html. I disagree that I was trying to write Rust as if it were some different language -- my frustration was in part because I was trying to write things with Rusty ownership types, etc, and getting that to work properly was so frustrating (and the error messages so opaque). If I just stuck managed boxes around everything my life would have been simpler -- but then I would not have been programming in Rust, really.

I'm interested to hear more discussion on the primary point made in my blog: is Rust supposed to be 'safe and fast'? Or just 'safe'?