Better code highlighting on the web: rehype-tree-sitter | Hacker News Reader

Better code highlighting on the web: rehype-tree-sitter

haz.ee

61 pointstodsacerdoti27 comments

I'm looking into client-side syntax highlighting as well at the moment and lezer popped up: https://lezer.codemirror.net/. It's directly based on tree-sitter but tailored to be more web friendly (and written by the same author as codemirror) - https://marijnhaverbeke.nl/blog/lezer.html

I think I'd prefer Lezer in a javascript-only setting. The results are compact and you don't have to deal with WASM.

I've written a small tree-sitter grammar, with off-side rule even, and had a bunch of issues with the javascript bits not being the same version as the WASM bits. I ended up having to get just the right version of emscripten installed (and maybe a few other tools). The process felt a bit fragile to me.

Another contestant in this realm is Bright[1]. It runs entirely on the server and doesn't increase bundle size as seen here[2]. Regarding parsing speed tree-sitter is without a doubt performant since it is written in Rust, but I don't have any problems "parsing on every keystroke" with a setup containing Marked[3], highlight.js[4] and a sanitizer. I did however experience performance issues with other Markdown parser libraries than Marked.

[1]: https://bright.codehike.org/

[2]: https://aihelperbot.com/test-suite

[3]: https://github.com/markedjs/marked

[4]: https://highlightjs.org/

Nit: tree-sitter has first party Rust bindings (and its WASM bindings use Rust wasm-bindgen), but the core functionality is written in C.

While I have no doubt tree-sitter is technically superior for the specific task of syntax highlighting, Shiki has some other appealing qualities:

- Built in support for TextMate themes (and therefore also themes for VSCode, probably Sublime as well)

- Shiki Twoslash[1], which supports a variety of really cool features for highlighting subsections of code, static types and type errors, module relationships

1: https://github.com/shikijs/twoslash

I was looking into implementing syntax highlighting for my hobby language and man, it’s surprisingly hard. Either you use tree-sitter which has limited support, or TextMate grammars which seem to be quite terrible. And to get JetBrains support it seems like you need to add a whole other lexer integration.

Can’t we all standardize around tree-sitter?

I badly want this but in LaTeX. Minted is alright but it suffers from the same pitfalls as other regex-based syntax highlighters, and while you can extend the Pygments' lexer with more keywords, it's a losing battle. Having good syntax highlighting in academic papers would be amazing, since many subfields have their own DSL-like notations.

Unfortunately, writing new LaTeX packages turns out to be quite difficult, especially if you want it to integrate cleanly with \usepackage and environments.

Shameless plug for the Rehype plugin [1] I wrote based on Starry Night [2]. If you're interested in syntax highlighting like GitHub or customize it with CSS properties (for example, to support multiple themes), this is one solution.

[1] https://github.com/Microflash/rehype-starry-night

[2] https://github.com/wooorm/starry-night

Definitely considering switching over my blog to Astro since I write my posts in MDX and using this plugin to avoid doing this work on the client at all (currently use highlight.js and Next).

Lovely work by my friend Haze :)

NextJS does however offer an experimental Rust implementation for parsing MDX which makes it a lot faster and less buggy to use than setting it yourself with various Rehype and Remark plugins. Using Rust parser with Bright[1] (if you don't need other plugins like table of contents or Github Flavoured Markdown) is a potent setup.

[1]: https://bright.codehike.org/

Never heard of Astro[1] until today. I was looking for something better than Hugo and ended up using Nuxt content. How is this better than nuxt content :D ?

[1]: https://astro.build/

I haven't used 'Next Content', but one big difference is that Astro is framework agnostic. I'm not a huge web dev so I wouldn't know how different these frameworks are, but not being locked in/being able to try another framework is cool

It looks like it's another framework to me. The basic example looks exactly like Nuxt to me as well! (component, pages, and layout folder!)

To answer my previous question, they have a page talking about the similarities and differences here: https://docs.astro.build/en/guides/migrate-to-astro/from-nux...

They look really really similar to me (Nuxt content and Astro)

Astro is akin to other “meta-frameworks” with SSR/SSG like Next/Nuxt/etc, but it’s client-framework agnostic so its meta-ness is too. What primarily sets it apart is that it’s designed to ship just HTML/CSS by default, with any client side JS being opt-in where a client component is invoked (termed “islands”), and those components being authored in any client component library of your choosing (if an adapter for it exists, and many do). Or even vanilla JS, but then Astro is more akin to Jekyll or other similar SSGs.

True to their word the page loads fine without any javascript.

The HTML is even somewhat readable if you ignore the embedded styling. Though this blog is a lot better in that regard.

Haven't heard of it before either, but their own homepage (which I assume is dogfooding) has unnatural feeling (vertical) scroll-jacking, and small amount of horizontal scroll due to incorrect width. (Mobile, Firefox, Android)

As someone who is firmly ( _firmly_ ) in the "nobody can write good manual memory management code" train, zig is probably a better fit than rust here - and this is one of the extremely rare circumstances where I would pick zig over rust.

Tree-sitter is written in C - the memory benefits of rust are completely moot. Zig also has a built-in C++ compiler, so you could generate the object file with a single toolset.

I chose Rust here because that was what the command line tool that I referenced were written in. I'm not going to argue on HN about memory management (because I too, prefer Zig over Rust) because it's all about the program you're trying to write.

In terms of generating the object file with a single toolset, It's just not something I care about. This was like, a two day project. I don't need to make it more complicated than it needs to be

Is syntax highlighting considered difficult? A regex loop that outputs HTML to a pre is a few lines of vanilla JS.

Please share.

I’m on my phone, so no code sample, but put a choice regex in a variable and loop with .exec(), emitting that text surrounded by span tags into an array that you then join(). It’s really almost a one or two line script.

and for languages that have contextual keywords or semantic highlighting needs?

You can either use CSS where it makes sense or you can keep the regex results to refer to as needed instead of throwing them out.

Consider:

  `5 + 3 = ${5 + 3}`;
  const foo = async () => {}
  const foo = async function () {}
  const foo = { 'PI': 3.14 }
  // etc.

And then also remember that HTML can embed that, and CSS. And then highlighting something like TSX which can highlight HTML etc. It quickly becomes a recursive nightmare.

And all this is coming from someone who thought similarly to you. And now about 3 weeks of full time work (plus using a pre-built library oniguruma to do a lot of heavy lifting) and I just barely have a performant syntax highlighter. There are still a lot of edge cases I'm working through though.

It's not a nightmare, just a very small bit of mechanical work. I realize HN hates it when people don't talk about JavaScript like it's sex and quantum mechanics so I'll just back off and stop.

I'm only talking about JavaScript because it was the most difficult language to correctly parse so far. The examples I chose highlight that it's not as simple as a couple regexes. Notice how curly brackets can indicate: additional JS code, objects, or interpolated strings. The objects require different syntax highlighting than the JS code. This means you need some complex regexes to provide additional insight into what your highlighting.

And as someone actively working on this, yes. It is a nightmare. No, it's not trivial.

This is essentially the bulk of grammar tools like TextMate (though it also has the very useful begin/end token construct).

The hard part is likely doing it efficiently, and standardizing on a set of token names such that a theme ecosystem can exist.

TextMate grammar is the only syntax highlighting tool I'm familiar with. I'm curious why these others might be preferred. I only know TextMate since it is what VS Code supports.

Reply on news.ycombinator.com