Lezer: A parsing system for CodeMirror, inspired by Tree-sitter
marijnhaverbeke.nl
marijnhaverbeke.nl
The level of depth he dives into to perfect his projects is insane. For example, lezer is a parser generator( which by itself is not a trivial feat with novel ideas like incremental computations applied to parsing) to power his mainstream project which is CodeMirror.
Like this Prosemirror too has some insane levels of engineering underneath married with thoughtful architectural decisions.
Apparently, a big fan of his works.
Slightly off topic; is anyone aware of any Lezer grammar for regex? I've not been able to find any in the FOSS world. I suspect Regex101 has one, bit it's sadly closed source.
The next day we just had to update our package to the latest version and marvel at his response time.
You can put a basic code editor on your page like:
<cm-editor></cm-editor>
<cm-editor>
<cm-lang-javascript typescript></cm-lang-javascript>
<cm-theme-one-dark></cm-theme-one-dark>
</cm-editor>
How about a dist folder with a plain js file (or a js file and a css file) that you can load directly?
I am sick unto death of Node and all its associates.
I publish to npm because 1) Package managers are a fabulous idea. They're an easy way to download a package and it's dependencies, and update them over time. 2) it's not a good idea to push build artifacts to your repo, 3) This depends on CodeMirror, which is published to npm. Otherwise I would have to provide it too. That's too much work. 4) you don't need to be a Node developer to use npm.
Yep. Let me stress that there's absolutely nothing wrong with this if that's what you want to do -- but it's less than optimal for the person who just wants to add a JavaScript file for the editor, with maybe a JS/CSS combo to support syntax highlighting for a specific language. Codemirror 5 was like that, pretty much.
You basically have to set up a whole independent build and packaging system to configure Codemirror 6, and a lot of people just don't want to deal with that.
It's a pity it's the only JavaScript-based editor that works at all reliably on mobile (I would be delighted to be proven wrong about this).
It seems like for the past few years many high profile JS projects increased the complexity of using their software for (to me) unclear benefits.
https://github.com/inro-digital/lang-traindown
I did notice that it seems (rightfully) very focused on the exact usecase of syntax highlighting. Do people also use this kind of system for quickly building ways to parse data from arbitrary text syntaxes?
There’s a kind of importer thingy here but it doesn’t work well for complex grammars: https://github.com/lezer-parser/import-tree-sitter
I would be curious if there's been an effort to get tree-sitter working on the web.
I just discovered lezer and am tinkering with integrating it into my projects, but maybe tree-sitter would be a better fit?
(Speed could be a determining factor, but working with wasm can also make things slower, if you often have to get data in and out)
And since I use codemirror anyway, it might make sense to also use lezer, unless there are downsides I am not aware of.
When you need to build a general tool that supports a ton of languages, that’s when Lezer doesn’t fit as well since there just aren’t a ton of existing grammars out there.
Having used Lezer and CodeMirror a bit on a different project, they both seem like they are probably high quality and well thought out projects from a speed and reliability perspective, but I found the architecture and docs to be confusing and unergonomic.
The main issue I have with the docs/architecture (for CM at least) is that they use a concept called "facets" without really explaining what it means, and to be honest it felt like a level of abstraction/indirection that my brain couldn't handle.
The other issue with the docs is that they don't seem contextualised enough, somehow. They would list the functions/methods of an object but not really explain how it fit into the system as a whole. Maybe this is to do with not understanding facets, or CM's extension architecture, enough, or something, but it was definitely a recurring theme of my experience. I also found the CM API slightly confusing in that (almost) everything you do is via a function that takes some state object as input, as opposed to via calling methods on that object, but that's more of a style issue once you realise that you have to `import {doSomething} from "codemirror"` and `doSomething(state)` as opposed to `state.doSomething()`.
Source: https://github.com/tree-sitter/tree-sitter/tree/master/lib/b...
NPM: https://www.npmjs.com/package/web-tree-sitter
Download from the latest Github release: js file (https://github.com/tree-sitter/tree-sitter/releases/download...) and wasm file (https://github.com/tree-sitter/tree-sitter/releases/download...)
It's great to see upstream maintaining the bindings. I maintain a Typescript/Empscripten/WebAssembtly binding for quickjs and it's more involved than I would like.
Besides providing a WASM build, Emscripten can build a lot of C code to straight JS, which you can use in non-WebAssembly browsers, or in runtimes like Cloudflare or Vercel Edge that support WebAssembly but it's a pain in the behind. Unfortunately the tree-sitter upstream doesn't output such a build but given they already maintain the bindings it wouldn't be a lot of effort to pass the option to emcc (set -s WASM=0 -s SINGLE_FILE=1 instead of -s WASM=1)
if (true) something()
// Comment
otherStatement()
Why not? A comment seems like it isn’t functionally of the syntax tree, but, to the extent that one treats it like it is, it does seem like it’s in the middle of an if statement.
if (true) {
something()
}
// Comment
otherStatement()
https://github.com/lezer-parser/lezer/issues/8
The dev seems mildly hostile to outside involvement too, so I moved on. These days I use Chumsky which is Rust rather than Typescript, but also way more awesome, if you can deal with the often incomprehensible compilation errors at least!
"It's rarely a good idea to jump in and start a big refactor of someone else's library without first discussing the direction you're going in."
But I agree, not hostile, just unwilling to do something, he does not want to do. Even if the other one already put work in it.
E.g. here (I'd forgotten about this actually): https://github.com/lezer-parser/generator/pull/6#issuecommen...
Here https://github.com/lezer-parser/lr/pull/64#issuecomment-1802...
It's nothing major but just emanates "difficult to work with" vibes so I didn't want to spend my time working with a project like that (looks like not many other people do either).
So sure, taking part in the project would mean accepting his way of things.
So difficult to work with? Depends on your expectations I guess.
And the comment above could have been maybe written nicer, but I see it as a defense of "no, I don't need to accept your PR just because you put work in it, you should have checked with me first, before doing it"
Because he had those discussions quite often I think (I follow decelopement from time to time since quite a while)
Which is a bit rude IMO, and not really in the spirit of open source. I never demanded that he accept it. A PR is like "here's some code, let me know if it's ok" not "you should merge this code without question".
Here is what I would have written:
"Hi, thanks for the code but I don't think I want to go in that direction because X Y Z." (he didn't give any clear reasons so you'll have to imagine those).
Anyway he's free to do his thing. I was just explaining why I moved away from that library.
Other people did before you. But I can see how you perceived that reaction to you as hostile. It was definitely a bit rude. I disagree a bit though, that it is not in the spirit of open source. Open source does not necessarily mean open for collaboration. He does accept PRs, if they fit his vision .. and get defensive, when they don't, because he don't want to spend energy arguing why not. He does not have to. So yes, it would be nicer, if he would be nicer. But he is the way he is and still he managed to build a great open source product.