Chat with Andreas Kling about Ladybird and developing a browser engine | Hacker News Reader

Chat with Andreas Kling about Ladybird and developing a browser engine

igalia.com

312 pointssamwillis65 comments

This is a really insightful conversation, it's so interesting how much success they have had by "just"* translating the html/css specification directly to code. The "debugging" of the docs that this is achieving is so valuable. Other larger browsers are going to also gain so much from all this insight.

Well worth a listen!

Slight aside, Andreas has had a bunch of donations and sponsors in the last month (over 300k), and just today announced he has hired a full time dev to help on the browser engin:

https://twitter.com/awesomekling/status/1676896595936198657

> Thrilled to announce that I've hired @KalenikW to work full time on @ladybirdbrowser

> Alex has been with the project since last year, and already done great work on CSS layout, testing infra, and more. Lately he's digging into JS performance and making great progress!

* massive understatement

It’s really important - and people seem to be unwilling to recognize - the huge amount of work done by browser vendors 10-15 years ago to actually make the specs usable for the purpose of implementing a browser (the whatwg work and es3.1 were all about actually building out correct and complete.

The w3c specs, and similar era es specs could not be used to implement a browser, and a lot of the browser engineering work of the time was basically trying to find site breakage and then reverse engineering whatever gecko and/or trident did. It meant things that today might take just a few hours to implement by following the spec could take weeks. At one point I spent a month working out what the correct behaviour and sequence of events should be fired when pressing keys on a keyboard because what the specs said both under specified what should happen, and also was just wrong in some edge cases. It was a truly miserable era, and the people who seem hellbent on bashing whatwg and what not for some perceived slight remain absurd to me.

That's interesting. The people working on these specs seem to be doing an idealistic work even though their employers' interests are more to keep their moats from having the, more or less, two only "HTML engines" viable on the modern web. Is it really idealism or are there interests at play that I don't know about? Is that stuff too low level to be used as a competitive advantage?

Around 2004-2007 most of the key web specification writers (From WHATWG; Hixie, etc, etc)

a) were employed by Opera Software

b) were surprisingly young (often around or even below the age of 18)

Then Google poached most of these people (from their partner; Opera had by then invented the placed browser search engine business model for them; allegedly Google is paying Apple $15B/yr these days.)

A few years later the scope of the web started growing in an insane manner, spearheaded by Google.

This later period was also the same time that the web got deeply javascriptified and largely turned from documents into apps that you download and execute. Before that it had been documents with sprinkles of javascript.

That's some revisionist history.

The bulk of the work was people from Mozilla, Apple, and even Microsoft (those that was mostly in the context of TC39/ECMAScript). Opera did contribute - for similar reasons to Apple: the existing specs were abysmal and making usable browsers was hard simply due to the need to actually work out what a browser is meant to do. Some folk like Anne, Hixie, etc were employed by opera at the time, but to portray that as if that is the majority or "key" work is disingenuous.

> Then Google poached most of these people

As much as I dislike google, claiming you can poach people implies that they're property, and speaks to the anti-poaching lawsuits of a decade or so ago.

> A few years later the scope of the web started growing in an insane manner, spearheaded by Google.

Google does take a shotgun approach to specifications, and one of the biggest issues when I left (the frustration of dealing with google engineers significantly impacted the burn out I experienced) was them spewing out piles of half assed "specifications" that helped whatever individual project they were working on at the time rather than well thought out specs that provide general solution to the general problem space. The constant battle against that, especially when in a world where forums like HN are filled with people just parroting AlexR style talking points of "Apple is fighting [BS half assed spec] because they hate the open web".

> speaks to the anti-poaching lawsuits of a decade or so ago.

There's a large difference between

a) colluding to not hire people between equally sized competitors

and

b) aggressively reaching out and poaching a (foreign) partner's key people, especially when that partner is 50-100x smaller

I'm sure you're aware of this so I'm left a bit confused here.

> Google does take a shotgun approach to specifications... in a world where forums like HN are filled with people just parroting AlexR style talking points of "Apple is fighting [BS half assed spec] because they hate the open web".

Umm... Okay.

> b) aggressively reaching out and poaching a (foreign) partner's key people, especially when that partner is 50-100x smaller > I'm sure you're aware of this so I'm left a bit confused here.

So to clarify, if you ever work for a small company you can't move to a bigger competitor? Or are you suggesting legislation that prohibits large companies paying more?

I'm really not clear what you think is correct here?

> Umm... Okay.

Literally I was agreeing with you about the workload that comes from google's approach to spewing "specs" is excessive, especially when coupled with the attitude of chrome fans.

> I'm really not clear what you think is correct here?

Perhaps I'm saying exactly what I wrote and not your explicit misinterpretation of it.

Edit: @refulgentis: Google launched AngularJS in 2010, for example.

You're not exactly wrong if I parse it carefully with rigid dedication to your desired interpretation. The overall hyperbole of the argument prevents that interpretation from being natural to the reader. (websites in 2011 were just documents with a little sprinkle of javascript here and there is where you 100% lose me)

The people working on browsers care about their users being able to use the internet in their browser. Absent the specification work that just wasn't possible - it took years to get webkit to the level that you could reliably expect all pages to work, even when spoofing the user agent (sites used to user agent gate access to only Firefox and IE, nowadays they just block every that isn't chrome[1]). The specification work helps everyone - without it you have to spend immense amounts of time reverse engineering whatever features other browsers ship if sites start using it, and similarly you don't have to worry as much that if you ship some feature some other browser will implement it slightly differently and then you have to reverse engineer what they did in the feature you shipped without specifying. That used to happen a lot. Similarly producing a specification and going through the standards groups can help identify issues and prevent them becoming a problem before the feature is shipped. It's super easy to go "I have a problem" and then make a feature that solves your specific problem rather than the more general issue you're working with. Back while I worked on webkit, Google engineers were chronic for this, and the problem is they would then frame any pushback as being "you hate the web" rather than "oh yeah, maybe we should have one complete spec instead of dozens of slightly different overlapping ones".

Overall I'd say the bulk of folk I worked with from pretty much every company I interacted with cared specifically about the web as platform independently of whatever their employer might be doing (Google, Microsoft, Apple, Mozilla, Opera, Yahoo, Facebook, are all companies I recall having significant amounts of communication with, as well as various academics, oss projects, etc and there are very few times I encountered bad faith behavior).

[1] Commercial site developers, especially "enterprise" ones, are absolutely abysmal developers. The only thing that has changed in the last few years is them switching from IE only to Chrome only.

I think 'chrome only' is offset a little from stakeholders who use apple devices, so safari gets some attention.

The WHATWG approach is not without its downsides, even setting aside the ideological points (e.g. that with no fixed releases you can never be done in any sense):

The spec is good and makes you life easy (well, as not-tremendously-difficult as it could ever be) if you go with the LibWeb/Ladybird approach of transcribing it into code line by line; then there is a yawning abyss of performance engineering and conformance bugs a mile wide; then there are all other real implementations, which do not follow the spec word for word for reasons of efficiency. There are a myriad ways to do an LL(1) parser, all well-known to be equivalent; but you are on the hook for proving the validity of every little discrepancy with the WHATWG algorithm that you’ve had to introduce for one reason or another.

Speaking as a person who has implemented numerous web specs, you generally do start with a close approximation to the actual spec - I believe my original Map/Set implementations in JSC were exactly that.

Once you have a complete and correct implementation, then you can actually measure things and work out the critical parts and make them perform reasonable well.

But it’s really hard to do perf work if you don’t already have a reference/known good implementation.

And that’s what LibWeb is doing, which is giving them a workable baseline. Certainly enough that I’ve seen a few Kling videos where he abLe to focus on perf issues. A lot of the core performance bottlenecks devolve to “cache more”, and then devolves to those most hallowed of problems: cache invalidation.

> Andreas has had a bunch of donations and sponsors in the last month

I believe that Igalia was the source of $100k of that! Awesome that they, as an open source browser consultancy, can give back to projects like Ladybird.

What would be their incentive to shell out 100k like this? Appreciation?

Of the 3 announced, 2 of them were anonymous and the 3rd was Shopify.

Oops, that's embarrassing. I must have been thinking of sponsorship for Web Engine Hackfest...

There's an Eric Lippert post from a few years back about working on the C# compiler, where he followed a similar spec-guided approach, with simpler benefits for both making the code clear and making sure the spec was reasonable:

> [...] I wrote all the conversion logic in Roslyn, and I did so for the most part by carefully following the specification. This had two purposes; first, to be sure that I was implementing the actual specification, and second, as yet another chance to ensure that the specification made sense. For much of the conversion code I have methods where each method implements one line of the specification, for this reason.

https://ericlippert.com/2015/05/26/a-different-copy-paste-er...

Wasn't one of the lessons of The Mythical Man Month that building from a spec produced good software? That was talking about System/360, my recollection is that most of the effort went into creating a very detailed and internally consistent specification/manual for the system. Implementation followed the spec, and the results were long-lasting and solid.

It’s also “conventional wisdom” that (especially?) with the web, there’s The Way It Should Work, and then there’s The Way It Actually Works, which is all the quirks, etc.

I’ve not completed an entire server or browser, so not speaking from implementation experience, just the word on the street.

Is that stale information now?

Mostly stale, yes, thanks in some part to the solidification of the specs that the interview touches on. “Quirks mode” is still a thing to some degree in browser engines, but you’ll get farther than ever adhering to the spec.

Good to know.

Listened to the full interview, and definitely rooting for Andreas et al with Ladybird.

The whole whatwg ES3.1/5 saga was dealing with the specs saying what they wanted the world to be (which they did badly and incompletely, and frequently in direct contradiction with) and what browsers needed to do (the “quirks”). The spec as produced by whatwg and the browser vendors actually matched reality, didn’t have ambiguity (or rather any ambiguity that was missed is considered a spec bug/error) and could actually be used to make a browser.

That’s heartening to hear. I appreciate you taking the time to point this out.

I thought it was the opposite, with Fred Brooks championing “build one to throw away” until software processes had developed to a more flexible, iterative model. But that’s because usually there is no trustworthy spec, and in this case, they can observe a working web browser in the wild to get requirements nailed down.

Right - I’m sure he’s the source of the aphorism, “plan to build it twice - you will anyway”.

I think having a great, self consistent spec to build from is awesome, but it’s rare that a spec doesn’t require revision after contact with reality; and at that point, it’s just another kind of software.

> But that’s because usually there is no trustworthy spec

This is the key - most of our writing on modern software comes from people building things with requirements that are unclear. There are pockets of the software world where detailed specs work wonders.

One of my professors in college had made a career writing these sorts of specs in a certain niche of robotics. It'd worked fabulously for him precisely because nothing was new or innovative, it just required exacting craftsmanship.

If you can explain a complex system precisley to humans then it is trivial to explain it to a computer.

Fundamentally defects are deviations between a user’s expected behavior and actual system behavior. That presumes there is an understanding of what users expect, and while a spec can model that, a spec can also bake rules in stone that break user expectations. So I would argue it is not the spec as such that increases quality, but rather having a documented reference of how the system ought to behave, whether written beforehand like a spec, afterwards as documentation, or during in the form of high quality tests.

If there is no separate record of how the system should behave there is no clear line between bugs and features, and the software will grow warts based on the vagaries of the ticket system. Often at that point someone proposes a rewrite to clean things up (aka “build it twice”), but unless the second time a proper spec is created the end result after a few years is often equally filled with warts. Quality is a process and not a feature, and the same process will usually produce a similar quality. I’ve seen the same software rebuilt four times, and only the fourth time was it preceded by a proper spec resulting from extensive user conversation. That fourth castle stayed up.

> Google throws hundreds of engineers at this [building Chrome] and they have full-time people working on it around the clock.

It was many hundreds already about a decade ago. I haven't done any recent research (the Chromium Git repo is where you'd get the source data to analyze), but I assume it's in the (several?) thousands now.

Edit (ran into some problem replying to the comment below):

An in-depth analysis performed today would need to consider that some of the active chromium.org committers are working on non-browser-related things like Chrome OS.

That said, I'm assuming 75+% of active chromium.org committers don't work on those non-browser parts. I also consider the Chrome Web Store a part of the browser - but I can't imagine that needing lots of developers in comparison to these large numbers, even when you start thinking about the abuse aspects.

Having said that, it should be easy to filter out those by just discarding certain directories.

The way lots of things are integrated into "Chrome" as an organization and a product definitely means a lot more people touch something you can call "Chrome" now. Chrome OS, Chrome Web Store, etc.

Last time I looked the 7 day active committers was about 750. So with holidays, people who don't commit for a week, PMs and other management etc, my guess is around ~1000 people, maybe a bit more. I doubt it's several thousand. Maybe if you include ChromeOS?

Thanks! That is still a lot to compete with, but it's not that much larger than what they had a decade ago.

And loads of them were certainly working on mostly homegrown web standards that went nowhere.

I think Kling and his growing crew has a shot here.

I am one of those hundreds. We work with a lot of issues smaller browsers might not have met yet. We could also be very inefficient.

(Sorry [for the following]...)

> We could also be very inefficient.

I really hope so - for the sake of the web.

No need to be sorry, I love the work and get a great satisfaction from protecting our users. Might not be what you want to do, but for me it is great.

Andreas Kling is a true hacker, one of the remaining few. A beacon of the hacker spirit. May he sail long in the winds of the incessant specs and guide us. I hope he inspires thousands more to eschew LeetCode and be a true hacker.

I'm looking forward to the Servo-Ladybird browser wars.

(Irrelevant post, but this was a missed opportunity for an article with the title "Kling on Ladybird")

Chrome and Firefox are absolutely gargantuan in terms of their functionality and also the decades of fine tuning.

Building a new browser seems akin to building a Saturn V rocket.

To be practical at all you’d have to lop gigantic trunks of functionality off it.

I'm not sure I agree with this metaphor but it's too tempting not to propose:

Labybird/LibWeb is to Chrome/Blink as Saturn V is to SpaceX Starship...

Not sure where that puts Andreas, hence my hesitation.

Maybe you mean the SLS (or Rocket Lab) instead of Saturn V

An observation shared by many.

If by web browser you mean web engine + chrome, than it is a big task, but notas big as Saturn V. Have a look at Netsurf, Ekioh Flow. Their teams are small, yet still capable.

Today i learn you need to explain a reason to the NYT before you can cancel a subscription! Incredible!

Q: Why are you canceling our wonderful subscription?

A: Because I don't want to be billed or to pay the subscription fee anymore.

The answer isn't the issue. It's the fact the question needs to be asked as a requirement.

Just yesterday I was thinking about Ladybird, and I claimed that it was a fool’s errand, for reasons I’ll explain in a bit. What I didn’t know is if Kling recognized what he was doing, and was just not taking it very seriously, or if he was way over his head and was kind of oblivious.

I enjoyed reading this interview because it makes it very clear that Kling has a lot of experience doing this, and understands that’s it’s difficult, and is doing it because he enjoys it.

The reason I don’t think Ladybird will ever really get to the point where it can render an arbitrary site perfectly, is based on my experience as a web developer. I haven’t been doing web development for very long, but there have been a couple times in the last couple of years where I’ve stumbled upon browser layout bugs where Chrome/Firefox/Safari render something different. These are normally not issues where a browser completely misses something (ie, every browser supports flexbox these days). Rather, there are edge-case bugs that appear just by combining simple web components in simple ways. Chrome and Firefox handle a table with percentage height differently. Or a margin on a child which is larger than the parent’s width. And these are bugs that have existed for years. And for every edge-case like that where Chrome and Firefox differ, there are 10 more where the behavior is unintuitive or not defined by the spec but Firefox and Chrome have standardized.

But maybe, since Ladybird doesn’t have an existing buggy implementation, they’ll be able to implement the spec properly quickly the first time. I think that’s an interesting perspective that Kling brings up in this interview.

The other thing I want to comment on is that it’s really hard to tell how much progress they’re making.

> We've taken a very vertical slice approach to adding functionality to the browser… We tend to find a webpage that we want to improve for whatever reason. Maybe it's cool to get the New York Times to render, and then we just spend a bunch of time fixing as many issues as we can for that site… And that might not be the most structured approach to it, but it has been very, very good at keeping people motivated and excited and engaged in the work

This means that Ladybird “supports” a ton of features. They support flexbox! They support grid! But pretty much all of these features are only partial implementations. Kling says in the interview they support svg, and then later explains that they’re going to have to re-write the text-rendering system to get text-on-curves working. Well, to me, “supports svg” means “we have implemented the entire svg specification.” When Kling is using it to mean “we have some portion of svg’s work. We don’t crash when we see an svg element.” I’m not saying Kling’s definition is wrong, but it’s another factor that makes it difficult to judge where the project is.

To start to wrap up this (long) comment, let me restate the four claims I’m making. (This is not the conclusion, this is the summary before the conclusion, hang on.)

* Ladybird is developed differently in two ways: vertical slices, and being implemented from scratch in modern times.

* there are people who are misled by the vertical-slices and Kling’s use of “support” to believe that Ladybird development is a lot farther along than it is.

* There are people like me who think that Ladybird will never complete with existing browser engines (we also have a hard time judging Ladybird’s progress for the reasons stated above).

* These same reasons may allow Ladybird to reach compatibility with other browsers.

I have a lot of respect for Kling, and I think it’s cool that they’ve been able to accomplish so much “for the fun of it”. As mentioned in the interview, this will have short-term positive impact even if it never matches Chrome. If it wasn’t clear from how long this was, I’m actually really enthusiastic about the project, and I’ll be looking at ways to contribute in the future.

You should check out his browser hacking videos. It's pretty shocking how much they have already and how fast it's improving. I have no doubt they will eventually be able render the vast majority of websites correctly. The real question is whether they will ever be able to make it performant enough that people who don't care about browser hacking will choose to use it. That's where a lot of the engineering hours in browsers goes.

> It's pretty shocking how much they have already and how fast it's improving.

My experience with large software projects is that there is an 80/20 effect where the first 80% happens really fast and easy and then the last 20% gets harder and harder and harder.

Most of the challenge of software engineering is dealing with the scale and complexity of the codebase itself. So greenfield development is always going to be faster at first because there's simply less code to go around. But as you keep pouring code in, you are creating the complexity that makes software development hard.

I definitely wouldn't do a linear projection of their current velocity and assume the whole project will keep moving that fast.

All very good points. Still, I hope they pull it off for the shear inspirational value.

I say something similar to my friends and colleagues all the time regarding my “professional” software projects. It takes 20% of the time to do 80% of the work. But that last 20% of the work takes 80% of the time.

Essentially the core functionality doesn’t take very long to build out. But the final polishing, details, edge cases, and bug fixes soak up a huge amount of time.

I suspect that the momentum behind the project is having a group of core developers (Kling foremost) that have complete knowledge of all the moving parts. This allows a small team to make changes quickly while keeping the whole cohesive and without the slowdown of deliberative bureaucracy. Maybe it’s like the human organizational equivalent of L1 cache. It’s like riding a wave, a situation that can’t last forever. I have been part of teams that did the work of much larger teams. It’s great when that happens. The last mile might be less fun, but this entire OS is a labor of love.

> The reason I don’t think Ladybird will ever really get to the point where it can render an arbitrary site perfectly, is based on my experience as a web developer. I haven’t been doing web development for very long, but there have been a couple times in the last couple of years where I’ve stumbled upon browser layout bugs where Chrome/Firefox/Safari render something different.

So what does perfect even mean in this context? Who's right, Chrome, Firefox, or Safari?

I 100% believe Ladybird will eventually be able to render arbitrary sites in an acceptable format at some point (potentially within the next year or two). It's pretty remarkable how many sites look very close to acceptable already. Any weird edge cases where Ladybird renders stuff differently from a big browser won't mean it's rendering the site "imperfectly"; just like how when Firefox and Chrome do different things, it doesn't mean either one is "correct".

Well, that's the thing when we're talking about hobbyist FOSS development. There's no bosses and no money involved to get a developer, point them at specifications, and say "implement this top to bottom." Instead you're going to get people implementing the parts important to them and/or fun to implement and/or relatively easy. Can a new browser engine compete with the big boys when it's developed like that? No, but maybe it can get good enough for casual use, and since it was mentioned in another thread here that Kling actually has hired a dev to work on Ladybird/LibWeb, maybe the whole thing's moot anyway and we'll start seeing things like text on an SVG path and other stuff which I imagine is not so fun to implement get done in the near future.

There is money involved (oddly, you go on to mention this in the comment)

> There are people like me who think that Ladybird will never complete with existing browser engines

Assuming you mean "compete": AFAIK this is not a goal of Ladybird.

> The reason I don’t think Ladybird will ever really get to the point where it can render an arbitrary site perfectly [...]

I similarly don't think 100.0% bug-compatibility with Chrome is a goal.

> we also have a hard time judging Ladybird’s progress for the reasons stated above

There are plenty of available test for web browser stack, allowing to compare browsers and their implementation of standards, including Ladybird.

  * Acid3: http://wpt.live/acid/acid3/test.html
      * All browsers score 100/100
  * HTML5test: https://html5test.com/index.html
      * Chrome: 528
      * Firefox: 491
      * Safari: 471
      * Ladybird: 266 (in december 2022, I couldn't find more recent figures, but I wouldn't be surprised if it were significantly higher today)
  * Test262 (JS engines): https://test262.report
      * Chrome (V8): 86%
      * Firefox: 85%
      * Safari: 85%
      * Ladybird: 87% (behind on language syntax, significantly ahead on built-ins, internationalization, and AnnexB)
  * Web Platform Tests: https://web-platform-tests.org/
      * I don't have figures for this, but this is integrated in Ladybird's CI pipeline
  * Probably others

The point is: it is quite possible to judge and measure overall progress, and the vertical slice approach they use lead to constant overall improvement.

I will still not try it out.

There is too much security issues by just browsing the web.

Security is hard and thanks to browser we are moving a lot into quite secure sandbox.

Reply on news.ycombinator.com