Ask HN: Why don't browsers pre-include popular JavaScript libraries? | Hacker News Reader

Ask HN: Why don't browsers pre-include popular JavaScript libraries?

15 pointsdavnicwil16 comments

I've often wondered why there isn't a standard for browsers having a pre-populated local cache of very popular JS libraries. To name a few examples: jQuery, Lodash, React.

It seems like the benefits of this would be significant for both user and developer - faster loading, lower bandwidth, quicker builds, etc.

It also seems eminently doable, technically. One possible high-level implementation off the top of my head:

- A standard index of the most popular libs is maintained.

- You pull them in via the script tag, with a new lib="name@version" attribute. Older browsers can use the src tag as a backup.

- SPA build systems like webpack et al are aware of the standard and can just not bundle libs pulled this way, while keeping the require() and import syntax.

I do, however, see some pretty big not-strictly-technical difficulties:

- The index would have to be administered by some entity. Who would that be, and who would pay for it?

- Who defines what the most popular libs are? Whatever system is used, how do you prevent it being gamed to the advantage of particular libs?

- Trust and security. A similar set of issues npm faces. Although that being the case, is this really introducing a new problem? It also seems like a much narrower problem if we're only talking about a small set of very popular libs.

- Maybe some issues around licensing, bundling code into browsers etc. But again, if it's a smallish set of only very popular libs, this seems like it could be dealt with individually for each one and its authors, maybe as a prerequisite for getting included in the index.

These difficulties are not insignificant, but nor do they seem insurmountable and also the benefits do seem genuinely worth it.

It just appears to me to be something that should so obviously have been done, that there must be really good reasons why it hasn't been - maybe including those above, maybe technical, maybe others.

Does anyone know if any serious attempt has ever been made at a proposal to do something like this, even if in a non-standard way by an individual browser vendor?

Extensions like Decentraleyes and LocalCDN are doing things like this:

https://addons.mozilla.org/en-GB/firefox/addon/decentraleyes...

https://chrome.google.com/webstore/detail/localcdn/njdfdhgcm...

The aim is improving privacy - your browser doesn't need to request the libraries from a CDN, so you can't be tracked by them.

This does also bring the performance gains you mention.

I’ve thought about this a lot, but mostly in the context of reducing bandwidth use of mobile devices.

Bundling with the browser wouldn’t be enough in my opinion— the ecosystem moves too fast and you’d risk bundling outdated code.

Instead, cache common assets you’ve downloaded naturally while browsing. Use some heuristic to keep the most heavy and frequently used assets cached.

I don’t think you’d need an index. You could instead use a cryptographic hash of the asset’s content as a cache key, and the page can specify further parameters for ensuring the integrity of assets cached from external sources.

Browsers already do cache assets, but to my knowledge privacy concerns result in this cache scoped per-site.

My understanding of one of the primary privacy concerns is that you might cache a dependency which either intentionally or unintentionally acts as a tracker. If site A and site B share a dependency C, either site may be able to deduce that you visited the other by observing whether you had to retrieve C or if it was cached. I’m sure a dedicated party could craft a system which fingerprints users by giving them unique dependencies to cache. I think coming up a solution to this problem would open the door to something like this becoming a real possibility.

On using the cache for tracking, I don't think that'd be an issue with what I'm thinking about here because all the libraries in this index would be pre-populated in this special browser 'cache' (maybe cache is a misleading term here, store or repository might be a better one).

Or said another way, there'd be no information to be gained from combinations of cache hits & misses because there would only ever be hits.

There can be hundreds of different versions for each popular library, and as a dev, I need customers to use exactly the version that we tested against. There’s a whole ecosystem and tool chain around lockfiles and deterministic builds for a reason.

And specifying a version range isn’t sufficient, because humans will and do get it wrong - either the one specifying the range or the one publishing a new version.

Ultimately, browsers cache static assets anyways. So this idea only helps the user’s first page load for a given website.

Edit: I’ll add that at my company we looked into preloading assets that are commonly used across teams’ web apps. And this is why we didn’t build it.

The many versions thing does seem like a problem just in terms of size on disk but perhaps a one solution to this would be storing the libs as a series of git-style deltas starting from a root version which are then 'resolved' on the fly.

I feel like the lockfile etc situation is pretty solvable just by making the relevant tools aware of the new standard. I.e. just a different implementation of an npm-style interface.

On the existing browser cache point, I guess the difference here would be that this special cache would never be cleared, and also being a standard would probably mean better reuse across apps. It's true though that otherwise it's not too far away from everyone just agreeing on a standard CDN or something, just that maybe making it an actual web standard as part of the script tag itself would make this more likely. In fact, I guess you could even just implement it like this under the hood!

>The many versions thing does seem like a problem just in terms of size on disk but perhaps a one solution to this would be storing the libs as a series of git-style deltas starting from a root version which are then 'resolved' on the fly.

Just sending the scripts with the site is still far, far simpler, and guaranteed to work (provided the user hasn't turned off JS.)

What little, if any, gain in performance doesn't seem worth the extra complexity.

Definitely agree here that the complexity / benefits tradeoff is not totally clear cut in favour of this, and it could well be the reason nothing like this has been proposed (that I know of)!

That said, aside from the UX-y benefit of load time, under performance gains I would also include saving the processing and pushing around of all those bits at web scale. That's got to represent incredibly impactful absolute energy savings, and bandwidth cost savings.

You still have to do extra processing to know whether the end user has the right libraries installed, and to deal with fallbacks if necessary, that's going to involve at least an extra round trip.

Also, all of this has to occur before the page starts rendering, which may not be an issue if the user has the necessary libraries, but if they don't, you still have to send them anyway. They could have them cached from a previous visit, but that's how browsers already work. The fact is, most JS libraries are small enough that they download almost immediately, it's what they actually run that's slow.

All that said, I do think something like this would work great for Webassembly dependencies. What's being described is basically a dependency or package manager type of thing. I can see browsers in the future shipping with JS, Python, C or whatever runtimes as WASM blobs. All of the same issues still apply (including latency issues when downloading as a fallback,) but programming language runtimes and libraries tend not to move as fast as javascript frameworks.

But imagine the possibilities if you could essentially embed the source code of any application in any language and have it interpreted or compiled and run in the browser, maybe with little more work than some script tags?

Just to be clear the proposal I'm making would be that all these libs are pre-included in the browser. This wouldn't just be a special non-deletable cache where the libraries are pulled lazily on first use. It'd be an entirely separate concept from the current browser cache of JS files, in fact. Potentially 'cache' is a bit of a misleading term here.

There's probably a bit of a question of what to do when new versions and/or new libs are included in the index (probably download new ones periodically or upon a notification) where there could feasibly be some lag time where that lib/version isn't available yet. But that'd be an edge case situation.

Perhaps as an orthogonal feature, implement a distributed CDN using all the browser cached assets across other browser instances (using an approach like IPFS or BitTorrent) to parallelize the object fetch using a DHT.

It could potentially use the HTML <link> tag's integrity attribute See: https://developer.mozilla.org/en-US/docs/Web/HTML/Element/li... for content addressable DHT lookups

Every reasonably complex site I've worked on uses some form of bundler so even if a site uses a bunch of common libraries they are all compiled together into a big JS file.

I think that a lot of the way websites are developed would have to change to support such an initiative.

Webpack allows you to get some dependencies from global window but IIRC it is tricky to set up.

That said if it were a web standard then webpack or one if the newer bundlers could figure it all out automatically based on your dependencies.

The problem is “who is the source of truth”.

A hash comparison system “get me the library that hashes to xyz” might work, and then you pick your “provider” as a fallback but the browser can pull from cache if the hash matches (even if it came from another provider)

Just thinking of that the problem then is which to cache. How to know “React” is important? I guess browser could keep tally and keep the most often requested ones cached.

yep, the way I'm thinking about this it'd certainly be the case that bundlers etc would be aware of the standard and act accordingly.

That's the most unique idea I've seen today.

It's is the same concept of OS drivers but for the browser ?

Like linux including python by default ?

Pretty much exactly this, and it's a good way to think about it: the browser as an OS and this as just pre including popular 'userland' software with distributions.

this is actually an interesting idea

Reply on news.ycombinator.com