Hacker News Reader

Comment by spiderfarmer | Hacker News Reader

One of my clients loads a 4.5 Mb bower.js (including Angular with a lot of components and jQuery), they also include an extra jQuery script, a full jQuery UI and several other scripts on each pageload. Nothing is minimized. The bower file alone has 300k in comments.

The CSS file is also nearly 1 Mb.

It's just a simple website with some forms.

They have 2 developers working on the site, a scrum master, a project manager and 2 testers but somehow they can't find the time to find out which legacy code they can remove.

It's only after I pointed out that they have an exceptionally high bouncerate on their (expensive) Adwords traffic and the slowest pageload time of all their main competitors that I managed to get some priority to optimize their site.

Serious question: are there any tools that can scan a full website for unused code and unused CSS?

View on HN

Chrome 59 has a coverage tool built-in: https://developers.google.com/web/updates/2017/04/devtools-r...

That's interesting! Do you know if it is possible to access this data from an extension? I would love an extension that crawls through the website and combines the data for all pages.

Note: The color-coding is likely to change in future Chrome releases. <-- Well, someone eventually found out that some people cannot distinguish red and green, but it was too late :-D

I am a little surprised to find such an issue in Google software as it is a topic for first semester CS undergraduates ;-)

Wait what? CS students learn color about color blindness now? I really wasn't in the good classes!

I think it's a joke.

I actually learned that as a CS student.

If it was actually part of the curriculum I sure hope that it was in some design centered elective class. IMO if your CS degree spent time teaching that as part of the core curriculum, they missed an opportunity to put more Math, PL theory, and interesting algorithms in there, because there's more than can be sanely covered in any one curriculum.

Since it's accessibility based, it's more laudable than teaching CS students how to center a div, but it's not like it really requires a mentor of some sort to express the nuances of, right? Or even if it does, it's still design.

It's some years now, and there were at least two courses in which it could have been. The one was elected, HCI. The other was not, it was an introduction into graphics and audio, and since you need to understand basics of human perception to understand compression in that area (jpg, mp3), they talked about stuff like that.

A good CS degree definitely has the space to teach some basics in that area. To mention Gestaltgesetze, to explain human perception a bit, and give an introduction into usability. You do not get a useful developer in the end otherwise

> You do not get a useful developer in the end otherwise

Not all developers do stuff with UI, and of those that do not all do anything with a UI that is actually graphical beyond a terminal.

> since you need to understand basics of human perception to understand compression in that area (jpg, mp3), they talked about stuff like that.

That is a good reason to teach it, and counters my overly assertive original comment.

I learned that in my 1st year HCI module at University nearly 20 years ago. It shouldn't be a joke. It should be normal.

I taught a ton of a11y stuff in my intro to web classes. Even did screen reader demos. I wasn't the only one. There are lots of folks teaching this stuff. Not a critical mass, but I'm so thankful it's happening. :)

I didn't learn anything graphics-related in my CS degree. Seems like more of a UX or software development topic than a CS topic.

I learned it in both my computer graphics course and my user-interaction design course as part of my CS degree (German university). None of these courses are required, but they are popular enough that the most important lessons were pretty universal knowledge among students.

Seen the same on maps where green is a good route and red is a dangerous route. Switched to a blue -> light blue -> yellow -> orange -> red -> black scale with great success. I also believe chemistry uses blue to mean safe, not green.

On a minified app it only seems to show line-by-line coverage of the minified files and not the source maps :(

You can pretty-print the file through the Sources panel.

Off-topic, but:

> Full-page screenshots. Take a screenshot of the entire page, from the top of the viewport to the bottom.

I'm surprised to see the Chrome developers not know what 'viewport' means.

Even more off topic but I'm surprised this isn't a consumer feature. There are so many sketchy chrome extensions that try and do this.

Even more off topic, but after Firefox (before Chrome) had introduced this feature in dev tools, they also recently introduced it as extension (and it might even work on Chrome)

I don't think more powerful tools are the solution here. The problems and solution are obvious. The root problem is that many "product" teams will take any 10x improvement from tooling and adapt by being 10x more inefficient. Because they can.

Hardware (laptops, phones), and browsers js and rendering engines, are all 10x faster than they were 10 years ago, giving 100x to 1000x overall speedup. Think about that, it is truly incredible. But web pages are also 100x larger and less efficient than 10 years ago. This is absolutely not justified by productivity increases. This is just how the world / humans seem to work.

"Work expands to fill the time allocated for its completion." This holds true for computer scripts executing a workload as well.

Developers tend to over time expand their programs to fill available memory and take up available CPU, too. My currently open text editor's memory footprint is apparently 36.6MB, and it is displaying a single file composed of ~1000 characters.

The scary thing is that I actually consider that to be remarkably small. My current Emacs session (recently opened) is just over 200 MB is size. And Emacs isn't even considered a hog anymore compared to the real hogs like Atom.

Remember when Emacs meant Eight Megabytes And Constantly Swapping?

> They have 2 developers working on the site, a scrum master, a project manager and 2 testers

... and daily standups? so busy working that there is no time to do the job

Exactly. And the people around the developers are busy coming up with things that make them look busy.

> Serious question: are there any tools that can scan a full website for unused code and unused CSS?

From my limited knowledge of your situation this seems like the last move to take. I would try the following, in roughly this order:

Serve libraries (Angular, jQuery) from a CDN to at least have the chance of hitting a browser cache on a visitor's first encounter with your site.

Use a minifier to strip all the unnecessary comments and whitespace (in addition to other benefits from minimization).

use gzip to compress the remaining minified files.

Those alone will likely significantly impact your load times. At this point, the remaining dead code should be relatively small and you can look into tools for surgically removing it.

As for JavaScript, Google Closure Compiler does some dead code elimination (DCE, a.k.a. tree shaking in Lisp lingo) as an optimization but it unfortunately doesn't tell you which parts of the code is dead. It also does not to DCE alone but does some aggressive size optimizations with it so, again unfortunately, it cannot output otherwise-identical code with only DCE applied.

This part is more of a shameless plug, but I believe public/industry/developers should be aware of what tools & techniques academia develops so that the researcher's efforts can be more useful. There are some static analyzers for JavaScript such as JSAI, SAFE, and TAJS (disclaimer: I'm working on a project related to JSAI on the lab which developed it). These static analyzers can discover the dead code if given all the entry points however:

- They don't have facilities to report the dead-code AFAIK (this is a trivial thing to add _in theory_, if there are parts of the program _unvisited_ by any of these, that part of the program is guaranteed to be dead). If somebody wants to add such a facility to any of the tools they are welcome. In case of JSAI, I'm willing to provide them all the information I can.

- They are conservative (e.g. sound) tools and may find too few dead code results.

- They don't play well with `eval` & friends in general although they try their best[0], the web frameworks rely on eval-like highly-dynamic approaches a lot to be generic enough and that hurts precision of such analyses dramatically.

[0]: https://pdfs.semanticscholar.org/8140/feec021818815c55a43b54...

[1]: https://www.cs.purdue.edu/sss/projects/dynjs/eval-TR.pdf

> Serious question: are there any tools that can scan a full website for unused code and unused CSS?

We're starting to get there.

JS Modules gives us static exports/imports, so we can statically analytics the dependency graph and identify functions that aren't used and do tree-shaking to remove them. Rollup and Webpack2 already does this, but there's more to go.

CSS Modules creates a contract between the your CSS and what uses it. As you have to import it, and exports are static, you can understand which style files are imported where. Next step would be to understand which classes are used inside that module and strip those that remain unused. This is theoretically possible (providing you use a more 'statically safe' syntax), but tooling hasn't yet popped up to do this.

Yes, it's called "tree shaking"[1], modern Angular apps can be cut down to a around 50-100k (assuming nothing heavy feature set). Find build tools that can do it, like Webpack or Rollup.

[1] http://blog.mgechev.com/2016/06/26/tree-shaking-angular2-pro...

Not really an answer to your question, but this might be useful: http://youmightnotneedjquery.com

Yeah. Except you need to do Ajax and fetch doesn't cut it. So new lib. And manipulate the dom with something better then the browser API or you loose your mind. So new lib. Then normalize browser events. Oh wait, you can do that manually. But you are writing a new lib. Eventually the code will grow to be the size of jQuery anyway. Only not as well tested, documented and cached.

> Yeah. Except you need to do Ajax and fetch doesn't cut it. So new lib.

Why doesn't fetch and its polyfill cut it?

Polyfill. So new lib.

And even then. You need to encode params manually.

And of course no middleware, so for special decoding or repeating headers, you end up writting a wrapper.

Programming is mostly creating functions and interfaces to decompose problems into reusable parts where reuse makes sense. Nothing wrong with creating a simple function for making XMLHttpRequests or for fetch API. Most of the time there's no need for the level of generality of jQuery interfaces.

Just by making a function you're not creating a library. Sometimes I find DOM API verbose too. Nevertheless the fix does not need to be a interface like jQuery to completely hide DOM API under another interface. It may be enough to extend the HTMLElement prototype a bit or write a function to "compress" the code a bit and make it more readable.

Just by making a function you're not creating a library.

Have you seen what passes for a library on npm?

Good point. :D

> it may be enough to extend the HTMLElement prototype a bit

You lost me here. Anybody extending standard objects cannot be taken seriously.

Do you realize JavaScript is a prototype based programming language? It is a major and very useful feature of the language. There's no reason not to use it. There are some things to be careful about with interop if you need it, but that's it.

We've already learned from our past mistakes changing the prototype of the standard library - the Ruby community learned the same thing with monkey patching their standard lib too. The maintenance overhead far outweighs any immediate benefits.

What maintenance overhead? Current and future costs are reduced by making code simpler and cleaner in the first place by using prototypes efficiently.

Also, to be clear, I'm suggesting adding new methods to DOM object prototypes, not changing behavior of existing methods.

Applications may be able to do this in a controlled manner. Library authors never should.

No, a polyfill is not a library. It's a dozen lines, not a thousand.

> No, a polyfill is not a library

Yes, it is.

> It's a dozen lines, not a thousand

A polyfill can be any size, but there's no size minimum to be a library.

Some people are mistaking a book (actually a module) for a library.

Overly-literal response.

A polyfill is a short snippet to implement a missing feature. Notably almost all recent additions to JS and DOM have been designed to be polyfillable where possible. Generally that means under a thousand lines, by the way.

A library in this context means a set of functionality built on top of the platform, like jQuery, with its own API, etc. "Use a library" and "use a polyfill" are not the same advice.

A polyfill is not a library.

> Polyfill. So new lib.

So, you don't like the polyfill, created to the specification. Fine, but depending who you target, you might not need it. Plenty of modern browsers have good support.

> And even then. You need to encode params manually.

Or use the Request constructor.

> And of course no middleware, so for special decoding or repeating headers, you end up writing a wrapper.

Request constructor will let you handle the headers quite nicely.

Also, vast majority of users will have a cached copy of jQuery from Google CDN already.

Now bundling it with the rest of your JS, that's indeed a mistake.

CDNs are a lie. The privacy implications outweigh the performance benefits since there are many, many, many CDNs and many, many versions of jQuery. It's rare for users to benefit from caching.

I believe this is true, but the world will only be convinced if there are metrics for this. Are you aware of anything that verifies it?

Also, the performance benefits from free CDNs that you have no service level agreements are unsubstantiated at best, but you'll notice very quickly how much your site is broken when your CDN is over capacity.

> Yeah. Except you need to do Ajax and fetch doesn't cut it. So new lib.

Ah, so this is how front-end got the way it is.

You can implement an minimal ajax interface in js (no jquery) in less than 30 LOC. But I grasp what you mean, it not just that. There are other concerns and requirements and you do not want to reinvent the wheel everytime...

I am so sick and tired of this argument. Having done it several times, no, you do not end up with something as big as jQuery, not by a long shot.

As big as jQuery? Gzipped it's 27k, a ridiculously small size.

Ridiculously small? That could take 10 s to download on a 2G network.

Yeah, sure, most sites today are totally browsable in 2G.

Event the damn native APP for hackernews, using only JSON, is not usable in 2G.

For me HN seems to be one of the few websites that is usable on 2G.

Network bandwidth usage is also not the only metric of size we care about. After it's ungzipped, it needs to be parsed by the JS engine, and more code = slower execution.

Sure, if all you do is some onclick() and a few appendChild().

Plus for the time you spend chasing compat issues, writting wrappers, choosing small libs and packing it all together, you could have done your site. For less than 40ko.

People are not blinking at including react ecosysteme, which is huge. But attack jQuery ? Seriously ?

> chasing compat issues, writting wrappers, choosing small libs and packing it all together, you could have done your site.

I don't know about you, but usually my projects last longer than a week, so those sorts of savings don't represent a significant chunk of my overall development time. Also, I'm not a shit programmer, so I can actually write a for loop that does what I want on the first try. I've spent more time trying to figure out jQuery's goofy syntax and what the hell it's doing with AJAX queries than it has ever saved me in development.

>Eventually the code will grow to be the size of jQuery anyway.

Um what? jQuery is huge. You won't possibly use everything in it.

Well, define "huge." It's like 28KB that your user may already have cached if you use a CDN. https://mathiasbynens.be/demo/jquery-size

I meant huge in terms of functionality. It supports so many things that you can't possibly need everything.

Oh. Well that's true, but I think the point is more like you end up implementing some significant subset of that, except with homespun code that isn't tested as well.

Event + Ajax + DOM is most of jQuery. And is most what your webapp is. Now I'm more of a fan of Vue.JS + axios myself. But if I have to go minimal, 40ko of jQuery is nothing and save so much trouble. All the "you don't need jquery" thing is done by people doing prototypes that are not tested on most browser or assess in term or ressource vs reward, which is a lot of the projects in the JS community.

> Except you need to do Ajax and fetch doesn't cut it

curious as to why you think this? fetch is pretty convenient to use. We created our own wrappers around it, but that basically just consists of ajax = (url) => fetch(url).then(res => res.json())

Exactly, you had to write a wrapper. And the next project will do so. And if you want to save time, and improve reliability, you'll test it, and write doc for the 3rd. And the new team member.

Or you can use the few ko of jQuery.

Good luck supporting old android browsers, IE and dealing with the inconsistencies between browsers. It only takes one look at the jQuery source code to understand why it's not a good idea to "roll your own jQuery".

Simply don't target old browsers. If people don't use JS, or don't have a modern browser, feel free to go elsewhere. I don't target edge cases as I don't get paid to.

I really envy the industry where you work on. In my industry you can't just ignore half a billion of outdated Android phones and Government seats.

I presume you get paid for this specifically?

uncss[0] works well for me. It removes the unused styles and then I run the css through a minifier to finish the job. The css filesize is greatly reduced.

[0] https://github.com/giakki/uncss

Have a look at PurifyCSS: https://github.com/purifycss/purifycss

For unused CSS, Helium https://github.com/geuis/helium-css

Lol @ testers. How about getting rid of the project manager and testers, and hire 2 decent developers who have some experience with automated testing, continuous integration and continuous delivery instead? Productivity rises 300% I would presume.

I think the best tool for this job in this specific scenario is simply "rm -rf. /"

Reply on news.ycombinator.com