Google, Twitter and Publishers Seek Faster Web

1. The ad/tracking industry seems to be incapable of writing JavaScript that is not awful. I don't have insight why that happens, but all the scripts I had to deal with were layers upon layers of document.write, iframes, redirects and copy&pasted JS from the Netscape 4 era. Even when somebody tries to fix it, the best they can do is to add one more layer of scripts on top of the rotting pile.

2. Business deals are made with no consideration of 3rd party's code quality and performance impact. Once the deal is signed the developers are just tasked with integrating all of the crap the 3rd party vendor wants. It's impossible to object: the vendor is not going to rewrite all of their shitty scripts, especially after the deal is already in place. Even when it's really bad, arguing about it just results in endless meetings, conference calls and at best the 3rd party vendor saying they'll think about putting some little tweak in the backlog for some future release.

Well, the same problems that beset big enterprise are also found in "big publishing" which is even more severely amplified by the need to have everything live yesterday.

> Given the wide array of compression tools, caching (including service workers on >40% of browsers, appcache on >90% and pretty much everyone having HTTP caching) and the sheer amount of information on improving website performance that's out there, making news websites load fast really should be a solved problem that doesn't need Google to solve it.

Correct, but you are assuming that most developers are a) interested in learning these things b) possess even the fundamental of CS knowledge and skills. By reading HN, I would assume that both a and b apply to the majority here, however, in my own cynical outlook, most "developers" are really just out there winging it and cannot be bothered to learn more than what's provided on w3schools.com.

We offer analytics with a similar business model to companies like MixPanel and NewRelic -- that is, not analytics for ad serving and buying, but for improving user experience, user engagement/loyalty, and informing content strategy. Like those companies, we charge money for these analytics -- because they are valuable to the site owners, are hard to calculate, and help them improve their sites over time.

I'll try to summarize what I see happening:

When I first entered this industry (~5 years ago) and launched our first JavaScript integration, the state of JS in the industry was poor. Ad tech companies embedded clunky iframes that would sometimes load entire web stacks in them. Loading a single publisher website was often like loading 10 websites in parallel in a single tab. Many of these ad tech vendors haven't upgraded, so this kind of stuff is still floating around.

I made a focused effort to make a slim JavaScript integration to collect the data we needed for useful reporting to the site owners, using as few bytes as possible. That meant writing all the JS code "from scratch" (no dependencies), forcing async-loading, and beaconing back the minimum needed data.

Over time, we've slimmed down our integration considerably as more engineers have come on-board and came up with ideas to make our JS have less and less impact on page load performance. We've kicked tires on various code minifiers to find the right one, refactored code to simplify it, etc. But my company is rare in this respect. Most vendors (many of them ad tech) simply don't care. Once they integrate their crappy code on the site, it's frozen forever, and they don't touch it.

As one small example of this, my company's analytics system collects data about pages via a backend web crawler that integrates the Schema.org JSON-LD web standard, the same standard that the search engines use. What does this mean? Rather than beaconing back ~50kb of article metadata on every page view of the article (which is what most page analytics vendors do, e.g. Adobe Omniture or Webtrends), we beacon back 1kb of event data on every page view, then crawl the other 50kb from our backend servers. In this way, we don't punish every user, over and over again, for the data we need to collect. This requires us to do a streaming join of these data sets on the backend, which is complicated. But it's complicated for us, the engineers building the analytics system, not for you, the typical web visitor. That's The Way It Should Be.

There is also a shady thing going on where many "data brokers" offer publishers a single JavaScript integration, which starts off slim. But as the publisher's data gets "sold" to other data brokers and ad exchanges, the JavaScript integration gets fatter, because it's now "JS embedding JS embedding JS". You get another (recur) of the JS embed for each additional bizdev deal struck between any two data brokers.

As a company, we have taken a firm stance that we are not an ad tech company. We charge our customers money every month for our services (reporting dashboards and APIs), and thus we don't need to "monetize on the side".

But you have no idea how against the grain this is in the media industry. On the web, "monetize on the side" is the primary business model. In fact, the CEO of a major media site once told me, "My only problem with your service is that you charge money for it. I don't think we should pay for services. Everything should be free, and you guys should figure out how to make money on your own."

With bad actors among the ad tech industry popping up every day who are happy to take advantage of this mindset, it's no surprise that the page load situation has gotten out of control.