Hacker News Reader

Comment by near | Hacker News Reader

The web has been engineered to a complexity level so grossly obscene that it is all but impossible for independent developers to produce fully independent web browsers anymore. You would need hundreds of millions of dollars in capital and strong leadership to produce a browser that would be successful in today's world.

I could dedicate the rest of my life working 100 hours a week on a browser, and I would die before I had something that could compete with browsers of 2017, let alone of the browsers far into the future at my demise. The standards grow in complexity faster than a small team of developers could possibly keep up with.

The best we have are attempts at forks of major browser engines, which will never gain any serious traction to have any power over the direction of web standards.

If people want to do something about it, they need to fork the web itself. HTTP, TLS, HTML, CSS, JS ... the entire stack needs to be scrapped and replaced with something sensible with a focus on simplicity. If a single highly talented developer working full-time can't implement a reasonable browser within a year, then the standard is too complex. This will of course never happen.

And so, on the subject of DRM, we've officially lost the war today. And combined with the impending loss of net neutrality, brace yourselves everyone: it's only going to get worse from here on out.

View on HN

If we want to win that war, we should focus on content not technology, Mosaic got a lead in the game because, despite the 'techies' seeing no use for images, image support was added, with images came the transition of media, users were au fait with articles containing text and images and could now experience similar on line, this alone was not enough, content creators starting creating content that people wanted to consume, that's what drove people to 'get online', they were missing out.

Don't get me wrong, it's still a long shot, but if an organisation could build a raft of talent, get that talent to make content that was only available on an open and progressive platform, then you might get some traction. So what might that look like, well, say I had the funds to secure 100 of the worlds best financial journalists and only delivered that content on a novel protocol, do you think I might be able to persuade some people to run a novel application to access that content, oh hang on, did I just describe Bloomberg...

I agree.

Back in the dark ages, when the battle of www vs Gopher was still on, I sided heavily on the side of Gopher because it didn't get in the way of the content. Gopher was a means to navigate the maze of content and most of the content was textual, postscript and neatly typeset. You could sit down with it and read it focused or print it out.

We've regressed to the point where 5KiB of text comes with 2MiB of crap, meaning a noise to signal ratio of 26dB which is insane!

Really Gopher is dead though. We could do with a fresh start. A search engine like WAIS, a menu system like Gopher and a standard document format that allows you to get quality content on everything from your e-reader to paper without having to read some fucked up ebook flow layout conversion. And not postscript. Something you can author by hand.

I've even had enough of building things on the web. It is a horrible horrible thing to have to do. I envy system programmers who never see the pain of trying to keep a very large web application secure at even a basic level.

> We've regressed to the point where 5KiB of text comes with 2MiB of crap, meaning a noise to signal ratio of 26dB which is insane!

Sadly the average clickbait article contains a lot less than 5000 characfters of actual text.

I don't see why these webpages need to be so huge. If websites used HTML5 properly and used CSS more extensively, a lot of the crap I feel could be removed.

Seriously, aside from graphics and video, what is taking up the size of these pages? Mountains of JavaScript?

As someone who used to work for a web optimization company, yes mountains of Javascript. Javascript for analytics, javascript for transitions, javascript for DRM, javascript for updating pricing in real time, javascript for fonts, javascript for colors, javascript to lazy load the images....six different style sheets and images that are downloaded and resized client side...and this is just the navigation bar.

Just for fun I like to pull up the browser diagnostics on pages, and watch the absurd number of requests build up. It's insane how much junk is downloaded on seemingly simple sites.

This is why I usually use just lynx.

I may be wrong, but isn't connecting to a web server by lynx illegal (considered "hacking") in some countries? I remember a case in the UK around the time of the London olympic games.

It can never be illegal to use any tool. It is the intent and action carried out with the tool that decides whether it is illegal.

Consider the difference carrying a knife, being a hunter, and carrying a knife being a bankrobber and going into a bank to use it.

Oh it most certainly can... it SHOULDN'T but it absolutely can.

Perhaps the knife was a bit exagerated. It will be considered a weapon in many jurisdictions. Lynx however will/should hardly be considered a hacking tool. curl has been in deep waters many times, equally perplexing as the lynx scenario. https://daniel.haxx.se/blog/2016/01/19/subject-urgent-warnin...

Compare Linux Reviews, "Solaris-user arrested by British Police for using Lynx" -- http://linuxreviews.org/news/2005/01/28_0001/28_0001.html.en

That is scary indeed. Was he convicted? Police do make mistakes and one can only hope that they are corrected.

I faintly recall reading (to my utter astonishment) that he was at least not freed from the accusation as you would have expected. I tried a search for any follow-ups, but to no avail. Maybe due to the "right to forget." (Information conveyed without warranty, liability or claim of correctness. It has been 12 years since, I may be in error.)

He might have never been charged, and being released without charge is rarely reported.

> It can never be illegal to use any tool. It is the intent and action carried out with the tool that decides whether it is illegal.

Since illegality is, literally, whatever is contrary to the law, anything at all can be illegal. We can hope that crazy things won't be illegal, and decry their illegality—but, since it's convenient for a government to impose sufficiently many laws that anyone can have some infraction pinned on him or her (https://www.youtube.com/watch?v=JwsLAqjqnxo), to say that any particular thing can't be illegal is probably wishful thinking.

I'm not familiar with that case, but I don't think an accusation that using a text mode browser like Lynx constitutes "hacking" would stand up to any type of scrutiny. (I'm not a lawyer and I'm in the US; things may differ in the UK).

Lawyers will make the most egregious leaps and grasps to paint the opposing party in the worst light possible, especially in the early stages of the case, because if they don't claim that Lynx is a "hacker's tool" in the initial pleadings, then it may be looked upon with suspicion if they try to raise that point later. So they are essentially trying to reserve as many potentially beneficial claims as they can at once, including (and especially) any claim that may portend that the other party is a devious scoundrel with the worst possible intentions.

Lawyers will thus front-load with every kind of tenuous, ridiculous accusation they can (without presenting a significant risk to the licensing), expecting some of it to get "laughed out of court", just so that on the off chance that it doesn't, it will help their case.

I'm sure a real lawyer would contest that to some extent, and I'm sure there is slightly more nuance to it than that, but that is my layman's assessment and I think it is roughly accurate.

The takeaway is not to take legal pleadings too seriously, because the goal is not to find justice per se. It is for your side to win, to get what it wants, under the assumption that that automatically represents justice.

Please note that this wasn't related to some lawyer actions, but the Metropolitan Police's Computer Crime Unit acting on the basis of suspected offence under the Computer Misuse Act. The offensive act occurred while the browser was used for making a donation at a tsunami related website of the Disasters and Emergency Committee, where the browser showed up as an "unusual event" in the access logs.

I believe, the case is part of a more general theme (which is, indeed, related to the EME topic): There's a perceptible will to regulate any kind of browsing into consumer-style app usage. E.g., manually entering a URL ("URL hacking") has been considered a fraud attempt; using a browser which only supports the basic protocol set and not all of it is considered as possibly related to intrusion and terrorism (oh, that T-word again). In the end, we may wake up with the law enforcing a certain, standardized life style. We may wake up with a deregulated companies and strictly regulated citizens.

I understand you're referring to a criminal prosecution. Criminal prosecutors are, from my limited observations as a layperson with a minor interest in law, more guilty of this than anyone. They go hyper-aggressive on charges, expecting them to be pared down. Some of it is a ploy to get a quick plea. There is an element of calling the bluff, and the defendant's attorney would have to evaluate the situation and advise whether or not that was smart in the given case.

This is not to say there aren't ridiculous prosecutions that run their course, especially in niche areas that aren't widely understood and thus hard to create public outcry over, like insider trading or computer crimes. It's just to indicate that using Lynx, by itself, is not considered criminal by any sane person. It's a nit the prosecutors pick when they're trying to nail you on something larger, a detail they exaggerate to attempt to create the impression of one's villainous ways.

Just don't want people to uninstall Lynx in fear that they'll be arrested for firing it up. :)

EDIT: From re-reading your comment, it seems you're suggesting that this wasn't part of a larger operation, but that actual charges were brought because of a single aberrant entry in the access logs? I find this hard to imagine, but if true, I would guess that this is more along the lines of "abuse of police power to harass an enemy" than "literally a random dude who just visited our site in a text-based browser". I've read access logs for many years now and even small sites get all kinds of weird things in there, it doesn't seem plausible that something like that would flag off any kind of detector. Do you have a link to more details about the case?

I'd hypothesise that anyone skilled enough to be using a text browser is also skilled enough to be a "hacker" and will most likely have a computer filled with "hacking tools" and other "circumvention devices".

Basically anyone that can use a text based browser also is using some form of *nix and knows how to handle a command line application. They probably tinker with computers and so have many low level tools at their disposal.

I would but hardly anything works in it which is the problem.

It's only insane if you subscribe to the idea that the 5 kibibytes of content is the purpose of a news or other site. Maybe as a user that's what you came for, but from a business perspective the purpose of the piece is to sneak 2 mebibytes of crap into your life. The snr on that is much better.

It's only insane if you have the tools to look at what's happening under the hood. Most people don't have any idea.

Hah... the _content_ is the noise in that case!

It's insane if you're paying for mobile data.

> We've regressed to the point where 5KiB of text comes with 2MiB of crap, meaning a noise to signal ratio of 26dB which is insane!

Unfortunately that's what the average layman wants. They want the SPA experience. That's what they want to pay for. What kind of professional am I if I don't give people what they want to pay for?

> I've even had enough of building things on the web. It is a horrible horrible thing to have to do. I envy system programmers who never see the pain of trying to keep a very large web application secure at even a basic level.

I build and maintain full stack web applications for my day job. Man, is it a pain. Between the database setup, server setup, back end application, front end application, security concerns, and browser compatibility issues, I rarely get to spend any time at all working on the business logic. Seriously, modern web development is 90% chasing down bugs, security enforcement, and fixing compatibility issues. Writing the business logic is the easy part.

>Seriously, modern web development is 90% chasing down bugs, security enforcement, and fixing compatibility issues.

That's true of most modern development. It might be more painful on the web, I wouldn't know, but systems programming is not really any different.

I think the only real difference is that, generally speaking, systems programming standards and best practices don't change weekly. Or, maybe they do. I don't work in that field professionally.

> Unfortunately that's what the average layman wants. They want the SPA experience.

What about that experience do they want, and how do you know they want it? Engagement metrics can make auto-playing video look like a great idea, but people dislike it enough that browsers are starting to block it.

> And not postscript. Something you can author by hand.

How about TeX? Many sites already use js libraries that integrate latex.

I hope this was intended as a joke? Most users struggle with simple typesetting conventions like underscore/asterisk. HTML tags (or CSS) are out of reach for the vast majority. TeX isn't even part of the known universe for users!

> HTML tags (or CSS) are out of reach for the vast majority

CSS sure, but most people can learn the basic html tags in a minute. Same for the basic latex conventions.

I think this is a case where struggling == "doesn't want to learn". It's important to distinguish so you don't waste time trying to help the latter.

Not the worst idea. However not the best. Something between that and markdown.

Markdown is a bad joke.

It's really fine for casual documentation. I wouldn't write a book or a paper in it though.

R Markdown is a joy to use and people even do write books with it!

For a straight-text book, actually, Markdown is excellent. You can include images, or for technical content, tables, easily.

Technical papers with formulae and complex references are slightly more detailed, but that ends up being gloss. Even LaTeX for a simple text is quite minimal (though you need the starting template).

Source: I've made something of a hobby of marking up source docs into HTML, LaTeX, and Markdown. Of the set, Markdown would be my first go-to, and I've used it for some book-length projects.

Example here: https://ello.co/dredmorbius/post/lhw2eq4qmnnwxijlcrfyba

> For a straight-text book, actually, Markdown is excellent. You can include images, or for technical content, tables, easily.

I would probably use org-mode for that.

Another solid option, yes.

I don't see any way to win that war. The IP oligarchs have won it. They own the W3C now.

But there is clearly a demand for DRM-free browsers. And if the worst fears are realized, and mainstream browsers won't display media without DRM, or become snooping devices for IP oligarchs, that demand will increase.

Sure, it'll never be more than a niche, but DRM-free browsers will be there for those who want them.

I've no crystal ball, so looking back, monopolies and incumbent monoliths encounter resistance as they mature, there are more examples of the status quo being that they wither under the weight of complexity, this is certainly not over while consumers consume content, consumers own their time, content creators own their content, everything else is middleman stuff.

> But there is clearly a demand for DRM-free browsers.

There is not a demand for DRM-free browser.

I don't know anyone in the real world that demands a DRM-free browser.

They probably won't tell you what they really want ;)

This is a paranoid slippery slope.

EME existed for several years already, and was implemented in browsers. Standardization doesn't change much.

And no one except the big movie producers even wants to use DRM. If you don't watch movies on the web, you'll never see DRM.

Firefox is DRM-free by default. It will download Widevine CDM if you start watching Netflix, but you can untick a flag in the settings to completely forbid it from downloading the CDM, if you're afraid of accidentally signing up for Netflix :D

> Firefox is DRM-free by default.

It is now. But next year? In five years? And if it's gone, then what?

> This is a paranoid slippery slope.

Well, it's a slippery slope. And I do tend to plan for the worst. Plus a safety factor.

Open source software can never be completely "gone".

The "OMG EVERYTHING WILL BE DRM" panic is not new. We've had widely available DRM back when Flash was super popular. And still, to this day, the only DRM'd thing is movies.

The only thing? What about computer software?

Maybe you meant to type 'the only thing I care about'.

I actually don't care about movies, very rarely watch them at all. The vast majority of video I watch is YouTube.

I meant to type "I never actually saw DRM on the web in person".

Also, proprietary desktop software is still usually "protected" by just a serial number :D

> Also, proprietary desktop software is still usually "protected" by just a serial number :D

The ineffectuality of most DRM isn't really an argument for it (or even just an argument against arguments against it), though.

> Also, proprietary desktop software is still usually "protected" by just a serial number :D

Not really; there's a lot of sophisticated DRM going around at least since the 70's..

There's an easy way to win that war: diversify the content you consume. If you're willing to step outside of the immensely popular, there are great treasures to be found.

IP owners are protecting their assets which are worth a lot. The irony is that the value of those assets come from us content consumers. And I'll be the first to admit that I'm a lot less interested in the local amateur hockey team than the national league team I follow. I chose to consume that brand. But I don't have to.

Yes I lament the loss of a vision for free-flowing information. But the society in which we live, unfortunately, values and protects powerful asset owners. Technology cannot fix a society problem.

For individuals, sure. But the war was about norms for the Web. And that's lost. Again, by the way. The first time, as I recall, being in ~1995, when the commercial takeover really got started. So it goes.

> mainstream browsers won't display media without DRM

Seriously?

Nothing has really changed, has it? Video services want to use DRM so they developed or bought tools to do so. A de facto standard emerged around the frameworks and the W3C essentially ratified the standard. If I understand it correctly, the standard itself isn't DRM, rather it's a place where DRM controls can be installed.

I get that the EFF saw this as an opportunity to ask for a promise from DRM companies to not sue researchers and I applaud them for that. But now that they lost that battle, I think it's entirely appropriate for them to drop out and apply their resources elsewhere. For example, if they can get rid of the ability to sue researchers in the first place, then none of this really matters.

I don't think there's a demand for DRM free browsers but there is a demand for DRM free content.

You wouldn't need to scrap all of it by a long shot. You could take a small subset of the stack and make something enormously simpler to implement. Start with supporting only one version of HTTP, TLS, (X)HTML, CSS and JS and fork them if necessary. Even drop CSS to start with and finally start relying on semantic markup and good browser defaults. Use existing open source libraries where available, such as for TLS and markup/media parsing. Pull out libraries for everything so the browser is easily composable and forkable. You don't want JS support at all? Compile without it.

Now that's an idea I could get behind !

Not many would be interested in an un-designable web. It bares no comparison to what browsers are capable of delivering for UX currently.

What you are suggesting then, a non-comparable browser experience, already exists and has for years, yet to no avail. You would get niche technologists and die hard enthusiasts only and the rest of the world would trudge happily into the future, DRM and net nutrality distant memories.

What you and I want, what HN and friends want, is not even a debate most realise is happening let alone that it affects them and will affect them profoundly in the future.

The masses are deciding what our internet will look like for they comandeered it years ago.

Yeah, yeah, the world is going to shit and nobody cares. How about doing something about it instead of telling everybody it can't possibly work? And why do we need the rest of the world on the bandwagon for it to work? The web was built without the masses, and still is. The open web still vastly outweighs the closed parts in amount of content. A bunch of freedom stealing technologies have died silently over the years despite often resulting in "cool" UX wholly different from plain HTML, such as Flash, Silverlight, and Java applets. EME is a fundamentally different wedge inserted between users and their machines by evil corporations, but not one that is in any way unbeatable. So screw 'em, and let's fork the web!

Forking the technologies that underpin the platform of the web I think tackles the wrong problem. The problem I see facing the open web is the consolidation of users and services into single platforms. Facebook is an easy example. You can build a Facebook alternative with any technology you like, it would take a completely different force to change the social paradigm around it.

The great thing about the open web that you rightly mention is still kicking, is that it's full of places and services that are driven by humans and passionate communities. Nothing about EME stops those communities existing, but nothing about those communities is going to stop Netflix being the most popular platform and using EME/DRM.

If you will excuse my fluffy analogy, the web is a farmers market that has a shopping mall being built right next door. What I think would help the web, is educating people that it's better to go to the farmers market so that we can support entrepreneurial and local community endeavors rather than stuffing corporate coffers and enabling that consolidation.

> Start with supporting only one version of HTTP, TLS, (X)HTML, CSS and JS and fork them if necessary.

The support for different versions that does exist in browsers tends to be minute: ripping out support for older versions would cut out almost no code, because there's basically nothing specifically handling them. (Heck, the only forks in CSS are for quirks and limited quirks mode, and both of them affect very, very little of the layout engine.)

> If people want to do something about it, they need to fork the web itself. HTTP, TLS, HTML, CSS, JS ... the entire stack needs to be scrapped and replaced with something sensible with a focus on simplicity.

That's flash.

> That's flash

Comedy aside, CSS isn't the problem, but if an experienced web-dev can't make a site without learning for 5+ years and then feel like they have no knowledge of web development then we have a problem, and that's where we are now.

The problem with the long list of frameworks and libraries available is that there is no easy way to make a web2.0 site without years and years of learning and then the resulting site is a mish-mash of what works without it being easy to maintain. Sitting here looking at .NET & angularjs 1.x and wondering where it all went wrong (actually, not the.,NET part, that seems to work).

CSS makes semantic markup work. CSS is not the issue, the issue is getting a decent graphical editor that does it well.

What do you mean by "make a site"? I made qemu.org last year with 10-years-old knowledge of web development and, apart from the obviously not-done-by-a-graphics-designer theme, it _is_ a decent site, responsive and with a 2010s-ish look.

99% of websites can be done with either a static generator (lean) or WordPress (somewhat bloated, but really mostly standard libraries that can be cached and/or served by a CDN). The remaining 1% are the really bloated ones and sure they are the ones that we visit all the time (medium, BBC, whatever). But it's certainly possible to make a site with not too much knowledge of web development.

edit: ignore the huffy tone this came out in, it wasn't meant that way, early morning.

I was talking about work as a professional developer. We can all 'make a site', there exists any number of solutions for that, CMSes like Wordpress, Drupal etc don't count in this argument unless _you_ wrote Wordpress or Drupal from the ground up. Being a wordpress hack isn't being a developer. You won't ever be asked to write something straightforward in enterprise, you'll need to know at least HTML5 & CSS & angular 1.0 & 2+ (or similar) & docker & gulp/grunt & backend & databases & jenkins & MVC & node/npm & version control & bootstrap (or similar) and and and.

99% of sites outside of a company that makes real money and doesn't just serve a page are sites that aren't what developers actually do at work.

Correction: its the loss of _American_ net neutrality, places such as the EU for example already has enforced net neutrality for all members.

I don't think that's strictly true. For example many mobile ISPs here meter data differently depending on whether it's from their fav websites (Facebook, Snapchat, Netflix etc.) or from the lame weird web where nerds go.

Agree. Here's how to approach it, imho:

1. Assume we will eventually converge to a virtual machine target like WASM. Thus, eventually, browsers will only have to implement the low-level instructions of WASM, which will make building a browser much simpler. Note that as a side-effect, browsers may also become more secure.

2. Start with a working webbrowser, like Firefox or Chromium.

3. Step by step, move functionality from the browser into the browser's user-space. Eventually, JavaScript and the rendering of CSS and HTML will run in the browser's user-space.

Should a single developer working full time be able to implement an OS within a year?

The web is pretty much an operating system.

Ideally, yes. It may not've been popular at that point, but Linus did. But what about the next Linus of 2017? Something like that's not going to be possible anymore. Just the basic necessary hardware drivers alone. The hardest challenge for Linux was just avoiding Winmodems. These days the entire nouveau project with years of research can't even manage to change the clock speed of your GPU. VESA's been all but abandoned so there's not even a way to set a widescreen framebuffer anymore for pure software rendering.

Nothing is gained from having hopelessly complex technology.

The next Linus of 2017 will be on a completely different platform, not PCs or smartphones. That's the nature of disruptive innovations.

You say that like Linus didn't write linux on the same platform everyone else was using

I think you're right, but only for a narrow definition of "browser". Sure, something that looks and works exactly like Chrome, Safari, et. al. would be a gargantuan effort for a sole developer. But an 'HTTP client' can take many forms and still offer value, even advantages over the existing browsers which serve a very specific remit.

A basic text browser is the kind of thing that's buildable in a year. Moreover, once you start building that, maybe you have an idea for certain features that current browsers don't do well (bookmarking, navigation) or at all (change monitoring, api integration, editing).

I'm optimistic that there is a lot of unlocked potential in the HTTP/HTML platform.

We should all move back to gopher. Seriously.

I call dibs on the first gopher based social network...

[edit] So I've read the RFC (https://tools.ietf.org/html/rfc1436) looks like it should be fairly simple, we can run gopher servers on both user machine and remote servers, so I can store my personal details locally, then give permission to other remote gopher servers to access that in a granular manner, third parties can operate gopher servers I can interface with to access 'cloud' style computing, storage, search, and manage contacts/calendars, sorted :)

[edit 2] Looks like Gopher has a pulse, you can access around 4.9 million unique 'files' using an HTML>Gopher tool run on this site http://gopher.floodgap.com/gopher/ here's a nice Ars write up https://arstechnica.com/tech-policy/2009/11/the-web-may-have...

[edit 3] My build of firefox does not seem to have Gopher support, it recognises the link (chrome just does nothing), but asks you for a local application to fulfil the request :( [edit 4] It's now a plugin for firefox, https://addons.mozilla.org/en-US/firefox/addon/overbiteff/

Cool, I'm searching Gopher, and it works :) Nice and quick, no tracking, no ads, just need something to consume, anyone want to help me whip up an HN > Gopher widget ;)

there's nothing new right, gopher://hngopher.com/1

I am all for text-only, structured web; but gopher transfer protocol is broken mess (which is an achievement considering simplicity of it's purpose).

Why stop with creating a browser from scratch? You'll need hundreds of millions of dollars in capital just to build your own compiler technology that would be successful enough and incorporates enough instruction set optimizations present in commercial CPUs that your custom browser can be built upon.

Maybe you'll need your own hardware as well...

I am hoping feature-testing continues to be a good practice - I don't expect to have webgl, the webusb, or webaudio APIs available...

ehhh HTML parsers/DOMs are trivially easy to write. after that you need to go get a JS lib (like ducktape) and write a css and json parser.

After that all you need is a layout engine, which GUI tookits like tk and gtk+ already have for text editing.

Parsers for valid HTML are trivially easy to write. The hard part is dealing with garbage HTML tag soup and still rendering something that looks good. Browsers have to figure out what the page author intended and work around all the content defects.

Parsers for invalid HTML are trivially easy to write: you literally just implement the spec, and that's not hard, and then you implement every major browser.

You're joking right? Sometimes it's hard to tell...

I'm not joking. It's not that hard. You literally pick up the spec and implement it, it's mostly just fairly boring menial work.

I think you misunderstood. There is no "spec" for handling all of the invalid HTML out there. That's part of what makes writing a browser so hard. It's far from being menial work.

AFAIK all major browsers have converged on the parsing algorithm described in the HTML5 spec.

It‘s super lenient and very messy but at least it‘s consistent.

IMHO this was one of the biggest benefits coming out of the HTML5 efforts.

Indeed. 15 years ago it was a huge amount of work reverse-engineering other browsers around parser behaviour; 10 years ago it was a huge amount of work finding where the draft spec broke sites; today it's pretty much just implementing the spec.

as the standard gets bloatted, new players could leverage this weakness and build something entirely new. yes, reinvent the wheel. =)

Reply on news.ycombinator.com