Hacker News Reader

Comment by kephra | Hacker News Reader

Lets look, how this new wiki behaves without JS, from the viewpoint of a search engine.

e.g. compare http://c2.fed.wiki.org/methodology-subsets.html with and without javascript. The new wiki is doomed by design, because content can not be found by Google or any other search engine. And a wiki where the content can not be found is for the trashcan.

View on HN

It has also joined the ranks of sites that don't work at all without cookies. Go ahead, try it.

This can be solved using any of these methods: https://developers.google.com/webmasters/ajax-crawling/docs/...

I maintain a client side rendered website and while SEO can be problematic it can also be done.

I don't see C2 using any of these methods at first glance tho.

It can more easily be solved by just not crawling sites that have mandatory javascript, because chances are their content is going to be crap and not worth the time if they are that hostile to basic usability.

This is a specific issue to be solved with standard REST techniques. Do you know if the team is working on it? Maybe it's on a timeline, scheduled to be implemented next month? I haven't looked into it, but the current lack of this functionality does not mean that the "new wiki is doomed by design."

Google executes JavaScript.

That can impossibly be true. JS is turing complete, turing completeness means that the halting problem is undecidable, which means that at best Google will execute JS with some random resource limit to avoid infinite resource consumption, which in turn means that whether Google will actually see the content is ultimately undefined. And even worse: other search engines might, due to lack of standardization of the available execution resources, see a different picture. That's just a braindead model for information storage and exchange.

I don't know what that was all about, but Google does, here's a blog post about it:

http://googlewebmastercentral.blogspot.com/2014/05/understan...

And webmaster tools has a tool to verify that content is scraped correctly.

Not saying there aren't other good reasons to do server-side rendering, but SEO is not a real one.

You're trying to be too intelligent for your own good.

Fact of the matter is the new wiki UI is horrible. It doesn't have to be spelled out any more complicated than that.

Everything you said applies to humans using normal web browsers as well. And yet people still fill pages with JS, and browsers still execute it, and people still see stuff.

With one minor difference: Humans at least have full-blown human intelligence which they can use to heuristically determine when the execution is "complete". Not that that makes all that much more sense ...

Yeah, that does make it better, but it's still not perfect. Witness the complaints right here about how you can't tell whether you got an empty page or it just hasn't finished loading yet.

Trouble awaits when you try to wedge a general-purpose application environment into a page-based document viewer.

I doubt Google is worried about unfairly weighting pages it indexes with Javascript that takes so long to run, hence the cutoff. It's more a question of where to place the cutoff than whether to have one at all, which is absurd.

No, the question is neither of those. The question is how to define the cutoff in a portable way that enables interoperability and long-term stability. A document format where every implementation has its own secret cutoff that also probably changes all the time is just idiotic if your goal is interoperability.

You seem to be saying that page load is not deterministic but that is not true. A special version of a browser can easily keep track of http requests and other async operations and consider the page to be complete when async ops = 0 and the last paint is complete. The only hard part I can think of is a page that is recursively calling setTimeout endlessly, but even that can be coded around.

No, it cannot, that is called the halting problem.

Right, so like you said they have a reasonable cut-off for this one most likely ultra-rare situation. It's not going to be perfect but it doesn't have to be, in order to be useful.

It doesn't matter if the halting problem is solvable or not, there would exist a cut-off anyways (even if they had a super-Turing machine).

The exact same argument could apply to pages that take forever to load, because the web server sends you a chunk of bytes every two seconds. Would it be braindead for Google to apply an arbitrary timeout without standardization?

Nope, but it would be braindead for google to then just index however many bytes it has received up to that point as if it were the full document.

OK, so we have a pretty straightforward rule. Providing no further input to the program once it begins,

1) If execution terminates by some timeout T, where T is at least several seconds, then index that.

2) If execution has not yet terminated by T, whether or not we have any idea whether it will terminate in the future, don't index.

Tune T so that it will get the vast majority of reasonable web pages. (Hypothesize, and test, T = 5 seconds or 30 seconds or something. Have a human look at the highest-PageRanked timeouts to figure out what's going on.)

This applies whether "the program" is server-side or client-side. The halting problem cannot be reduced to the timeout problem (since the halting problem asks if it ever terminates), and the timeout problem is pretty clearly computable.

Reply on news.ycombinator.com