Does Google execute JavaScript?
stephanboyer.com
stephanboyer.com
That is, new and low traffic sites are crawled by less intelligent bots, and as the site gets more visitors or better rankings, more complicated and resource intensive bots are deployed.
How this might work with the most popular sites out there, the Amazons and Wikipedias of this world - I'm not so sure about that. If I were in charge, I'd be tempted to have customised bots and ranking weights for each of these exceptional sites.
Sadly the chances of getting a real answer on this in my lifetime are close to zero.
How this might work with the most popular sites out there?
We see it in on-page answers that provide extracts of pages with the answers to questions asked in search phrases that include a reference to the document they were sourced from.
Matt Cutts used to qualify sites like Wikipedia as "reputable" to the eyes of the search engine.
I'd expect that there are also other heuristics and different strategies for crawling to better handle eg. content presented by one of the popular CMSes.
heuristics to see whether
executing JS would be worth
it, would yield additional
content.
If you look at Google's cached version, you can see that the JS is executed (although it fails trying to download the actual data): https://webcache.googleusercontent.com/search?q=cache:hN5yCk...
Edit: as has been pointed out below, the cached version is just the same as the original and the JS gets executed on your end. This doesn't show weather Google also executes it during its crawl.
Correct me if I am wrong but when I look at the cached version of the homepage (http://webcache.googleusercontent.com/search?q=cache:hN5yCky...), I don't see that the JS has been interpreted.
Second, thank you for responding to someone pointing out that you were wrong without putting yourself on the defensive.
I see this entire sub-thread as a positive; glad we all could learn along with you!
It also explains why they effectively killed flash and Java applets. They were competing technologies that weren't owned by Google and not crawlable. If they would have taken off, Google's position as top search engine could have been in danger
The offical reasons were speed and security, which was obviously a farce since all the major browser vendors wrote their own PDF viewers which is both slow and known for tons of security issues as well.
They could have just as easily wrote their own fast and secure implementations of flash and Java instead of disabling them. Especially for Java given Google's deep experience with it and that they already have their own runtime.
Apple may have fired first on flash but google recently disabled all NPAPI plugins in chrome as well. The reason... "Security". I guess code not made by Google isn't secure enough to be used on the internet?
I do agree with you that Google's security model is broken in many ways, both on the web and in Android.
As in most things there's probably some truth on all sides.
http://www.operasoftware.com/press/releases/mobile/opera-del...
https://en.wikipedia.org/wiki/Nokia_N800
It actually ran quite well! I used to play flash games all the time.
Google went to significant efforts to sandbox it, but that still caused problems, not least with additional complexity. As for this idea that NPAPI should continue to live... Yes "security" is a problem with NPAPI.
As for Google fixing Java... You may have heard of Oracle? And the fact that they are .... not nice?
Yup, Google should:
1. Reverse Engineer a proprietary language/API (Flash).
2. Write a secure VM for it (Flash and Java), trying to hit a moving target.
or
Get everyone to use an open, standardized tech (HTML5/js).
Can you give me one reason Google should prop up Flash?
Security wasn't a farce at all. "Flash is a big hole" (which is true) translates to "Chrome is a big hole".
What do I mean with “the threat of Facebook”? In the old days, before today’s large “social media” sites, people made their own web pages on places like GeoCities or on simpler social-media-like sites like LiveJournal, etc. Those sites all had content and linked to each other. This is the web which the Google search engine and its algorithm was meant to find things in, and it worked very nicely, as it took advantage of the links other people had made to your site as a proxy for relevance in search results for your site. People making small web pages about their favorite topics (with lots of links to other people’s pages, since information was hard to find) could slowly and easily transition into making larger and larger reference web sites with lots of information, thereby attracting lots of incoming links from others, which in turn enabled people to find the information using Google’s search engine.
Compare this to now. Firstly, people having a Facebook account have no place to simply place information, no incentive to simply make a web page about, say, tacos or model trains, because that’s not what Facebook is about. Facebook is about the here-and-now, and whatever is yesterday is forgotten. As I understand it, there is no real way, in Facebook, to make a continuously updated page with a fixed address for people to go to as a reference point about some subject, or at least people are not directed towards doing this as part of their online activity (as opposed to in the past, when it was basically the only thing which people could do). Secondly, this makes it so that people have no natural path going from using Facebook to creating a larger web site with information, and there are no smaller model train or taco Facebook “pages” which could have links to your larger site and thereby validate its relevance. Thirdly, even if this second point was false, Google could not use these Facebook pages, because Google cannot crawl them. These pages are all internal to Facebook, and Facebook has every incentive to not allow Google to crawl and search this information. Facebook would much rather people used their own site to search, and thereby gaining all of Google’s sources of income: User monitoring and advertising.
Putting walled gardens around large portions of the net will inevitably result in endless mergers and an eventual "website fee" that users pay to one of three highly entrenched monopolies that control all the content. It will be back to the cable TV days in no time
It makes sense to me that now someone who used to be motivated to build a site about their life or a topic of interest now may often just sign up for a service like FB and occasionally do a post with an article, picture, etc about themselves/their interests. It seems to require much less effort for folks, which is perhaps why they do this. I lament the change somewhat and wonder what the future holds for this type of thing.
It's also much easier to find readers.
Compare:
Facebook: become friends with your co-workers, now they see your picture
Blog: Please go to jupiter90000 (that's how many zeros?).blogspot.com to see my once a week pic updates!!
Java applets just always sucked and only masochists/sadists ever used them.
It's also a good reminder of the positive influence Apple has often had. Remember that "no flash" was the "no Esc" of its time, and that they had to go all-out to defend it (http://www.apple.com/hotnews/thoughts-on-flash/). 6 years later, it stands as one of the rare cases where a proprietary technology was replaced by an open standard.
If Flash didn't kill your battery and expose you to new security vulnerabilities every few weeks, it likely would have continued to this day.
But Adobe touted Flash on mobile as some kind of essential technology, when it was 100% crap.
If Apple offered it on iOS, Adobe would have been milking it to this day...
http://www.adobe.com/devnet/air/air_for_ios.html
So iOS do support Flash, just not its browser.
Executing JS everywhere would require a lot of CPU time and I think Google prefers not to do that when possible. And indexing a JS app is a very complicated task anyway (it is difficult for a robot to even find navigation elements if they are implemented as div's with onclick handler instead of links) so you better use sitemaps to make sure the bot can find content.
And I don't think it is necessary to index rich apps. It makes no sense to index a ticket search app (the data become outdated too fast) or an online spreadsheet editor. Just make indexable pages as server-rendered HTML pages and put their URLs into a sitemap.
Also Google looks for strings in JS code that look like URLS (e.g var url = '/some/page') and crawls them later.
(Disclosure: I work for Google, on unrelated stuff, though I didn't at the time I wrote that blog post.)
Why is it that Google doesn't get flak for not discovering content that's engineered to send the absolute minimum over the wire, cache intelligently in localStorage and IndexedDB, and scale well by distributing the appropriate amount of rendering work to the client agent? Why can't I expose a (JSON/)REST-API-to-deep-link mapping and have Google just crawl my JSON data and understand (perhaps verifying programmatically some percent of the time) that the links they show in search will deep link appropriately to the structured JSON content they crawled?
It's such a waste of talent and resources to force server-side rendering. There's obviously the resource cost of transmitting more repetitive content over the wire, and requiring servers to do more work that the client could do. (Yes, even with compression this will still be a higher cost, because more repeated sequences reduces the value of variable-length encoding). But more than that, what bothers me is that there's this false truth that server-side rendering is a requirement for modern architectures, which must result in hundreds of thousands of wasted engineering hours trying to enable the idea of server-side and client-side rendering with the same code.
This is not about time-to-first-byte either. Yes, the user-perceived latency matters, but the idea that server rendering even solves this problem is again utterly false. Sure, the time to very first byte ever may be faster, but that's not a winning long-term strategy unless you never expect your client to request the same content twice (or come back to your site at all). When properly cached and synchronized, the client-side-only app has many orders of magnitude faster TTFB, because it's coming from disk or even memory, and can be shown immediately. The only thing left to do is ask the server "what's new since my last timestamp?"
All of these benefits seem to be completely disregarded 99% of the time because the golden "SEO" handcuffs are already on. I really hope we can get away from this mindset as a community and rather let the better-engineered and sites with the best and fastest UX over time will start driving search engine technology, instead of the other way around.
It's fairly clear that just by developing those webmaster tools, Google is effectively (and understandably to a point) saying that they won't try to create engineering solutions to certain problems. Or that at this point it's not a sound financial investment because, after all, people will come to them because they're the biggest game in town.
If you're referring to my alternate crawling strategy suggestion of mapping a deep link structure into a structured REST URL structure, that's just an optimization I'd love to see. Really, I just think that search indexers should index what my users see, regardless of my engineering decisions.
It this problem really that difficult? Why?
Why should your code care if it is running on my computer or yours?
Isomorphic JS has been around for years. Build your product on bloated tech stack relying on a increasingly poorly planned web of dependencies, and I'll agree it could be challenging.
> Why can't I expose a REST API to deep link mapping and have Google just crawl my REST API
They do crawl REST APIs. Specifically, ones using Hypertext Markup Language.
> cache intelligently in localStorage and IndexedDB
Speak of hundreds of thousands of wasted engineering hours...HTTP caches are simple and straightforward. IndexedDB leaks memory in Chrome so badly that Emscripten had to disable it (https://bugs.chromium.org/p/chromium/issues/detail?id=533648 https://github.com/kripken/emscripten/pull/3867/files). Mozilla advised developers not to adopt Local Storage due to the inherent performance issues. (https://blog.mozilla.org/tglek/2012/02/22/psa-dom-local-stor...) And how many wasted hours went into WebSQL?
> utterly ridiculous that engineering and efficiency decisions are so deeply affected by whether or not the largest search engine will properly index your content
Actually, it makes a lot of sense. Content needs to be discoverable. Hosting a complex language in a VM where the slightest deviation from the 600-page specification (and that's just for the core language...not the browser APIs) causes failure -- that's not "discoverable". It's like putting up a billboard with one giant QR code, just because that makes it easier to develop the content.
> Why should your code care if it is running on my computer or yours?
It shouldn't. But my users already care about perceived latency, and that is directly limited by the speed of light. My users want feedback as quickly as possible that their input has been received, and that something is happening in response. Thanks to the speed of light, this would ideally take place instantly right in front of their eyeballs. That can't happen yet, so as much as can realistically happen on my user's CPU, memory, and storage is the next best thing.
> They do crawl REST APIs. Specifically, ones using Hypertext Markup Language.
What I meant to say was JSON, so I'm contributing to my own pet peeve of saying "REST" and meaning "JSON." :-p
HTML is awesome and does a wonderful job of letting me mark content in a way that it can be efficiently rendered, semantically, and be both human-readable and marginally machine readable. There are two problems, though. The first is that full documents (since the article points out AJAX is not performed by Google) are incredibly repetitive and wasteful, especially when retrieving the same content fragments multiple times.
The second is that it is strongly coupling content and presentation, two orthogonal concepts, much earlier than is optimal. Sure, you can cache full documents and display them when requested again, but the more common case is that a large subset of what I just displayed to my user will be displayed again, with one new item, but has still invalidated my cache because the granularity is at the full-page presentation level, and not the business domain object level. If, instead, I cache and render business objects on the client side, I can be more intelligent and granular with my caching strategy, react much more quickly to my users' feedback, and have a much smaller impact on their constrained devices. Not only that, but transmitting structured business objects instead of presentation-structured content lets me more efficiently reuse that data across devices for which HTML may not be the most effective way to present the data to them.
My personal architectural bents aside, the truth remains that content discovery agents (e.g. indexers) should not be treated as content delivery agents with such a huge influence on content format. This ends up creating (IMO) too much influence over external engineering decisions, rather than allowing engineers to think critically about the right architecture that gives users the best possible experience.
Most importantly, I'm not saying that all the engineering effort should be placed upon the discovery agents. Of course there are limits on how much they can discover on their own, and (as always in matters involving many parties) there need to be good conversations about the state of things, and what we think is the right direction to go to support each other and our users. It's just been my opinion lately that this is not so much a conversation anymore as a unidirectional stream of "best practices" coming from a single group.
Closure Library/Templates was meant to render server-side and bind JS functions after render, or create client-side dynamically. (Interestingly, the historic reasons were performance, not SEO.)
React and Meteor have good server-side stories. Angular 2 is getting one.
I would say there is a lot of low hanging fruit in just avoiding most client-side JS. Take http://wiki.c2.com/ -- the "original" wiki. That should all be static. Same with blogs, documentation, and lots of other public, indexable content.
And I don't expect Google or anyone to be able to support every edge case, either. I really would just like some sort of better solution that involves a global minimum of effort to achieve the same thing -- indexing what the user actually sees (non-private info, at least), and helping users discover sites that will give them a great experience and not just sites that give indexers a great experience.
All this anger aside, I'm actually pretty impressed with the world we live in and proud of my company. Think about how far we've come that merely crawling and indexing the vastness of the internet is so mundane now. Now we should expect the whole internet to be downloaded and executed. That's got to be a great security and integrity problem. Surely someone had tried to break out of the sandbox. Can that be abused to affect SEO of other sites. The easy answer is "spin up a new VM for each page" but that would slow the indexing process down by orders of magnitude.
Of course, now that they're gigantic, they now are the primary force that's adding unneeded engineering complexity to keep content in a format they can already read.
So who's really helping whom?
Offtopic, but Google's propensity to migrate anything popular to their own hosted content is a better example of this. They find ways to present the good stuff without the end user ever visiting your site. At some point, this starves off the sources.
Again ironically, this will put Google out of business if they keep it up, unless they can start to collect all that data on their own or otherwise incentivize content producers to allow them access.
It's utterly ridiculous that engineering and efficiency
decisions are so deeply affected by whether or not the
largest search engine will properly index your content.
(Disclosure: I work for Google, on unrelated stuff.)
I browse without JavaScript by default, and if a page doesn't load properly because someone decided to implement not a web page but a web page viewing client-side web application, I usually just leave. Then there are terminal browsers like lynx.
Moreover, reimplementing a web browser's navigation logic for a specific site is silly. It will in all likelihood be less reliable than a web browser's navigation logic. Moreover, it will always be slower for the initial page load than just serving a normal web page.
Yes, maybe you'll make things slightly faster for subsequent page loads. But consider that initial page loads from search engine referrals may well be the most important case, latency-wise. And if you do server-side rendering with progressive enhancement, you can have your cake and eat it if you really want to implement your own navigation logic with pushState, etc.; serve the static page and enhance it with an async script.
I would argue that it is nothing to do with Google.
Google's official line is that that will index your content, even if your entire frontend is a big client side only SPA. The explicitly tell you to focus on having good content that people actually want to read, and little else.
As others have suggested in this thread, it is likely that they developed Chrome for this reason. My understanding is that they basically crawl the web using a headless version of Chrome, and take a snapshot of the augmented HTML straight away, after 5 seconds, and after 10 seconds.
Other search engines aren't so clever, and so the work to have your important content available even when Javascript is disabled is done so you will show up on Bing, Baidu, Yahoo, etc also.
To go further, I reckon Google would love it if you didn't do this, as it would give them an edge over their competition which has less advanced capability to crawl the web.
https://www.google.nl/search?q=site%3Adoesgoogleexecutejavas...
If you can get Google servers to execute Javascript, that sounds like a possible attack vector. It's likely that Google runs these in a proprietary feature-sparse interpreter.
The lack of AJAX would make it difficult to leak information about the black-box interpreter.
If you have a professional website you want to be found also by these other engines. Until also these support javascript you may end up with a hybrid SEO architecture anyway which means nothing was gained?
Do you really? Except if you are interested in the Chinese market.
In fact the inverse is probably more true: if you run one of those other search engines, then you want to be as good at indexing any particular site as Google is.
Second supporting other platforms is also important, think of FB, Twitter, etc.
So either this really becomes the new standard that everyone supports or you may end up with a hybrid or traditional approach if you care about other platforms as well - imho you should.
If I click the link within that page that leads to the exact same webcache url, I get "yes, embedded and external but no ajax".
If I google the site, the preview text is the non-changing portion of the text only ("This is an experiment to...") - not even a "No".
I think Google is just trolling us.
Google is crawling and indexing them with zero issues.
Through Google Console/Webmaster Tools, you can tell Google which query params your website uses. But for my modest website, google only uses the page query (?page=3) and I don't notice it using the other queries.
I'll mention that there is a ule that was added a few years back, in backbone.js days, that urls with /#!route anchors will enable (read: force) ajax requests and JavaScript from the spider. Still remains a helpful way to force caching/indexing of JavaScript-only pages in Google.
I have literally heard every combination of practices with regards to SEO and have no idea what is truly correct. Every source contradicts each other, Google employee statements contradict those, etc.
tldr; google indexes js generated content.
I have no other reason to believe it is true other than it's what I would do to distribute the job of crawling the web to my users if i were Google :-)
But Google could use this to discover new URLs that are not linked anywhere.
Also if a user follows a link from such page, the URL would be leaked via Referer header so it is not secure anyway.
I believe it was this setting (I have it disabled as well as other options - however I believe it's only this option that sends data back) [1].
Yes.
There are sites that can't be loaded without javascript that are indexed fine by Google. The only explanation is that they run some javascript.