Hacker News Reader

Comment by TimJYoung | Hacker News Reader

This is what I tell potential customers that are evaluating our product that creates single-page web applications where the JS dynamically creates all content: if you want search engines to crawl the content, then you're not using the right product. It's a surprisingly easy decision when viewed in those terms. Most SAAS products that likely would be implemented as SPAs do not care about public search engine access, and are only used by B2B customers where you can dictate a certain minimum standard in terms of browser versions (IE).

Everything else should be a traditional static web site with "decorative" JS, or "progressive enhancement".

View on HN

The "holy grail" architecture whereby the first request sends a fully formed HTML response which is then immediately hydrated with a JS frontend app that takes over routing and data gathering is now pretty technically feasible. It's the best of all worlds ... fast static content for the first load, and then fast frontend routing thereafter. If you ward your clients away from this path you're doing them a disservice. With this stack JS web apps no longer need to be app-like, they're well suited to content-heavy sites too. They even work with JS turned off :)

> pretty technically feasible

Not only feasible, it's extremely easy to do these days with things like React (+Redux). We're about to launch a fairly large project for an Australian broadcaster using this approach and it's been incredibly simple to do.

And there's nothing ~magic~ that React does to make this possible - it would be quite trivial to 'roll your own' framework for this. Best of all, you could start with doing it just server-side and then roll out the JS to 'the other side' later on.

Seems like Ember is working towards this: https://github.com/tildeio/ember-cli-fastboot

Yeah, the cool thing about React is that its fairly pluggable. So stuff like server side rendering is not "React" specific but more so Node specific.

Seems like its what Redux/React is doing at the moment with server side rendering

http://rackt.org/redux/docs/recipes/ServerRendering.html

On server request, render the first snapshot of a React app into pure HTML and send it out, and then that HTML snapshot uses a script tag to get the actual JS file of the react app.

Google can and does crawl Javascript driven web apps. Last year they even deprecated the AJAX crawling scheme that they used to recommend on the basis that they can now crawl web apps: https://googlewebmastercentral.blogspot.co.uk/2015/10/deprec...

Yes, but in the case of SAAS apps, what exactly are they crawling (if allowed) ? In our case, it would be a bunch of edit controls, grids, buttons, containers, etc. that don't really make a whole lot of sense in the context of a search engine.

I think it's important to start recognizing that there's two distinct types of web "applications" starting to emerge: the more traditional kind of "application" that is actually more like a web site, and the newer kind of application that is more like an actual application (desktop/mobile).

Edit: Just read the new guidelines, and they are now considering serving up different content for the crawler as being "against the rules". So that probably cements the idea that you shouldn't allow Google to crawl anything that's mostly content-free and just an "app".

"Don't let Google crawl your site if it's not appropriate" is great advice, that's true, but it has nothing to do with whether you're serving up JS or HTML.

Yeah, I'm conflating the two, based upon my personal experiences.

But, are people really writing applications that generate all content completely on the client side via JS, but want the end result to be more like a traditional web site ? That seems weird, even to me. :-)

Tons of news sites are doing exactly this.

I can't tell you how many times I follow a link from Facebook or other social media, only to be brought to a page where a spinning gif spins for several seconds while JS loads the article.

> Just read the new guidelines, and they are now considering serving up different content for the crawler as being "against the rules".

That is called "cloaking", and Google has said this for a very, very long time. At least 10 years.

They had an exception for it for sending Googlebot to a 'static' copy of your site for rendering SPA.

Google crawls Javascript sites, but it doesn't execute AJAX calls so most SPA's still won't be crawlable by Google.

In theory it works, but in practice it awfully degrades the page load time. The way it works is that it will call the init() method of JS app (ready() in jquery, main() in Dart etc). And it is very sub-optimal compared to server side generation. This is because it has to execute the ready()/start() method, which may spawn of some async requests to get some JSON, on getting which successfully you recreate your DOM.

So too many possible failure points, and for these reasons it will timeout many times, resulting in increased crawl errors, apart from page load time degradation. I learned this the hard way, and had to revert to server side generation of crawlable pages. And now looking at something like react/redux to achieve what I originally wanted (which is basically again server side generation, but while maintaining the quality of the served pages).

Can't upvote hard enough. Applications on the web are amazing, and developers must embrace JavaScript heavily.

That said, the apps do need to remain accessible to those with screen readers, vision issues, motor impairments, and cognitive issues. They need to be accessible to those with slow or spotty connections. But search engine optimization is about content sites, not necessarily apps.

Static pages are perfect for those cases, and I am thrilled to see more people coming around to that.

>Can't upvote hard enough. Applications on the web are amazing, and developers must embrace JavaScript heavily.

So how's it going in JS land, are you guys still reimplementing Windows 3.1's message passing loop and hailing it as a revolution ? :^)

Or maybe you're still busy pulling in literally 10+MBs of code on top of a 100+MB browser to fix the shortcomings of what is a terrible language ?

Or maybe you're busy trying to get performances that matches what a 68060 could do thirty years ago. Oh right, browser vendors had to agree on WASM to get OK performance. So basically, not Javascript.

But hey, I guess it's crossplatform. Kind of.

But promised, once transpilers are good enough, once I can use a good, strongly staticly typed language, once the APIs are stable and the tooling becomes tolerable, I'll join the SPA side.

So what you're saying is you don't know very much about JavaScript but you're willing to learn some once it becomes obsolete like Java?

Stable and field tested, with a majority of the marketshare, as opposed to using the flavour of the day because that's what's hip?

Where do I sign up ? I mean, I do C# day to day, so boring is perfect :)

Also, much to my dismay, I do try things outs, maybe not extensively, but enough, before criticizing them. I've used Angular, React, Ember, JQuery (yes, working on a 100% jquery SPA. It's as fun as you can imagine), then other things like Vue.js, Knockout, and quite a few more. Even Vanilla JS. So, while I will not pretend to be an expert, or even having advanced knowledge of Javascript, I'd like to think I kind of know what I'm talking about.

That's good to know, and I promise I wasn't (just) being snarky! The problem is that unless you're working with something low-level like C, by the time it's stable and field tested, the language has also gotten kind of brittle, there's a mountain of bad code, and new features don't undo the damage done by the bad patterns of the past.

Javascript also has mountains of bad code and bad practices, but a lot of it's being accumulated in frameworks which are getting created, tested, found wanting, then dropped, with the good parts surviving on, either in new frameworks or as part of the standard.

I won't lie though, I'm as framework exhausted as everyone else right now and it can't go on like this for much longer. Using something like C# (which I see as "good Java") would be a nice change of pace. Definitely check out Ampersand and Webpack with React, they're kind of coalescing to provide a saner, "use only what you need" approach.

I agree with all of those points.

But meeting the needs of customers is what I get paid to do. And customers don't want server-side rendered stuff. They hate losing their place when they add a row of data.

So what are ya gonna do?

See, I hate JS. I think it's terrible. If there were any other choices out there, I don't even think it would still be a thing if there were any other options.

But it's a requirement of a modern web developer to know how to do it well. And I'm a web developer. So I'm going to be good at it. At least as good as I can be given the tools I use and constraints I have.

> But meeting the needs of customers is what I get paid to do

That was like being slapped with a good dose of pragmatism – I love it!

Wrt the later part of your post, try Dart. It's annoying in entirely new ways, but it does away with a lot of JS pains; it's also mature, powerful, and production-oriented. It even comes with a VM to run it in, so you can pretend it's a real –boy– language. Its obscurity also means that it can be hard to come across help/tools (emacs completion, anyone?), but despite all the shortcomings I enjoy using it, and am happy to back Google's bet on Dart.

Thanks for the recommendation. I'm currently annoying myself by learning Elm. :)

> But meeting the needs of customers is what I get paid to do. And customers don't want server-side rendered stuff. They hate losing their place when they add a row of data.

>So what are ya gonna do?

Both.

Do the initial render (well, HTML generation) server-side, and update the state client-side when the user performs an action (or receives an event from the server).

Use real, accessible URLs for links, and use pushState when updating client-side.

I'd love to see books, screencasts, or tutorials that talk about doing this beyond the very trivial. I hear people talk about it, but whenever I ask, they can't show me code because reasons.

This makes me skeptical. And based on what I see in the wild... lots of loading progress bars, etc, I don't see widespread use of this technique.

"This makes me skeptical. And based on what I see in the wild... lots of loading progress bars, etc, I don't see widespread use of this technique."

If you're looking for a "big" site that does this, then: Yelp. All of the content on yelp.com is statically rendered, with progressive enhancement. Turn off javascript, site still works. It looks nearly the same, in fact. So go inspect source to your heart's content.

There's really no magic to it, and nothing of which to be skeptical: you first generate static HTML and CSS, and you treat javascript as an enhancement to that content, rather than as as the content itself. This is the way things were done for years. It's only since 2013 or so that people have lost track of this "ancient" art.

No, no, I get that. That's how I do my stuff and have always done it. We used to call that "progressive enhancement".

But it's easy to say "ehhh just render the whole page statically first then re-render parts in JS".

I'd like to see people demonstrating this with modern frameworks and tools. Because without people leading, it'll be JS-rendered static content for years to come.

The GP wasn't saying that, as I read it. It was saying: "use progressive enhancement".

When you insert new elements in the DOM with JS, you are re-rendering parts of the page. I see no other, non-invented reason that anyone needs to render static content with javascript. Devs do it primarily because they're annoyed that they have to think about AJAX logic, and want to write simpler code -- at the user's expense.

This post is dangerously approaching flamebait.

Doesn't necessarily make it false... JS is such a piss-poor language. I'd rather write QBASIC if it ran in a browser.

Agreed, but ES6 is a big improvement. I've just gone back to it and I must admit, it's not actually too bad now if you use babel + webpack.

>QBASIC if it ran in a browser

http://tayeandjoe.com/qbarcade/games/qbasic.html

This is probably going to send this whole thread on a wild tangent, but here goes...

I've posted this before, but check out this link:

http://www.elevatesoft.com:8081/maxgridtest/maxgridtest.html

That app was created with our web development product (check the site if you want to know more), and has the following characteristics:

Two files, one HTML loader and one monolithic JS app, so latency during loading is minimal. The HTML loader is ~189K and the JS app is ~462K, and includes the entire runtime and UI layer, and a lot of the control/component library. Both the HTML and JS are aggressively compressed/obfuscated by a compiler, and the coding is done in a statically-typed, OO/procedural language with RTTI and other nice things. The UI was designed using a WYSIWYG designer with two-way tools (code-behind).

So, there are products/tools out there that will do something along the lines of what you want. And the existing JS engines are very good in terms of performance, so all that developers like us need to do is some quick compilation to JS and we're all set.

However, I do agree with you on two points:

1) JS, by itself, just isn't structured enough for large-scale applications.

2) The push towards libraries and away from frameworks was misguided because JS, by itself, doesn't have the means to allow for this approach to be successful. Instead, what we have now is every single small library reproducing the same functionality over and over again. Case in point: I was looking at writing an external interface (tells our compiler how to type-check external JS code) to ChartJS this week (great little library), and started looking at the code. 80-90% of the "common" code in the library was code that was already present, in some form, in our UI/runtime layer, and that was around 70K right there. Multiply this by the number of small libraries, and you end up with a lot of duplication of functionality that is, essentially, dead weight. I don't know if it's 10MB of dead weight, but it's pretty significant.

190KB of maxgridtest.html took 1.44 s to download

462KB of maxgridtest.js took 2.63 s to download

5.2MB of datasets?method=rows&dataset=IPCountry&Country=%27United%20States%27 took 21.22 s to load

"39246 rows load in 40948 msecs" - 41secs total just to see something!

This is not an awesome presentation, and misses some of the stuff the previous commenters mention about "windows 3.1" like "don't load an entire dataset in memory, but show a small slice of it at a time". At the time, memory was very limited, so developers were forced to be efficient, and it was good for users. Even on my old 486 DOS machine, I could load up a spreadsheet application, and navigate through it (with way more than 40k rows) at nearly instant speed because it was smart about what it was doing and what it's limitations were (memory, disk access speed, etc)

Our product is for developing applications, not consumer-facing web sites. It is expected that the first hit on the HTML/JS is going to take more time than normal (we offer progress options for this). After the initial hit, the browser will cache both, and the load is instantaneous until it is updated again.

Also, the total size/download time that you're seeing isn't for just a combo box and a grid, it's for the entire UI layer and a lot of the component library. IOW, as you add more and more to the application, you're only going to see very minor increments in the total size of the application because most of the code is already baked in. Most extensive client-side JS UI frameworks are at least 300K or so, minified.

In retrospect, it was probably a bad idea to post that particular example without an explanation of why that app was created. The whole point of that particular example is to show that you can handle large numbers of rows in JS apps without killing the browser if your UI framework does smart things like implement virtual grids. Many other frameworks don't handle things very well when the number of rows grow beyond 1000 or so, and memory consumption gets out of hand very quickly.

After the initial hit, the browser will cache both, and the load is instantaneous until it is updated again.

Ok. That's pretty much the definition of normal HTTP caching policy though.

Many other frameworks don't handle things very well when the number of rows grow beyond 1000 or so, and memory consumption gets out of hand very quickly.

Fair enough. Obviously you've got the UI down as it's butter smooth once everything is loaded.

But many of the popular JavaScript grid systems already have stuff like virtual grids, and have had them for a while (scroller[0][1] for datatables was released in 2012 for example). They may not be quite as smooth as yours, but the DataTables example[0] (timing from the moment each has started loading of the data until there is some visual data on the screen) is less than 300ms, whereas yours is still over 10 seconds (not a 100% comparison as yours is 5MB of data). Definitely rather have people working on actual product, rather than waiting 30 seconds to do something (and each time the dropdown is switch to another country and back it's another 5MB download and 30 second wait).

we offer progress options for this

Then please, please show this. Showing people a demo of good code doing smart things is a great first impression. For example, a demo of 5 million rows of data via server side processing with that butter smooth animation would be amazing. Then when people are hooked, and start asking about doing stuff like loading 100% of their giant files into browser memory, you can impress them more with how there is no slowdown other than the pain they cause themselves by not loading data piecemail.

But ultimately, any demo that takes 40 seconds to load is just going to be unsavory without a lot of context.

[0] - http://datatables.net/extensions/scroller/examples/initialis... - client side scrolling through 50k records (generated inline rather than downloaded, making for an unfair comparison)

[1] - http://datatables.net/extensions/scroller/examples/initialis... - infinite scroll on server side processing of 5M records

Thanks for the links.

Just to clarify: I would never recommend that someone actually try to load 30K+ rows into a grid in a web application. So, most of the countries not called "United States" is probably more representative of the typical usage of such a grid. The majority of the time spent is not the server request, but rather the loading of the incoming JSON data into what we call a "dataset".

Having said that, we've had a lot of requests for more incremental row loading, so that will definitely be in the works at some point in the future. Initially, we tried to make the "datasets" as dumb as possible, but information about primary/unique keys are available from the back-end, and it is possible to progressively load the rows as necessary (without requiring manual pagination). But, again, loading that many rows isn't a typical use case, and we normally advise against it.

It might be great for the developers, but frankly, as a user, I find that demo very disagreeable. It breaks multiple useful UI conventions that work fine in native browser components, and it still took 5s to download a dropdown and an empty table (for comparison, the HN frontpage takes <1s here).

Please see my reply above about the loading time.

Also, which UI conventions are you referring to, specifically ?

Also, which UI conventions are you referring to, specifically ?

I'm from Portugal, so I click on the dropdown and press P. The standard behaviour is to jump to the first result, but this one automatically selects it, closing the dropdown from under me and starting an update.

There's no horizontal scrollbar even when the content doesn't fit, so if I reduce the window horizontally, it becomes unusable.

When I click on a row of the table, nothing changes; I must move the cursor to see it has in fact selected it.

And the text is not selectable.

Thanks for the feedback, it is very much appreciated.

The near-search is just how it was coded (it's responding to a selection by loading the rows). It is a bit off-putting, though, so I'll change that.

The horizontal scrollbar is the same thing. You can specify how you want the surface (body element) to behave with respect to scrolling, and the I turned off the scrolling.

As for the clicking, that's the "hot" state over "focus" state preference in the UI layer. We're also thinking of changing that: it used to be "focus" over "hot", but it was changed, and I'm not sure that it was a good decision because of what you describe.

I've updated the example to fix the issues that you mentioned.

Also, the text isn't selectable on purpose. There's an option for our grid control to put it into "row select" mode where it is only used for navigation/selection (similar to a listview control under Windows).

It's definitively much better now, though it still has a weird behavior when pressing a letter to jump in the countries list: if I press P repeatedly to get to Portugal, only 1 in 5 presses actually register, the others are ignored. It seems there's some "cool-off period" after a keypress.

Still, on a larger point, it's not just important that one can configure the software to behave "well"; programmers are lazy, so the defaults are crucial, and as bad as browsers can be, the basis are pretty much solid nowadays, so it's hard for me to be confident that these will be well handled by new frameworks.

By the way, how is the software on acessibility? Can a blind user understand that there's a dropdown and how to select an option?

Yeah, there's a near-search timeout involved in order to allow multiple keystrokes to be used, and it was set a little high as the default (it's configurable as a property). I've bumped the default from 500ms down to 200ms and that seems to be a better fit.

Accessibility is so-so. We use text blocks for any labels, grid headers, grid cells, etc., but I still need to audit the UI completely with the ADA helper tools that are available. As for the combo box, the answer there is "I don't know" until I complete that audit, but it probably won't be as usable as our other combo box options that are actual edit controls (you can select text, etc.). The reason for the combo box that you see here is touch environments, specifically those that automatically pop up a virtual keyboard. It's used for situations where you don't want the keyboard constantly popping up, but you need a button-style drop-down list control.

The problem with the current HTML incarnation, and why stuff like the above is done, is that it's just not flexible enough to handle real-world applications (equivalent to desktop apps) without some serious compromises on controls, or by subverting the HTML semantics. We chose the latter. It's the only way to do things like virtual list controls (the combo box has a virtual list control associated with it). If the browser offered a way to define semantics for custom controls, it would help immensely.

Another minor ones: 1. If I click on the background with the mouse so the input lose the focus, I cannot use the tab key to bring back the focus on any of the input elements (in Firefox)

2. Cannot use Ctrl + mouse wheel to Zoom the page, but Ctrl + "+" works.

3. The dropdown list opens using any of the mouse buttons not just the left mouse (should not open on right click or middle click)

4. Cannot select the data in the list with mouse (but can select whole page text with Ctrl + A)

Also it's strange that you are talking about fast load time but I see the opposite: 660KB loaded in 4.5 seconds for the inital page load.

Some performance tips:

It should be obvious: use HTTP compression, a quick check shows: the total size could be reduced to 27% ! of the current one. Maybe your custom server ("Elevate Web Builder Web Server") doesn't support it yet? This could help with the HTML and JS load times.

In the case of the countries JSON it's not the transfer size that is slow but the response time of the server: 600ms for a 8KB JSON is not so fast especially if the data is not changing and easily cachable.

If it's a single-page app then you could embed the JS in the the HTML so all the data comes in one continuous request, and if even the initial JSON data (for the countries) is included then additional more than 1 second can be saved that is currently there between page loaded and starting to load that JSON.

Also loading big dataset by selecting United States from the dropdown freezes the browser and Firefox shows the unresponsive script warning (maxgridtext.js). There could be some processing in the JS that may not be necessary to do for the whole dataset but do it only on the visible part of it or it could be done on the server.

I post this because I see you care about the UX and performance and I hope it helps you a little.

Thanks for the feedback. I fixed the focus issue, and I'll look into the keystroke/mouse handling.

As for load times, I never stated that the initial load time was fast - I said that the latency was low. As you say, combining the JS into the actual HTML file would improve it even further.

Yes, the server doesn't support gzip yet, but will soon. It's basically our "here's a web server to get you started" web server that we include with the product, but you can use any web server that you want. We actually didn't even plan on including one, but you know how plans go..... :-)

The JSON isn't cacheable, exactly, because it's the result of a database query. So, the response time you're seeing includes setup/teardown time for the database connection, etc. Again, this isn't production-level stuff here, just a one-off coded to show something in particular.

As for the loading of the "United States": the server request is actually very fast, but the loading of the JSON takes some time. We do custom JSON parsing in order to validate the JSON properly and allow for missing column values. We investigated the built-in JSON parsing, but we are still left with the same issues in terms of finding whether certain column values exist in the resultant JS objects, and would end up using almost twice as much memory because the resultant JS objects would still need to be copied into different target structures.

«But promised, once transpilers are good enough, once I can use a good, strongly staticly typed language, once the APIs are stable and the tooling becomes tolerable, I'll join the SPA side.»

Typescript is a plenty good enough, statically typed language that transpiles to whatever flavor of JS/ES you need, including ES3 if for some reason you are stuck supporting IE < 8. Typescript has a good ecosystem of tooling (try Typescript in Visual Studio Code, for instance).

As for API stability, it largely depends on the SPA Framework you want to try. Yes, the problem is that there are so many to choose from and some of them do wonky things like reimplement the old GUI message passing loops and call it progress.

If you want my biased opinions on SPA frameworks: These days I use CycleJS when I get the choice, as it is simple, gets out the way, and built on top of RxJS for true reactive programming. It's simplicity provides a very small API overall and thus a considerable amount of stability for said API because of its smaller surface area. If you need something a bit more time tested with the broadest classic browser support, I think Durandal (Knockout-based) is a stable, well worn SPA framework. Having used Durandal in the past I admit that Aurelia, it's successor, is likely to have a similar stability, as it matures.

TL;DR: The transpilers are good enough if you give them a shot. Typescript is a great statically typed language for the web. SPA APIs are crap shoot, but there are good options out there, especially if you stray just a tad out of the way of the hype trains.

It's not very fair, I feel, to drag browser binary size into it. Those hundreds of megabytes give you a UI. If you have a QT or GTK app then the dependency you will drag in will not be small (and heaven forbid you depend on some Gnome or KDE lib).

I do agree about the browser UI performance though - why I can't scroll a page without the tearing effect in 2016 on an 8 core machine with 16GB of RAM is beyond me ...

I didn't realize perf was that bad. Apparently, a 68060 could run stereo 3D content on an Oculus Rift.

Why does it have to be one or the other, though? You can have static, pre-rendered landing pages for search engines to crawl and index, and your actual application can be an SPA. Right?

I had always assumed this was just the norm.

The vast majority of applications don't need to be searchable, the website that shows off the app does.

Absolutely. In our case, it's a non-fit because of the client-side content generation via JS (our apps are more akin to desktop/mobile apps with a monolithic JS app). But, if you already generate most of the HTML/CSS from the back-end and just sprinkle in some JS here and there, then you can certainly do a hybrid approach, like you suggest.

Universal applications get the best of both worlds. That is why we created GlueStick https://github.com/TrueCar/gluestick/

Reply on news.ycombinator.com