Static site generators focus on the wrong thing
blog.pankajmore.in
blog.pankajmore.in
CMS designs tend to fall somewhere between two extremes: completely static (grinding out HTML files) and completely dynamic (generating new pages for each request). Static is appealing because it scales like crazy, but it's also unappealing because it requires a "compilation"/"rebuilding" step that gets longer and longer as your site gets bigger. Dynamic is appealing because it requires no "rebuilding" step, you just click "Save" and your words immediately appear, but it's also unappealing because "live pages" equals "lots of hits to the database", which makes scaling to handle even moderate amounts of traffic a problem. In other words, there is no "silver bullet" approach, just a set of tradeoffs.
The market oscillates between these two extremes, usually on the basis of whatever product is fashionable at the moment -- a new It Product gets lots of buzz for the positive elements of its choice of static/dynamic, which are apparent immediately, but its newness means it hasn't been used long enough for people to experience the downsides. Those come later. And when they do, people flee to a new It Product on the other end of the spectrum, where the process repeats.
Such a setup will scale like crazy and skip "compilation" steps
SSGs are just a rerun of the 90's and don't solve anything that Varnish doesn't already. Pages are an increasingly quaint notion on an increasingly dynamic, personalised, multi-device web. Time to publish is ever more important as Google SEO measures news sources in minutes and publishing workflows often require instant publishing and internal search where there are handoffs (e.g. a subeditor and picture desk).
The two real problems I see is that ESI is our best solution to rendering (and is a clunky 90s-style design), and that object cache refresh has a tendency to combinatorial explosion.
Thoughtful articles require time to write and edit, reviewers will spend time on them. Re-running a static site generator to publish them is a fraction of the total time it requires to write a good article.
If, on the other hand, you are not in for quality, then yes, time to publish may be important for you.
The argument is sometimes made that this doesn't apply to simple blogging and that SSGs of course aren't aimed at pros, but I can say with some certainty that the segment of bloggers who can and want to publish with Git etc is infinitesimally small and quite possibly 100% represented on HN :) That's fine and great if it works for your use case but it pains me a bit to see people re-discovering the very old and disproven model that SSG represents. It's a) highly niche and b) counter to many current and new requirements in content publishing.
The main argument for it is performance but I don't see how it improves on Varnish - albeit w cache warming - and tellingly edge cases of cache warming are getting replayed in SSG increasing complexity. In fact dynamic systems often forgo cache warming as not worth the result much of the time (and can do it selectively), but you don't get that control at all in an SSG system.
Just to single out a shortcomings I've recently found: multiple authors with multiple languages, defining short and long version of bio/tagline. The tagline is displayed at the article, which may be tagged with multiple contributors with different roles (author, translator, reviewer), while the author page displays the long version.
I've spent days to figure out what plugin/theme combination (Wordpress or Drupal) would enable me this single feature, and I couldn't figure it out.
Yesterday evening I gave up, and now I have <200 lines of code in Dart (using markdown and mustache), generating static pages that were annotated with metadata, doing just that. To be honest, I'd trade it any time for a decent CMS, I just can't find any.
Clearly static doesn't suit all sites but if the site is infrequently changed and not too complex or dynamic it is sometimes the best choice. If you have highly dynamic content with multiple authors static is clearly a terrible choice. This is not a zero sum game, there is room for both approaches.
Is it common for people to set up a database backed website without caching? The framework I'm most familiar with, Django, will cache the results of database queries, so if, say, an article on the site becomes popular the database is hit once, and thousands of views are served from the cache.
Edit: Jweb_Guru, thanks for that clarification.
This is particularly true of the current plethora of 'static site generators', many of which are not really improvements in any aspect.
If you build a static site, you can concentrate on making sure a rebuild is quick; if you build a dynamic site, you can concentrate on making sure the data structure is robust to scaling.
You might never be without these problems, but you can minimise them to the extent that they are no longer costing your users excessively.
It's not that we need to go back and forth: either solution could work perfectly fine for all but the heaviest sites.
Another feature for the (not to distant?) future will be generating a TF-IDF index in javascript so that a client-side search functionality could be introduced.
I also find Gus to have a pretty low bar to publishing. I write in markdown/textile/reStructuredText, build, and rsync. Granted I haven't used many other static generators (the ones I did look at didn't seem to have the features I wanted like tag or date indexes, plus it was one of my first python projects).
I thought a number of SSGs already did this.
The OP complained about it and I was just pointing out that it doesn't have to be that all pages are recompiled.
Dynamic sites run so much code during page generation that there are a lot of ways in which you could try to use all that code to do something unintended. And some of those ways pay off.
A static site cleanly separates the moment of code execution and the moment of serving up the content in time, does not even have to happen on the same server.
So from an operational security perspective, if you can run a site statically you probably should. And what with blogs outsourcing comments to disqus and the like quite a few of them are candidate for being served up statically.
As for the performance element, if a site has 1000 pages then re-generating that site will be the same load as serving up 1000 pages when the caches are cold. So that's 1000 visitors. If that's slow then it is annoying to the site admins, not to the visitors. It's an off-line process, it doesn't have to be fast to work. All that matters is that the end product can then be served up blazingly fast.
Any website that does not require user generated input or logged in users is a really good candidate for this workflow.
It's very easy to bust the cache (simply affix a random bit of nonsense to the end of the urls) and force dynamic execution of every request you make, opening you up to all the security issues that a direct dynamic front-end would give.
If you don't then the front-end cache will have to play it safe and will have to pass the request to the backend if it does not know the URL.
Any kind of variability in the request that causes a conversation between the front-end and the back-end and the cache can be busted, or is a potential DOS vector.
Please show me a caching front-end with a dynamic back-end that you control and that you think is protected against cache busting and we'll discuss that off-line. I'd hate to expose you to some joker taking you down just to make the point.
A cache can even become it's own enemy, for instance, if it passes requests on to the backend server if the GET contains url parameters and these make the request unique in the eyes of the cache but not unique in the eyes of the back-end then the cache could easily eat up a ton of memory even though from the back-ends perspective the same page got hit over and over again.
Caching is tricky.
TFA argues exactly against this point. All that matter (or rather, what matters more) is minimizing the friction for the blog author in order to stay motivated, not if it takes a fraction of a second more for a page to load.
When I tried to decide between a Middleman app with minimal configuration work and easy deploy (Heroku), or Ghost, Wordpress, or something like that, one consideration was page generation time.
I calculated that most optimistically, I'd probably write at most three articles a week (which is very optimistic). That means approximately 150 articles a year. That's pretty quick to generate, so I could even leave the generation process to heroku on first page refresh if I wanted.
It's no different to 'clear the whole cache every time I change any content and store it all in memcached' - you're just substituting 'file-system' for 'memcached'.
Of course - SSG do solve another problem - that of needing complex software running on your server - you can run SSGs locally or on a centralized server - but that's not what is being discussed here.
There's no reason why the SSG needs to rebuild the entire site if you've only changed one thing.
On the other hand, if you start with a SSG, your initial state is that every page is cached and works, and you have to think carefully of adding some kind of pagination mechanism.
As the quote goes, "There are only two hard problems in Computer Science: cache invalidation and naming things." The big benefit of a SSG is that you 'solved' the cache problem, and will immediately be aware of issues there.
It doesn't have to be either/or and I don't think the market really "oscillates" in that regard, in my opinion it will eventually provide users the best of both worlds (and maybe already does), at least for more or less static websites.
I'm guessing most of the static site generators probably fulfilled the needs of the programmer who wrote them (which is totally fine and kudos to them for releasing their work so others could benefit). However, static site generators are not what you'd call "user-friendly" (in my view) unless you're comfortable with the command line and like markdown. (Wordpress isn't particulalry user-friendly either, but more so than static site generators in my opinion.)
Unless the generator can keep track of what has and hasn't changed and only rebuild the changed stuff.
Easy solution would be to make a duplicate page like /latest.html which updates as necessary.
Deployment was usually frustrating and broke in non-obvious ways. Deploying a non-trivial Collage site was like gambling.
The experience of switching to Drupal was like night and day. Any bullshit concerns about performance were taken care of with just Varnish or a caching plugin. I can't imagine going back to an SSG and an ecosystem of disparate webapps to make it a usable website. Heck, even my side projects with Wordpress are like switching from a Model T to a Tesla compared to dealing with an SSG. Maybe SSG's are fun for sites with low requirements, but once you have a non-trivial set of features to support, then you're doing extra work and experiencing extra pain that a CMS wouldn't give you.
Also your coworkers wont be cursing your name after you leave them a patchwork of barely held together code in your wake to make the SSG based site do dynamic things CMS's find trivial. Especially if they come from a CMS background.
Can you provide an example of something that CMS's find trivial that is difficult to do in an SSG?
Although, I just switched over from Wordpress to Jekyll, and I find it there to be a lot less friction for publishing. I talked more about the new and exciting workflow of Jekyll at http://rmorabia.com/redesign
I really don't agree with the author here. Hackability is what's aided my decision to publish more. It's just that a lot of people get caught up in the hackability more than the publishing.
Also, there's no data backing this post. I've seen plenty of active Hakyll sites. http://gwern.net is my favorite example.
"However, by posting or transferring content to Medium, you give us permission to use your content solely to do the things we need to do to provide Medium Services, including, without limitation, storing, displaying, reproducing, and distributing your content. This may include promoting your content with partner companies or services for broader broadcast, distribution, or publication."
(They do say they won't sell your content to third parties without your consent, which is good.)
And you don't control if and when the terms of service change.
I do think that most of the static site generators focus on the wrong thing. The main appeal of a SSG to me is that you can focus solely on the content and none of the other crap typically associated with blogging (updating wordpress/security issues, trying to write in some stupid WYSIWYG web form, dicking around with formatting tags, setting up caching plugins because my blog falls over when it's on HN) - all I do is write some Markdown and I'm done.
Can't upvote this enough. I want my blogging tools to just get out of my way. (I ended up writing my own for that reason, because none of the ones I tried did it well enough: https://github.com/pdonis/simpleblog3 .)
I want to be able to write on my computer, in Markdown, using the tools I want (I blog in either Emacs or iA Writer, depending on whether I want to be in "programmer brain" or "writer brain"). I guess I spend enough time in Markdown that I don't need to preview that much, so I don't find the switch to the browser and reload the page cycle especially frequent or onerous. Definitely less onerous than the click preview and wait for it to load cycle I had on WordPress.
Having to log into a browser and type into a crappy little rich text editor would lower my (already extremely low) blogging frequency to non-existence.
But hey, that's my choice -- you should use whatever works for you. I'm just glad that we have the plethora of options.
The truth of the matter is, for most web writers, any CMS is plenty performant. You probably won't even need a cache, and if you end up needing one, it's usually pretty straightforward to add it. And it's pretty self-evident to me that updating one document and saving it is easier than re-generating the updated parts of your website with some script before git-this-git-that.
One good argument against CMS is security: many of them have a history of security woes. This is partly why I ended up writing my own very minimal CMS engine. All it does is read some markdown files and render them. For me, that achieves both security (or at least if not, it's easier for me to go through my 100 lines of Ruby code than the Wordpress codebase) and low friction.
When you do this, you work with the platform as long as you want. Then you can walk away from it - all your content is in your git repo and up on your web host as though the middle layer never existed. Another service can pick up on the same task, which you then point to the same endpoints to continue. If you know how to use git and your host directly, great. If you don't, who cares, these services take care of it for you as a transparent endpoint.
In fact you could even charge money for this kind of thing. Or even make it more open ended than just blogs and web pages, but that's where my thinking got jumbled on this idea and I only got as far as buying a domain for it.
Obviously sites like medium and everyone else would never do this, because it means you can leave their platform immediately. People who are capable of executing this idea pretty much don't need it, and it's otherwise hard to monetize. The desire to lock people into things is really what ruins everything.
My thought is to write a book and sell that. It would be aimed at those who want to create content-rich websites (less blogs, more lasting and organized). So the middle man server is just a bit of quick glue and easily replaced/localized.
The goal is to help people get their stories out there without a lot of irrelevant technical choices. Just start writing content and the system gets it out there.
I also like the idea of version control, issues, and wiki planning the github gives. And with prose.io, editing markdown files directly in a github repo is very pleasant.
"Not often updated and mostly abandoned" describes most blogs, no matter which technology is used to create them.
It's true that static site generators are not frictionless, but I think they have the least friction of any non-hosted alternative---at least if you can keep from screwing around with the generator all the time. Hosted alternatives are a completely different matter.
However I just found this site: http://staticsitegenerators.net/ , which is an impressiv elisting, so possibly I'll find something that fits my needs or gives me a starting point.
Do people that use Haskell self select for doing new things?
I love the idea of a static site generator, but they're really, really reinventing the wheel...
Sure, if you're willing to use make as your SSG. :-)
Also, I'm not sure how well make or rake would work with version control, since make/rake, AFAIK, look at file modification times to detect changes, and DVCS pulls clobber those. In other words, suppose I make a change on machine A, re-generate the site, then push; then I pull from machine B. Make on machine B may not be able to tell that the site was re-generated, because the file timestamps are not preserved by the push/pull, so machine B is seeing different ones than machine A did.
Anyways, my point wasn't that make is sufficient (it isn't), but that a lot of SSGs have been pretty embarrassingly feature-sparse. Understandable, I suppose. :)
I use an SSG because they're simple, don't require dynamic page loads and are more secure. However, the 'simple' part of this is wearing thin. Like so much of modern web software the developers only add things to it. They never remove or refactor. So you end up with massive dependencies and libraries from everywhere. It's crap software.
I still use it because I'm too busy to migrate to something else. Also, because I realize that whatever new tool I migrate to will eventually face the same fate. Eventually any compact SSG will eventually bloat just like Jekyll has. It's the nature of development these days.
Markdown -> script to generate a page with a header and footer and basic navigation.
Another separate script to invoke lftp to sync with remote server when ready to publish. Lftp looks after incremental updating.
In emergency, I can ssh into server and edit the html with nano. As I have shell access, I could just run the page generation script on the server I suppose.
* Have blog posts be plain-text files, in Markdown or some other such similar human-readable format. Not locked away in a MySQL database that can't be easily versioned with Git.
* Versioning: plain text lends itself really well to versioning. Diffs are meaningful. It's easy and light-weight, and pragmatic to version a bunch of simple markdown files.
* Finally: fine-tuned output control. This is what Liquid lets you accomplish. You can specify your HTML pages in Liquid, and easily give your blog a new look. The simplicity, ease, and freedom that this system gives a person with basic web dev skills, shouldn't be ignored.
1) Only compile newly updated items.
2) git commit -am 'new post'
3) git push origin master
The first step needs to be built out in the tool itself, but the rest is a simple bash script, really. It's not even something that needs to be monitored. If you did want to monitor it, it's a few simple `curl` commands away from pushing to prowl or NMA. No need to sit at your computer waiting for it to finish.
Seems like a good weekend project - build a set of tools around Jekyll (and/or any number of other static site generation tools), and let the laurels rain down.
prakhar1989.github.com git/master*
$ rake
rake default # List tasks
rake draft # Create a new draft in ./_drafts
rake post # Create a new post in ./_posts
rake preview # Live Preview (use mode=drafts for drafts)
rake publish # Push to github
https://github.com/ojilles/jilles.net/blob/master/_bin/wget-...
(Poorly implemented but works for me)
I came across Hugo (http://hugo.spf13.com) and have been super happy ever since. Here's some of the reasons why:
1. Really easy to install. One binary, no dependencies or package managers
2. Really fast. Renders my entire site with hundreds of posts in less than half a second. Even changes to templates or themes (which affect every page in the site) render in milliseconds. Wordpress couldn't even render a single page that quickly.
3. Because it is so fast the live reload feature Hugo has works really well. Keep a browser window open and see a nearly instant preview every time I save a file.
4. Deploying is as simple as running a 3 line shell script I wrote in a couple minutes. Since I'm already in the terminal running the script is trivial. I can even run it from within Vim. Thanks to Rsync it takes around 3 seconds to deploy my content. Again, less time that it takes to hit preview and then save in Wordpress.
And all of these are the default.
The hosted version of Ghost is amazingly simply to use. Sign up and publish. Done. The download and self-host is a bit more difficult, but nothing any tech-savy person shouldn't be able to figure out.
To me, the simplicity of Ghost is the best feature and trumps any missing features.
And it's 822 LOC, so it's easy to write, easy to handle. No big issues, no PHP security hole, no Wordpress monsters to administrate. Another 508 LOC give users comments with Markdown using Djangoproject.
And, last but not least: I prefer writing my blog articles partly automated using vim. YMMV.
But hey, if you don't like it: just write your own one.
Not necessarily. In most cases, they simply move the external dependencies from the server side to the client side.
jekyll --limit_posts 3
http://scottpatten.ca/2011/11/speeding-up-jekyll-generation....
Does this also update 'latest posts' boxes and tag clouds on all the other pages?
What about the homepage?
(I used OctoPress, not Jekyll but I suspect there is a way in there to use this same trick)
I guess I should put the full Rakefile up on that post. Here are the relevant parts:
desc "generate and deploy"
task :deploy => ['jekyll:generate', 'deploy:deploy']
namespace :deploy do
task :deploy do
`rsync -r _site/ #{USER}@#{HOST}:#{DEPLOY_DIR}/`
end
end
namespace :jekyll do
desc "start the jekyll server in auto mode"
task :server, :num_posts do |t, args|
num_posts = args[:num_posts]
cmd = "jekyll --auto --server --pygments"
cmd += " --limit_posts #{num_posts}" if num_posts
puts "running #{cmd}"
exec(cmd)
end
desc "generate the site one time"
task :generate => :clean do
`jekyll --no-auto --pygments`
end
desc "remove the generated site"
task :clean do
`rm -rf _site`
end
endMy blog isn't my business, although I certainly use content (blogs, newsletters, eBooks) as part of my "personal brand".
For most of the technically-inclined, the bottleneck is the writing process. The friction in the commandline-compile-and-push stage seems to me like a minor concern...because if it took you 20min. to 20 hours to write the post, what difference does it make that it then takes 1 or even 10 minutes to push the post to S3?
And consider one of the benefits of most static blogging implementations, in which the writer can choose his/her text-editor of choice, which is especially useful if you are using Markdown? Another useful feature is that with all of the blog posts hosted as flat files, you can easily grep/replace across an entire folder or even all of your posts, in a way that is significantly more difficult in a database-backed site, in which you have to destructure the schema.
What kind of idiot blogger would put something in a bunch of posts only to later decide to grep/replace it all?...well, it could be something as simple as someone wanting to add affiliate links to all their existing URL references to Amazon pages? Or it doesn't have to be a replace function...sometimes I want to use grep to find terms/concepts I've blogged about before (most search utilities on dynamic blogs do not support regex).
My main blog is a self-hosted WordPress site. I'm OK with it but I find that there's a lot of friction to just build out a post. This is my workflow:
1. Draft and write a post in Sublime Text and Markdown
2. Convert to HTML (self-hosted Wordpress does not yet have a Markdown editor, and I'm loathe to install a plugin)
3. Paste HTML into WordPress body
4. Hit Publish
Simple enough, but it's almost never the case that my text is perfect...and so now I have to either go back to the original Markdown file and restart the steps, or edit the WordPress textbox which means that my canonical version of the text now exists on WordPress.
And did I mention that WP is dog slow on shared hosting? I've setup caching, which has allowed my site to stand up to the Reddit and HN front pages, but when I'm editing/adding content, it can take 5 to 10 seconds to move between the admin pages.
All of this friction means that I rarely update my own blog, even though I have dozens of draft posts in my Dropbox in Markdown format...most of that is because I just don't finish those drafts, but some of that negligence is because the thought of logging into WordPress and going through the process of simply publishing is enough to put me off.
On the other hand, I use Tumblr on a very regular basis, even though it's simply to post photos and captions. Yes, part of my ease here is because I'm just posting photos, but Tumblr's relative speed makes it a service I want to post photos to...And of course, the tradeoff is now Tumblr owns all of my content and I have an even harder time mass-managing it.
So it's all tradeoffs...the OP is right, but not right for many of the kinds of people who have the expertise to build and deploy a static site.