Hacker News Reader

Comment by OJFord | Hacker News Reader

If it were my site, I think I wouldn't even bother with the search: just stick it all on S3, and have one of those 'Google custom search' or similar boxes, so it's static as far as your site's concerned, and just redirects to Google with the `site:foo` filter.

I don't really have a handle on what S3 costs 'at scale', but I think I'm willing to bet it would knock at least the 0 off the end.

View on HN

Search could easily be done if content is indexed in json files.

CDN hosting of a static site is nearly $0 so def the best option in this case. Plenty of providers give free PRO service to OSS projects as well (i.e. netlify)

Lunr.js is designed for exactly that use-case and the example shows that workflow pumping a bunch of documents into an index and saving it as JSON which is loaded client-side:

https://github.com/olivernn/lunr.js

Another JS search for static websites is http://www.tipue.com/search/

It works very well for me on a site with about 300 pages.

Probably the best/cheapest solution is:

A) Get a Linode VM, put elastic search on it and have it load the text. Probably $20/month with that little text, tops.

B) Use something like KeyCDN to cache everything for long periods of time.

I doubt it'll cost $50/month.

A linode vm's cost is not measured in its price tag but in how much effort it takes to maintain. Updates, security advisories, hypervisor crashes, crashes under sudden unexpected load, crashes because you're on vacation and you are having too much uninterrupted fun, ...

As a famous writer once said: "ain't nobody got time for dat."

How much money is git-scm going to lose if the search function is temporarily down?

Maybe not enough to afford a solution with more nines.

The money it costs to get someone to fix it. That person isn't free, unless this is their hobby.

It's not about uptime, it's about the service existing at all. Even if you are fine with an hour downtime per day; at the end of that hour, you'll still need to bring it back online. The "aaS" part is about taking that load off. The nines are a corollary.

Concretely: a dedicated server leaves you with a lot of extra work. A packaged, managed service doesn't. E.g.: s3 for static files vs an nginx server on a VM. Even if they cost the same, s3 would still be a better option for git-scm.org.

> A linode vm's cost is not measured in its price tag but in how much effort it takes to maintain. Updates, security advisories, hypervisor crashes, crashes under sudden unexpected load, crashes because you're on vacation and you are having too much uninterrupted fun, ...

If I had a choice between that and ~$1800/year?

If you don't want to do sysadmin work at the rate of ~$300/hr, that is your business.

The generated Rust API docs, which are uploaded to https://doc.rust-lang.org/std/ (but also the same ones you can download), do something where they generate an index of the entire site as a JavaScript object, so searches can happen client-side. So it's a static website, but search functionality works.

See https://doc.rust-lang.org/search-index.js for the messy back-end.

sounds nasty for blind people andor people using lynx

Blind people don't have javascript?

Why even bother paying for S3? Just use a static site generator and put it on github pages, github is already paying for the heroku vms so I bet they'd be happy to pay less to use their own infrastructure even if the git-scm.com/org site uses more than their bandwidth requirements.

Github pages just seems a step too far - sponsoring the hosting, fine, but I don't think anyone should go out of their way to make git depend on GH.

Well, I was thinking the same, if open source projects from facebook have all the documentation hosted using GHP why not git? And for the search option they use algolia and I think it's free for documentation projects.

I've used Algolia [1] and Swiftype [2] in the past for search on static websites, but recently experimented with lunr.js [3] for our company playbook (built with GitHub pages): http://playbook.wiredcraft.com/

I still want to separate the index in its own JSON file, but so far, so good. Search is fast and index is rebuilt automatically.

[1]: https://www.algolia.com

[2]: https://swiftype.com/

[3]: http://lunrjs.com

Why bother with S3? I'd buy a Raspberry Pi, plug it in at home and call it a day.

As much as I enjoy the Raspberry Pi, I doubt if it could handle the traffic or if it would last long with the limited write cycles of an SDcard.

You could put the site itself on an external drive via USB - and ideally most of it the access be read-only for a static site, so the life of the sdcard shouldn't be as big of a concern.

That said I'd agree, Raspberry Pi's a great but not quite fast enough for serving a high-traffic website.

I know someone who optimized searching the leaked Adobe database (of a few hundred gigabytes if I remember correctly) on a Pi to a sub-second search. That was super impressive and the method used wasn't even obscure (binary search). The same doesn't apply here, but I'm trying to say a Pi isn't entirely worthless.

For example, what pages get accessed the most anyway? I'm guessing the latest source code and maybe the latest release, though most people probably just apt-get git instead so it's probably mostly the source code. Then there are man pages and some other info pages, if I remember correctly. Sounds like the latest release + 90% of those text pages can easily fit in RAM. So memcached? Nah, the Linux kernel happily caches the files that you read from disk.

I don't know the actual numbers but it doesn't sound infeasible to me. A $230 hosting bill is very heavy though, I guess you'd need some serious fiber as well to provide the uplink. But again, without numbers it's all "maybe" and "probably".

Yeah, that was my project, it worked quite nicely. However, the biggest reason was that my Raspberry Pi was my only server at that point, and besides, tranferring 10GB out of my home connection to a VPS would've been too slow for my impatient self. I think the file was stripped of all other data (which brought it down to about 4GB) and I put it on a thumb drive instead.

Nowadays I would have used a VPS for that. The point with S3 or any other cloud solution is that they make sure you're up and running. Even though it might still be useful, the need for good service monitoring is as good as gone when moving to one of these cloud based platforms. And then I'm not even taking into account the time you have to invest in setting up and configuring a RPi properly vs just pushing a repo to Github Pages, or uploading a zip file to S3.

Heroku or other cloud platforms can be crazy expensive, but for static file hosting, S3 or Github Pages is more than enough and quite affordable.

Raspberrypi.org hosted the entire site using raspberry pi's for a while.

I've just checked and it looks like they nolonger do so though...

    $ curl -I https://www.raspberrypi.org 2>/dev/null  | grep X-Served-By
    X-Served-By: Blog VM 2

Fun while it lasted!

A raspberry pi is also not a CDN. Why even manage servers when all you need is a static site? Deploy to a CDN and if properly configured, everything just works.

Especially with the site being considered in 'maintenance mode', I doubt they want to manage a server aside from the other things they need to do.

Not sure if it's a sarcasm, but upvoted just in case if it is.

That's not sarcasm. It's not like anyone here has numbers to say I'm wrong, and I'm not hearing any good arguments as to why it's a bad idea. I've had my blog (which does 3 mysql queries for each pageload) hit the HN front page with 105KB/s uplink and either a Pentium 3 or Intel Atom as CPU (I forgot when I switched servers). In any case, something much like a Pi can handle HN front-page traffic with a database. And people are downvoting hosting a website that's even less heavy on a Pi? Unless it's 500 hits per second, I don't get it.

Well connection reliability is an issue, for a start.

High availability is the standard these days and you don't get that with a single Raspberry Pi.

Pedantic analysis:

At 5 Watts and $0.30 per kilowatt-hour, a Raspberry Pi would cost $1.08/month to run.

With 1 free GB and $0.09/GB-month, S3 would be able to deliver 13 GB/month at a cost of $1.08/month.

So, RasPi at home gives you "unlimited" egress and a fixed cost, but you have increased latency, a rather small outgoing bandwidth (most likely), and all the downsides of running your own server.

S3 gives you unlimited bandwidth, low latency, and no server maintenance, but it's only competitive on price if you don't exceed 12GB of egress.

Overall I prefer S3, even if I think their egress prices are ridiculous. RasPi at home has some geeky cool factor, though...

NOTE: $0.30/kwh is basically what I pay (California) for any additional usage. These equations will favor home hosting if your electricity is cheaper.

I'd lower that 5 watts estimate. RPi3 idles at around 1.25W and goes to 3.75W at full load.

> I'd buy a Raspberry Pi, plug it in at home

I don't know how I'd feel about a public resource being at the mercy of some user's comcast connection.

It's usually against the EULA of internet providers.

You can buy a hosted moral equivalent of an RPi for peanuts, e.g. https://www.scaleway.com/pricing/.

Reply on news.ycombinator.com