"The key to performance is elegance, not battalions of special cases."
"The key to performance is elegance, not battalions of special cases."
Elasticsearch - for searching/recommendations
Redis - hot data (certain data is only kept in redis)
Postgres - for the rest of data
Clickhouse - analytics
Most of the system is written in Go. Whole system was tuned for performance from day one. As to latency, data from the last 21 million requests today:
p99: 17.37 ms
p95: 6.86 ms
avg: 2.37 ms
I'm not actually sure, just guessing...
Balancing between user experience and cost.
Why should adding features make an app crumble on a single server?
I think the point is that good software is able to serve a lot of users on a single server.
A great example imho is Blender. Features are added constantly but because the software is modular it doesn't have any impact on the overall performance.
Today the problem is that adding features means: adding the latest and greatest lib while having absolutely no idea about the inner workings.
Yes it takes time to write your own libs. But when performance is an issue you will either have to write your own lib or take one that is good and tested.
It's not inherent, but obviously as you have more developers working on more and more things independently, each with different needs, tolerances, and deadlines, it becomes increasingly unreasonable to presume it can all be managed well on a single box.
If they went and added chat, or Twitter-like features, or subreddits, or similar, it might be a lot tougher to keep it all on a single box. It's a lot easier when we're all looking at the same top 30 stories, and pretty limited in how we interact with them and each other.
As someone who used to hack on Blender all I can say is it's a big ball of inter-dependent modules all with interlocking dependencies. The only thing that really keeps it manageable is the strict adherence to MVC which, I suppose, does make it modular.
Most of the work is server side, e.g. voting ring detection. We only notice indirectly, when the quality of the site goes up.
Some feature requests were a matter of days, like user profiles: https://news.ycombinator.com/item?id=481
A brief essay on some HN design decisions: https://pastebin.com/bSW5dfRQ (from https://news.ycombinator.com/item?id=8424502)
I think the arc codebase is worth studying and understanding, primarily so that you can extend its simplicity into your own projects. The reason HN was such a success is because it handles so many cases in the same way: Stories, comments, and polls are all the same thing: items. If you want to add a new thing, you just create a new item and add whatever fields you want.
These rapid prototyping techniques have downsides, but the cure is to keep in mind what you can't do. (For example, you can't rename item keys without breaking existing items, so be sure to choose good names.)
Like emacs, HN's design is borne out of simplicity and generality. It's what you get when you write the next most important feature as quickly as possible, then cut as much code as possible, every day. Both halves are equally important.
It's fine to say that our modern applications are so much more complicated that the old lessons don't apply. And in extreme cases, that may be true. I don't think SpaceX has the luxury of rapid prototyping their software.
But the typical app is CRUD. For those, data processing flexibility is perhaps the most important factor in whether you can write new features quickly. And since code is data, a lisp master can write systems with a shocking number of features in shockingly few lines of code. (See Jak'n'dexter: https://all-things-andy-gavin.com/2011/03/12/making-crash-ba...)
Are you talking about the source code for ARC, or for Hacker News?
It would be interesting to see how ARC is being employed on such a high profile site. I expect that some algorithms won't be freely available so as not enable people to game the site, but the rest would be interesting to see.
I could find any source for Hacker News though.
> And since code is data, a lisp master can write systems with a shocking number of features in shockingly few lines of code
You don't even have to be a master. So much bikeshedding has been spent over the decades. We are still going back and forth on data interchange formats...
Current hn code (as in the actual code that delivers this comment) isn't open AFAIK (partly because of the shadow banning, filtering etc code.
But there's a full "news" site in Arc source - old and more maintained/evolved:
https://github.com/arclanguage/anarki/blob/master/lib/news.a...
pg's and rtm's original arc3.1 is on the "official" branch: https://github.com/arclanguage/anarki/tree/official
news.arc is the old HN source code. You can run it by following the steps in how-to-run-news.
(Run it with "mzscheme -f as.scm" though, not mzscheme.)
That they keep changing (many times against express complaints of their users) is not reason for excusing them a bad experience.
Incomplete. They are fronted by one of the largest CDNs in the world, on whom they rely for most traffic.
Well, I actually don't know the numbers, but I do know that for popular posts HN admins (a) try to break the conversation up over multiple posts and (b) plead with users to log out so as to allow Cloudflare to handle the load.
The biggest lesson HN teaches for designing large scale systems is "use a large scale system someone else has already designed".
They now have a few more bits: https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar...
I guess the advantage of not relying on your site for profit is that you can afford to not worry about a bit of down time so much...
When HN breaks into the top-100 for the world, let us know what it runs on.
FreeBSD 10.1
Nginx 1.7.10
Racket 6.1.1 with some HN and FreeBSD specific patches.
2x 3GHz Intel Xeon-IvyBridge (E5-2690-V2-DecaCore)
8x16GB Kingston 16GB DDR3 2Rx4
SuperMicro X9DRI-LN4F+_R1.2A
Adaptec 71605
9x 1000GB Western Digital WD RE4
2x 200GB Smart XceedIOPS SSDYou can do all of it locally (except multi zone and multi region reliability) but it would take a lot of configuration, skills and you wouldn't have the same Gui for everything.
AWS is not cheap but companies don't care, because without it, you have to pay to find experts and keep them happy, which is a lot harder.
Some older discussion on the topic here: https://news.ycombinator.com/item?id=3165095
FreeBSD 11.1-RELEASE-p4
CPU: Intel(R) Xeon(R) CPU E5-2637 v4 @ 3.50GHz