How to build a HackerNews Clone using Redis | Hacker News Reader

How to build a HackerNews Clone using Redis

developer.redis.com

75 pointsgoogleroadies49 comments

I have also seen some videos on Youtube sponsored by Redis, they want to hammer the point that they're not just a caching tool, but a more serious DB.

https://www.youtube.com/watch?v=OqCK95AS-YE

Can someone speak more to this? I've also seen it stated that it's possible to use redis for durable data storage, is that really the case? What do you lose out on?

They have two different approaches to persistence, with different tradeoffs: https://redis.io/topics/persistence

Either would have the generic disadvantages of:

- Dataset can't exceed available memory

- The general disadvantages of NoSQL vs SQL around queries, transactions, etc, if you're expecting relational type functionality

> Dataset can't exceed available memory

From my understanding, they do support sharding

Sure, but still limited to the total amount of memory of all machines. You can't page in/out from storage.

You can store Terabytes of data in Redis. Just google it.

Yes, in Terabytes of RAM, or some separate commercial offering with flash. Not sure I get your point.

Redis supports clustering and sharding: https://redis.io/topics/cluster-tutorial

There are also managed solutions that implement Redis with partial flash storage. Redis Enterprise and AWS ElastiCache both do this.

From my experience it works pretty well, the issue is that the idea that it's temporary storage is so prevalent that I wouldn't risk it.

From what I saw, people think two times before doing `truncate table` but not that much with `FLUSHALL` in redis...

Also, I don't really see where the usecase would be as a permanent datastore compared to something like postgres.

I have found this to be useful: https://github.com/fpereiro/backendlore

It is more about writing backends generally, but in it he uses Redis as a primary database and walks through durability and scalability.

Broken and laggy scroll, hijacked back button. Why can't we have normal web pages again?

Hmm, I am not experiencing this though I am using FF with some anti-click jacking plugins...

I'm just using uBlock Origin, what do you use?

I use the following, in addition to NextDNS.

  - uBlock Origin
  - Privacy Possum
  - DuckDuckGo Privacy Essentials
  - WebGL Fingerprint Defender
  - Font Fingerprint Defender
  - AudioContext Fingerprint Defender
  - CSS Exfil Protection

Opened in FF and Edge, and am not experiencing a hijacked back button, nor laggy scrolling.

If I were to build a Hacker News clone, I'd probably do it in a seriously old-fashioned way:

* MySQL for a datastore

* All rendering done server-side

* PHP for the language

* As little JavaScript as humanly possible, and only vanilla, no libraries or frameworks

* As few images as possible

So basically, what the current HN is sans lisp.

Yeah, pretty much.

An actual clone, no real value add. But it'd be fun to do, regardless.

There’s a full feature clone in Python on GitHub, in case anyone actually want features, not just tech learning: https://github.com/datatau-net/datatau-net

You know... Hacker News is built with a pretty-obscure framework IIRC that is little-used elsewhere, and we're all about open-source. It would almost make sense to reimplement HN in something more common and then open-source it so that regular developers could make improvements.

I'm solidly in the 'if it ain't broke don't fix it' camp on that one.

I'm solidly in the "Hacker News is the Craigslist of web forums" camp. It may not be broke, but there is a lot of room for improvement...

I actually love the design of Craigslist. it works well enough for you to find what you're looking for (if it exists), but it does not invite you to linger afterwards.

I have to be honest, the 'craigslist of web forums' is the entire appeal for me. It's simple, fast, low clutter and in my opinion benefits from great moderation and a general involvement from the community to avoid treating it like a gutter. I could be suffering from limited imagination, but I'm not really seeing major opportunities for improvement. Change? Yes, for sure, lots of ways to change the site but I don't know how they would result in an outcome of improved community or experience.

improvements, like, making yourself supreme ruler?

> Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith. https://news.ycombinator.com/newsguidelines.html

It's fine as it is, do we really want it to go the direction of Reddit with broken navigation, live chat, typing notifications?

I maintain a pretty high quality implementation of Hacker News. https://github.com/tensorfork/tlarc

It has bcrypt, HN's current CSS, and so on. We use it to manage our TPUs: http://www.tensorfork.com/tpus

Improvements welcome. The nicest part is how quickly you can iterate on a webpage. The production site pulls from github every 10 seconds, and everything reloads on demand, so I can just push new code to master and see the changes instantly.

(No need to push to github just to see changes locally, of course. Every time you reload, it checks whether any loaded files have changed.)

How much does it cost to run a website like HN?

A lot using redis as the backend.

Why would using Redis add significantly to the cost?

Because Redis' database is stored exclusively in RAM, which is an order of magnitude more expensive than even SSD.

Also, long-term maintenance effort and costs would be substantially higher. (Redis persistent-to-disk options, namely AOF and RDB, are notoriously unreliable.) These issues would persist (ha) even with a clustered setup (actually, especially then.)

I've never heard of them being unreliable, and certainly not notoriously so. What sources do you have on this? My own impression formed from reading HN is that opinion around Redis is nearly entirely (almost cultish-ly) positive.

I'm just checking if there is something I missed.

> Because Redis' database is stored exclusively in RAM, which is an order of magnitude more expensive than even SSD.

Maybe, but still very cheap. Unless you are talking about insanely high priced cloud services.

Something like Hetzner would provide you with 128GB of ram for 60 euros / month or 1TB of ram for around 500 euros / month.

Not exactly true. There are things like "Redis on Flash" that redis.com offers for large sets of data. If you enable it, most of your data gets stored in a Flash memory and hence super cheap, in fact even cheaper than things like DynamoDB and stuff.

I don't know how much traffic HN is getting, but i think you should be fine on a 10$ VPS for the most part

Probably surprisingly little. Also, the publicity HN brings is most likely orders of magnitude more valuable and that makes it much less important to try and optimise the service.

The pages are very light and the algorithms seem to be designed to be quite light, too.

Now, I know neither Arc nor the traffic stats, but I would not be surprised if one or two machines were able to serve entire traffic.

2018 is was around 6m requests/day https://news.ycombinator.com/item?id=26469566 Hardware used at 4m/day was two machines but it runs on one machine https://news.ycombinator.com/item?id=16076041 So maybe double that now.

(update: I mixed up M5 hosting company with AWS M5 instance types)

Do you have to look at the peaks and design for those? (The hardware link accounts for this of course, but not sure the hardware scales linearly with average load, because the peaks may not scale linearly with average load)

Related question: is it possible to run HN and serve HN-level traffic on any of the major cloud providers' free tier?

Maybe Oracle Cloud. They give you up to 4 cores ARM64 instances with up to 24GB total RAM.

I kinda feel that I should point out that if you want to build a complex app on top of just redid you can as well use zeesql that I develop for exactly this reason.

https://Zeesql.com

I should make it open source...

This is interesting, I've seen stuff like RediSQL as well. It would be nice to see this OSS so people can contribute to it!

Ahahah it is the same, I just rebranded RediSQL to ZeeSQL when RedisLabs got the trademark for Redis.

Reply on news.ycombinator.com