Thredded Forums – An engine for Ruby on Rails | Hacker News Reader

Thredded Forums – An engine for Ruby on Rails

thredded.org

173 pointsksec39 comments

Did not expect to see this show up on HN today (or ever for that matter).

In any case, I wrote the original code for Thredded and it's since had the stewardship taken on by two very smart and capable gentlemen named Gleb and Tim.

In recognition of the 1.0 release I wrote up a short blog post with a little history and background on the project (spoiler: the first version was born 21 years ago!), and congratulating the 2 guys for their work the last 5+ years. They've done an incredible job, and I only wish we'd all met 5 years before.

https://joeloliveira.com/2022/03/04/weeknotes-for-the-week-e...

How do you think you compare to Discourse?

Discourse is a large, huge, very successful project and company but the direction they approached the goal of "open source messageboard/forum" was very different than the way Thredded did. I had hoped to make Thredded a drop-in solution -- they, Discourse, did not. I had hoped to make Thredded as vanilla and "boring" as possible -- they did not (a SPA [ember] on top of rails).

In terms of features, the breadth of their product is larger, more complex, more opinionated. Thredded, on the other hand, hopes to get out of the way if and when you want it to (just look at what @druseph did with notebook.ai/forum).

They did a great job, though. They built a sustainable business on the foundation of "internet messageboards" and for that I applaud them. That was always my goal and I never got there, sadly. That was my dream.

In all fairness, it's been years since I've looked at Discourse so my assumptions may be out of date at this point. Take all this with a grain of salt.

Thanks for the reply. I'm tentatively in the market just now for a discussion forum app for a client and this posting on HN was somewhat serendipitous.

If you end up trying thredded out and need any help or extra set of eyes, I'd be happy to help. Just lmk :)

I will do, and appreciated. Just waiting for the client to firm up their interest before spending any more time on this.

I've been using Thredded for years now on a relatively popular site (~250k users on Notebook.ai; our instance at https://www.notebook.ai/forum) and have been extremely happy with its performance, scaling, and featureset. The devs behind it are super smart and care about code quality. Would definitely recommend if you need a drop-in discussion engine for your Rails app.

How much caching are you doing and what are the CPU / memory / disk specs of you server if you don't mind me asking?

I ping 15ms to your server but page load times are 500-600ms on most threads (10 or 3,000 replies doesn't make a difference). This is checking out the Rails app server response in dev tools.

Page load times around the site are probably pretty high because I don't do any caching (or, really, prioritize load speed in code), but right now I'm scaling between 1-4 of the $250/mo Performance-M dynos on Heroku (2.5GB RAM but other specs are hand-wavy) whenever my p95 goes above 1000ms.

I have had some occasional backend issues in the bigger threads from things like notifying all users of responses in a 20k-post thread, but I think that's more on me for not having anything to scale up more workers as needed. In general though, most things Just Work.

Thanks, that's impressive and exactly what I was hoping to see. Pretty much zero effort into making it faster but it's still decently fast and could probably get literally 10x faster if you wanted to cache a few things.

Also surprisingly not that much RAM usage. Are you using Sidekiq too where this is included in the number of dynos you run? What's neat is (and I'm not recommend switching) it sounds like you could host a massively popular forum and throw it onto a single $48 / month DigitalOcean server with 8GB of memory and 4 CPU cores if you wanted to. I get wanting to use Heroku but for folks who are ok with self hosting that's really affordable for such a popular site that has quite a lot of writes.

Thanks! Glad the details are helpful. There's a lot of optimizations to be made on my end, for sure.

I isolate Sidekiq to a separate worker dyno (a shared Standard-2X dyno with 1GB RAM for $50/mo) and typically run just 1, but up it to 2 if I ever notice a backlog of jobs that need done. Most of the main dynos's RAM needs are for the main site outside of the forums, which is way more resource-intensive.

On Heroku: I'm admittedly very terrible with devops and their UI has made it really easy to scale resources for viral shares without the pressure being on me to figure out how to scale properly (and their support has been super helpful in cases where the DB failed, certs expired, etc). Thredded itself hasn't really required much resource scaling, so Heroku's probably overkill for it unless the rest of your app needs it. :)

Having just added some fragment caching as well as cache headers (basically cache do...end around view fragments and some fresh_when in controllers) - at a glance it looks like there's a lot of low-hanging fruit for increasing performance in the thredded code-base.

The project i adjusted was an old rails 3.2 code base, and in addition to fixing some missing db indeces - the caching sped things up significantly (10-100x, from atrocious to acceptable).

I sometimes get the feeling that no-one using rails ever read the cache section in the guide.

https://guides.rubyonrails.org/caching_with_rails.html

Caveat: there may be callbacks and authorization hooks in thredded that hamper this approach in practice.

For a new project i would probably try and go with "fresh when" (http caching) and varnish or fastly.com in front.

But fragment caching can make a lot of difference too. More than I expected.

Ed: although I can see why it might be missing - the upstream demo feels plenty snappy - so for most uses, extra caching might just be extra complexity for little gain.

I was expecting a much much faster site. Instead I thought the site was really slow. StackOverflow manages sub 20ms response on nearly all pages time with zero caching. While I dont expect Ruby Rails to ever achieve that performance, 500ms response time is just painful. The bar needs to be far higher. Even for non-caching web site. ( May be I got too used to HN's speed )

Personally if it were my site I would put in some time to cache a few things. Rails makes this fairly easy with https://guides.rubyonrails.org/caching_with_rails.html. I say "fairly" because caching is never really easy because of what could go wrong but I would say this is as realistically easy as it can get with Rails.

A cold page load on any old StackOverflow answer page is about 50ms for me. Keep in mind the hosting environments between both sites too. The forum site is on Heroku. StackOverflow is running on https://stackexchange.com/performance, basically hundreds of CPU cores and multiple terrabytes of RAM on dedicated hardware. There's no doubt .NET is going to crush Ruby even with the same hardware tho.

> StackOverflow manages sub 20ms response on nearly all pages time with zero caching

I'm not sure where you read they don't cache anything? They cache a lot of things. They have an in depth article on this at https://nickcraver.com/blog/2019/08/06/stack-overflow-how-we.... They're also serving a lot from a CDN to give end to end low latency responses across the globe.

The 20ms is straight form the Stackexchange Performance page in your link, at the bottom of the page. That is server response time. I assume your 50ms is with latency, and paint delay included?

They only cache static asset like images, they dont cache page in CDN. [1]

There is probably a better tweet which I cant find right now, but they dont cache page on their server either, every thing is generated on demand. [2]

StackOverflow is quite famous for not doing any cache and dont use CDN. HN doesn't use CDN either and uses minimal cache ( for non-logged in users ). But yes the hardware is quite far apart. My opinion is that I dont think we should settle for 500ms response time as acceptable. But then it is quite costly on Heroku......

[1] https://twitter.com/Nick_Craver/status/1318296453266231297

[2] https://twitter.com/Nick_Craver/status/1410549840493350915

50ms includes just latency (not paint).

Yeah I don't know, their blog says they have almost 100GB of cached data and it's slow without it (their direct words from the post). I'm not sure why they would write about something but tweet something else, it's the same person too.

> My opinion is that I dont think we should settle for 500ms response time as acceptable

Sure, it's a perceivable amount of delay. Fortunately with Rails you can achieve <= 150ms response times without too much trouble which I think is fast enough where no one is going to notice a difference between a 125ms and 25ms back-end response when the total end to end time painted in their browser might be 750ms (back-end response + latency + paint). The total end to end time is what the user experiences.

>have almost 100GB of cached data and it's slow without it

They dont cache the the generated pages, but they do cache the common backend Data and some keys required for DB. They are used more like an In-Memory DB rather than Cache.

Some of those cache are not longer required once they moved to .Net Core 6.0. But Nick went to work for Microsoft once StackOverflow were sold to some PE. So we may never get any updated figures.

Is HN employing TikTok style algos now?!

Literally have been discussing with our devs what we can use that's as off the shelf as possible for a HN style community.

They said it would be best if it was built with RoR. I see this post randomly when I can't sleep at midnight.

You might want to take a look at this project as well: https://github.com/lobsters/lobsters

It's the software that runs https://lobste.rs/ - a smaller community, similar to HN.

Thank You. I didn't know lobster is open source and it is running on Ruby Rails as well.

Why is it called an "engine"? Is that some new type of gem or package in Rails?

https://guides.rubyonrails.org/engines.html

Rails has the concept of “engines,” roughly small Rails apps that can be embedded in / composed into larger Rails apps.

https://guides.rubyonrails.org/engines.html

They are like gems, but larger in scope and specific to Rails. You might call them plug-ins

Confusingly, Rails also has plugins as a separate concept.

Not since about a decade. Plugins were Rails 1-2 era.

True, I have not seen them used anymore but the documentation still refers to them (https://guides.rubyonrails.org/plugins.html) and the rails plugin new command still exists (on rails 6, at least. Don't have 7.x on this machine yet).

Thank you. I love discovering things like this on Hacker News.

For anyone (like me) who prefers a quick video demo: https://www.youtube.com/watch?v=vakfo0HqpQE

I use the length of a demo video as a first-order approximation for how much work is involved in getting something up and running. I happen to subscribe to DR, and can see the full video demo is 9 minutes, meaning it's probably quite painless.

I will be trying Thredded soon!

Despite the name this doesn't actually support proper nested threading in the way we think of it here or on Reddit, does it?

To this point I ask - are Twitter "threads" nested?

They are necessarily hierarchical, which is to say "yes, technically." I think the way that Twitter deals with this without any kind of expando functionality is one of the more interesting problems in mainstream UIs nowadays. This is not to say that it's working well or is using the best model, but it's a complicated problem.

Heh, wonder what they used to generate the fake posts/threads in the demo forum.

> Use the online RAM firewall, then you can override the neural protocol!

> You can't parse the JSON without calculating the auxiliary AI pixel!

Super clean and I always love the idea of drop-in forums that integrate with your application.

> Heh, wonder what they used to generate the fake posts/threads in the demo forum.

Just by eyeballing a few of them it looks like the Ruby Faker gem https://github.com/faker-ruby/faker. There's direct references to it such as the Stormtrooper line in https://github.com/faker-ruby/faker/blob/master/doc/movies/s....

Can anyone recommend a similar rails tool for events management, attendance/ ticketing?

Thanks - it looks pretty but in terms of usability I prefer USENET news via Gnus client.

Where do you use USENET? Do you host your own and if so any resources about getting started?

Edit: My curiosity regarding USENET was piqued after seeing DFeed on here, a forum front end to USENET, made for and in use at forums.dlang.org.

https://github.com/CyberShadow/DFeed/

Not the parent but I use aioe.org for USENET and gmane for mailing list archives via NNTP.

I think it's an interesting idea to expose webboards with NNTP as an additional method to, say, RSS.

Reply on news.ycombinator.com