Static HTML Comments | Hacker News Reader

Static HTML Comments

sive.rs

113 pointsarticsputnik83 comments

I can tell it's Friday because it took until the section ID until I realized this was going to be about user comments on a website not  LOL.

If you've got Ruby listening for updates with a postgres database all tied into a content generator it's probably time to admit you don't have a static website anymore and just look for best practice ways to go about this (with a bit more security!).

Same here... and I wouldn't call this a static page anymore if it runs JS on load, calls a HTTP endpoint which makes DB query which returns HTML that modifies the DOM.

but if you can host your site from an S3 bucket, would it not be called static? to me, static means no server side languages necessary to render the page. if your statically served JS can call a remote host to pull down some data that updates things client side, it's still considered static, from a hosting perspective.

Wouldn’t that definition would apply to standard SPAs as well?

I guess it depends on perspective. What a host service that will charge you more money for a non-static site, a client-side JS app can still be considered static.

We might also be blurring lines of static sites vs interactive sites. You can have a static site that does nothing but allows the user to scroll to view it. You can also have static sites that change and do things based on the user's input. Entire games have been made using client-side JS but is still served statically.

IMO if a web user triggered event triggers a server side page regeneration they see when the page reloads what you have is more a dynamic site with caching not a static site. "It's a write-to-file-before-serving site" isn't really a meaningful distinguisher but "the site functions the same when you only have a copy of the website files" is.

The JS doesn’t trigger a DB query, it loads a static comments file previously generated by Ruby at the time the last comment was inserted or modified. It could be replaced by a server-side include.

Yep, it's not static because data in the DB is changing the file.

I’d regenerate the complete HTML page server-side whenever a comment changes, instead of only regenerating the comments section, to obviate the secondary loading via JS.

There is some advantage in splitting it in that the page itself stays cached even as the comments change and the page doesn't need to wait for the comments to be transferred to start rendering. You could keep this advantage with <iframe> if you really wanted to avoid JS though. It's a shame html imports died, they were a bit more lightweight.

Which advantage is that?

Cached pages load faster?

If you're just slinging static HTML, there's really not that big of a difference, and if it's a frequently accessed resource where a difference might be expected to matter, it will be cached regardless.

Hilariously enough, that's no longer true in many parts of the world; according to firefox telemetry, on-device storage is frequently much slower than the network.

Do you happen to have a link for that? It seems pretty counterintuitive given the state of modern PC hardware.

https://bugzilla.mozilla.org/show_bug.cgi?id=1392841 is the bugzilla bug to enable the feature by default.

https://simonhearne.com/2020/network-faster-than-cache/ has some discussion.

> the page doesn't need to wait for the comments to be transferred to start rendering.

It doesn't. Browsers will happily render half an HTML page as it's streamed in. There's an interesting HTTP 203 on this. https://web.dev/shows/http-203/LLRig4s1_yA/ if you're interested.

Interesting, will have to give that a listen. Thanks!

Server side includes work nicely for this, I think nginx was what I used (long time ago now). You can even inline them on load but reload htmx-style with minimal effort.

It's saturday here now, I've slept plenty and I still interpreted the headline the same way. But the topic is plenty interesting too, I'm curious how far the SSG movement will take it.

So, you’ve got a static-ish page, a Ruby server, Postgres and NGINX. The page both POSTs and GETs comments from the backend via an Ajax calls. All very conventional, but then you’re worried about traffic spikes, and to deal with that, the comments on the server are stored in pre-generated HTML files, updated by some wild PG triggers setup, and served up by NGINX. IMO you could go way more traditional, with the same result.

Worried that Postgres will tip over? Just read from the DB, but cache the results in memory with a short TTL on your Ruby server.

Or, if you’re worried about not just Postgres, but also your Ruby server getting overloaded, just go with a fully server rendered page, and cache the whole page. Could use NGNIX cache, as you’re already using NGINX, or a CDN, or Varnish, whatever. No JS loaded comments at all.

Either way, caching should be able to get you tonnes of scalability, easily avoid hug of death, without resorting to PG triggers calling a Ruby server that writes files to disk.

With that being said, this approach works too, and if it’s just a personal page with you as the only dev, you understand it, so it’s all good. But I’d certainly avoid this approach at a company, largely because it’s unusual and will take new devs longer to wrap their heads around.

The approach in the article is functionally different though, in that the cache is always immediately invalidated when a new comment comes in. So it’s not the same result as using a cache with a TTL.

You can always set a super short TTL like 15s or even 1s. It's still going to make scaling a non-issue like that and at some point the caching will be fresher then the html generation, as that needs to be done too (and read after by the http server).

It's fun to make architecture like in the article though, I'm guilty of it as well with previous hobby projects

OP’s approach has fast cache updates, but it’s still eventually consistent. Like, the whole flow of PG trigger, handled by Ruby server, then writing to a file, that happens outside the flow of POST comments, and could sometimes finish after POST comments returns to the caller.

Also, with my suggestions, you can cache invalidate too, if a short TTL isn’t good enough. Based on writing files to local disk, it seems like a “single server” scenario, so invalidating the in-memory cache on POST comments is dead simple, and can be truly immediately consistent. Likewise, for full page caching, if using Varnish as a cache, Varnish lets you invalidate easily too. NGINX and CDNs don’t (generally), but if a short TTL isn’t enough, and you need instant invalidation on new comments, then just go in-memory cache or Varnish.

The big idea is to write the comments as static HTML, only when comments change, instead of doing a database query to display them every time. This prevents the “hug of death” if you get a burst of traffic.

I believe this was the technique used (and still being used?) by 2chan, the ancestor of the infamous 4chan. It didn't even need a database, as it was entirely file-based and simply appended each post to the appropriate file. Visitors were served static HTML with no JS required.

Yep, I'm surprised the OP doesn't use such a scheme, possibly with something SSE-ish so you don't mix the content and comments on-disk but they're both retrieved as a single request.

This way all you need is a POST endpoint (which might just be a cgi script) to receive the comment and update the comment file.

Gets more complicated if you support threaded comments or pagination, but for a linear comment system it should be more than enough.

The main 4chan open source clone project (tinyboard -> vichan -> infinitychan) works this way.

I implemented static HTML comments on my website by tail'ing the /var/logs/nginx/access.log with a perl script. No CGI, no database, just the actual text of the comments stored as a single comment per line in a .html file.

To comment a visitor takes any url on the domain and appends "/@say/". Like http://notmyurl.com/somepage.html/@say/Some response to somepage. Or "...lakephoto.jpg/@say/Cool fish! How long was it?"

The perl script sees the /@say/ in the logs and adds the parsed out and sanitized comment to an .html file. There's some nginx location hijinks for matching /@say/ URLs that goes to a confirmation page and redirects to the comment listing page.

what executes the perl script? the POST doing a CGI type thingy i have so forgotten how to utilize from not using it since the 90s? on first read, i missed that you're modifying the contents of the html and was concerned about log rotation before realizing that was just dumb to be concerned about.

I have it start when my OS starts. It runs continuously thereafter (~5MB ram). The webserver does not know or care about it. The script leans heavily on perl's POE::Wheel::FollowTail module to get updates to nginx's access.log automagically then I abuse regexes to note log entries with /@say/, extract what URL-file-path commented on, what the comment is, and prepend the line to the algorithmically appropriate .html file with some html mark-up. The webserver serves the static html as it would any html.

You've told everyone on HN how it works, have you been root-ed yet?

I've used this comment system on my tor onion services sites for the last decade. I get plenty of people trying to exploit it. It's kind of fun. If the Tor folk haven't pwned it I doubt the HN folk will. Not for lack of skill but mostly a lack of motivation relative to the tor folk.

Minor pet peeve, but it is so much nicer when people use https://example.com for example links.

I clicked on your link expecting a real site (my monkey mind having skipped over the actual URL contents) and I was redirected to some other odd Chinese (?) URL with spammy ads and NSFW content.

This is how the first discussion board on the web was written to work. Search for Matt's script WWWBoard. Perl script added the posted comment with date into the HTML file's insertion point.

At that time in the 90s this was heavily deployed everywhere. This was before PHP-BB and other database based boards got popular.

This was my first memory on reading the post too. Ah, Matt Wright, and FormMail, WWWBoard and even… OMG… guestbooks (remember those?!).

Yes, and we have 100x the DRAM capacity, 1000x CPU Performance if we include MultiCore, and 10,000x faster IOPS on SSD. We could have gone back to the old ways of doing things and it would ( or should ) actually be more scalable.

This used to be relatively common. Messageboards and forums (going all the way back to Matt's cgi board, which didn't even use a database) would cache static HTML pages and only regenerate them when necessary. I think Wordpress' cache plugin works in much the same way.

I guess it's a completely foreign concept in the age of SPAs when everything is dynamically generated in the browser.

The main point of the article is using the Postgres notify/listen mechanism to trigger the update.

That's the least interesting thing about it, though.

Yeah my guestbooks from the 90s and early 00s were all just cgis that directly edited the html with the new comments. Great times.

Is there a point in using NOTIFY in PostgreSQL, instead of just generating the HTML file when saving a new comment? The performance cost of doing a simple SELECT and writing a file in the request handler is neglible (especially since you are talking to the database anyway), but a NOTIFY-less solution is far simpler and will work with any database, even sqlite.

That's similar to what was my firtst message board done. It was a static HTML file with a form. The form submitted to a separate PHP script that appended the comment to the original HTML.

It looks like you could trivially exploit this with XSS? The example code has glaring vulnerabilities in terms of taking user input and then inserting it into the DOM verbatim.

Anyone who copy/pastes that example is going to have a rough time.

I call this hybrid static websites. It allows you to statially generate pages for even people who are logged in. It means the generation step is powered by interaction with the static site. I use a special api.subdomain.tld proxy_pass to a server to trigger the site generation.

I wrote a forum that regenerated the static files when a post is added. Including the user's own account pages, those were statically generated too.

In theory a change to a user's account should only change that user's account pages.

I wrote a medium article about it. I was trying to get as many requests per second while logged in. It's an extension of the cache invalidation problem.

I got to 3288 requests per second on a virtual machine. A static site should get at least 12,000 requests per second. For reference, Visa and Mastercard process 40,000 requests in total per second for financial transactions. I use an nginx authenticator and Rust authentication program that talks to Redis to check the IP address of a user's session token in the URL. You could probbaly improve the performance further by using an embedded Lua script in Nginx and using that to authenticate IP address rather than doing a HTTP request for each request that the nginx authenticator does.

You can see the static site as a cache that needs to be updated based on new inputs.

https://medium.com/@samuelmichaelsquire/1633-requests-a-seco...

Ok, but instead of inserting into db, it'd be cool to have the script commit the comment into source control and have your ci rebuild/deploy your static site

That's what I did: https://greenash.net.au/thoughts/2022/03/email-based-comment...

Why not use a blockchain if you want an append-only store?

Didn't say anything about an append-only. Blockchain is the answer to exactly 0 technical problems.

Source control repos are effectively append-only. How else could you recover earlier revisions?

This doesn't deal with the deluge of spam you'll get once any page on your site gets even modestly popular. (But it seems great for simpler use cases).

It would be so nice to have a comment system like Disqus but simple like described here, that also had an easy tie-in to a spam prevention service (or would work with a trained spam prevention model).

The Famous Article is just about the content management.

A separate authentication layer is entirely possible, at the loss of anonymity.

You could also do it by chucking something like a Varnish cache in front of it, and invalidate a page on a new comment.

This seems similar to Incremental Static Regeneration (ISR) provided by NextJS/Vercel

https://nextjs.org/docs/basic-features/data-fetching/increme...

I have written and maintained a blog post about different methods and tools to include comments on a static website: https://darekkay.com/blog/static-site-comments/

Have you tried Hypothesis[0]? It also seems like a nice solution that offers other features in addition to comments.

[0] https://web.hypothes.is/

Thanks for the suggestion! As I understand Hypothesis, it is a social tool for users to annotate any web page; as a website owner, there is nothing I need to do, right? In that case, it doesn't match the original intent of my blog post, but I think I should still mention it.

You're totally right that this would be something that the users would be using... perhaps the only requirement as a website owner would be to properly configure the Content Security Policy (CSP) HTTP headers [0].

[0] https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP

> Another interesting approach would be to write the comments into each HTML file directly, instead of in a separate file, so you wouldn’t need JavaScript at all.

A better approach IMO. Why require js for something as trivial as adding a small block static html to a page every time it loads? Makes no sense.

This is another cool way of doing it /w Elixir + Lit https://launchscout.com/blog/embedded-web-apps-with-livestat...

I've got an idea to allow comments to be appended to the HTML and counting on the fs implementation synchronization, so only allow comments around 4k (or whatever is prooved to be safe) for a complete single append. It would require the page to not have a footer and closing tags for </body></html>.

Here is an incomplete try: https://github.com/hadrianw/compend

Lock and unlock is there for moments when someone wants to modify the content or moderate comments while still allowing commenting.

> The big idea is to write the comments as static HTML, only when comments change, instead of doing a database query to display them every time

I'm using Cloudflare to do this, and any kind of db updates will update the cache. I'm new to all this, but wouldn't this be spiritually the same, e.g. if the cache would serve the latest db content instead of hammering the db? (I'm doing this because I'm using Airtable as db, and it's incredibly slow, rather than a "hug of death" kind of scenario)

Web users can insert, the author can moderate, and database changes are reflected on the cache instead of being generated on demand. It looks useful.

One thing to know if you’re using this Postgres feature is that the max payload is 8000 bytes [1]

That was the initial implementation for supabase’s real-time updates (afaik) and for this reason they had to resort to read the WAL.

[1] https://www.postgresql.org/docs/9.4/sql-notify.html

Looks quite overengineered. Most modern web frameworks already do it aswell, and using a much nicer static generation approach.

If you already have a DB and backend app just use a CDN.

> When comments change, PostgreSQL trigger sends NOTIFY

If your application is down when the NOTIFY is triggered that comment is lost, isn’t it?

There is a queue that will prevent further writes when full. https://www.postgresql.org/docs/current/sql-notify.html

The normal way around this in Postgres is to use NOTIFY to merely signal that there is something new, and use a separate (perhaps transactional) work queue to claim a row to process. You’re correct that relying on parameters in the actual notification will lose work.

If you go this route isn't your server spending a lot of CPU cycles generating pages that will never be accessed?

You're only regenerating pages once when necessary, then they just sit on the server as flat files until someone reads them. That takes up fewer CPU cycles than dynamically generating pages for each request, unless you're rebuilding multiple pages at a time, as with a messageboard or forum. But even then I would expect file I/O to be a bigger bottleneck than the CPU, or race conditions if posts are still being sent during a rewrite.

Have the static comment creation trigger an upload of static html to cdn and you will have a hug safe solution

HTTP caching + conditional requests are a much easier approach.

This is a very specific implementation of a very general idea.

This has to be a joke. If the Ruby implementation that saves the comment would simply contain the logic to save the static HTML file - the obvious implementation - no Postgres events are required.

Having a database separates the view from the model. If you want to modify the generated HTML, it's much easier with the comments being separate from the HTML itself.

I don’t say to get rid of the database. Of course it makes sense to store the comments there. I say get rid of 1) the database events and 2) the Ruby logic to listen to these events. This can be done by the Ruby saving logic. It’s just over engineering to use events.

Where are you are sanitizing the comments? You have literally provided a perfect SQL injection mechanism here and allowed arbitrary HTML to be inserted onto your page.

HTML injection, yes. SQL injection, no. The insert is parameterized.

Ok, I wasn't sure if DB.exec_params handled that

Hm. Is it really a "static" website if you have some JS constantly listening for updates?

JS isn’t listening for updates, but simply pulling in a static html file filled with comments.

    document.getElementById('commentlist').innerHTML = xhr.responseText

ah, ok.. so it just dynamically loads comments on load?

Static as in "the page is pre-generated, not generated on every HTTP request"

It's the right setup for write rarely, read often scenarios.

Reply on news.ycombinator.com