Comment by sillysaurus3 | Hacker News Reader

Comment by sillysaurus3 | Hacker News Reader

Parent Full thread

Even though HN is a community, the front page is essentially the mods' personal judgment. They are the editors, and HN is a newspaper.

I like it this way. They have excellent taste. The community has repeatedly proven it cannot be trusted to upvote intellectually gratifying content. pg's original HN code contained a flag for "rally", meaning the story is a rallying cry, and is therefore penalized because it's so easy to mindlessly upvote it.

The results speak for themselves: HN is now almost a top-1000 site by traffic.

They had to be phenomenal. I don't think many people appreciate just how difficult this was to pull off, or how much they work their asses off every single day. If they ignored HN for even a week, it would deteriorate noticeably.

Yet technology is at least as important. The fact that our upvotes do matter makes the situation unique -- although the mods can determine the relative weights of stories, we can still push a story upwards if enough of us are interested, and the story fits the site's original goal of gratifying intellectual curiosity.

There is no company anywhere in the world like this. At this point, HN practically defines our culture. When news stories say things like "so-and-so story about <technology> has been making the rounds on social media," at least some of them are talking about HN now. And your coworkers probably hear about the tech they use at least partly from HN.

The sole reason they're in this position is because they deserve to be. I hope they stay selective. As to being secretive, we should let them choose. If they want to reveal, fine. If not, fine. They've earned it. Even still, they've been remarkably transparent.

HN's cool but it's just one forum among a million others. The moderation is good though sometimes arbitrary. That's a lot easier for users to accept when they know why they are getting positively/negatively moderated, which is a good reason for more openness. But at the end of the day it's a forum, the moderators have the final say, that's the contract you agree to in using it.

If you don't like the way this site is edited, stand up an m1.large, hook Rails up to an sqlite database, and build your own. You'll be two generations of technology ahead of HN.

The freedom to take other people's private belongings and repurpose them to your own ends is no real kind of freedom at all. But that's ultimately what you're yearning for when you allude to the Empire of Napoleon.

I'm not complaining. HN is about the best of what we've got. My Napoleon reference was the slobbering nature of the comment, which is so pro-moderation as to lose objectivity.

Sorry, it's probably the tenor of the whole thread I'm responding to, not you in particular.

Something I find frustrating in discussions like this is a tendency for expressions of gratitude or appreciation to often be dismissed as uncritical, as if the only meaningful act of criticism is that of finding faults or flaws. In such an environment it can be very difficult to express reasons why someone likes something or finds benefit in a way that others read as something other than fawning: particularly when you know that's how it's going to be read.

There are reasons people continue to frequent HN as a news aggregator and comment on the site: it has some uses and benefits to them. 'sillysaurus3 does list a number of reasons supporting their position. If they're expressing a bit more joy and appreciation for your taste, dismissing that as "slobbering" seems quite harsh and unnecessary. Phrasing it as "pro-moderation" in an arguably pejorative manner likewise derails. A quality site, one I find personally valuable, requires moderation. If I'm in favor of a particular site and value its moderation efforts, I'm going to be "pro-moderation". If I actually find it valuable, enjoyable, and a benefit, I'm going to be appreciative, and maybe at times effusive. Have I lost objectivity in such a case? I'd like to think we're more human than that.

You're right. I edited the original to be kinder. Thanks.

Try not to remove context like that. My reply looks quite strange now. :)

Antirez tried, with lamernews: https://github.com/antirez/lamernews/blob/master/app.rb

Diff that vs HN circa 2008: https://github.com/evanrmurphy/SweetScript/blob/master/arc3....

Lamernews seems like just another HN clone, but back when it was launched it was a serious attempt at being a competitor. They probably came closest to pulling away a significant chunk of HN. I started to use it.

Then I ran into missing features. Little quality of life stuff. I forget what they were, but the whole site felt noticeably harder to use. So I stopped using it.

It's so tempting to think "HN is so simple. Just throw Rails at the problem and you can wipe the floor with them." But Lamernews was 2,964 lines of Ruby, and only managed to contain about 60% of the features. news.arc is 2,616 lines. And I don't care what anybody says: if you sat down and forced yourself to read that code for a week straight, you'd find it is actually quite readable. It's terse because it's for getting shit done.

I'm not saying you'll be as successful as HN. Lobste.rs, which is run carefully by thoughtful people using software which is probably better than HN's, is much less successful. But that's the point.

You said you'd be two generations of tech ahead of HN. I know it was tongue in cheek, and I'm not calling you out. I'm nerding out about how much functionality pg managed to pack into so few lines of code.

And it all worked! No data loss, no massive security problem, and the last time HN was down for >24h was because pg manually edited the database and accidentally created a circular comment chain.

One way to look at the situation is that pg's original vision for arc failed. Arc was supposed to be what Clojure turned out to be. But even though HN alone is the one significant Arc project, what a project! The world was made materially better thanks to it. And I don't think they could've pulled off HN in some other language. There was too much work for one person to do to be distracted by the thousands of details a modern webdev needs to stay on top of.

I think it's possible to do the same thing for modern webdev. Arc's goal was to be a reliable foundation for building webapps. It comes with a user+admin system out of the box, for example. Every Arc app does.

I've been playing around with hooking up AngularJS to it, because unlike React or Angular, AngularJS can operate purely in HTML. You normally never want to do that -- usually you want to set up controllers and such. But when you control the language, you end up being able to give it the Babel treatment: the output looks complex, but you get all the features of a modern frontend site. Initial experiments were quite promising.

And from there, it's easy to just emit JSX instead. React? Angular? Sure, whatever. It's treated as compiler output.

(Indirect sourcemaps are funny... JSX compiles to minified JS, whose sourcemap points back to the JSX. But if Arc spits out JSX, you'll want a sourcemap from JSX -> Arc as well. It's sourcemaps all the way down.)

You can call me out any time you'd like. I think an actual database backend for this site is only a couple years old. Or do we actually have one yet? I know the API has one, but HN doesn't run the API. It's possible this thing is still relying on flat files.

Flat files FTW. Proof that a simple solution can scale. And not just in a "Why the fuck would you subject yourself to this torture?" kind of way, but in a "This actually simplifies everything" way. It's why Arc was so fucking fast: remember how impactful memcached was? Arc had that from day one.

Here's how it works. Everything is loaded into memory at startup. You can think of it like a bigass JSON tree. Anytime you change the tree, write the whole thing out to disk. Since all the data is loaded from file "foo", and you write out to "foo.tmp" followed by `mv foo.tmp foo`, there is no risk of data loss, ever.

So what about sharding? Well, yeah, that's a problem. But it's not a problem that HN had to deal with. You can see how effective it was to throw a single 64GB RAM monster at the problem rather than a cluster of servers. The moment they hooked up Cloudflare to HN, they got all the same benefits a big cluster would provide anyway.

People can be prejudiced against this simplicity all they want, but it's hard to argue with results. You don't even need to know SQL to start using it.

"But migrations --" well, Arc makes it super easy to define a "template", which is just a set of fields, and each field is whatever you want them to be.

https://github.com/evanrmurphy/SweetScript/blob/6b4ed26acc9c...

  (deftem item
    id         nil
    type       nil
    by         nil
    ip         nil
    time       (seconds)
    url        nil
    title      nil
    text       nil
    votes      nil   ; elts each (time ip user type score)
    score      0
    sockvotes  0
    flags      nil
    dead       nil
    deleted    nil
    parts      nil
    parent     nil
    kids       nil
    keys       nil)

So yes, theoretically if you wanted to rename "kids" to "children", you're gonna have to process all the old data. But 1. then don't do that, and 2. even still, it's not hard to do that anyway.

It's sort of similar to mongoose record definitions.

But look, HN is proof positive that this system can work and can scale to a top-1000 website. 99% of webapps won't even come close to those performance requirements. And when you do approach them, you can find ways to make it scale.

My main focus has been updating Arc to run on JS, and to add keyword arguments. That way you'll be able to use the whole JS ecosystem too.

For what it's worth, the published Arc/HN code is rapidly approaching a full decade out of date.

When you get the design right, there's no need to change it. :)

Jokes aside, I refuse to let Arc die. There were genuinely good ideas embedded in it, amidst the rough edges. People just never forced themselves to use it. It's like if Vim was invented but had no documentation, so no one ever realized it's powerful if you embrace it.

But yeah, I imagine my version will have a different name and a shiny website. Being a decade out of date does matter, especially for hiring purposes.

I think I'm just saying that I'm not sure you can conclude they don't have a real database backend now from the news.arc code.

Ah, yeah. (It's odd that Arc went closed-source but kept improving. It's maybe one of the most important custom languages in the industry, relative to HN's impact on the world.)

Did Arc go closed-source or does Hacker News run a closed-source fork?

As far as I can tell, the arc forums site is still up and running, and a lot of people seem to be focused on Anarki[0].

[0]https://github.com/arclanguage/anarki

Hacker News has been on a closed-source "fork" (it's really the upstream) since the day news.arc was published.

>"But migrations --" well, Arc makes it super easy to define a "template", which is just a set of fields, and each field is whatever you want them to be.

That's... just a struct or a map in any other language. Convenient but not necessarily a point in Arc's favor per se.

>So yes, theoretically if you wanted to rename "kids" to "children", you're gonna have to process all the old data. But 1. then don't do that, and 2. even still, it's not hard to do that anyway.

It's still an argument against using flat-files. Updating a site with a proper database isn't typically tied to disk read/write speeds - at best renaming "kids" to "children" means changing a field in a model and maybe a template - and that's a fixed amount of work regardless of the number of records you have.

>My main focus has been updating Arc to run on JS, and to add keyword arguments. That way you'll be able to use the whole JS ecosystem too.

You mean Node/NPM on the backend? That might be interesting. I'm not familiar enough with Arc to know (I've only played with Anarki a bit, but I have too many random projects on my plate as it is) but does it even have dependency management?

Direct answer: no, probably no database. HN forwards info to Firebase which provides an API. But assuming Arc is still running on Racket, there aren't a lot of options for database bindings (https://docs.racket-lang.org/continue/) and flat files work well enough.

EDIT: Actually, you may be right. They already forward data to Firebase, which means they could just store the data in Firebase directly. Firebase is pretty great, so it's possible they did this.

One bit of supporting evidence: there seems to be no startup time. Loading GBs of data from disk takes a little while, so restarting HN would be painful. We'd notice it as ~30 seconds of downtime. Especially now that HN has grown so large. I haven't noticed this in a couple years (but I'm not omnipresent).

So it could go either way, and there are a lot of advantages to the Firebase approach.

I guess I take back everything I said. But only because a lightweight alternative exists, not because flat files are inherently bad.

Lots of database options! See https://docs.racket-lang.org/db/

>I think it's possible to do the same thing for modern webdev. Arc's goal was to be a reliable foundation for building webapps. It comes with a user+admin system out of the box, for example. Every Arc app does.

Forums written in other languages do as well - and any decent framework can just include user management as a dependency. To me it doesn't seem like an unqualified good idea for Arc to bake all of that into the language or the default app.

It's pretty handy. A few times over the years, I've needed to throw together a standard "manage this data" type of app. I just extend the standard blog.arc app that ships with Arc: https://github.com/evanrmurphy/SweetScript/blob/master/arc3....

Bam, you get CRUD for free, you get special superpowers if you're logged in as an admin, and you can put in access controls so that other people can log in as other types of users and see a subset of the data. I dunno, it's pretty great. You even get a REPL at /repl in case you need to hop on and run some code on production. (If something seems amiss, rather than bother with grepping logfiles, you'd head to /repl and start asking questions about the current state of the app. This tends to be a lot faster.)

Depends on what you consider successful. If the metric is raw numbers and impressions, HN is definitely the winner. If the metric is signal-to-noise for technical content, Lobsters is probably ahead. If the metric is user count, HN wins. If the metric is moderation transparency, Lobsters wins.

Just depends what you're looking for.

I spent some time on Lobsters and found the signal/noise content for technical stuff only marginally better than that of HN, and many of the same pathologies were evident, just at a lower volume (because the site sees overall a much lower volume of comments). Moreover, while the variance on technical badness on Lobsters is marginally lower, so is the variance on technical excellence. That, too, is a straightforward consequence of lower volume: the original authors of important software packages are more likely to comment on HN than on Lobsters.

I like Lobsters a lot, and I like a lot about what they do differently than HN, but we shouldn't pretend that it's something it isn't.

If Dan and Scott were kings, they would be fucking epic. Best monarchs ever. It's hard to think of anyone who cares more about fairness, but clever enough to decide for themselves. And they're usually right. I think I can count on one hand the number of mistakes they've made that even remotely mattered. And from a strategic standpoint, that number is effectively zero.

There are probably a bunch of people behind the scenes, and they deserve lots of credit too. But there is something incredible about being able to post a dozen moderation comments every day for years without ever being so mistaken as to provoke community backlash. It would be really easy for them to offer an offhand opinion, or just make a bad call, and suddenly have people up in arms about it.

They're also some of the most humble people. Both of them have done some really amazing work and have cool hobbies. But I've sort of had to pry that info out of them over years.

http://github.com/gruseom/numen - Dan's project. JS REPL for emacs. It took another four years for anyone to even come close (https://github.com/NicolasPetton/Indium) as far as I know.

https://github.com/sctb/lumen - Scott's project. A self-hosted Lisp that compiles and runs in both JS and Lua. The whole codebase recompiles in about one second flat when using LuaJIT.

But like, that's maybe 5% of the cool stuff they've done. There's just no outward sign -- they don't bother with blog posts or any kind of self-promotion.

So yes, all hail the kings of lisp.

Was not aware of those projects or their backgrounds - thanks for those links.

I'd rate Dan and Scott highly as rulers, and I've been banned before.

I have some qualms about some systematic things, but it's a broad philosophical dispute and not the running of HN. In governance - there is no way to 'win' anyway.

I wish every leader had to be a forum moderator of very different groups of people, but we seem to select for whatever the opposite of that is no matter the political predilection.

(FWIW, the parent edited their comment significantly. They made a comparison to Napoleon / kings. I didn't just say that out of nowhere.)

https://github.com/sctb/woe

Indeed this is cool.

To compile on BSD, I used

   gcc -static -std=c89 -Wall -pedantic -o woe woe.c -lm

Do you mind opening an issue or just emailing me (scott at yc) the error you see?

Edit: fixed, thanks!

Reply on news.ycombinator.com