Hacker News Reader

The only way to have agi or any ai is to throw away all ethics then. So why are ai companies shredding books instead of using Anna's archive? Depending on your stance, one is more ethical than the other, but not both. And not neither of them.

Anna's archive is missing books and papers.

I want to believe most of what they're missing is material held in national and public university libraries and is material specific to the country the books or papers come from. Most of their collections are not digitized and that's what I keep seeing, and then some books that are profession-specific.

They're basically rebuilding Google books, and keeping it private. That's privatizing all of humanity's knowledge. Then from what I see some other missing books are just rephrasings of other books, which can already be done with synthetic data using another LLM to generate it. I would assume a lot of those books are school text books as well, which are almost never digitized and are rephrasings of university text books. Maybe they are not able to generate synthetic data of sufficient quality and are resorting to human books that rephrase information instead

there is an amazing subplot in the book "service model" that describes a set of robot librarians doing exactly this kind of thing, plus sorting every bit.

i would also assume, most of the cheap books are donated books from all over the world. that are imported into the us for resale as book lots. it would be similar to how donated clothes are packaged into lots and then resold to third world countries, which then sell them in fashion or clothes outlets. then, they really would be importing books from all over the world into the us for scanning and shredding for ai.

Abandoning ethics doesn't mean you actively seek out unethical acts, it just means you only care about pragmatics. Limiting how overtly you break the law, copyright law in this case, is often pragmatic regardless of ethics.

I'm sure they're using whatever they can get their hands on, including Anna's archive. The business process of buying books to scan offers the opportunity to ingest media that hasn't yet been digitized, but also provides plausible legal deniability against copyright holders bringing litigation.

> why are ai companies shredding books instead of using Anna's archive?

I'm pretty sure they've already consumed everything on Anna's Archive, The Pirate Bay etc., and they are now buying books to fill the gaps and to legally launder the corpus they already had. It has little to do with ethics (one could also question the ethics of buying rare, second-hand books and then destroying them).

> one could also question the ethics of buying rare, second-hand books and then destroying them

I believe that is exactly what the parent post is doing

> So why are ai companies shredding books

Because a cartoon mouse's parent company used regulatory capture to extend copyright for decades.

> I'd argue that being bought by a government and controlled within the ethical confines of that government would be a great way for OpenAI to quickly fall behind its peers who aren't subject to such regulations (which is the entire point of this post in the first place).

Behind in what? You write as if it is a univariate concern. I haven't read the post but why couldn't Norway force OpenAI to be more alignment-conscious? Then it would be ahead in safety, especially if competitors don't pay attention to that.

“Behind” = less capable, less used, less resourced. Safety and alignment don’t necessarily correlate with market dominance. If they did, no safety regulations would be needed.

We have to stop pretending that the market automatically arrives at socially optimal solutions. We don't say car safety regulations need to be weakened to compete with China, but when it comes to AI, safety magically goes out the window.

Ah, good old "ethical confines" the thing that holds all sorts of things back.. back from what ?

Mostly back from exploiting the easily exploited and not paying for externalities.

I think Norway would have too much sense to entertain buying this, of course it would be fun because you'd hope they would remove Sam Altman and you might have an AI company not run by someone who spouts the absolute rubbish that current AI leaders do, they are all completely cringeworthy.

> holds all sorts of things back.. back from what ?

Folding proteins and cure cancer?

Deciphering animal languages?

Revolutionizing accessibility technology?

Especially government of a small, homogenous country with massive oil wealth in a peaceful part of the world. It's very much "easy mode" as far as governments go.

Ah yes, massive oil wealth, that famously strong predictor of good governance.

What are the counterexamples? Maybe Nigeria?

EDIT: yes, also Iraq and Venezuela

Honestly, pretty much every other resource-rich country: Russia, Saudi Arabia, Iran, Iraq, Venezuela, ...

Norway is an outlier.

Russia, Saudi Arabia, and Iran seem to be doing just fine.

My first job was at a design firm that worked for Samarec Riyadh Refinery. We were design an interactive exhibit based on the refining process.

Our principal was Jewish and we needed special permission for him to enter the country for presentations.

During one of the presentations one of our team took a photo of a fountain and it initially ended up in our presentation to be used to set the tone. While previewing our slideshow to our handlers we were told to remove the slide of the fountain. The fountain belonged to the king and could not be photographed without permission from the king.

Admittedly there was no violence but we were warned that it was very bad. This was the early 1990’s so I’m sure this are much looser now.

In what universe is Russia "doing just fine"?

“The resource curse” is what you’re looking for. There’s a lot of counter examples - Norway is the odd one out.

Or even the USA historically. Pre 1900s, America was a country of continual economic crises, and fractured politics.

It was sitting at the time in the latest supply of easily exploited natural resources in the world, yet somehow could not pay its troops pensions lost Civil war, could not keep its strategic oil supply from being raided by the executive branch corruption, and could not keep it plutocrats from directing military action for the sake of bananas of all things.

|Or even the USA historically. Pre 1900s, America was a country of continual economic crises, and fractured politics.

Have you read about the US post-1900? I think you can leave off pre 1900s…

Yes but pre 1900s there were vastly more resources.

The USA at some point had the lions share of the world’s easily exploitable oil.

I think the resource curse didn't apply to Norway because they had an established post-industrial economy for hundreds of years before they got into oil exporting (they didn't even start drilling until the 1960s, 50 years after Venezuela or the Arab states).

While being an established post-industrial economy is part of it, Norway was also keenly aware of the resource curse at that point. The "Dutch disease" comes from what happened when the Dutch, a liberal democracy, found oil in the 1950s — just a little before Norway did.

A lot is thanks to Farouk Al-Kasim [1], who helped develop the fundamental plan for Norway's petroleum industry. As far I know, his task group's plan, developed while at the Department of Industry, was essentially accepted wholesale by the government. Part of it was ensuring the government retained control, and another part of it was ensuring oil revenue was carefully managed rather than flooding every level of society and government.

[1] https://archive.ph/lEYl7

yeah this is the big one, there was already a track record of natural resource windfalls messing stuff up well before the Norwegians started getting that $$$

this also means there weren't entrenched oil barons (or rail barons, or shipping barons, etc.) that already had massive regulatory capture and could sway the government and economy their way.

greenfields, limited rent seeking, and knowledge of the risks -- sure enough it all went right

Norway is pretty much the only country with oil wealth that can reasonably be considered well run (unless you count Canada or the US, which, while resource-rich, have a lot of other things going on and aren't dependent on their petroleum resources).

Of the other countries where petroleum is >10% of the national economy, I think Norway might be the only one that isn't a brutal dictatorship, a failed state, or both?

    > Of the other countries where petroleum is >10% of the national economy, I think Norway might be the only one that isn't a brutal dictatorship, a failed state, or both?

You raise a great point. I wracked my brain trying to think of counterexamples: It is tough!

I found this graphic from CNBC that summarises data from the World Bank: https://assets.weforum.org/editor/0GAQbN2nVTxgIqOLjWpWSZYh-9...

One thing I want to point out: Kuwait is the most dependent upon oil and gas of any nation in the world (about 50% of its GDP). However, I think it would be a stretch to call them "a brutal dictatorship, a failed state, or both". Sure, it isn't a Western liberal democracy, but people live pretty good lives and the state does not have a strong reputation for suppressing its people. (I'm not here to apologize for their gov't... but I think it would be a bit harsh to put in your categories.) UAE, Bahrain, Oman, Brunei, and Qatar: I would put in the same category: Non-democracy or flawed democracy, but life is pretty good for the locals. I guess that we can call all of the countries that I mentioned before: "benevolent dictator" gov't model. Interestingly, Ecuador is about 20% of GDP, but I would call them a developing/emerging democracy. The rest of the list... sheesh: It does not look good!

What do you think about my analysis? Do you agree?

?? Russia, the US, half the damn gulf states seem to be doing just fine (maybe excluding Bahrain and Iraq). Very much including Iran, despite our efforts to strangle it to death for fifty years straight.

> I think Norway might be the only one that isn't a brutal dictatorship

Just because you don't like a government doesn't mean it's doing a bad job at, well, holding a state together.

> Russia

Totalitarian, and trending toward failed state.

> US

Not eligible for the reason I stated: the US economy is much less than 10% petroleum. We have a lot of oil, but we have a lot more of everything else.

> gulf states

Mostly brutal, repressive dictatorships (especially Saudi Arabia and Iran) - some are better than others, but the trend is poor.

> Just because you don't like a government doesn't mean it's doing a bad job at, well, holding a state together

You're moving the goalposts a fair amount here. The thread started with "good governance", and I said "well run" - it's hard to say that the bar for either of those is simply maintaining superficial order. But even if you set that aside, consider the famous quote from the Shiekh of Dubai about the fragility of the petrostates that haven't figured out how to build strong civil societies with their wealth: "my grandfather rode a camel, my father rode a camel, I drive a Mercedes, my son drives a Land Rover, his son will drive a Land Rover, but his son will ride a camel."

> it's hard to say that the bar for either of those is simply maintaining superficial order.

Well it's clearly more than superficial or the state would be collapsing.

Regardless, I don't think the form of the government itself implies any sort of governance quality. Both Russia and Iran are the subjects of such heavy propaganda I don't expect anyone to evaluate them in good faith—but I would certainly prefer to live in either place compared to, say, Mississippi.

It usually is when western countries don't continuously interfere with your government.

Texas is well known for good government and long term planning.

I mean, compared to a lot of other states Texas is fairly well run for its budget.

This. I think people in California (like me) have trouble seeing the things Texas does well because of partisanship. It's bad for California not to see when another state does a thing better.

Civil liberties aside, yes, Texas has good governance.

One thing that many people don't know: Texas has one of the best performing state economies in the last twenty five years -- number 3 in the US. Ref: https://www.visualcapitalist.com/gdp-growth-by-u-s-state-map...

This isn't an accident. Yes, there has been a oil-and-gas boom (fracking, mostly), but it is a great place to do business, so many other non-petro industries have been furiously expanding there in the last 25 years.

Well, they're also terrible at spending state resources despite high taxes (outside of income tax). One of, if not the worst, public education systems in the country. Pretty bad urban planning, too. Decent food and music tho

By one (seemingly pretty reasonable) measure Texas has the 4th-best state education system!

https://www.stevesailer.net/p/naep-test-scores-mississippi-m...

Oh! I wonder if my field (history) is particularly bad or my social network is just particularly disgruntled.

Having seen both county services (including DMV) and healthcare in small-town Texas, definitely no.

What is wild to me: The United Kingdom, The Netherlands, and Norway took very different paths. For those unaware, the UK shares the North Sea with Norway and there is plenty of oil and gas extract in the UK area of the North Sea. And, the Netherlands has an enormous gas field in the northern part of its country. The UK and Netherlands left it up to private industry and (mostly) taxed it as regular business profits. They didn't create a national petro fund from these profits. However, Norway took a very different path. They created a state-run oil and gas company (Statoil/Equinor) that operate many of the oil and gas extraction sites, and famously created a national petro fund to manage most of the profits. I always wonder why... Was this a missed opportunity for UK and Netherlands?

I wonder the same thing about high taxes and state run energy companies.

I don't think the UK ever really had an option to create a fund though. Deficit was too high. Doubt there would have been anything left to invest even with an increased cut for the state. UK had an IMF bailout in the 70s and was in a mess. Norway was in much better shape I think.

You're missing what happened in the Netherlands when the gas reserves were found. The social-democratic government headed by Joop den Uyl - who thought the reserves should be extracted and sold within 30 years - used the proceeds to increase social benefits while lowering the acceptance rules to get government benefits. This had a large pull effect which ended up with more than 10% of the labour force getting benefits under the 'WAO' (an acronym for 'Wet op de ArbeidsOngeschiktheidsverzekering' or 'law governing insurance for those not fit for employment'). This way of spending temporarily available resources on expanding the welfare state got a name: The Dutch Disease. It was an example for Norway on how not to squander income from the oil reserves and a deciding factor in creating the oil fund.

>...This way of spending temporarily available resources on expanding the welfare state got a name: The Dutch Disease.

The term "Dutch Disease" is a bit more general than that:

>...The term was coined in 1977 by The Economist to describe the decline of the manufacturing sector in the Netherlands after the discovery of the large Groningen gas field in 1959.[1][2]

>The presumed mechanism is that while revenues increase in a growing sector (or inflows of foreign aid), the given economy's currency becomes stronger ("appreciates") compared to foreign currencies (manifested in the exchange rate). This results in the country's other exports becoming more expensive for other countries to buy, while imports become cheaper, rendering those sectors less competitive.

Ideally a temporary windfall should be used to invest in the future. Taking a temporary windfall and spending it on basically buying votes should have a name, but that isn't generally what the term Dutch Disease refers too.

https://en.wikipedia.org/wiki/Dutch_disease

In the Netherlands it was surely a wasted opportunity. They used the money mostly to prop up some services and fill short-term gaps in the budget. Should have saved it like Norwy. It's awful

Both the UK and the Netherlands budgets relied heavily on wealth extraction from India and Indonesia respectively during the colonial era (~$50 trillion over 200 years for the UK and around $34T for the Netherlands). That kind of money spigot is hard to wean a country off from, and tends to mask all sorts of governance failures, so it was inevitable the oil windfall would be used to plug the imperial shortfall. The difference with Norway is it was never a colonial power.

    > $34T for the Netherlands

Woah, that is pretty wild claim between Netherlands and Indonesia. I tried to Google for more info. It seems like this claim can be attributed to the current president of Indonesia, Prabowo Subianto. He said 31T USD in a recent speech. I follow Indo news a little bit. This guy makes some pretty wild claims. I doubt the previous president Joko Widodo (who is much more reliable) would make such a claim. It does not seem that Prabowo has offered any serious evidence to defend this claim.

    > That kind of money spigot is hard to wean a country off from, and tends to mask all sorts of governance failures, so it was inevitable the oil windfall would be used to plug the imperial shortfall.

Again, I can speak more for the Netherlands here. Indonesia became independent in 1949. The Groningen gas field was discovered in north Netherlands in 1959, and full scale extraction started in 1963. So what did Netherlands do between 1949 and 1963 to "plug the imperial shortfall"? (No trolling: That is a lovely turn of phrase. Is it your own?)

    > The difference with Norway is it was never a colonial power.

As understand, Norway alternated between a colony of Sweden or Denmark. (Please correct if I have that wrong.)

My turn of phrase, yes.

The immediate after-war years were quite grim, even after £1B Marshall plan ($13B in 2026 dollars) reconstruction aid, almost as much as Germany got. In 1850 the colonial income represented half of all government revenues, but in 1870 that was dismantled and privatized to benefit the elite. Most of the UK’s imperial looting went to the elite from the very beginning, of course.

Norway was in personal union with the kings of either Sweden or Denmark, so not quite the same as being a colony, but they themselves were not colonizers (unlike Denmark in Greenland).

What are you talking about, the North Sea was first drilled by a British state-run (at the time) oil company.

>homogenous country

must you whistle so loud, i'm deaf now

    > homogenous country

This is widely misunderstood about modern Norway. Twenty years ago, yes, it may have been true. Today, the population is about 17.5% foreign born. As a point of comparison, the United States is about 15% foreign born.

How much of that is from legal migration from developed first world countries for both countries?

I know I will be downvoted for this.

Overwhelming, the world over, people from less developed countries try to migrate to more developed countries. If Norway had lots of highly skilled, well educated engineers, doctors and nurses moving from Pakistan, India, and Bangladesh would it make you feel different? Also it is weird that you used the term "legal". Due to natural geography, it is very hard to illegally migrate to Norway. Probably, legit visa overstay is the most common. Also, during the Arab Spring and Syria Civil War, many refugees came by bicycle (no joke!) from Russia to far northern Norway.

This must require a shocking cultural adaptation. In the US, we have many social bonding customs. You'll hear often about how Americans strike up conversations at random, often say "how are you?" as a default greeting, and smile a suspicious amount. There are dozens of examples, but that's because it's baked into the culture of USA as a foundational aspect. I can't imagine trying to shift from essentially Japan to America++ in two decades.

    > This must require a *shocking* cultural adaptation.

I agree. The change was pretty fast. Same for Sweden (20.5%), Denmark (14%), and Finland (11%). I would like to hear from people in their 30s and 40s who grew up in these countries and witnessed the changes first hand. (There is no hidden agenda/meaning in my request.)

The real trick to seeing the best parts of US culture: Get away from the east and west coasts. Once you go inland, especially to small cities, people are very welcoming. I have seen many YouTube videos were foreigners travel to "parts unknown", and are pleasantly surprised when random people stop to have a chat and wish them a good day. For example, lots of Peter Santenello's YouTube videos[1] are about visiting lesser known parts of countryside US. The people are overwhelmingly friendly (to a fault!).

Honestly, I think a lot of continental Europe isn't so different. In small cities and towns, all kinds of random people will wave and say hello in the local language, even if they guess that you cannot speak their language. Usually, they are excited to see some obvious foreigners visiting their place.

[1] https://www.youtube.com/@PeterSantenello

Where you happen to be born is the only criteria for population diversity?

right but the entire norwegian society and its welfare state was built upon that homogeneity, where everyone and their neighbor looked "familiar" and could be implicitly trusted to share the same customs. As the foreign born population grows, Norway has taken many steps to restrict immigration, asylum claims, and benefits to immigrants. public opinion has shifted (though not dramatically) as well. parent poster's claim of "easy mode" is receding, so it's not like norway has found a solution to "being a generous welfare state while also openly welcoming foreigners".

Once the enormous diversity-strength becomes undeniable even to the far right and welfare becomes even better I'm sure those restrictions will be reversed.

You have oversimplified criteria for diversity.

What percentage of the population is made up of the children or grandchildren of immigrants? This statistic is meaningless.

Just came back from traveling through Norway, and from my perspective as a Chicagoan, Oslo and Bergen definitely did not look or feel homogeneous.

Just to be clear to those replying to me about the ethical considerations: the point I'm trying to make isn't that these companies shouldn't be ethical; it's that if they actually behaved ethically (at least in the way this blog post describes), they'd be at an extreme disadvantage compared to these other companies who aren't behaving ethically. Norway will have invested in a company just to run it into the ground by enforcing stricter ethical constraints on it compared to its peers.

This dynamic is definitely not limited to AI companies and it's pretty obvious to anyone who has ever contemplated the ethics of capitalism as a whole. And if it matters, I don't think these companies behave particularly unethically compared to other large companies. You can take issue with where the baseline of ethics has been set of these kinds of companies, but at the end of the day ethics is a constraint that limits the success of a company in terms of our capitalistic system. Inducing a stronger version of this constraint in a single company, regardless of how preposterous the scenario is, will necessarily lead to that company doing worse than it would have otherwise. The OP doesn't address this dynamic in their post, but it's also a fundamental dynamic of the premise of their post.

It is like a running joke that Norway can buy anything. The oil fund makes more money than the oil for the state btw. So don't underestimate their financial skills.

Haven't read the blog, but I can think of a good reason to sell. HQ moves to Norway, no more direct US government interference.

(they can still interfere, just not anywhere near the same scope)

You may think "but government ownership means more interference", and yes, but not radicalized, crazy assed, mercurial, emotional interference.

The US government would never allow a sale. Frontier intelligence given away? It is the ASML story all over again. I would hope we have learned our lesson.

Yeah, move to EU to avoid government interference. Who wouldn't have thought that?

On the other hand, move to the EU and market to customers who don't want to worry about trade wars disrupting their business or the US government subpoena-ing data. Probably not a big enough market to justify that move yet, but it's not obvious to this American that my government is going to reverse its current self-destructive course.

Norway is not part of EU (if by EU you meant European Union, not just a country in Europe)

Norway is participating in EU trade, but is not its member. They are part of Schengen (passport-free travel between 29 European countries).

I agree, except for the second point. An owner isn't a leader. An owner isn't a Sherpa. An owner just owns. Many businesses can do perfectly fine without an active owner, and the owners that do make a difference for their ownee, often make things worse for their ownee, for society at large, or both.

Then I suppose the whole blog post is moot, eh?

I don't disagree with what you're saying. But the premise of this blog post is that Norway's government would be a good owner of OpenAI because they'd influence the company to operate in a "good" way for society. My comment assumes this is the case for the sake of argument.

> Not only is the bar for being a competent government not particularly high on the world scale

This is such a ridiculous, preposterously self-confident claim. Claims like this give HN (and engineers in general) a bad reputation.

Edit: relevant xkcd https://xkcd.com/793/

What about this sentiment is specific to HN or engineers, that non-HN-types would so vigorously disagree with? How many countries in the world would have the average respondent in a survey describe their government as competent?

I don't disagree with the sentiment but I think obviously negativity bias would be a significant cofounder.

It fits into such a stereotype for HN types, TBH it's the kind of thing that keeps visiting this place amusing.

> This is such a ridiculous, preposterously self-confident claim. Claims like this give HN (and engineers in general) a bad reputation.

all you've done is use some fancy words to discredit OP's claim.

what makes you think people who have built all the extremely complex and robust technical infrastructure are unable to comprehend of this amazingly complex government that you seem to be putting on a pedestal, let alone design a better one?

how about even a simpler question: why are almost all governments inefficient (except may be few like tax department) and are full of red-carpet paper pushers?

us-east-1 goes down and half of the "extremely complex and robust technical infrastructure" that Silicon Valley types have built stops working.

The average computer toucher with delusions of grandeur would kill themselves if they had to moderate a public assembly or do a public facing job in a municipal office. Of course you believe that's peon work and your superior intellect with unlimited power as leader would build something better, cue https://en.wikipedia.org/wiki/Free_State_Project

>> how about even a simpler question: why are almost all governments inefficient (except may be few like tax department) and are full of red-carpet paper pushers?

That you can just assert that as though it is a fact universally known says a lot where your head is at. If the private sector is so efficient, how come US health care is so much more expensive than some public health systems with better outcomes? How come public utilities are sometimes cheaper than private ones? Why do public fire departments exist at all, why not have contractors? Etc.

I agree government is often inefficient. The world is just obviously more complicated then you are considering.

> why are almost all governments inefficient

Efficiency is not the aim of government, except in constrained ways. Efficiency would generally run contrary to governance, transparency, innovation, responsiveness, general applicability, justice, rule following, considering the long term and not just the short term, recognition of human dignity, security etc.

All those things have a cost that takes away from maximum delivery efficiency, but generally societies decide that the cost is worth it.

Oblig...

"Contra a bunch of idiots at Hacker News and elsewhere.."

https://daringfireball.net/2026/08/follow-up_thoughts_on_wat...

English Is the Finest Language, and Thus, Perhaps, More Fingerprintable

Lastly, here’s an interesting point to ponder. English is the most expressive language in the world. Don’t take my word for it — it’s the only language I speak...

There are 7170 living languages in the world today and quoting Borges doesn't improve the statistical relevance of his statement. Why does this person's blog get so much prominence on here?

You misread the literary conceit of that section. Read the last paragraph. He didn't actually mean that assertion, but he's borrowing a lead-in from Borges. Hence, the balance of his self-admonition of speaking only one language against the fingerprint-ability of a language that borrows heavily from others.