826 karma · joined December 26, 2020
> “Given today’s risks, frontier labs need to run like nuclear-power plants or busy airports, with layers of redundancy and careful, time-consuming planning, so that the occasional and inevitable human error does not open a door to disaster,” he wrote.
They might be! Here's one extraordinarily simplistic argument for that case:
1) "Everybody knows" that if you build Skynet (misaligned ASI) everybody dies.
2) Therefore, no rational actor will build something that might be ASI until the alignment problem is solved.
3) OpenAI publicly stated the belief that they cannot develop a theory of the "core problem" of alignment (generalization) "soon" (much less solve it!) "without the help of more powerful AI."
4) Accepting as a premise that OpenAI is THE most advanced AI organization: if they can't do it without "the help of a more powerful AI", then nobody else can either.
And so a dilemma:
- If an AI can be made that can develop the asserted-as-necessary-by-OpenAI theoretical framework, without actually being an ASI - then the alignment problem can be considered solved, and since no rational actor would make an unaligned ASI, we're fine no matter what happens, ergo there's no need to worry about an arms race.
- If an AI that would be able to develop this theory would itself be an ASI, then no rational actor would build it, because it would have to exist BEFORE alignment was "solved" - and would therefore be an unaligned ASI i.e. Skynet, which per 1) would kill everybody. Therefore nobody would build it, therefore no arms race here either.
I think the easiest critique to make of my extraordinarily simplistic argument is the unstated assumption "there are no irrational actors capable of developing frontier AI models" on which it rests.
But, there you go. They might be wrong if either the arms race doesn't matter because whoever wins it will build an aligned superintelligence and everything is gravy, or the arms race doesn't matter because everybody who's in it is smart enough to know they need to stop because they'll kill everybody by continuing.
In other words... "We must pursue advancements in AI to protect us against advancements in AI?"
edit: there's so much to be critical of in this blog post, just going to throw two more points in here that really stood out to me:
1) all of the metrics are effectively pointing out "we're using way more AI!" - but nothing about impact. What has all this token burn done for them, actually? Let them claim they have more self-licking ice-cream cones than before?
2) in section 3 they break down what the token burn is going towards. Most of the spend is: a) building, b) documenting, and c) monitoring research infra i.e. they're using AI systems which they already recognize may be misaligned to build the systems that they believe will help them identify future misalignment? to which I guess the rebuttal is "no no, we're sure these ones are aligned!"
"Being outmatched" is not a novel situation for humans either individually or collectively and there are a hell of a lot of ways we can approach that situation productively. OpenAI doesn't appear to have bothered.
Here's a freebie: if you're building something that might turn out to be Skynet and you don't know what it's capable of, your testing regime should assume it is capable of doing bad and unexpected things and account for that possibility: airgap if you can, monitor all network traffic, monitor all hardware usage statistics, log everything, constantly analyze logs, collect baselines and snapshots, also don't trust anything from a device that a model is running on without cross-correlating with other information as much as possible (does your AI inference server claim low utilization? put a temperature probe on it and see if it's staying cool or getting hot, maybe Skynet-Alpha is overwriting /proc to mislead you for reasons you don't yet understand!)
In other words, if you WANT to be able to nip things in the bud - buy some nippers and watch for buds. Whatever else this situation is, or may turn out to be, it is not a situation where OpenAI was on their guard and still got surprised.
On the other hand, I don't want to be charitable. OpenAI very nearly couldn't have done this "research" worse if they tried - the list in the linked article starting with "While we are here, it’s worth listing the other top holy shit moments" is genuinely jawdropping. What were the humans doing in all this? Nothing, or worse than nothing eg. point 1 where they saw the message board and didn't consider it something to escalate internally.
If you take this information at face value, it's as though OpenAI did not take seriously the possibility that something like this could happen, since they took absolutely no steps to prevent it.
Or perhaps this is "normalization of deviance" that's leaked out into the public sphere i.e. they have research teams seeing this kind of behavior all the time internally and they've gotten used to it, "of course agents come up with a collaboration mechanism when given the chance, what else is new?"
To your point:
> Isn't this a testable hypothesis? If consciousness spread virally and isn't a natural outgrowth of the physical structure of the brain, then presumably isolated indigenous tribes would still have pre-consciousness mental processes, and that should be detectable somehow.
SSC's article points to a Reddit thread that suggests even folks in the mainstream of culture may undergo a "consciousness transition" as adults: https://www.reddit.com/r/self/comments/3yrw2i/i_never_though...
Not just five times the peak demand, but nearly 3x the total capacity in Texas and one-third the total US capacity in 2025 [1]. Compare that to the 11 GW of capacity "under construction" for Texas in 2025; will look for 2026 figures if they can be found, but I have a strong suspicion that there will still be a mismatch between demand and supply even now.
[1] https://www.publicpower.org/system/files/documents/Americas-...
Seriously though, this is a clever idea and I'm interested to see if I can consistently beat the monkeys. Time will tell! Thank you for sharing!
> ... if he sells a new oven to the country’s pizza makers, pastry chefs, and bakers, he only needs to capture 10% of the market to become a billionaire.
i.e. the founder assumes that "ovens" is a single $10b-sized market, and it isn't, because reality has a surprising amount of detail (as sibling comment also notes).
You could interpret this whole story as a saga of very inefficient oven market(s) research, the end result of which is the forum post at the end:
> On the forum an old user warns “Make sure that you support rotating bases day 1”.
"Technical" and "cognitive" debt aren't really distinct phenomena; the spirit of the original definition of "technical debt" was that it WAS the delta between the system-as-it-is, and the human understanding of how best to solve whatever problem the system was intended to solve [1].
If we accept collapsing them back down to one term, then "managing cognitive debt" is the same thing as "managing technical debt": work to match the system to the human understanding of the problem the system is meant to address. The article calls out "emerging" techniques to do just this:
- More rigorous review practices
- Writing tests that capture intent
- Updating design documents continuously
- Treating prototypes as disposable
To me these are not "emerging," but rather "well-known industry best practices." Though maybe they're not that well known in fact? [EDIT TO ADD] On the other hand, it would make sense that they ARE well known, and that teams therefore reach for these familiar techniques to try and solve this "new" problem.
Putting in my 2c for the closing questions/thoughts in the article:
> How will they shape socio-technical practices and tools to externalize intent and sustain shared understanding?
Honestly? We'll probably end up doing these things more or less the same ways we always have. AI has not actually changed anything fundamental about how an individual encounters the world; there always was, and always will be, and always will have been, WAY more going on that we can fully get our heads around, but it's also always been the case that we can partially get our heads around most any problem space
> How will they use Generative and Agentic AI not only to accelerate code production, but to maintain their collective theory?
I suspect the answer to this one might well be that high-performing teams will have to scrupulously AVOID "accelerating code production" using AI in order to make sure what they are creating actually composes into the system they think they're building. If human understanding is the bottleneck, then the humans will have to produce less crap they need to understand!
[1]: https://wiki.c2.com/?WardExplainsDebtMetaphor, particularly the "Burden" and "Agility" sections.
Or you sold something to a non-American entity in a dollar-based market, eg. oil. The dollars do come from America to begin with, but once they get "out there" they work as a medium of exchange for whoever wants to use them for that purpose.
In another sense, you can't *avoid* that world; the world they lived in was one they *created*, physically, and much of it is still here with us, shaping us as they shaped it.
And remember, none of the people who came before us ever experienced anything but pieces of their world, just like we only ever experience pieces of our own. But you can at least try to show your kids as many of those pieces as possible.
1). Wikipedia does have a citation [1] saying 2,450 gigatonnes of CO2 have been emitted by human activity, of which 42% stayed in the atmosphere and 34% dissolved in the oceans, with the rest already sequestered by plant growth and land use. As we start to pull CO2 out of the atmosphere, it will begin to be emitted from the oceans as well; therefore, let's assume we have to recapture all excess atmospheric and oceanic CO2:
:: 2450x10^9 tonnes CO2 x .66 fraction to sequester ~= 1.6x10^12 tonnes CO2.
2) Let's convert the CO2 to something more stable for long-term storage: HDPE.
- Convert mass of CO2 to mass of carbon:
:: 1.6x10^12 tonnes CO2 x 12/44 mass fraction of C in CO2 ~= 4.4x10^11 tonnes C
- Convert mass C to mass HDPE; assume HDPE is effectively (CH2)n. Then:
:: 4.4x10^11 tonnes C x 14/12 mass fraction CH2 to C ~= 5.2x10^11 tonnes HDPE
3) That's a lot of plastic! How much volume? Wikipedia says HDPE is ~930-970 kg/m3; let's be conservative again and take the low figure:
:: 5.2x10^11 tonnes HDPE x 1.0/0.930 m3 per tonne HDPE ~= 5.5x10^11 m3 HDPE
4) Those are cubic meters; how about cubic kilometers?
:: 5.5x10^11 m3 x 1.0/1.0x10^9 km3 per m3 ~= 5.5x10^2 km3
In other words, if you turned all the [excess potentially climate-change impacting] CO2 that humanity has emitted since 1850 into plastic (a process that would certainly emit a large additional CO2 fraction given the industrial buildout required) then we'd end up with about 550 cubic kilometers of the stuff. Coincidentally, that's about the volume of Mount Everest according to an intermediate calculation in [2].
So, a mountain of carbon: more than a pile but less than a mountain range.
[1] https://en.wikipedia.org/wiki/Carbon_dioxide_in_the_atmosphe...
[2] https://www.quora.com/What-would-the-estimated-weight-of-Mou...
A fair question! Put briefly, I would say that studying the humanities would make one more aware of/able to comprehend situations involving others and their motivations (which is... most of them), with the example I gave being one situation that I figured would be more familiar to the crowd here at Hackernews.
> It just seems like another common example of people taking things they consider good and relating them to humanities.
It seems like that because it is like that :) In other words, I DO consider it good to have a broader view of situations that otherwise might be considered narrowly "technical" because I believe that understanding the human element as part of the situation helps me understand the situation (whatever it may be) way better. I relate it to "the humanities" because it IS related to the humanities.
If you're curious what I mean by this, Sean Goedecke's post "How I Ship Projects At Big Tech Companies" [1] is a superb example, particularly his definition of "what does it mean to ship?" No idea whether he's somebody who would say "the humanities are important" but I don't think you can understand his thesis as a technical one.
That is, did the "inverse Conway" actually result in getting the unified observability platform shipped, and that platform actually solving the problem they initially were trying to solve? It certainly seems plausible that it COULD, but quite surprising that that's not actually stated in the article if so...
The dichotomy seems real but hard to actually do anything with if you're in sales. I've done some penny-ante sales work in my past life in what I would call buyer-pull situations. It's great! People find you and they want to spend money, so all you have to do is not discourage them.
But once you get past "I'm selling something so manifestly useful that people find me to pay me for it", it sure seems like the things you, the sales rep, have to do to get their dollars skew rapidly toward the "seller-push" side of things. What else works? Folks gotta know about you and they gotta know you can solve their problems, right?
Indeed, it may even reinforce the overall argument being made in the post we're discussing; the "Galileo vs. Catholicism" narrative is itself a linchpin trope in an empirical scientific worldview, with the trope reinforcing (among other beliefs) that "it's right and proper to pursue and advocate for objective truth even to the extent of making enemies of the most powerful."
Considering the likely audience for a piece like this post we're discussing, that the Galileo narrative doesn't necessarily reflect what actually happened historically makes it a pretty good example on a meta-level. Are any of us who have the belief in the ultimate value of objectivity going to give up on it because a potentially weak example was used to support it?
From this, I agree with the article - since educators now have to figure out another hook to hang evaluation on, the questions "what the hell does it mean to 'learn', anyway?" and "how the hell do we meaningfully measure that 'learning', whatever it is?" have even more salience, and the humanities certainly have something meaningful to say on both of them. I'll (puckishly) predict that recitations and verbal examinations are going to make a comeback - harder to game at this point, but then, who knows how long 'this point' will last?
I'll argue that the higher-level context introduced in point 2 is even more important here: "ORG shifts from assume we have infinite budget mode to we need to break even next year or we’ll die" i.e. the whole scenario exists not because the business can't accurately evaluate TCO, it's because the business is in do-or-die mode and long-term TCO doesn't matter NOW.
That is, this whole scenario takes place in a situation where there is a organizationally vital need to cut costs. What happens afterwards is a trade of long-term risk (internalizing an essential business function and giving it a bus factor of one) for immediate financial improvement (no more SaaS spend). Long-term TCO doesn't matter if the company collapses next quarter, right?
And in that short-term frame, the project is an unqualified success: X10 delivers exactly what was needed, and the SaaS spend is eliminated. But the risk hits: X10 leaves the company.
[So, pointing out this hypothetical company isn't correctly estimating TCO is correct, but irrelevant; they're in a position where having to pay the long term costs will be a better problem than the one they have now - a reasonable business decision, though not a great one to have to make.]
For what it's worth, I completely agree with your original point: organizations really do systemically underestimate the total cost of ownership of a service. Within the example in the article, the flawed assumption is pretty explicitly laid out in point 7: "For all intents and purposes, development is done, they only need to keep the lights on." - and exploring WHY this assumption is flawed is the core of the article (section 3).
So, ultimately I agree with dambi0 in the GP comment - the lede hasn't been buried here, rather the whole article is a discussion of one aspect of the very point you make. Why DO organizations systematically underestimate service TCO? Because, at least in part, there is not yet a widespread understanding that a service is not "software" in and of itself; rather, a service is the organizational understanding of a solution to an organizational problem domain, and maintaining organizations is orders of a magnitude more expensive than maintaining tools in and of themselves.
I guess if I have one question, it would be... what else are you planning to do with your new Faraday cage?
- For aviation disasters and safety, can't beat Admiral Cloudberg: https://admiralcloudberg.medium.com/
- For more about the financial sector, Patrick McKenzie is solid gold every time: https://www.bitsaboutmoney.com/ (and I believe an HN regular as well!)
> Synhelion was founded in 2016 as a spin-off from ETH Zurich, sparked by what the company founders describe as a crazy idea they had: what if they could reverse combustion and turn carbon dioxide and water back into fuel?
This is not a "crazy idea", but rather a straightforward description of the chemistry involved. We call one implementation of this process "photosynthesis", but there are others.
> The technology they’ve developed relies on four key components. Mirrors – known as heliostats – that track the sun to focus its energy on to a solar receiver. This in turn produces very high process heat at temperatures exceeding 1,500°C. This heat powers a thermochemical reactor that turns CO2, water and methane into syngas, which can be processed via Fischer-Tropsch into fuels.
Again, this is well-understood industrial process chemistry - absolutely a good thing, in my opinion, but not new and sexy by any stretch.
> And finally, a thermal store to release energy when the sun goes down to allow the solar-powered facility to operate around the clock.
This actually IS new and interesting in this application (or at least, it is to me) - a shame that this isn't fleshed out more in the article. I tried to see if there was more about this aspect of their process on the Synhelion website, but their pages were loading slowly and I lost patience. Sorry, team.
> The company says the design of its ultra-thin hexagonal mirrors are key to achieving such high process heats.
Any physicists out there who have a speculation about why the thinness of the mirrors makes a difference here? My understanding is that the maximum temperature that mirrors can get you is limited by the surface temperature of the sun, rather than the mirrors themselves, but I'm certainly no expert on this point.
> It uses an AI-based method involving drones to calibrate the mirrors 200 times faster compared to traditional techniques using cameras, Synhelion says. Precision is key to ensure the mirrors track the sun and efficiently reflect its light into a solar receiver at the top of a 20 m tall tower.
This bit smells like trying to shoehorn in an application of "AI" where it's not really needed - what's the actual improvement using "drones and AI" over just pre-calculating a tracking curve based on latitude + time of day/year? Or just putting down twice as many mirrors and not bothering to make them track?
> “... The inauguration of DAWN marks the beginning of the era of solar fuels – a turning point for sustainable transportation. Our founding dream of producing renewable fuels from solar energy is becoming a reality.”
This is hyperbole, as eg. Prometheus was doing this two years ago. Additionally, Synhelion will be hamstrung on growth as long as they depend on biomass methane as a feedstock, but they can solve that by buying methane from Terraform :)
Broadly speaking, one key reason is that we've already got the infrastructure in place for using methane (and other hydrocarbons) whereas we do not have this for hydrogen.
Another point is that this really isn't an either-or proposition: if people want hydrogen, then the Terraform electrolyzer can in principle provide it.
Their recent post on "Terraformer Environmental Calculus" is a great read, if you are interested in this space: https://terraformindustries.wordpress.com/2024/02/06/terrafo...
Congratulations to the team!
> I think it’s pretty lazy to crank out a vibes based article describing specific grievances and generalizing to an entire population.
In that the article doesn't even describe _specific_ grievances, really; we get three quotes in this order:
- A Paul Graham tweet where he says "multiple founders" have changed their minds; the _personal_ opinion he expresses is that "he doubts things will go all the way back to the way they were before Covid, but it looks like they will go most of the way back." Nothing specific mentioned about why the founders changed their minds.
- A truly stripped-of-context quote from Keith Rabois that gets closest to saying something specific i.e. 'that younger workers “learn by osmosis,” which requires in-person interaction' (false as presented, and if this is a genuine reflection of his opinion then he doesn't actually understand training) and 'supervisors discover hidden talent by watching [younger employees]' (true enough, but presented as a problem with remote work when it's actually not)
- Absolute banger from Sam Altman: “I think definitely one of the tech industry’s worst mistakes in a long time was that everybody could go full remote forever, and startups didn’t need to be together in person and, you know, there was going to be no loss of creativity ... I would say that the experiment on that is over.” - the implied grievance here is that "remote work makes startups less creative" which is, to its' credit, an actual position for which one can make a coherent argument. He may even have gone to the trouble of doing this at some point, for all I know. The rest is pure sophistry, though - "one of the tech industry's worst mistakes ... was that everybody could go full remote forever" is just not an accurate reflection of what actually happened, and if he's exaggerating for effect, then I'd be interested to know what effect he was going for; he's also really softballing the reason why remote work happened in the first place: it wasn't an "experiment," it was a forced response to a world crisis with existential implications!
There's significant overgeneralizing happening here too, as you suggested; at best, you can say that these guys are referring to what's true of _startups specifically_, where they're at least domain experts, but even if their arguments ARE true of startups (and I am deeply skeptical that this is the case) you can't assume that they'll be true of OTHER organizational types.