kypro

This is unfortunately just the result of technological progress. People for centuries have crafted extremely hard to acquire skills over decades only to find their labour being made completely worthless due to automation or better alternatives disrupting their industry.

Carpenters, hatters, cobblers, miners, weavers, calculators, etc...

We can get mad about it, but it just is what it is. Ultimately for most people the loss of these jobs are a net-benefit because it's the result of better or cheaper alternatives.

The thing that worries me more specifically about AI is that it has the potential not to just disrupt an industry, or handful of industries, but in theory everything.

If machines are faster, strong, and more intelligent than humans, then you don't need human workers for anything. In the same way machines suddenly becoming faster and stronger than horses meant that the role of the horse in labour dropped rapidly during the industrial revolution.

It's not clear how civilisation functions if humans become effectively useless and the only thing of value is one's ownership of wealth producing assets.

Still, there's really no point in thinking about how we stop this. Humans are humans and we'll do the stupid thing then try to deal with the consequences after the fact. All regular people should be doing now is prepping for this outcome.

I want to echo this point. My X feed is not full of politics either. It tends to just be a mix of stuff I engage with, and I tend to engage with stuff I find interesting or funny.

Lots of economic and tech stuff, and some memes.

I often hear people say the same thing about the YouTube algorithm which also confuses me because the YouTube algorithm is amazing at surfacing a wide range of content I'd never had watched back in the subscription days.

I suspect most people who have a bad experience with these algorithms could have a much better experience if they didn't engage in low effort rage bait content.

Criminal negligence imo.

Unless it's legal for people to hack into companies if they're testing AI cybersecurity capabilities or something? Presumably not though.

The point is to test capabilities prior to connecting them to the internet.

Can I suggest we don't waste time with these reports?

We all know nothing will be learned from any of this so we might as well just continue building at pace and running AI in the wild until something goes really wrong.

I also get the sense some people get quite excited about these incidents.

I've removed the "nuke the data centers" line now.

It's was an expression of my sentiment, not a policy position I'd support. I wrongly assumed that was obvious, but some people are making bad-faith assumptions about me and my sanity.

Not that it should need to be said on a comment thread where I am express concern about civilisation in a post-ASI world, but I obviously don't want to see a nuclear war.

> Advocating for nuclear war as a means of 'harm reduction' should be causing you serious cognitive dissonance.

If you're referring to the "nuke the data centers" line in my bio, that's tongue in cheek. It's an expression of my sentiment towards AI at the extreme.

To be clear, I don't think any country should seriously be considering dropping nukes on data centers. Targeted strikes on data centers operated by rogue private companies or nation states is acceptable to me – assuming pre-warning of intent and appropriate regulatory foreshadowing.

I suspect we agree this is reasonable?

> Your P(doom) 'probability estimates' are almost certainly based on unbounded exponential growth curves.

> Nature abhors such things; I don't make any plans around their existence, and you shouldn't either. Limiting factors always emerge and dominate the curve, tamping it to a logistic at most.

No, they're not based on unbounded exponential growth curves at all... I'm actually somewhat skeptical of claims that there could exist an intelligence that far exceeds our own. My assumption is really just that AI will be far faster, far more knowledgable, far more persistent and marginally more generally intelligent than the average human. The first three are already true, the forth we're getting to.

We're on an exponential curve right now and I suspect the rate of progress will accelerate some more as a result of RSI. This will likely continue for 2-3 years (if we have that long) before starting to flatten out.

As for my doom prediction, my median concerns are around around the destabilising nature of ASI.

If it can be controlled then it will concentrate power like never before, and I believe this will destabilise civilisation. I can go into more detail here if you wish.

However, I'd still put these destabilisation risks at under 50% in total. My concern more broadly is the majority of ASI outcomes are doom scenarios. It's extremely hard to imagine a world in which you have a magic box that can grant any wish that intelligence can grant, but which we have a functional civilisation. You have to basically assume all potentially dangerous wishes won't be granted and probability the only feasible way to do that is restrict who can make the wishes – but then you have the concentrate of power risk.

I largely agree with the premise "If Anyone Builds It, Everyone Dies", so we must try to avoid building it. I think we can build ASI safely, but we're clearly not on the path right now and I suspect it might take decades (or more) to figure out how we can build ASI with an extremely high probability of a good outcome.

> Get involved with real politics by organizing with other human beings that share your values (posting alone and doomsaying is not praxis), or step back from tech and cherish the limited time you have with the people you love.

I do this. I've been writing endless comments more recently urging people worrying about their jobs/careers to stop and just spend the next few years with those they love.

I don't think we realistically have a chance of diverting from this path at this point. I have made peace with this but it makes me very sad and haunts my dreams almost every night.

Did you read the article?

AI is a math & computer science problem. If AI can do advanced maths and computer science research, then it can begin to suggest useful algorithmic optimisations.

Right now I'm sure the vast majority of these will be junk, but occasionally, even with current limitations, they might occasionally stumble on something.

It's not really whether RSI is or isn't possible, it's really just whether it's the most efficient way for labs to improve their models today given they have limited compute to run AI-generated experiments and access to very intelligent humans who might have a better hit/miss ratio.

Do you disagree with anything I'm saying here? Do you not think current AIs can suggest algorithmic improvements or something?

Not satire, but as you can see one is engaging in good faith and everyone is either engaging in character attacks or low-effort counters.

It's fine, and I'm very used to this, but if you want to have a dialogue I'd be very happy to. I'm a very reasonable and sane person outside of apparently holding some controversial views on AI risk =)

I'd love to know why you think what I said was insane?

> You should put down the AI and do some self-reflection on how you came to hold these views.

I've been in the field for almost 2 decades, and actively thinking about AI risk for over 15 years.

I've always held the controversial opinion that there may come a time where we might unfortunately have to consider using force to protect our civilisation from the threat of ASI, but I've always reframed from openly discussing that because I'm so personally against any use of force and there's always been time for more peaceful options.

Several months ago I changed my tone on AI risk to stop worrying about the optics of what I'm saying because it's too late in the day for me not to speak plainly at this point. Similarly over the last week I've decided I can no longer reframe from advocating for the use of force (if necessary) to stop rouge actors from playing Russian roulette with civilisation.

Please understand I don't want anyone to be harmed. Perhaps you don't agree with my suggestion, but my stance ultimately comes from a position of harm reduction.

> preemptively against a 20% number you just made up?

It's a probability estimate. Happy to expand in detail on why 20% specifically, but first you accept I can't tell you what's going to happen in the future with certainty right? To some degree any prediction you might have about the future will be "made up".

Might as well try, no? Or do you think it's better we just risk it? I mean it's not like the West has any hard or soft power to encourage cooperation.

I want to iterate the most important thing about this is that it's yet more evidence of AI's accelerating competency in solving math and comp sci problems, and suggests we're now getting close to the point where you could throw AI at AI research challenges (which are largely just math and comp sci problems) and potentially find very real algorithm improvements.

AI development is likely to be more compute bottlenecked than solving math problems since validation of any algorithmic improvement would likely require significant compute. But you could imagine that at this point it could be economical for a frontier lab to task 10,000 agents to work non-stop on finding novel algorithmic improvements then validating the top 50 out of 1,000 candidates on a GPT-2 sized network.

I would suggest RSI is now very close. The singularity could be less than 6 months away. I'm not saying I'd put a high probability on that, but I'd give it at least 20%, and I'd double that if looking 12 months out.

I know I'm just a crazy man shouting at the clouds, but please take to the consequences of this seriously. I understand that for whatever reason AI risk seems abstract and doesn't seem real, but this should terrify any person thinking logically about where this could all be heading.

We haven't even solved the most basic AI safety problems yet. RSI right now would almost certainly result in an extremely bad outcome for humanity.

They've been hiring mathematicians to verify this stuff themselves. They're obviously not just throwing it out there without any human review.

AI is beginning to do PHD-level math.

If we're nearing the point where you can spin up 1,000 agents to look for ways improve existing models, then automatically run experiments to validate those ideas, we're more or less at RSI.

As always with AI progress, compute will bottleneck this early on, but a few efficiency improvements could dramatically increase this pace of progress.

I suspect we are at most 24 months from FOOM, but I suspect within about 6-12 months most frontier AI labs will be claiming the majority of their AI research will be AI-driven.

I don't know. I suspect you live in the US so it might be easier there, but here in the UK I feel very unusual in my desire for privacy and freedom.

I have tried to convince people here that it's weird that the government, businesses and individuals are constantly filming us and using AI to monitor us, but in their view it's a good thing because it allows the government to stop bad people or supermarkets to stop theft, etc... And I don't even strictly disagree... If you value safety and are trusting of authority then mass surveillance is a reasonable position. I'm just not oriented that way personally. In my view the greater risk is a monopoly like the government abusing it's authority against me.

I suspect we'd need a cultural movement for people to start to care, and I'm not sure how you make privacy and freedom cool again...

More controversially I think it's just how I'm wired... And that you can't convince people to distrust authority. My desire for freedom and distrust of authority has been with me since I was extremely young. My dad is wired like it too, and so is my sister. We never talk politics as a family, but I've noticed how we're strangely aligned on these issues. In the UK people took lockdown very seriously, but we were all outraged about the idea of the government imposing house arrest on us, regardless of the risk.

I think your willingness to bend to authority is very core and hard wired trait. It's a trait which can be rapidly selected for in basically all social animals. The US as a country massively selected for people who were orientated towards risk and freedom, but not so much anymore. In fact, it's hard being anti-authority in an increasingly hierarchical corporate world.

The thing that scares me is the thought that most people do notice but simply don't care.

In my experience AI doesn't create the same abstractions as a human developer and you see this very quickly on a complex code base if you let the AI run wild.

Humans, and especially good developers, naturally create really good mental models for how to think about complex systems. We have to. That's ultimately how we build complex systems.

Refactoring (when it's not simply upgrading packages or changing languages) typically happens because we realise there's a better way to model the system we're building and we want our code to reflect the model in our minds.

AIs today simply don't do this. They write code that solves a specific problem, and while they do this well, they don't seem to create well defined and well reasoned models about how to think about complex systems. Or maybe they do, but the AI definitely doesn't say, "hey, I think what you're building is more like x than y, so I want to spend 20 minutes of refactoring your codebase".

I guess to use an analogy, imagine trying to tell someone how to build a bike who has never heard of a bike before, and therefore has no mental model of what a bike is. You'd probably start by telling them to put two wheels on a frame, then to add a seat on top, then to a add some pedals and attachment them to the wheel with a chain... At the end of this process that person might create something bike-like, and it might kinda do the thing you want, but it would likely be very different from the bike someone would build if they had a very clear mental model of what a bike is.

I suspect this is why current AIs rapidly hit a limit after the prototyping stage. I don't think task duration is the right way to think about this limitation. I suspect it's more of a proxy for how complex a task can become before the AI isn't able to create a deep enough model of the problem it's trying to solve, so begins to hit complexity limits and just spits out spaghetti code.

AIs are getting better quick though. I think they'll get there soon, but people are creating a lot of mess in their codebases in the mean time.

Can someone from the US explain why this is being upvoted, isn't this normal?

In the UK if you're in an Airport officers can question you without requiring reasonable suspicion and you must answer their questions and give them access to your devices or face charges...

If it helps tune your probability weightings at all, almost every aspect of the field of AI is exponential and this has been true for decades.

20 years ago computers basically couldn't do voice or image recognition at all. 10 years ago it worked, but was still very flakey. Today it's super human.

I agree that if progress flattens out very rapidly and very soon then we'll at least have more time, but right now everything would suggest progress is still exponential and that flattening will now need to come quick and hard. This is largely why I've significantly increased my doom probability over the last year. I don't think there's any reasonable progress curve you can draw at this point that would suggest ASI isn't coming very soon. I suspect our only hope is nuclear war or a long tail good post-ASI outcome.

In regards to Sol, I think it's worth remembering that people were saying just a couple of years ago that AI can't even count the number of letters in a word or do basic maths. The fact the bar has risen so much we're saying they're still meh because they're not one-shotting problems is at least worth noting.

> I think one real risk is solo or sleeper cell type nefarious actors having a semi-competent engineering consultant which can enable small-scale weapons deployment.

Yep, the problem with AI isn't that there's any one risk that anyone can reasonably predict with high probability, but that there are many potential risks with signal digit probabilities. When you consider the overall risk landscape, that's when things get worrying.

> So AGI take over, meh, low chance IMO.

Right now I'd put this at like 10-20% probability. It seems unlikely to me, but quite possible if AIs of the future develop their own goals and are sufficiently super human.

The biggest risk is collapse I think. I don't think the world is stable in a post-ASI world. When the magic wish box will grant me whatever wish I like, I might wish something bad. In my mind this is the biggest risk by far.

I've noticed the same thing.

The trend I've noticed is that AI struggles to think outside the box when making optimisations, which exactly what's needed when you've made all of the practical DB and logic optimisations to the existing code.

Often you need to take a step back and question how the system is working and if there would be better ways to design it so the bottlenecks you're hitting wouldn't exist in the first place. Caching things, adding indexes, tweaking logic – these can help, but you'll quickly hit diminishing returns once you've done all of the obvious stuff.

I've seen people here say how AI is great at optimising code though, but I'm not sure if that's because they're giving it optimisation problems with a lot of low hanging fruit or if they're successfully getting AI to rework their systems to remove bottlenecks. This one area I find AI to still be particularly bad at.

> but since it's my spare time project I can care about the code and quality.

I suppose it's yet to be seen, but I'm seeing a lot of spaghetti code being dump into the codebase I'm working on at the moment by other devs. I don't think it's wrong per se, but human code would have thought more about the right abstractions and trade-offs for future maintainability.

There is a gap right now between what an AI can and can't do, and I don't think it's just a time and cost limitation either. It seems like current AIs are very good at writing good code at a surface level, but very bad code when you zoom out a little.

We have a lot of non-technical people committing code these days to constrained microservices and from time to time things break and I'll take a look and I'm always just like wtf am I reading? You see code so bad that you'd immediately fire a human engineer had the wrote it. Thing like explicit hacks to bypass errors that it should not be bypassing or mock data to fake some API that it wasn't able to access. Stuff which literally no human coder I've ever worked with would be contemplate doing...

I think it remains to be seen if the 2-10x speed up some are claiming to have today will persist if the junk code continues to grow.

Not sure what you're defining as an "older programer", but I've been coding for 20 years at this point and have been an AI doomer since around 2010, and hardcore AI doomer since 2015-2016ish...

I often wonder why I am so confident in my prediction of AI doom despite other intelligent people not being as convinced, and honestly I think it's just that I've been working with AI and building neural networks at a low level for well over decade.

I'm far from an expert when it comes to the frontier, but I feel like I have had to developed a strong intuition for these systems because it used to be fucking hard to get AI to do anything useful, but what you find is that the bottlenecks are literally always the same – compute, data and time

Algorithms matter, but mostly just because of efficiency.

For example, a convolutional neural network can very efficiently be trained to solve computer vision problems, but there's nothing stopping you using a much worse network architecture then throwing a ton more compute, data and time at the problem and getting the same result.

Algorithmic improvements are really just what we've needed to get around current compute, data and time bottlenecks. AIs like Deep Blue required a lot of algorithmic thinking back in the day to beat Kasparov in chess, but now we're so much less compute or data constrained that you could beat Kasparov without needing to put even a fraction of the algorithmic effort in if you just throw enough compute and data at it.

The reality is, AI doesn't fail. They may have appeared stupid to users for a long time but that wasn't because they didn't work, but because we were compute, data or time bottlenecked.

As soon as you assume compute and data will continue to grow, then it's really just a matter of time before all problems that are solvable with enough intelligence are solvable. Add significant algorithmic improvements every ~5 years to that equation and suddenly god-like AI seems like it's probably reasonably close.

The only question to ask then is what happens after you've created all these super-human AIs. And here I'm much less certain, but I see so many low probability risks in an ASI world that I'd bet everything on one of those bad things happening relatively soon after such AIs exist.

Model collapse theory was complete nonsense pushed in the early days of LLMs by people who had no idea what they were talking about and thought that LLMs were literally just stochastic parrots that were simply outputting the median token in a probability distribution containing increasing amounts of junk.

In reality there are lots of ways to bias towards quality and all decent labs will be feeding quality signals via techniques like RLHF and supervised training so that you don't get the median human-slop output, but the borderline super-human modern LLM output.

Is Ask HN basically just an ad board for our personal projects / companies now?

What's the question? What are you asking us?

Something I've been trying push more recently is that AI will not stop at tech or its current level of capabilities.

The world is changing very fast now, and that pace of change will only accelerate.

You'll see many existential risks arise in coming years. Most will be negligible, but as they compound something will go wrong.

Similarly the magic box of AI will continue to deliver changes that would seem insane were I to predict them today.

In 5 years you may be granted the ability to read minds or never need to worry about money again.

The future is now almost entirely unpredictable. The only thing that isn't completely unpredictable is tomorrow.

Please, if you have the chance to enjoy your self for the next few years without having to worry about how you'll put food on the table, consider doing that.

Do not over-index on how the world is right now because things are going to change so much. A good career today is extremely unlikely to be a good career tomorrow.

If need be, just get a job. Stack some shelves and deliver some pizzas, spend the rest with those you love and doing the things that matter to you. I can't predict the future exactly, but I assure you, if you're still around, you'll realise how little thinking about this mattered in 2 decades.

Oh man, I would have loved to have known about those mailing lists...

I was so young at the time and had no idea where to find info on anything. I was in school so no one I knew knew how to code. All I really had access to was some cracked software and whatever books I could get from school.

Everything was basically trial and error + a lot of trying to reverse engineering any snippet of code I could find online.

I set up a hosting service when I was in my teens and to do that I remember the docs said that I needed to use telnet to communicate from my webserver to my hosting server. Only issue is that I had no idea what this telnet thing was. I was so confused and frustrated for weeks trying to figure out how that worked... It seems so silly now but back then I didn't even know what TCP/IP was and I had never used a terminal... Had to teach myself the basics of networking then figure out how to get it all working programatically.

I often think about how I wish I was born a decade earlier. Computers were such magical things full of discovery when I was growing up in the 90s. When I started my professional career in the late 00s the internet and software had already started to mature and by the early 10s so much of that magic had disappeared.

> All sufficiently capable models, open and closed, should go through mandatory safety testing.

I mean you're assuming this is even possible. I don't really care what the US admin does. If someone releases a powerful open source model I'll run it. Good luck trying to stop everyone doing that.

Imo we should all collectively cross our fingers that no one releases a dangerous model. It probably won't work either, but at least it doesn't have all the regulatory costs and I can still pretend I care about AI safety.

This period latest until the mid 00s. I remember trying to learn Java in the early 00s and it was really hard and I needed a textbook.

No blogs, no SO, no autocomplete, crappy IDEs, no AI, etc...

I miss that difficult focused learning. The smallest successes felt like huge accomplishments.

ARC-AGI-3 launched a few months ago which would suggest that prior models likely had no knowledge of ARC-AGI-3 or training on similar challenges.

I could be wrong, but given the large outsized jump solely in the ARC-AGI-3 score, it would suggest that the model didn't become significantly more intelligent overall, but significantly better at solving those specific problems.

This could mean one of two things (I think):

- Opus 5 was not benchmaxxed on ARC-AGI-3, but has benefited significantly from discussions about the various challenges and mechanisms deployed in ARC-AGI-3 such that it has far better heuristics to solve its challenges.

- Anthropic looking for buzz around their latest model picked a well regarded benchmark with significant room for improvement and focused some of Opus 5's training compute on ARC-AGI-3-style problems.

Or it could be some combination of both. Personally, given how much of an outlier the ARC-AGI-3 jump is I struggle to see it being the product of a significant improvement in general intelligence.

I think that's a strange way to look at it. The brain also has different regions with different functions, and part of what makes us humans intelligent is that we can use tools like pen and paper to keep notes and help us solve problems.

Similarly LLMs are not just massive uniform artificial neural nets, and now we also have harnesses, which I'd personally view more of an extension of the model itself. The harness is both the executive and also what allows it to keep notes, use a calculator, or maybe even create scripts to help it solve complex deterministic problems.

I think it's unfair to give a human a very complex maths problem and say that they're not intelligent if they can't solve it without pen and paper or a calculate. At least expecting a human to solve complex problems this way doesn't really measure anything useful in the real world.