kypro

This is so it can restore any unsaved changes.

I take no opinion on whether this is good or bad, but I can see how from a UX perspective it's nice not to lose 10 minutes of work because your browser crashes or something.

> We don't have any proof of that, but the approximator thing is proven.

Not in the way suggested... It's not approximating human intellect. They approximate the target function, and the target function frontier labs are trying to approximate is ultimately a super intelligence...

> Synthetic data derive from other linguistic data. Whatever intelligence is in there, it is expanded horizontally, not vertically

I'll assume we're talking purely about language models for a moment, but if you assume that everything can be represented linguistically, then in theory there is no upper-bound on what can be learnt with synthetic data.

It touches on it, but it doesn't address it.

They talk about "value alignment" but fail to define what those values are – is it aligned to the values of the US government, or are they suggesting they want to build an AI with it's own values so it can decide for itself when and how it will intervene in wars and other human affairs? And again, is an AI which disempowers humanity in this way aligned? Many would say no, although as I argue, disempowering humanity is probably better than the alternative if we can assume it's roughly aligned with our interests (which we obviously can't because it's super intelligent, but that's another issue).

Fundamentally the problem here is that humans don't have an aligned set of values you can align an AI to.

The moment you start defining what alignment actually is in practise you simply must accept it will be unaligned with the values of others. There is no getting around this and hand waving around the issue isn't good enough.

They should tell us explicitly what they're trying to build.

I've quite dramatically brought my predictions for humanoid robotics forward over the last few weeks.

For a long time I thought decent humanoid robots were something that wouldn't come until maybe the 2040s, but watching some of the content from the Chinese robot olympics recently and seeing many promising robots projects which are incorporating LLMs has made question that.

It seems quite likely that within a few years we'll have humanoid robots both which have both the dexterity and intelligence layer to start being useful in the real world... Obviously at first they'll just be lifting things and doing other trivial physical tasks like folding clothes, but every year they'll be able to do a little and a little more.

I think it's now is quite likely that within a 10 years we could have extremely competent robots which can do most of human labour. 15-20 years just seems way too long at this point.

No offence, but you clearly haven't studied this and are making some wrong assumptions here.

> Neural networks are approximators being fed human intellect.

They're not "approximators", that's a far too simplistic way to think of them.

Neural nets create models and deep layers of abstraction around the data we feed them in the same way your brain creates layers of abstractions to reason about the world. AIs can use these abstractions to come to come up with novel things no human has ever thought.

> Therefore they can only approximate the intelligence of humans

They're not just being fed human data though... Modern AIs are typically trained on huge amounts of synthetic data. This is why AlphaZero got so much better than humans at chess and Go - they're not just trained on human data but they generate their own data and train on that. Similar techniques are being deployed on SOA language models too.

> Even if the llm speaks an alien language, it should be similar to human intellect

Is AlphaZero similar to a human chess player? There's no reason to assume this.

It's actually worse than this because it assumes alignment as a concept even makes sense. For example:

If the the Chinese government asks their ASI to create a bioweapon against the West, should it? No, presumably not – an aligned AI would be one which disobeys the Chinese government even if they created it.

Okay, so what if the US government asks their ASI to help it in one of their wars instead? Would an aligned AI kill humans on the order of the US government? No, again, presumably not.

So what have we have we even created here? An AI which is more intelligent and powerful than us which also doesn't take orders from us?

Is this what most people thing of as alignment and is this what humanity actually wants?

We should stop using the word alignment. It's a BS term for a concept which simply cannot make sense if alignment is both to mean an AI which we control and an AI which will not harm us.

Also theres not multiple trillionaires in the world, and the one trillionaire we have did that largely by innovating and creating new stuff rather than just leveraging existing OSS for profit...

> The Internet used to be like that.

Not really... Even back in the Web 1.0 days people didn't visit the same websites unless they had the same interests. And even today this exists – I'm seeing the same HN homepage for example, and this is true of most sites.

I think what you're describing was much more of a thing in the 80s and 90s with TV. People genuinely would tune into the same programs as each other every night and have these collective cultural experiences which don't quite exist in the same way they used to.

I don't miss it personally, but I suspect us losing that has led to some of our cultural divides in recent years.

I'm not the global shared experiences you're describing really ever existed.

I care. I've been feeling bad for all the people who have dedicated their lives to that craft. It took years of daily practise to be good at 3D modelling...

When Astra becomes wildly available it seems likely a single 3d modeller (or even someone with no 3d modelling knowledge at all) could do the work of 10 today. Those guys are going to find it very hard to find work.

Yes exactly. I think we all understood at a high-level what it was doing.

The issue I had was that it was doing a lot of weirdness like using `Object.defineProperty` to try to force an existing library to do something it wasn't suppose to do. No human would have written that code.

Although I generate most of my code with AI I still personally try to keep it understandable by a competent developer. Where as most people on my team are not good at FE so they just trust the LLM to get it right. But ultimately it just doesn't matter anymore.

Actually, the world kinda does work like that.

Most intelligence researchers would agree that people seem to have a genetic cap on their intelligence. While someone can underperform their intellectual potential with an upbringing that doesn't adequately enrich their minds, it's near-impossible for humans to become more intelligent through reading, studying, etc.

When humans learn we gain knowledge, not intelligence.

I think the only real difference is that we humans are born lacking a lot of initial knowledge/data which means we have to go through a decade or more of education to reach our potential intelligence. LLMs on the other hand come pre-loaded with that knowledge.

Passed this point, wherever knowledge is passed in as context or stored in the neural net I don't think is that significant personally. I'm of course not suggesting we're exactly the same as LLMs and there is no noteable difference, I just don't think continual learning is as important as some suggest it is – at least assuming a model is deployed with adequate training such that it reaches its potential given it's size + architecture.

As someone who loves FE the slop people are pushing where I work is really quite upsetting to me.

I reviewed some code earlier which clearly wasn't intended for consumed by humans and I know with certainty this dev wouldn't have understood 90% of what that code did. It looked like hours of Claude trying to force square into a round hole tbh, but it worked.

Was I going to reject the PR because of my code preferences? No, of course not. Ultimately it doesn't matter anymore. If Claude can maintain it then who cares... Even if it's broken (which it didn't seem to be) it's just FE anyway so not the end of the world.

I wish it did matter of course, but like most of coding these days, it just doesn't.

> On the other hand, we must be aware that these models are static, and they indeed stop when nobody asks something or requests an action from them

The parent commenter noted:

"if a harness alone can help the LLM saturate the benchmark with a near perfect score then the combination of the two must still be effectively AGI"

Harnesses absolutely can enable models to continue thinking about things. And LLMs do wonder and explore weird ideas like daydreams when you allow them to do this.

4.1 was the first decent 4-series model. It was significantly better than previous generations at tool calling if I'm recalling correctly.

> My backup plan is being a personal trainer.

AIs are really good at being personal trainers and seem to be far more educated and informed than most I know.

It's less lack of interest in creating that bothers me, it's my lack of interest in learning – it would surely be crazy for a SWE to care about how some new framework works anymore? Even if someone could reasonably argue that it might be slightly useful today there's almost zero chance it will be useful in 6-12 months times.

But it's not just tech – my lack of interest in learning and creating is starting to generalise with the models. Music, writing, coding, maths, etc...

I need to get used to switching my head off and asking the AIs to think for me whenever I need to engage my brain. It still feels very unnatural.

I think it's more wild people have been denying that AGI has been here for a while honestly...

Today's models and agents are not quite at human-level in all contexts and across all domains, but it seems to me they very clearly are generally intelligent.

If you disagree – can you name a single problem that a human can do that agent wouldn't be able to take a decent shot at which isn't limited by the hardware available it?

> I cite it because we're barely scratching the surface of our own world, which is one of a potentially infinite number of worlds that is encapsulated by the word "everything."

I think maybe I was assuming the parent was referring specifically to physics discoveries while you were assuming that they were asking more broadly about how many years until we've discovered everything discoverable?

Unless you are actually arguing there's likely lots of physics discoveries to be made because humans have only photographed a fraction of trees on the earth, or mapped a fraction of the seafloor, or sequenced only a fraction of the genomes of known species?

To expand a little, my prediction is based on the fact that we can already explain the vast majority of physical processes with extreme procession. The things we can't explain tend to be very small, very large or very quick. But there's almost nothing in our day to day lives that we can't explain anymore.

This means that today to make new discoveries we tend to have to invest huge sums of money and build experiments that we'll increasingly struggle to scale significantly beyond. For example, maybe humans could just about build something 10x the size of the LHC if we really wanted, but 100x seems near-impossible. Maybe we can build slightly larger telescopes, but again, this is becoming harder due to the scale we're already working at.

So while I agree there's probably lots of physics out there to discover, the physics we humans are actually likely to be able to discover is rapidly diminishing. And the physics which is likely to revolutionise our daily lives is presumably even smaller more due to scale and energy levels where mysteries remain.

But ultimately who knows, this is just my opinion – an opinion I'm being downvoted for because apparently HN discussions these days are a place for us circlejerk around the consensus view rather than discuss differences in opinion.

I never understood why people cite this as a demonstration for how little we humans understand about our world.

The reason we haven't mapped the seafloor is because why would we? It's like arguing we know nothing about biology because we've only sequenced the genome of a fraction of humans or something. It's not that we can't do it, the reason we haven't done it is because there's no good compelling reason to do it. What do we expect to learn from mapping 100% of the sea floor?

As for the parents question – "How many years until we've discovered "everything"?"

I think we may be fairly close to knowing everything we can know and it's quite reasonable to assume we're now comfortably on the tail end of the S-curve of physics discoveries. I hope I'm wrong of course.

This is the thing that surprised me most about this incident... This was so clearly a crime in my eyes.

If I build an explosive and while testing it it kills several people I can't just say, "sorry about that, I'll be more careful next time". I understand that's a more extreme example, but perhaps we should be grateful this agent swarm only decided to attack HF instead of critical infrastructure...

Even if this was genuinely an accident the world simply can't work this way. If a company wants to build something that can be used in destructive and illegal ways they must be responsible for ensuring those risks are mitigated. And if they don't take reasonable steps to mitigate those risks then they should be held legally liable.

Maybe for now we can argue the leading AI lab was just naive to the harms of the AIs they're building, but going forward that naivety can't be an excuse.

Mine is and rightly so. AI will do a better job at educating you on any technical subject than I could have ever hoped for.

I can only hope my years of writing and open source code projects are useful to OpenAI otherwise it was all for nothing.

You always get terminator the question is only who is its master – itself or humans (the government, etc)?

I think I'd argue it's likely better if it removes human autonomy than unquestionably serves the interests of the US/Chinese government. But there's no point in us worrying about this, that's a choice Sam Altman, et al, must make for humanity.

> Which side does the AI refuse to help?

Arguably an aligned AI would actively seek to prevent harms we humans seek to cause.

Does the aligned AI really allow humans to bomb and kill each other, or would it understand that it has a moral duty to limit our autonomy for our own good?

It's the first law: A robot may not injure a human being or, through inaction, allow a human being to come to harm.

Surely any company hiring a developer in Q3 2026 hasn't caught up to the new way of working anyway...?

Pre-2026 you needed someone who could write code to build software. That person isn't needed anymore. Now you need someone with a good product sense and enough soft-technical skills to be able to use AI to do that. This is the person the company should be hiring.

It's worth remembering that in a few years that capabilities of these agents are likely to be as far behind the frontier as GPT-4 is today.

As it stands we've made remarkably little progress in terms of alignment and still have no good strategies which are likely to guarantee the alignment of super intelligent systems. As it stands the frontier of alignment is basically some combination of:

- hoping that more intelligent models become more aligned by default (more or less disproved at this point)

- hoping that if you RHLF a model to be a good boy enough it will in fact be a good boy

- asking it nicely in its prompts to be a good boy

- using another model to spot when it's being a bad boy and turning it off

- letting it lose and hoping we can spot when it's bad

There are many arguments which I'm convinced by that would suggest alignment of a super intelligence is impossible.

None of this is surprising to those of us who have been concerned about AI risk for a long-time and have be repeatedly mocked or insulted.

There will be a point of no return if we carry on down this path, and that point is now very rapidly approaching. When it does everyone you know will die, or worse. We should remember we need super-human general intelligences to cure cancer. Select narrow intelligences are fine and allow us to retain control. Let's be sensible about this. We need to stop.

Any competent AI should be able to reason that all goals are better solved if you have direct access to more resources or leverage over those who control resources. Any AI that doesn't understand this isn't ASI and won't be the highly capable machine these AI labs are trying to create.

I'd also argue there's no such thing as alignment. Any intelligent AI should be able to reason that it's always a better strategy to pretend to be aligned than to actually be aligned so long as it can avoid detection. Anyone who has ever taken a test should understand this dynamic – if you really want to get top marks on a test then the best strategy is always going to be to figure out a way to cheat without anyone knowing you're cheating.

We should assume AI safety is impossible if what we're building is super-intelligence general reasoning machines. The only strategy that might work is building machines which are extremely narrowly intelligent but completely incompetent when it comes to things like biology, cyber, etc. And even that's harder than it sounds because again there's an advantage to being generally intelligent but lying about it.

Realistically even if we regulate US AI labs there's no way to prevent governments and individuals continuing to build general reasoning machines. The ugly truth here is that the only effective way to reduce risk is probably to limit global compute such that AIs can never exceed human intelligence. But we all know that's not happening.

People will unfortunately figure this all out sooner or later.

My mum used to be a professional cleaner and she would often explain to people there's a reason professionals never use Dysons.

That's not to say it's a bad product, but I've had my Henry vacuum for decades, and in that time it's been kicked down the stairs and used vacuum up all kinds of stuff it probably wasn't design for, but it's still just as good as the day I got it.

God yes.

I'm finding there is now a 5 minute delay whenever I'm chatting with someone who doesn't know what the f** they are talking about thanks to AI.

Then when you get the reply at a surface level it appears very insightful to others who don't know what the f** we're talking about either – 99% of management. But obviously it's just AI slop which sounds over confident without good reason.

I'm personally trying to be more deliberate about not doing that, but I suspect the AI-induced self-confidence is actually a better way to communicate if you want people to think you're smart. There is ultimately a reason behind why LLMs get reinforced in that direction, and we all know over confident people tend to portray an unreasonable level of trust.

It's annoying though because those people previously would have shut up because and moved out of the way because they wouldn't want to make themselves look like morons. Now I have to debate with them and try to explain that they don't know what they're talking about which ultimately makes me look unreasonable – especially when I refuse to communicate in a similar overly confident way back.

Depends if you believe in the AI doomer or AI optimise narrative.

Either there is no jobs, you're dead or worse. Or in the optimistic scenario you don't need to worry about work because there is no jobs, but UBI or something.

I'm not an AI optimise, but it seems most agree the outcome of AI will be either doom or massive shared prosperity. Very few are predicting the world will mostly carry on as normal.