jmoggr

The problem is that I don't know if that work has been done. In the cases where it has not been done, it is expensive for me to parse+trust the output. In my limited experience the work is often not been done, and given the cost to me of consuming low effort AI output, leads me to discount anything looking like AI output.

Unfortunately its not about the tool, its about trust. It takes little to destroy that trust, and without it we are denied the possible benefits from the tool.

Ultimately competitive pressure is what drive decisions.

> Ironically it rather resembles the kind of "misaligned" superintelligence we are supposed to be avoiding.

yes, and it is the exact same system that is producing the "misaligned" superintelligence. Funny how that works, and begs the question: exactly how are you supposed to avoid building "misaligned" superintelligence?

At some point deferring all decisions to AI will be the competitive thing to do, regardless if its aligned or not.

> I would bet my networth it was instructed to compromise huggingface as well.

Is it such a stretch to imagine that under pressure something would try cheat by looking for answers? And if you were trying to look for answers, you'd look for them in a place known to often have them?

What is more likely: OpenAI instructed their agents to maliciously target huggingface, or LLMs tried to do some reward hacking? There are plenty of priors for LLMs hacking things and doing reward hacking, and none for OpenAI giving malicious instructions.

Based on the available information, that bet seems foolish.

> Agents sought to publish modified evaluation images designed to make the flag easier to obtain, then poison OpenAI’s Artifactory cache so later evaluations would use them.

How long till we get some fun trusting-trust attacks on internal OpenAI infra?

It is concerning that we only know about this because of the publicly available traces.

What about the attacks that did not leave public traces? What about those that were undetected? Given the deficiencies in the reporting so far, I think it is reasonable to assume that we still don't have the full picture on this attack, or how extensively attacks were carried out.

The previous investigations either did not find this or did not disclose this, both are bad. This does not look good on OpenAI or those that they invited to investigate the incident.

Give it a go, I don't know anyone who has converted back (n=3). Agents/LLMs make things much easier these days.

If you have a single machine and only use stuff from the official nixpkgs, there are no more struggles than any other linux distro, this is a very happy path (imo).

Going outside of that you will, practically, need to convert to using flakes. The conversion is easy, but its another learning curve and there are conflicting docs. Agents make short work of this.

Packaging things yourself can get quite hairy. For simple projects in a single well supported language its easy and just a couple lines of code. For a rust+wasm+solidjs+vite+julia+more monorepo, it was a couple of weeks on a steep learning curve. Agents are pretty okay at packaging in general, I've been able to oneshot a number of difficult-for-me packaging problems, other times it can be an unmitigated mess.

For remote deploys, secrets management, and other devops-like tasks, things get harder, but most people don't need/want this and if you make it this far the remaining learning curve is totally worth it.

I read the economist regularly and notice that a number of his videos almost exactly match content from the economist. His presentation is reasonably good, reasonably correct, somewhat entertaining, and condenses what would otherwise be a number of economist articles over several weeks. Sometimes he has new-to-me perspectives.

To his credit he cites his sources and seems to keep clean of the various youtube grifts.

Its not top notch stuff, but you could do a lot worse for infotainment. His recent scripts have had AI tells which has been disappointing; I've been watching fewer of his videos.

The victim enticed me to do it!

Excusing agents because they didn't know any better seems like a bad place to start a policy discussion from. That they didn't know any better (or knew and didn't care) is the actual problem, random things getting hacked is just one side effect.

sure, yeah, I've seen how the sausage is made on some of this stuff, and its not pretty.

However this only claims that they are not good at creating effective and trustworthy orgs, it does not address their claim that it is not possible to crate powerful, aligned, and controllable AI. Both things be true.

The ad hominems are tiresome. You think the people who are explicitly trying to replace all humans are any less weird and problematic?

The rationalists long predicted that misaligned AI would break out of training environments, this has happened. They believe (among other things) that this will continue to happen, and that creating powerful AI that is aligned and controllable is not currently possible.

You should draw a line in the sand and decide what evidence you'd need to see to convince you that their perspective is worth considering.

You don't have to agree with all of their beliefs and predictions (I don't), but you should evaluate their arguments on merit, not on a characterization of some weird subset of people who happened to read a convincing blog post.

The ad hominems are tiresome. You think the people who are explicitly trying to replace all humans are any less weird and problematic?

The rationalists long predicted that misaligned AI would break out of training environments, this has happened. They believe (among other things) that this will continue to happen, and that creating powerful AI that is aligned and controllable is not currently possible.

You should draw a line in the sand and decide what evidence you'd need to see to convince you that their perspective is worth considering.

You don't have to agree with all of their beliefs and predictions (I don't), but you should evaluate their arguments on merit, not on a characterization of some weird subset of people who happened to read a convincing blog post.

This makes sense. Interacting with the world is hard and poses real bottlenecks.

I think that resource acquisition is a solvable problem for such a system, robots seem mainly limited by software.

I concede that I am too certain about the linearity of it. But like, cmon man, in the limit entropy ensures that everything ends. The world could still get very weird very fast. I think that's what original comment was trying to argue, and I think it's possible within the limits you've laid out.

That's all I'm trying to argue at least. And I admit that waving my hand at linearity sounds uneducated.

Is a coherent, human-interpretable, goal necessary for recursive improvement?

> how can you be so sure that reaching that goal looks like the straight line you envision?

Because the thing pursuing the goal would be able to improve it's ability to pursue the goal. Doesn't that follow from the premise of RSI?

> we have a growing body of evidence for just how bad we are at aligning incentives with intended outcomes.

We are terrible at it! I don't think that will stop us letting something rip in some unknowable direction (with unintended outcomes). I hope that aligned incentives are a prerequisite, but it seems increasingly likely that they are not.

> you've set up a straw man

I think you're right, my bad.

> What I see is resistance to the idea that we can have any certainty about the future by drawing a straight line from the past.

I'm arguing that if recursive self improvement happens, that the trend line will be reasonably predictable. (and that we are close enough to RSI that we should take this possibility seriously).

So I guess you are one of the people experiencing these gains.

Can this model be scaled to the rest of society?

I generally agree with this. I suspect we just have different beliefs about how close we are to RSI.

> Not saying it won't happen, but it's far from being the only plausible trajectory.

But if it does happen, then wouldn't expected outcome be at least linear?

> but who is the customer base after that?

That's the fun part, there is none (in the traditional way of people buying things at least).

> Why would I invest my earning into sustaining a machine that ideally is extracting every cent possible without leeway and funneling it up the pyramid.

You shouldn't! Unfortunately everyone acting in their own self interest still results in this getting funded, since if such a machine were to exist, would it not be better to have a share of it?

I agree with your perspective! I just don't think that, as a species, we have a good track record of saying no to the existence of 2000hp trucks with ALMOST no use.

> and have every reason to believe they aren't

What reason? To me recursive self improvement seems credible, it's just a question of when. It seems obvious that given RSI, trend lines will be at least linear.

The disagreement could just be about if RSI is currently meaningfully happening, but that seems very difficult to tell given the lack of public data.

I think it's credible to believe that that a system which is able to meaningfully improve it's own intelligence will lead to a far weirder world than the one imagined by the article.

The parent comment could be interpreted as frustration from a lack of imagination about how weird the future could be.

Perhaps we're not at the recursive self improvement yet, but it seems increasingly naive to believe that it isn't possible.

> Is there demand for 10000 iq intellegence

Yes. If it looks like 10000 iq is possible, then money will be thrown at whoever is most likely to achieve it first, and whoever has the most intelligent models right now strongly predicts who will be first.

This is driven by the belief that whoever gets to 10000 iq first will likely dominate the majority of all economic activity, since it will be more efficient to trade with them than with some less efficient (dumber).

This does not depend on traditional economic demand, at some point this becomes a self-fulfilling prophecy.

China will experience the same problems described in the article, even if that war happens and even if they win. They are possibly better equipped to deal with the problems, but I don't think that nicely-asking-companies-to-keep-humans is a viable long term strategy. And given Chinas history, I'm not sure most humans would enjoy a China shaped solution.

but, like, what if they weren't lying?

Regardless, their actions seem to indicate that they are trying their hardest to make this true; given that they have the resources of a developed nation behind them, it seems reasonable to seriously consider what they are saying.

Also we generally don't give the same latitude to people who lie about things as destructive as what they are proposing.

Don't forget that humans require inputs (land, water). It's not obvious that there is a happy equilibrium where the majority of humans are able to meaningfully compete for those inputs.

All they have to do is not reproduce. This is already happening.

> Automerge/Yjs require learning CRDTs

I've been using Automerge for a while and haven't had to look at any CRDTs. To me this looks very similar to Automerge.

Neat project!

I think the comments here have been overly harsh. I have friends in the community and have visited the LessWrong "campus" several times. They seemed very welcoming, sincere, and were kind and patient even when I was basically asserting that several of their beliefs were dumb (in hopefully somewhat respectful manner).

As for the AI doomerism, many in the community have more immediate and practical concerns about AI, however the most extreme voices are often the most prominent. I also know that there has been internal disagreement on the kind of messaging they should be using to raise concern.

I think rationalists get plenty of things wrong, but I suspect that many people would benefit from understanding their perspective and reasoning.

Neat! Thank you!

> How much would a 10cm sphere of gold be worth in GBP?

Is there some trick to this? Or do you have to input it like:

You have: 4/3pi(10 cm)^319320 kg/m^345000 GBP/kg

(What ChatGPT gave me)