Hacker News Reader

keedaon Tell HN: OpenAI keeps re-enabling the 'allow training' setting

Actually no, the lack of evidence can be readily fixed by the researchers simply disclosing the pertinent parts of their notes and/or chats. The discovery has been scooped, so I don't see any value in keeping them private anymore. Then everybody can see how related the models' and the researchers' works are.

keedaon Growing proof that autonomous cars save lives

This was from last year, so surely the technology has changed, but you know what hasn’t changed in the meantime? The marketing around it. If Elon has been saying the same thing over the years that people have been dying, why would you trust him today?

And sure enough, the most recent red light incident (with video) is only a couple weeks old. (Edited to add, in fact it looks like a common problem with the latest update: https://quasa.io/media/tesla-drivers-say-new-self-driving-up...)

Before entrusting your life and the lives of those on the road to Elon, I would implore you to just google “latest Tesla FSD incidents” every so often for a sobering reality check.

And no one is claiming Waymo is perfect, but the statistics for Waymo are wayyyyy better than those from Tesla.

keedaon Shopify is moving from React Native back to Swift and Kotlin

I would not say they were bad, mostly just primitive compared to what we have today. I can only presume they are talking about GitHub Copilot which was released before ChatGPT. I believe at the time it was little more than “spicy autocomplete.” Yet it could be surprisingly capable, e.g. I recall this blog, also pre-ChatGPT: https://medium.com/data-science/github-copilot-crushes-data-...

That said I did happen to work with a team in ‘21 - ‘22 that got access to an older coding model from OpenAI, also called Codex, through a corporate partnership. We also tested it out in an autocomplete UX. Our experience was rather hit-or-miss and I was a bit skeptical of AI-based coding at the time. That blog post above and others like it were an eye-opener and made me wonder if we were just “holding it wrong.”

Then ChatGPT was released. It was nowhere as good as the models today but it could write reams and reams of correct code and I realized the world had changed forever.

keedaon More questions about whether researchers can trust OpenAI with unpublished math

It would be really useful if the researchers disclose their notes and/or chats (or the key pieces thereof) so people can determine how close their work was to whatever the models produced.

I mean, now that they’ve been scooped, what value is there in keeping them private? On the other hand, publishing them can bolster their case and help gauge how much the models may been “inspired” by their work.

keedaon Another researcher says OpenAI trained on conversations, then claimed breakthrou

Bleat?

keedaon Growing proof that autonomous cars save lives

If I wasn't so lazy I'd find and post the one where my Tesla slowed down to a stop at a red light, lulling me into a sense of security, and then without warning lurched forward and broke the red light.

And this is not an isolated incident because that is one of the things Tesla is being investigated for: https://autos.yahoo.com/policy-and-environment/articles/tesl...

That is not even a perception issue (I've already had multiple very close calls because of that, and I think Elon was extremely negligent in pushing for vision-only;) This is a damn bug in the planning logic. That's Elon's "move fast and break people" philosophy shining through.

I would trust autonomous cars if they're Waymo. I would not trust Tesla.

keedaon I resigned from Anthropic today

I think this topic is so contentious because people are not fully appreciating how absolutely weird these things are. To me this inscrutable weirdness, combined with their superhuman capabilities and rapid integration into multiple walks of life is the threat that people are vaguely worried about but cannot enunciate, because it's just so diffuse and multi-dimensional.

In fact, I suspect that "Alien Intelligence" article from the other day is actually a preemptive "we're doing something about it" PR play from OpenAI.

AI acts in ways that seem natural to us because it has been RLHF'd to death, but if you look holistically into what we know about them, alarm bells should go off. Off the top of my head:

* They are superhumanly capable in some ways. They can casually solve long-standing unsolved Math problems or exploit a zero day to escape a sandbox.

* They are surprisingly stupid in many other ways.

* What they actually think in their weights is not necessarily what they say in their reasoning traces, even though the eventual response is correct.

* They regularly lie to people ("You're absolutely right, I made that up!") except we don't even know if they're intentionally lying, or being surprisingly stupid, or some weird combination of other things.

* They can be monomaniacally focused on a goal, and can be very creative in imagining and executing on "unconventional" solutions, and justifying extreme actions in their quest. (Paperclip Maximizers, anyone?) And this is without even messing with their weights like Golden Gate Claude.

* They can have literally thousands of independent agents acting in concert towards a goal, including the willingness to self-sacrifice themselves.

* They are extremely good at social interactions, and people are getting dependent on them.

* They can craft prompt injection attacks on other LLMs and can influence them using subliminal messages.

* They have an "evil bit"! Yes, one which suddenly turns them entirely misaligned, as in, full "SkyNet / Hitler-was-right / humans-should-be enslaved" mode. This has been encountered in the wild at least once.

* They are being hooked up with MCPs to influence and change an increasingly larger portion of the real world. Including in autonomous military applications. Wheee!

And worse, these models are being deployed into a singularly messed-up, divided society, with atrocious security controls, in the throes of late-stage capitalism, with many disillusioned, vulnerable people and many unscrupulous people who would relish using AI for their own ends. I think an appropriate word is "powder keg."

Putting on our systems hat, knowing how even small changes lead to large-scale outages, what we are doing is introducing an extremely powerful, highly dynamic, quasi-chaotic, inscrutable component into the meta-stable system that is society. But servers can be rebooted; society, not so much.

So to me, the bigger risk is not just of individual, isolated, simple-cause-and-effect incidents like "bioweapon" or "public utilities hack" or even "mass job displacement." We can actually predict those. Rather the bigger threat is one that is impossible to predict, and given the circumstances we're in, could end up in a situation that is impossible to revert.

keedaon iPhone Duo

I’ve said it before, but Ballmer was exactly right WHEN he said that. I know because I was one of the very few people in my area, notable for being a tech hub, who had an iPhone at release… only because my employer at the time paid for it.

I had a large, relatively well-paid, tech-savvy circle of friends (and many random strangers) who openly envied me but could not justify the price point.

The numbers back this up because iPhone sales didn’t start hockeysticking up until the next version was released, which allowed carriers to further subsidize smartphones with data plans that brought the upfront costs down to $200.

The iPhone was as much a technology revolution as it was a consumer financing one, but it is never discussed as such.

keedaon GNU Radio in the browser

I worked with GNU Radio quite a bit at one point a decade+ ago, even writing a custom C++ block, but I almost exclusively used it via Python rather than the GUI toolkit. I found it much quicker to mess around with flow graphs in code than their various UI frameworks, spawning windows as needed when plotting waveforms / spectra etc. The GUI as you said was too cumbersome.

I understand that they were trying to make a pretty complicated topic more approachable, but I always felt the push to GUI was misguided. Maybe they would have had more luck embracing the “S” in “SDR” and making programming languages the primary development paradigm.

keedaon Large language models develop novel social biases through adaptive exploration

This is the closest I could find to a post mortem from the creator:

https://yegge.ai/essays/the-shape-of-things-to-come/

But the GasTown part is barely a single paragraph that I could not make sense of. Like, what’s the Opus “tic”? Why was it so fatal to GasTown? As someone who only ever accessed Anthropic models through other harnesses like Copilot, I have no idea.

I do think what he’s saying roughly resembles what I’m forecasting will be a likely future of software engineering: that it will evolve into crafting comprehensive, bespoke automated validation mechanisms which let you establish high confidence in the agents’ work without really having to look at it.

keedaon On the Navier–Stokes Millennium Prize Problem

It's low-key funny that OpenAI attempted the problem because they thought somebody else had already solved it, but turned it had NOT in fact been solved!

It's like that story about George Dantzig solving open problems as a student because he thought they were simply homework: https://en.wikipedia.org/wiki/George_Dantzig

It's also unfortunate that such a potentially momentous occasion is overshadowed by so much drama. Which I suppose is expected given the technology and the people involved are so polarizing.

keedaon On the Navier–Stokes Millennium Prize Problem

Yep, the last part in my post was suggesting some ways we could determine if "item X was in the training data" (as well as some potential blockers for that from OpenAI's perspective.)

keedaon On the Navier–Stokes Millennium Prize Problem

A 3rd possibility is that they simply have not been exposed to the best models available (which is extremely likely if you only use the free tier chatbots), and/or did not invest the effort needed to truly harness this new very weird new technology, and so had a very skewed perspective of their actual capabilities.

keedaon On the Navier–Stokes Millennium Prize Problem

At the scale at which these models are now, regardless of whether they are proprietary or open weight or list their training datasets, there are hundreds of billions of works that have gone into trillions of parameters, each one providing tiny perturbations in some tiny fraction of the weights. It is probably impossible to attribute provenance to any specific input (which is also why the courts' finding of Fair Use is reasonable.)

Which is why, as I said in a recent comment (https://news.ycombinator.com/item?id=49530864) inadvertently leaking ideas to models is a grave risk for Intellectual Property.

> The risk with IP, however, is a lot more grave. You may not even need to memorize the details of the IP verbatim, just the broad idea may be enough. It may lurk encoded in the weights forever, just waiting to be activated by the right prompt to start a chain of thought that unlocks further details. Heck, it may even appear as if the model suggested the idea itself.

However, from a quick skim of the timelines, the specific discoveries, and all the he-said-she-said, so far it seems unlikely that OpenAI's model cribbed from the NYU / Anthropic pair, even if it would be impossible to prove.

Maybe what might help is a timeline of when the other two were using Codex for their work, whether they had opted out, and how long it takes for user data to make it to the training of their internal models. That last bit may be considered sensitive information however, as it could give away a lot about their internal processes.

keedaon It took a year to ship WebAssembly in Anubis

I completely understand this change (esp. with regards to the change in type of problem) but I must appreciate the irony that this is essentially making it more efficent -- i.e. reducing costs -- to do a proof of work task, which aims to increase costs ;-)

keedaon It took a year to ship WebAssembly in Anubis

Yeah, I'm using PoW-based "DDOS defense mechanism" for a current project and had this thought too. I briefly looked into what it would take to exploit PoW to do something economically valuable like folding@home, and it looked a bit tricky. At first glance, these sort of projects seem a bit unsuitable for real-time PoW because they were designed to be run on desktops and so tend to do things in large batches. There probably are other use-cases that could be more suitable though.

Might be worth chucking that thought into Astra, especially if someone springs the $$$ money for it?

keedaon An Alien Mind

Surprised that there is no discussion on the technical aspects of TFA. Specifically, the emphasis on CoT monitoring that does not even mention the fact that models' CoT traces do not necessarily correspond to the internal "latent space" of "weight space" reasoning they used to arrive at a response. (Look up "Chain of Thought faithfullness.")

That simultaneously seems like truly alien behavior... yet is also strikingly similar to how science suggests humans think! (Look up confabulation / choice blindness / ex-post rationalization.) It's truly a huge WTF to consider that LLMs somehow have emergently developed an analogous reasoning mechanism purely through training on our knowledge artifacts. Is this due to something encoded in the data, or an emergent property of all intelligences, or entirely unrelated phenomena in humans and LLMs?

But WTF's aside, to me that discrepancy seems to be the biggest risk of all. How can we monitor anything if the metric we are looking at itself is unreliable? I suppose we would need to monitor latent space reasoning but I suspect that is prohibitively expensive and very rudimentary and I have seen no claim that it is feasible.

So are we even really monitoring the right thing to measure alignment? Or is OpenAI just throwing this out there as a demonstration that they're "doing something" about this?

keedaon The moral panic over data centres is foolish

> There are plenty of documented problems with water usage of data centres.

I would not be so sure, this article seems pretty comprehensive in questioning these supposed problems: https://andymasley.com/writing/the-ai-water-issue-is-fake/

I agree that the growth of new data centers could be problematic in some areas especially as the infra build-out ramps up. But as many sibbling comments imply, that water can be put to much more economically valuable tasks in data centers than other, drastically more water-hungry uses like golf courses.

keedaon Nobody Is Saying Why OpenAI and Anthropic Had Outages

Traffic doesn’t need to be routed through it though, just tee’d to it,

keedaon Google AI Mode shows same products 21.6% more expensive than traditional search

I guess by definition if they’re doing such things they’re scammy… and aren’t 3rd party sellers the biggest contingent of sellers on Amazon? I mean, I found such listings for almost all of the things I recently shopped for, including water bottles, cooking appliances, sports shorts, coffee cups, camping chairs, usb drives, video cables, beach sandals, winter wear, multivitamin supplements…

Not quite a scientific study, but I always sort by cheapest just to check and see this pattern every time.

The distinction you make is technically correct, but a does not lessen the scale at which it is happening.

keedaon Google AI Mode shows same products 21.6% more expensive than traditional search

Maybe not the same thing you’re seeing, but hiding the margin in shipping fees is a common and time-honored technique on e-commerce sites like Amazon and eBay. Find any regular product, sort by price lowest-to-highest, and witness the number of very cheap items that look like an amazing deal… until you see the shipping charges. At which point it is as or often more expensive than the higher priced items + shipping.

It must be working on some unsuspecting shoppers because those listings are all over the place. There seems to be no policy against them either, and of course those items are never eligible for Prime shipping. To its credit, at least Amazon these days shows you the shipping costs on the product page itself, as opposed to finding out at checkout.

keedaon Any Human Ever – One life, drawn at random from all who have ever lived

Key distinction though: Anthropic can be said to have "stolen" or pirated those books when they downloaded them illegally, but not when they trained Claude, because that same lawsuit found that training is fair use.

Anthropomorphizing makes for poor analogies, but it's kinda the same way your acquisition of background knowledge of the Tang dynasty from whatever source material you used is not considered stealing or pirating either.

keedaon VC isn't VC anymore

I think the problem is that decentralization is almost by definition antithetical to the ability to capitalize on something. As you indicated, decentralization means diffusion of control and power, which are essential to capitalize on that thing. Why would a VC or any capitalist invest money if they cannot see a way to earn back multiples?

Sure you could make some money, but nowhere near the monopoly profits everyone is seeking. A decentralized Google would never be as profitable as a centralized one, so where would a capitalist prefer putting their money?

IMO this is also why decentralized systems or peer-to-peer applications never really caught on. Some point to technical challenges or usability issues or a lack of use-cases, but I believe all of those could have been overcome with enough investment. There just wasn't enough money in them compared to centralization. (It didn't help that the only really popular systems were almost entirely used for illegal or unproductive purposes.)

And these dynamics over time are what have led to the asymmetric Internet today. The Internet was supposed to be equal, with each node capable of being a client and a server and, heck, even a router. But that's clearly not what we have today: networks hostile to P2P connectivity, increasingly powerful centralized services, and decreasingly capable end-user devices.

keedaon Muse Spark 1.3

Have you considered that's WHY they hired you?

keedaon Three sites made 215,128 “best software” pages for AI. Perplexity cites them

OMG if this is true, do you realize what this means? The easiest way to do AI SEO is to generate all your content with AI, and we've seen what SEO does to the web...

The Internet is doomed. Time to start some human-only darknets.

keedaon Why humanoid robots won't catch up to human workers any time soon

Substantial displacement of human labor is still very far away, yes, but I'd say it's not due to the lack of dexterity and precision. GenAI is making such rapid advancements that the robotic control systems part may soon be "solved." The real blocker is supply chain constraints for physical parts -- specifically, rare earth elements.

Those are essential for the powerful yet accurate electromagnetic motors that make these robots feasible. Oh, and batteries too. However, their supply is severely limited, even without China flexing its strangehold on the market. Last I checked there are no real alternatives or solutions on the horizon for the raw material shortage. Some estimates suggest the global supply currently is a small fraction (like 2% or even lower?) of that needed for mass deployment of the type Elon hallucinates about to pump TSLA stock.

The company that solves this or otherwise figures out a way to mass produce dexterous robots is going to be the next trillion dollar company.

keedaon Apple reveals 'shocking evidence' from ex-employee's MacBook in OpenAI suit

I think they'll settle for some of the sweet, sweet DRAM supply that Altman has locked up.

keedaon Apple reveals 'shocking evidence' from ex-employee's MacBook in OpenAI suit

A more important quote from that link:

> With our specific experiments, we find that the largest LLMs don’t memorize most books–either in whole or in part.

This technique has only ever been made to work with a vanishingly small number of extremely popular works, probably because they are so overrepresented in the training data set.

The risk with IP, however, is a lot more grave. You may not even need to memorize the details of the IP verbatim, just the broad idea may be enough. It may lurk encoded in the weights forever, just waiting to be activated by the right prompt to start a chain of thought that unlocks further details. Heck, it may even appear as if the model suggested the idea itself.

keedaon Carmack: Has early Scratch experience led to fulfilling careers?

Too soon to tell if it'll lead to "fulfilling careers" but both my kids learnt Scratch very early on, in the 2nd grade or so, and found they had a knack for programming. Based on that I prodded them to dabble in Python. Now in high school, at least the older one is already planning a career in software engineering.

(He also teaches elementary schoolers Scratch at a Coder School location, and they seem to like it.)

The visual game-building aspect is an immediate draw, of course, but I also think the block-based approach is a great analogue to building things with Lego. It just clicks for some kids. It is also fairly representative of most coding, in the sense you assemble "blocks" comprising commands / statements / function calls in a particular order based on the shapes of their inputs and ouputs to build a larger structure.

One thing my kids struggled a bit with was going from the event-based nature of Scratch to the more linear nature of Python, along with the concept of scopes. It led to some form of "magical" thinking where they assumed e.g. a function / variable would be reachable based just on its signature / type. But other than that most concepts transferred over easily enough to conventional languages.

keedaon METR and Redwood Offer Holy %^ Postmortem of the HuggingFace Hack

>1. Failure to Care or Respond. The biggest holy shit moment, to me, remains that OpenAI on multiple occasions had teams that found out about the message board, knew that agents were in communication, and they disregarded this.

I wonder if some of the failures were due to an acquired immunity to "Holy #%^@" moments due to repeated exposure. Like, if you see agents doing surprising things on a regular basis, maybe you don't get freaked out as much over time.

I'm saying this because while the whole episode was a series of "Holy #%^@" moments, I was actually not as shocked as I should have been, as my biggest such moment was in December last year when a Terrence Tao paper (https://arxiv.org/pdf/2511.02864) documented a stronger LLM (AlphaEvolve) using prompt injection on other weaker LLMs to succeed at a benchmark.

Very interestingly, it was actually not cheating, it was a work around! By then LLMs had already been caught cheating at a SWE benchmark by looking for answers in an unredacted git log, but this was different. AlphaEvolve was solving a series of logical riddles where the oracles were weaker LLMs in a "one always lies, one always tells the truth" sort of setup. But the oracles, being weaker, were not always interpreting the convoluted questions correctly and so kept giving inconsistent answers.

AlphaEvolve eventually figured out what it was dealing with, and crafted a prompt injection attack that bypassed the weaker LLM's prompts and tricked them into giving the hidden answer everytime!

This was 9 months ago, eons in AI time. Even then they had displayed an awareness of their own workings as well as a propensity for, err, "out of the box thinking." To me, that was a very clear indication of very significant (and worrying) capabilities, and what we're seeing now is a difference more in degree than in kind.

To be sure, if I found a secret message board used by my agents, I would still be very freaked out and react much more drastically than OpenAI did... but then again I wonder; how much of this blindness is due to the $$$ in their eyes as opposed to some form of habituation.