flatline

Meta, ByteDance, ...

I don't think the discrepancy is in LLM capability improvements over the past year.

Correctness has never been a priority across an industry where rapid iteration and feature delivery drive sales. There's always some opportunity cost to doing things right, at the price of technical debt down the road. If AI is primarily used to produce fragile code, people will be wary of AI solutions. There's also ongoing public debate about AI safety and alignment. Deploying AI in safety critical applications feels riskier than ever in the current environment, even though it doesn't have to be.

SQLite FTS could have done this a decade ago. We've had good local search capabilities for two decades and they either been underused or abandoned (e.g. Google Desktop, Yahoo! Search). This may be a reasonable projection of where AI is headed. You can do a lot locally but there is too much incentive to centralize around cloud infrastructure, then the privacy concerns make that prohibitive and we end up with what feels like a false choice of cloud or bust.

Maybe it's LLM hype that will bring more powerful capabilities to the desktop?

I think that's a bit strong of an assertion. How many people use copilot daily under an enterprise agreement? I don't necessarily disagree in spirit, especially given the questionable data sanitization around the recent Navier-Stokes announcement, but most companies disclose huge amounts of data regularly to hopefully-trusted third parties. I think the internet -- and a good share of the world's commerce -- would grind to a halt if we suddenly stopped. Setting up, securing, and maintaining local models for even a small user base is non-trivial and there is way more demand than supply for that skillset right now.

Exactly. It’s still just software, which someone programmed and deployed to do specifically dangerous/malicious things. I feel like we already have legislation and case law surrounding this.

Does mathematics still have a practical impact without humans in the loop? I don't think there's one single answer to that question, but I think it's worth considering exactly what that impact may be.

Tech companies are as much the topic of this post as AI, I think that's the immediacy.

Isn’t the implication that the vulnerability had already been found and exploited instead of reported to Google, because its value to Google was so low? Whoever found it originally had more incentive to sell it on the grey/black market. Or is the incentive structure truly different for vulnerabilities discovered to be already in the wild?

It’s Foucault’s interpretation of the panopticon. If we know that we may be watched — and punished — we self-police into following the norm, even if we do not know when we are being watched or that someone is necessarily monitoring us individually.

When I was a kid in the 80s none of this was an issue and I was free to roam. Starting in the 90s this became an issue and not only will parents not let their kids roam but the kids themselves have less interest because the norm is to stay in or only go out when supervised. You know you are doing something slightly wrong. It only takes occasional reminders like this to reinforce the rule, once established.

Same, this is like blogspam in the days of old that doesn't add anything to the original content and is just there to generate revenue for this site. The linked posts are better and per HN guidelines the toplink should probably be changed to one of those.

General doesn’t mean it applies to literally everyone. If autism turns out to be a specific condition that is e.g. the result of abnormal neuron pruning during early development, it could have a wide array of symptoms, but that understanding may someday lead to genetic treatments for sever cases, better early childhood interventions, etc.

I do think that the current ASD umbrella as written in the DSM is so broad as to seem nearly meaningless when take in isolation. It’s not meant to be in isolation however, it’s defined first and foremost as a disorder, so the symptoms need to cross some threshold of tolerable. It’s also associated with a growing body of research as to how nervous system development can cause related clusters of symptoms.

Is autism as we understand it scientifically accurate? Probably about as much as anything hovering in the intersection of social sciences and medicine. It does have some predictive power. If you meet the diagnostic criteria for autism you are likely to have certain comorbidities for example. Having certain autistic traits are good predictors of certain outcomes later in life. And we do have better therapies especially for young people than we have in the past.

The fact that people can have non-overlapping clusters of traits and still be autistic does seem problematic. It may not be the right model. Just my $.02, I’m not the autism researcher in this convo, but I am autistic :)

This line of thinking diminishes the real struggles autistic people face. Not everyone is autistic, and it's not just a quirky set of traits. I do believe society could be better arranged for people who for whatever reason don't conform to the overculture norms, so we could talk about disability along the axis of how society privileges or disadvantages certain individuals.

The concern is that autism as a rigorous category may be useless, but that doesn't mean that autistic people are suddenly neurotypical -- a term that is more politically than scientifically oriented. People who are not autistic broadly are allistic, to be precise with language. They may be neurodivergent in other ways.

I’m not sure why you were downvoted. I have to use Windows for work. The barrier to entry is simply lower for many system administrators and users alike. The homogeneity of a Windows rollout is extremely attractive. You are going to have a hard sell getting your accounting team and receptionist onto Linux. Your developers will complain and make do. I love complaining about Windows!

Their sandbox just sucked, the agent found a route through artifactory which was also connected to the internet. After the agent breached it, they put it back in the patched sandbox and it found its way out with another artifactory exploit — which was still connected to the internet.

No attempt was made at an actual air gap as far as I can tell. OpenAI is dangerous and irresponsible and doing a lot of pearl clutching over how their cybersecurity agent is doing cybersecurity things.

I think I agree with you in spirit. I don’t like what these companies are doing, and Anthropic’s actions can for the most part stand on their own. Copyright law is just a special interest of mine, and I do think it’s important to recognize what external incentives exist and what they prioritize. Because other companies will act in similar manners under the same incentive structure, and the problem is going to cascade and magnify if it hasn’t already. There are active court rulings setting precedence for this behavior - take note!

Why would a company keep the hard-copy around at the risk of it being inadvertently given away, resold, etc.? It's a huge outstanding liability given that the illegal copying of works -- the other part of that case -- is what they settled out of court for some huge amount of money. Destruction is the only thing that makes sense.

I'm old enough to have been around when DCMA legislation was under discussion. Many people were dead-set against it and raised concerns over matters exactly like this. In Rainbows End (2006), Vernor Vinge wrote about a similar scenario where a robot went through the university library shredding books, and scanned the shredded pieces to recombined them into a digital archive.

Anthropic may be doing shady things and may have even done this on their own recognizance, we just don't know. As it stand, this is 100% a consequence of US copyright law, much of which was written by large corporations to protect their own assets.

They cannot scan the books then resell them or donate them under current US copyright law. It’s not clear to me that they could warehouse them if they wanted. In the recent Bartz v Anthropic case the judge ruled this destruction as legal, saying

> The print original was destroyed. One replaced the other.

So that there was still only one “copy” of the book. This is in compliance with the DMCA. You can make a personal digital copy of a work but then you cannot resell the hard copy and keep the digital one. Same principle applies here.

This is why we have hierarchies of abstraction. Pretty much every field of mathematics relies on constructing notations, models, and other tools to simplify things in a way that is verifiable. LLMs rely on the same basic technique, they can just pull from a wide variety of these abstractions at once. So far we've been able to understand their proofs just fine. Computer-assisted proofs in the past that relied on brute-force is where we have run into trouble. We cannot reason about millions of possibilities at once, and we had to trust that the computer program that analyzed them was correct, which is a really hard problem and leaves humans fairly unsatisfied. I think we are actually progressing in terms of understandability in computerized proofs.

You deploy an application that ostensibly has the desired capabilities but it fails at scale, or the architecture is not extensible to add new capabilities or fix bugs. Those are expensive mistakes and in safety-critical domains could be deadly.

The thing is I've worked on plenty of code like this written by humans. AI just accelerates the process and puts this type of code within reach of more people. It could be seen as a step backwards in good software engineering practices we've painfully learned over the last two decades. I see it as a cycle repeating itself in a new iteration, with the hope that we'll end up with still better tools and processes in another few years. But I agree it can and will result in real harm in the meantime.

The author kept acting in good faith till the other party stopped doing so, which I took as the actual resolution to this story. Maybe when you stop treating people like rational adults when they have demonstrated otherwise is where you find your real “no.” It’s a hard lesson I’ve learned many times, and it sometimes entails walking away from something to preserve your own sanity/integrity.

I agree about OpenRouter. The AI Gateway number [0] is likely the figure that was actually coming to mind. Moreover, Qwen models alone have overtaken the previously-dominant Llama models in hf downloads by quite a margin.

Real question, and a refinement to my previous statement: would you find it more surprising if over 25% of worldwide inference was running on Chinese open-weight models, or not? I personally would not be shocked.

[0] https://vercel.com/blog/ai-gateway-production-index-july-202...

I knew when I wrote that it was a bare assertion, based partly on memory. This is an approximation based on a few sources, the principal of which was this article, which pulls from a bunch of other sources in turn.

https://www.secondtalent.com/resources/ai-trends-in-china/

I watched the full video and their conclusion was: service providers need to be doing this type of agent red-teaming continuously to counteract the attack sophistication of systems like theirs that are either extant now or soon will be. “You must buy our top tier agents for the good of humanity.”

This is their only realistic counter to cheap open weight models. Usage of AI services has shifted dramatically to Chinese providers - from 4% at the beginning of the year to some 30% now. They cannot release their latest SOTA models to the public, due to government restrictions and possibly real risk of misuse. US labs face downward price pressure on one end and anxious government admins on the other. How will they pay the stupidly high cost of training the next SOTA models? This is their only avenue, and it’s questionable how viable it is IMO.

As critical as I am of NM, I don’t know that this is true. The state has been funding mental healthcare for 2+ years out of gross receipts revenue. A large part of this goes to Medicaid recipients - NM is one of the highest paying states for Medicaid providers, at least for mental health. The money from this may very well go into a public coffer used for similar purposes, ultimately paying out to private providers while mostly bypassing our broken health insurance system.

I do question whether therapy services and the like are the best use of public funds for the people of NM. A lot of the state’s problems stem from a lack of economic opportunities, therapy isn’t going to help that. The state also carries a heavy burden of generational trauma which this could go some ways towards addressing, I suppose.

I have still not gotten a Waymo even once. I am in the service area. I don't use the service frequently, just enough to be disappointed.

Impressive judgment from someone who presumably has no knowledge of the types of problems the lab was addressing, the scope of research, the actual individuals employed, or the prior and anticipated impact of their work. This was the whole problem with DOGE: cuts were indiscriminate and based more on feels than factual data. I won't posit that those cuts were all downside - but we will be paying to discover which were really needed for years to come.

Unscrupulous operators will just run up huge liabilities and close up shop by the time they can be identified and dragged into court, meanwhile having started another similar operation under a different name. That's basically already what's happening, and the speed differential between the technology and the law will be forever in their favor.

I don't understand how people fail to understand how public policy works. You can tell people to do something and they may or may not ignore you depending on how you say it, and what incentives and penalties you provide.

Being a parent or a teacher puts you in a position of fighting multi-billion dollar corporate interests that are trying to capture your kids' attention. It's also fighting their very successful advertising campaigns that makes kids, just like adults, feel FOMO for not having the latest gadgets, or access to the same social media outlets as their peers.

Any public policy trying to counter this needs some weight behind it, and a strong consensus as to what we value as normative behavior in society.

Every time there is an economic boom like this, you get unscrupulous actors coming in and extracting as much value as they can before the regulatory environment catches up. Yes, datacenters should be doing all these things, but that costs money, so they won't until there are local statutes and federal regulations that require them to do so. I expect the current situation to continue at a huge social cost for at least the next 5-10 years unless the bottom falls out of the AI boom before then.

Yeah I didn’t read it that way at all. I think that addressing mental health issues requires some frankness with yourself first and foremost. I know some people object to identifying with labels such as ADHD, autism, depressive disorder, etc., but I do not know if that is what the parent intended.