advael

I get why someone would want to tell people this, but I think it fails to be advice, as it describes the current state of affairs without much actual guidance for how to do better in it

Which, to be fair, no one really seems to be able to answer that meaningfully

It's sad that all I can think about this is that it's just another creep forward of the surveillance oligarchy

I really used to get excited about ML in the wild and while there are much bigger problems right now it still makes me sad to have become so jaded about it

The problem described here is a fundamental mismatch between the structure of our economy and the conditions that facilitate human thriving

I have no reason at this stage to believe that this will get better instead of worse

I think that a mythology about where the difficulty in working with computers lies has made the relationship between businesses and the people they hire to do this stuff miserable for quite some time

"Coding", as in writing stuff in programming languages with correct syntax that does the thing asked for in isolation, has always been a very dumb skill to test for. Even before we had stackoverflow syntactic issues were something you could get through by consulting a reference book or doing some trial and error with a repl or a compiler. That this is faster now with internet search and LLMs is good for everyone involved, but the fact that it's not what matters remains

The important part of every job that gets a computer to do a thing is a combination of two capabilities: Problem-solving, that is, understanding the intended outcome and having intuition about how to get there through whatever tools are available, and frustration tolerance: The ability to keep trying new* stuff until you get there

Businesses can then optimize for things like efficiency or working well with others once those constraints are met, but without those capabilities you simply can't do the job, so they're paramount. The problem with most dinky little coding interviews wasn't that you could "cheat", it's thst they basically never tested for those constraints by design, though some clever hiring people manage to tweak them to do so on an ad hoc basis sometimes

* important because a common frustration failure mode is repetitive behavior. Try something. Don't understand why it doesn't work. Get more frustrated. Try the same thing again. Repeat

It's clear you're missing what point it is that I'm making from these responses, but I'm unsure how to explain it better and you're not really giving me much to work with in terms of seeming to engage with the substance of it, so I think we gotta leave this an impasse for now

I mean by the same logic the only difference between a diffusion model and a VLM is that you put the spatial transformer on the other end.

Yes, one of the powerful things about every kind of neural network is that they're a very general class of function approximator. That we can use a similar toolkit of techniques to tackle a wide variety of problems is very cool and useful. Again, the analogy to statistical models is telling. You can model a lot of phenomena decently well with gaussian distributions. Should we report this as "Normal distribution makes yet another groundbreaking discovery!"? Probably this wouldn't have the same impact, because people aren't being sold sci-fi stories about an anthropomorphized bell curve. People who are using LLMs already think of "AI" as a thinking thing they're talking to, because they have been encouraged to do that by marketing. Attributing these discoveries made by scientists using this method to "AI" in the same way that we attribute answers produced by chatGPT to "AI" is a deliberate and misleading conflation

Yeah so like, this is a cool result, and it uses a transformer architecture. I actually do think that it's fair to say that transformers have proven widely useful, especially in tasks that look like sequence modeling. It's a step change akin to the now-pervasive use of convolutional neural networks that started in the 2010s, and is deeply significant of course. This is also really different from "this is an LLM"

The reason I want to specifically harp on this is because a lot of people are selling this narrative where "AI is becoming superintelligent" or whatever by making an amorphous blob out of a bunch of separate advances that use machine learning techniques. This has been happening for a while, is a great thing for science, and it's clear that machine learning methods are here to stay in science. I'm a machine learning researcher. I've understood, celebrated, and tried to help with this as best I can manage over the last 9 years of my life. And it's been going on for a lot longer than the general public has been in this AI hype wave. The entire modern field of bioinformatics is arguably built on the backbone of machine learning, and has been since before I went to grad school.

This is really different from "We fed everything into a language model and now it's superintelligent and is making scientific advances all by itself" or even "scientists just ask chatGPT shit and it figures it out for them". The breathless tech press really makes it sound like anything that happens in AI research, which increasingly includes the entire usage of ML toolkits in the sciences (Which is pervasive, and expectedly so! ML is an extension of statistics and statistics has been the basis of science for like a century) is just some amorphous force called "AI" that's suddenly gained this aggregate body of competency. Imagine if we anthropomorphized statistics that way. Or Math for that matter. This kind of narrative gives me the overall impression that this is not being talked about honestly, and it's clear that this is profitable to do. I don't have to use charged words like "con" or "fraud" to think this deceptive framing is not a great thing

Okay so because of the ambiguity of the other reply I'm just gonna say, I don't think we should be surprised that someone is trying to use LLMs to do basically anything. That's basically what prints funding money right now, so long as you're the kind of company or guy the VCs or whoever will believe in. The signal here is "does it do something to appreciably advance the state of the art over previous methods"?

Huh. I mean it makes sense to train end-to-end on all the interrelated tasks involved in driving but putting a whole-ass language model in the middle of that seems like a stunt. I wonder if it does better than like, any random transformer not trained on language first? Still, I hadn't heard that so I guess I was wrong about that one

I swear no matter how many times people say it people will still conflate all ML with LLMs. No, chatGPT is not driving advances in self-driving or weather prediction

Yea TBH you're not gonna sell me on a president past Kennedy looking out for the US taxpayer or the economy in aggregate. Neoliberalism is one of the worst kinds of parasitic meme

Here are some easy ones off the cuff: Kill a massive amount of wasteful private contracts. Remove the entire means-testing bureaucracy from every social program. Put more funding into stuff that actually gets ROI, like the IRS, the FTC, and the CFPB. Don't shoot ourselves in the foot by killing massively beneficial soft-power investments like USAID. That one in particular is essentially handing the mantle of cultural and economic hegemony to China on a silver platter. Maybe you don't believe America should be a hegemonic empire from a moral perspective, but it's hard to argue that it's not a beneficial position

The agenda here is just very clearly not efficiency, but cronyism, enabling autocratic control, and culture war vendettas. It really takes willful ignorance to think this is going to be good for the country overall. It will mostly be good for Elon Musk and his buddies, and maybe only in the short term

More broadly, I'd like to see the US return to a state of regulating finance and enforcing anti-trust in a sane manner. The boom-bust cycle and total crackdown on labor and consumer protection has turned the domestic economy into a casino that impoverishes the vast majority of its citizens, and the effect of bailing out this corrupt financier class every time they crash massive sectors of the economy has been the destruction of the real economy, an enduring public distrust of the government, and a class of insulated plutocrats who have been shielded from the consequences of their mistakes, believe themselves untouchable, and have now orchestrated a total takeover of the political process. We had economic stability before, and it was looted by the greed of our financial and industrial sectors.

Of course this all seems like a pipe dream. Those guys won. They have assumed control and they don't have to give a fuck what people on some forum say about them

Sure, but also hypocrisy only makes sense when applied to a single person or an actual organization that actually collectively takes positions. "Progressives" is an amorphous category that can seemingly include anyone we want to declare "woke" at any given time, and seems to span the political spectrum from even the most right-leaning democrat politicians to random tankie teenagers on social media. Such an incoherent group doesn't have collective positions or collective responsibility for each other's positions, and I think it's important to point this out because it's a very common rhetorical gambit

Wait so you're saying that anyone who criticizes Musk is in this bucket of "progressive" and therefore is accountable for the supposed hypocrisy of all "progressives"?

Actually this argument makes no sense. I know that it's commonly repeated in propaganda, but that doesn't mean it isn't stupid on its face. You can't just throw people into a collective arbitrarily and then hold them responsible for everyone else you'd call that's actions

Actually freedom of speech is a principle, which Musk and his ilk claim to care about, despite using their power to censor people through either direct control of information streams or threats. The first amendment to the US constitution specifically seeks to reify this principle in laws pertaining only to the government's actions, but freedom of speech as a principle can be supported or violated by anyone with power over other people. The people claiming to be "free speech absolutists" are hypocritical even if they didn't also work for the government and enact these vendettas against speech they don't like on a governmental level (which is also happening)

This whole framing is nonsense. The US is the reserve currency of the world, and its influence in both economic and soft power has had massive ROI for the entire economy for decades. There is no other government that plays this role in the global economy, and it's a ridiculously advantageous one. The fact that a balance sheet shows the sum of the investments to achieve and maintain that as a deficit is not a reason to freak out, this is a drastic misunderstanding of how nation-scale economics work that comes from people who are only familiar with household and business budgets

But if we were concerned about the budget, contractors are drastically more expensive than federal agencies and are much more dubious in their value proposition for the state or the public. It's just that the people involved made their fortunes on cushy government contracts and corporate welfare, so they won't go after it. The whole "fiscal conservatism" concept has always been a massive grift and this is the most blatant and obvious it's ever been

And you're absolutely right in saying that the Democrats are on the same grift, as their strategy has shifted to triangulation in the last 40 years, though the economic shocks they tend to produce over this insanity are generally smaller. The fact is, neoliberal economics has failed to produce widespread domestic prosperity, was likely an attempt to loot modern economies for an oligarch class from the beginning, and people keep accepting doubling down on these policies because they've been systematically misinformed about economic cause and effect through trite little analogies

Not even remotely. Same line since at least Reagan, never been honest, never will be. It doesn't even make sense. It's relying on laypeople not understanding the economics at play. Hell, going after stuff like the CFPB that are deeply revenue-positive and not things like DoD contractors makes this claim ridiculous on its face

Also, even if it was believable, that's not an upside, it's a claim of ideological justification. An upside needs to be an action they've actually done that has actual tangible benefits

I'm not even really seeing the short term gains here. It seems like despite making grandiose claims, the administration and the "broligarchs" are relying on cheap political wins ("owning the libs/woke") to placate their base in place of even pretending to accomplish anything that helps their constituents. This honestly surprised me. I'm not a fan of the MAGA set but I genuinely expected there to be some upsides here and not just an immediate private equity style looting frenzy

Baking in the assumption that cognition is equivalent to computation will tautologically lead you to this result, but this assumption itself is unjustified. Of course if you start with the premise that the brain is a computer you will come to the conclusion that the brain is a computer. You haven't justified the most important part of your argument, so I have no reason to take it seriously

Weird how when the rapacious whims of capitalists destroy things people value the economists posit that "those things weren't viable in the market, we have to be realistic" but when business models that capital finds valuable are under threat from technological commodification the line is "disaster, death spiral"

I have to say, my days of not taking economics seriously as a science are certainly coming to a middle

I think a lot of this is because the economic pressure is weak right now both on the side of labor and consumers, due to decades of severe upward wealth transfer. A lot of these companies are not improving or even maintaining their productivity or quality of service, and while there are probably some productivity gains for engineers, I suspect based on what I'm seeing that this is going to burn a lot of people out, as there is significant social pressure both from peers and employers to exaggerate this somewhat. People can have too heavy a workload for a decent amount of time before breaking.

There's just no countervailing force to make these decisions that immediately painful for them. Sectors are monopolized, people are tired and desperate, tech workers are in a basically unprecedented bout of instability.

The situation is super dark from a lot of angles, but I don't think it's really "the overwhelming usefulness of AI" that's to blame here. As far as I can tell, the biggest thing these technologies are doing is providing a cover story for private-equity-style guttings of various knowledge work verticals for short-term profit, which was kind of inevitable given that's been happening across the board in the larger economy, it's just another pretense that works for different verticals.

There are cases where LLMs seem really genuinely useful (Mostly ones that are for and by SWEs, like generating documentation or smoothing some ramp processes in learning new libraries or languages) and those don't seem to be "transformative" at scale yet, unless we count "transforming" many products into buggier products that are more brittle and frustrating to interact with

Anthropic pivoting to doing economics? Seems about as legit as most economics

Probably an overall smart move, since claiming to be doing economics sometimes leads to being positioned to make policy favorable to oneself

I love this approach. I already like doing this in systems above a certain complexity made with programming languages too. I think the functional programming community calls this "holes"?

Interface "not implemented" errors follow a similar logic but I honestly think the value of writing something trivial that gives a meaningless but valid output for each of a bunch of pieces that depend on each other goes a long way toward expediting the process of building those pieces. It makes it much more likely you can test and build one thing at a time and not need to write a bunch of separate parts before you can test any of them

Having this type-wise validity matter in scripting contexts is sometimes harder, as in the use case described in the article, as a lot of the effects of command lines are going to be "side effects" from a functional perspective, but it being sequential makes this a lot less impactful and the waiting prompts make it so you can still preserve order of tasks at the low cost of needing manual steps you'd be doing without the script anyway

Scaffolds are incomplete but they're still, fundamentally, useful

My days of not trusting a single thing microsoft controls are certainly coming to a middle

Yea I'll give you that. But many people seem to have the argument you've made - which is dubious on its own terms, by the way, as we don't really have a complete picture of human learning and the assumption that it simply follows the mechanisms we understand from machine learning is not a null hypothesis that doesn't demand justification - loaded up for these conversations, and it needs to be addressed wherever possible that the ontological question is not what matters here

This argument, by now a common refrain from defenders of companies like OpenAI, misses the entire putative point of intellectual property, and the point of law in general. It is a distraction of a common sort - an attempt to reframe a moral and legal question into an abstract ontological one

The question of whether the mechanism of learning in a human brain and that in an artificial neural network is similar is a philosophical and perhaps technical one that is interesting, but not relevant to why intellectual property law was conceived: To economically incentivize human citizens to spend their time producing creative works. I don't actually think property law is a good way to do this. Nonetheless the question when massive capital investments are used to scrape artists' work in order to undercut their ability to make a living from that work for the benefit of private corporations that do not have their consent to do this is whether this should violate this artificial notion of intellectual property that we have constructed for this purpose, and in that sense, it's fairly obvious that the answer is yes

The investment thrown at OpenAI seems deeply inflated for how much meaningful progress they're able to make with it

I think it's clear that innovative breakthroughs in bleeding-edge research are not just a matter of blindly hurling more money at a company to build unprecedentedly expensive datacenters

But also, even if that was a way to do it, I don't think we should be wielding the law to enable privately-held companies to be at the forefront of research, especially in such a grossly inconsistent manner

I'm strictly speaking never going to think of model distillation as "stealing." It goes against the spirit of scientific research, and besides every tech company has lost my permission to define what I think of as theft forever

Certainly most of the ones with green names for whom a reply to me criticizing a big tech company or disavowing them in aggregate is among their first ten must be

Unless we start chopping these tech companies down there's not much hope for the public internet. They now have an incentive to crawl anything they can and have vastly more resources than even most governments. Most resources I need to host in a way that's internet facing are behind keyauth and I'm not sure I see a way around doing that for at least a while