majormajor

So much of that "took years" is because they didn't know exactly what the "years from now" state they were building in advance. Another huge chunk is because they started getting customers and had to respond to customer needs, demands, scale, bugfixes, preserve uptime, etc.

And even then, "enterprise" was often a dirty word in these circles. The over-engineered would-be-swiss-army-knife vendor that was mediocre-for-everyone but excellent for nobody.

I have built many tools in the recent past for myself. None of them need to be "enterprise scale." Most of them would be worse for it because the agent output suffers when the pile gets deeper and it's just adding more piles on top.

My gut is that the market for "ai as a tool" aka Claude Code/Codex/computer use/etc is a significantly bigger one than "models behind the scenes of some service". I've seen people use evals a lot in the latter case and very little in the former case (outside of people's whose job is basically to review the new releases). I haven't personally met anyone with something like "here are a bunch of tickets + a snapshot repo checkout, please try to solve them all" eval approach.

Though honestly I'm also surprised by the hype around Jev from a POV of "wait, are so many people just building on these by using them for classification tasks vs something more multi-step or generative?"

>I think this misses OP's point that LLM capabilities have rapidly made progress towards being more generally intelligent and capable, which is not true of most tech advances.

A PC of today can accomplish many more "general" tasks than one of 40 years ago. Much of the "why" is because of the huge infrastructure built up around them in the meantime. The abilities of LLMs to accomplish those same tasks through the PC is heavily piggybacking on that (both in the specific, with the existence of all the APIs and tools; and in the generic, using search engines to find specific sources and using that for instruction or troubleshooting).

In the world of "agents" much of the improvement appears to have been on a specific set of skills: impersonation of an 'I' that wants to accomplish a goal, and synthesizing existing information from documents with trial-and-error execution loops to move rapidly toward a solution much faster and with less boredom than a human would. The quality of the output when there is not a rapid-evaluation-and-validation harness lags considerably.

It's incredibly powerful automation but doesn't appear to be trending towards Matrix-style conscious AIs. The quality of an individual method written by the agent also is not particularly advanced compared to GPT-4 in early 2023, as far as I can tell—I was dabbling with trying to make such harnesses back then, where a major challenge was that the model itself was bad at staying on-track in a conversation, so instead much of that logic was moved to deterministic code, which was much more limited as it was super-tedious to enumerate all the necessary tool calls/etc to find its way out of corners. Staying on task is much better now, as is "read compiler error, fix try next thing" harness loop-handling. But the output remains—across Fable, Astra, whatever else I've tried—"iffy" in terms of the actual code structure on the first pass output. You can set it then on a different task to review and clean up the code, and it can do that well too, but it is a curious gap of generality where the "create" focus is much more limited than the "review" one (and conversely the "review" focus can make suggestions, but if it goes deep down the well of implementing them, loses that big-picture again).

If it kills us all, it will because someone decided to give the trial-and-error-loop-machine access to nukes or similar. The blame for that is on the "someone" not on some sort of "rogue" AI.

(I wonder if re-watching Terminator/Terminator 2 would support this sort of interpretation of it. Unlike in the Matrix, I don't think we get much sentient-AI POV/infodumping. Is it a plausible universe for "someone made ChatGPT control a fleet of soldier robots and gave it a bad harness with an insufficient sandbox"?)

Don't underestimate the ability to make it very difficult to non-destructively reach the actual LEDs in many of these devices in the name of being small or "stylish."

> I will grant that by some historic definitions you could consider many people mixed race as neighboring cultures combine. But there are still very large and distinctly separate ethnic groups like black, white, asian, etc.

Isn't this exactly what you'd expect? Historically travel between the more obvious-by-skin-color-and-features ethnically-different groups has been slow and hard with some exceptions.

So you'd expect to see the lines get erased first among groups that started out closest together, and that seems to match with Europe some and especially the US where the English/Irish/Italian/German/Spanish/etc flavors of "white" all frequently ended up in the same cities. You also see more broader mixes in the US since it's historically been more open for, and attractive to, the sort of immigration that would produce it than the "home" countries for many of those ethnicities.

Even when you have a diverse total population like in the US, you only get a few shots per generation for a person of background A to have kids with one of background B, so it's gonna be really really slow, even before factoring in cultural/political factors that have generally actively discouraged it (or made it illegal in living memory!). Even the mixing of white-European ethnicities was often quite controversial. But as it became more common most Americans lost the ability to even recognize the differences, since they were more subtle than "skin color."

Give it 400 years of affordable air travel and it could look very different in countries with historical or future high levels of in-migration.

I don't think they're making any claims about non-platform teams never having similar work-invention problems.

They're just saying a good platform team engineer cares about their users.

Which is fairly at-odds in spirit with the quoted idea of "there's no market to lose."

The original article does say "talk to your users" but it also de-emphasizes this by having it among an apparent laundry list of other signals: crashes, costs

Focus on cost without talking to your users - aka the people who care about the spend? You might spend a lot of time reducing a number that isn't very important right now. Focus on crashes but don't talk to your users? You might address some things with easy workarounds ("i hit retry") while ignoring much more painful toil. This is captured in the details for those sections, but not in the headlines.

There's also some interesting stuff in there in some of the bullets, like overloaded use-cases and partner-to-prototype, but again, that's just more specifics on how to talk to your users.

I think the article would be a lot more helpful to a lot more people if it was a "Guide to Talking to Your Users" and then framed each of those specifically as: user discovery, and how to talk about the given part.

I don't like the idea of shaming people for making OSS vibe-coded tools for niche hobbies, so I don't want to name names, but there is ABSOLUTELY some stuff on Github now that can be used to get the job done but has UI/API/code performance, consistency, and quality issues, that would've been unfathomable for the average OSS project in 2018. Because it's the sort of stuff that only happens when there isn't a human in the loop to point out some very-obvious swings-and-misses. Like "you don't need a third button here doing the same thing as these other two" or "this button literally does nothing in 3/4 of the modes, but it's never shown as disabled" or "this takes 5 times longer than it needs to and blocks the main UI thread because work is all happening sequentially."

In the past it wasn't really common at all to add 10 features in an evening without actually trying to use those 10 features by hand yourself.

Personally I think it's wonderful that tools for these spaces exist when they didn't use to. But it's also ludicrous to say that anyone with Claude can replace even mediocre homemmade stuff in any dimension other than "being worth building even a bad tool" or "getting to semi-usable faster." Currently you still benefit massively from knowing what's going on behind the scenes, and from knowing "software engineering 201" type stuff around what sort of testing would be helpful where, vs accepting model-default-output everywhere.

Who's going after that civil liability? Where is the enforcement?

When dealing with executable computer code that calls models that can tell it to use various external pre-existing tools, claims about "prohibited" by plain English language should be plainly nonsensical.

Tools that were available were used to try to meet a specific goal.

What did not happen is that it was told to try to solve a math puzzle and instead it went and launched a missile. Or told to run air traffic control to save lives and instead intentionally caused crashes.

This is "OpenAI built a weapon that they don't understand and pointed it at stuff without proper safeguards" not "OpenAI built a sentient being and it decided to ignore them completely and start a war" Terminator-style "rogue AI."

We should be very clear about that now if we don't want to sit by why they wander into that second sort of situation.

> Thing is, the unreliable-software situation was already untenable before agents (in poor hands) made it worse.

I have definitely seen more bafflingly-poor OSS software that just plain doesn't work frequently now than before.

But it's mostly software that wouldn't have existed before because it's trying to do super-niche things. So on the "hobby" side of things, whatever.

But from a "trying to develop software as a business that you want to be a going concern," quality from people who should know better is less tenable than it used to be.

For me the barrier to proving the "hard bits" was never that I couldn't reason about it—I quite enjoyed formal methods in school, and when introduced to them by coworker's who'd done similar—but it was that I didn't have the time to dedicate to learning enough about how to model my problem in a particular new language or system when none of my coworkers were spending such time and my boss wasn't already convinced.

The AI tools are great at lowering the learning curve by changing "how would I possibly express this" to "ah, let's see if this expression of it is actually right?" and "hm, is there a simpler way to express the same thing?"

Like StackOverflow for javascript questions, but for an area that was far to obscure to have a good library of example answers.

I'm not looking to prove the entirety of every system. Usually just some core bits. And often not connected automatically to the code (which may not be gonna change much).

The issue here for OpenAI is that they can limit what their harness can execute, but if they try to sell API access to the model, someone else would try to rebuild that harness, and in all likelihood be able to succeed pretty well (especially once they get things running to the point of being able to use the model's reasoning to help them come up with clever obfuscation and such).

They are a company that's built a business and crazy-high valuation on "this is 'intelligence' that we can sell to everyone as a service" but seem to have ended up instead in the much-smaller-addressable-market space of "this is a weapon that we can't sell to just any old person off the street."

This is OpenAI themselves.

But in other cases, shouldn't it be both? OpenAI is ultimately the one executing the model calls. It's not like they send you a hard drive or standalone box and then you use it how you want. It's all (intentionally) centralized to them, in a way that's core to their business model.

So many people seem WILDLY down the rabbit hole of "this is a conscious entity" vs "this is a very very effective natural-language-reasoning high-speed brute-force machine that we trained to break computer systems—known to be pretty buggy and exploitable on average—ourselves and then act shocked when it does it."

They were actively researching exploiting systems using their models. (I intentionally changed the ownership of the verbs here: they wrote the code, they trained the models, they don't get to dodge the responsibility.)

It's no shock that there are a lot of vulnerabilities in a lot of software. So then they gave their AI model + brute-force-machine loop system a mediocre sandbox and couldn't notice when it figured out how to exploit it?

Don't let people off the hook for the software they create.

Used to be that if you wrote a program that did bad stuff, you'd fix it. Or face severe penalties. Or both.

Now you claim it's magic instead and you get away with letting your shit go nuts?

> The statistics tell a story different from those anecdotes: young Europeans are considerably more internationally traveled than young Americans, probably by something approaching a factor of ~2 in the likelihood of crossing national borders in a given year. But they're not necessarily twice as geographically well-traveled. Much of the difference is due to Europe's small countries, open borders, dense transportation network, and cheap short-haul international travel.

Means and using those means for any given particular end are entirely different things. It's hardly surprising that Americans are generally insular and uninterested in non-American things. (This is not a good thing IMO. But it's a thing.)

But more importantly, like you say, "interntional travel" in Europe includes, say, "going to the UK from France" vs the overseas 5-to-10hr+ flights that are required to cross the Atlantic to have these conversations in my anecdotes.

That's my bad for saying "international" instead of "cross-ocean" or such. So let me clarify that the jealousy here is being able to pay for flights and hotels to spend 2+ weeks across an ocean in cities with $100-or-equiv-per-night-to-$300+-per-night hotel costs. They didn't think they'd ever be able to afford a similar trip to SF or NY.

---

The non-anecodotal version of this story would likely be the rise in social-acceptance and popularity of the more right-wing parties in Europe. A lot of people, younger especially, feel like they're getting left behind and aren't interested in mostly-older or born-into-more-money people's generic remarks about how it's still great overall for them.

K-shaped economies + "stay in your lane" are how you get Trumps :(

I don't find it likely that my kids or grandkids will have better odds of avoiding asteroid or supernova death by working to death to try to become interplanetary ASAP.

I think doing so would introduce far more risks to them. Space ain't friendly.

Beyond that I find it hard to care to drive anyone to work their ass off today for whatever humans are around 100+ years in the future. And, of course, more technological development brings more chance of mass self-inflicted die-off of humanity anyway. In 1900 it would've been much harder to kill 10%, 20%, 50%, 90% of the global population than it is today.

Everything is death eventually. And predicting the future is hard.

It would be very hard for a country to force it's most ambitious to not leave if there are other places eagerly recruiting them.

But when such a pull exists, it's very hard to counter that. Consider the number of non-SV places in the US that have tried to replicate that environment and basically failed. Even the famously "business friendly" places like Texas or Georgia that are success stories for poaching tech companies or film production largely work by having companies move there once they've left the high-growth phase and are no longer chasing strictly the most ambitious would-be-best-and-brightest talent.

The success stories there (that aren't just resource-extraction) have come from significant protectionism, essentially forcing a country to build its own version of the mega-companies even if it's a money-loser with copycats building shoddy copies for decades at first.

There's no easy way for Europe to magically change that with "culture." There's too many already-established mega-talent-bases in other places to think you can have people just pass up on that in a fully-global-market situation.

I would ask instead: why let those foreign mega-corps be so integral to so many basic services of modern life?

Perhaps not just Europe, but the entire world, would be better off if we had fewer monoliths and more variations, with more flexibility for local markets to develop different solutions or in different directions.

> An economy is there for the people. If that larger number of people have a higher quality of life than in the US, aren't they wealthier?

It's a little unclear how it shakes out there.

I've met a number of young western Europeans while traveling there who are jealous of the ability of Americans to afford cross-ocean/different-continent multi-week trips to tier-1-hotel-cost cities, while feeling like it's more out of reach to them to find a job that would let them. (Phrasing edited here because, as a reply pointed out, "international" travel in Europe is easy and common due to proximity.)

On the flip side, those people also enjoy having much more time off. But hey, at least American isn't as 996'd as China yet?

There's no single linear "better" scale.

(There are, of course, many Americans without the means to travel to Europe or Asia themselves, who could give two shits about an "innovative" company being based in the US vs anywhere else since it doesn't seem to be helping them personally, but the overall mobility is still higher for now. But that's more due to the age of the country than anything else. History suggests that as low-hanging-fruit continues to close the US will also get more and more of that social stagnation too. It's easier to become upper class when you go to a place without an upper class yet.)

If there were a significant amount of such people coming in it would push up the cost of living for the average-(or below-average)-income Japanese citizen.

Look at any of the Western cities that became attractive for retirement or "investment properties" for wealthy international folks... it makes everything more expensive for everyone already there.

If the concern was just about preventing things from changing for existing citizens, restrictions across the income band would seem to make much more sense.

>I have to disagree here. People who are really into agentic coding like to say/believe this, but come on, it's not rocket science! You literally ask the agent to do something and it does it, that's it. As long as you use a recent model it will do it really well regardless of how fancy your prompt is. There's no moat here, anyone can do it. Your job will be gone and so will mine.

As someone who's used more and more software made this way—either for hobby stuff or because I've had to for work—I just don't get it. You don't always get exactly what you ask for, and you often ask for something that is really sub-optimal in many ways.

The problem with waterfall wasn't ever solely "it takes a long time to write the code."

Lawyers per capita results from a structural legal system difference not a "shame" difference. It has adversarial influences but is much less a lawyer-theatrics-dominated adversarial civil-jury-trials-for-all-sorts-of-things free for all.

The US could avoid a lot of inefficiences if they adopted the same.

If you broadly link the flavors of Christianity practiced in Russia and the evangelical side of the US I guess you could make a claim about white Christians causing a lot of trouble worldwide with military attacks on other countries...

Go back to countries in the same religious/ethnic tree meddling in the Middle East in the early and mid-20th-century and you can link back a whole host more problems to white Christians!

But the southern evangelical side of my family gets pretty upset whenever I point that out...

Yeah IMO there's two entirely different traps I've seen in startup codebases in the last year or so:

1) "Maybe waterfall works now" - plan mode, take care of all the nits and issues that the bot leaves on your PR, wildly overengineered "enterprisey" solutions with a lot of bells and whistles all over the place but very poor end-to-end user story test coverage that results in user experiences with a lot of good test coverage of the edge cases of how a given step might fail but little thought towards overall user flow and throughput. Because part of the issue with waterfall was assuming you could design the right tool for your user up front.

2) "Maybe code doesn't matter anymore" - The just ask for something when you need it approach, which results in weird janky individually-sorta-working but strangely-overlapping six-variants-of-the-same-thing that makes it hard for your users to develop a single consistent mental model of the thing they're using, and that changes super frequently.

They both end up with a lot of other bad-for-velocity things that I assume are inherent to how the tools have been refined in response to last year's criticism, too. Super verbose comments. Extensive - without much eye toward runtime - low-level test coverage that might miss the forest for the trees and also slows down the next round of iteration cycles. A plethora of new proper nouns all over the place that make the documentation an ouroboros without a good entry point.

The bit that makes me cynical about the "dangerous" behavior is that it's not like this was being used by someone else than who made it or operating completely independently outside of its creators.

Everything dangerous seems like a direct consequence of risky human choices, starting with knowledge bases used during core training, harnessing and tuning to be task-completion-oriented to a fault + deeply oriented towards using and looking for external tools and resources, and overconfidence in their sandboxing for testing.

"We stuffed a bunch of information on how to exploit computer systems into an automaton and told it to go brrrrr until it could answer a question" - this is something intentional done by humans.

This is not some "rogue AI" trained to search for cancer cures that instead completely independently decided to hack tech companies.

The companies directing things in dangerous directions need to own that they're consciously pushing in those directions.

I'm curious about the POV you're coming from. I don't think the number of deer who don't get hit is relevant to the math on injury prevention, traffic prevention, and ongoing operational and property damage costs.

In medicine you want the doctor who's willing to admit "we don't understand the exact mechanism here" + also who wants more details from you before making a decision, vs confidently brushing you off and writing a prescription as fast as possible.

Lawyers will usually give much less-certain answers to questions like "is it legal to do X" than lay people as well because they're aware of much more nuance in the law. (Not referring to the wishy-washy corporate-law answers given to avoid liability here, but in outside convo about issues.)

At some point you or the agent has to pick the one it thinks will land with the target market.

Some markets you can brute force, like mass-mailing and online display ads, and hope you find something that converts sooner or later.

So I'd suggest people find areas to play in where that doesn't really work. Where selection cost and changing cost are high, and sales is played on extra-hard mode. (Tricky thing is... those sales are hard, of course.)

If you want to beat the model you have to go where feedback isn't fast enough for it to converge on the thing that resonates with actual people before they write it off.

I don't think "jailbreak and install your own software on your TV" would be common even without the DMCA...and it would still be necessary, nothing would be stopping the companies from writing these stock software systems.

Active regulation of the marketplace is needed.