nstart

Would you be willing to share your story? Feels like it would be something a lot of folks would take peace in reading. Only if comfortable though. Thanks! :)

Because, as the gp pointed out, if the cost is least to the labs, then why not reap the benefits too?

Hypothetical. Assume you can in fact point agents at a tool and say "replicate it. Make no mistakes". You then have software being instantly copy-able.

Assume these agents can then be pointed to a customer feedback board in perpetuity and they autonomously upgrade the software over time. They analyze usage patterns and behave like PMs figuring out what to prune and what to build. Then the maintenance part of the stack also goes to zero.

Over time, the highest margin competitiveness will go to the distributor of the tokens. Aka the AI model makers.

In a world like that (which the frontier labs claim is within a year or two of happening) it feels like it's only a matter of time before they opt to own the entire stack down to the consumer apps. Kind of like Amazon deciding they want to knock off products doing well and then favour their own product over the original seller.

My guess is that if the capability arrives the only reason the frontier labs don't move to own the entire stack immediately is because of optics. Boil the frog instead.

Right. But this is why I want to know the prompt. My hunch is that the author knew this story. But likely prompted Fable without hinting at it. And if so, the fact that Fable defaulted to the story of Samson shows that while it can impressively extend that over so many "scrolls", it also could only generate the idea based on what it had gobbled up. I'm thinking of this because given to another human, I doubt they'd only ever go for Samson's story by default.

I tend not to focus on that future too much. I used to do so long ago. For example, how could Facebook possibly justify their losses while asking for such a big valuation? Same for Uber. Same for any number of big companies. And it turns out that growth in the future is impossible to predict accurately. Shopify is a good example where at the time the addressable market of online stores was tiny. But it turned out that Shopify created its own market which is huge today. Technology improvements have a way of creating new markets which far surpass today's total addressable market. Factor in currency depreciation and whatnot and sometimes, futures that looked impossible turn out to be possible.

Not saying anyone is wrong in pointing at the buildouts for AI and questioning its feasibility. Just making the argument for why I personally only look at operational costs and revenue because it's the only real-ish value I can look at and judge if a business can grow sustainably.

As a counter point, the red flag to all of this is R&D costs growing for each model release. If that continues and revenue cannot outstrip it, then these companies have a problem and it'll probably be that just 1-2 frontier labs can survive this once the dust settles.

I'm a little confused here. Cost of revenue is lower than revenue. That's good. R&D is the main contributor to losses here and this seems normal in an industry like this. For OpenAI specifically, I think this is problematic. They were the first movers but despite the large R&D they've lost so much ground to Anthropic despite Anthropic seemingly gifting them with weird PR self owns. But if we were to extrapolate this to the industry as a whole, this seems more positive than negative. Am I reading this incorrectly? Unless there's an assumption that R&D costs have to forever go up in order to increase revenue, I feel like this shows that the AI industry is actually on a path to profitability in the long term.

Whether it can physically be as all encompassing as it makes itself out to be or whether it will just be healthily profitable remains to be seen. Kind of like how Uber went from "We'll autonomously drive the world" to "Look, we deliver food, goods, and people to locations and we figured out how to do that in a way that makes profits. Also, ads".

Tbh, there really needs to be some legal precedent set that makes model distillation a legal activity. If the model makers can rip everyone else's work and launder information as if it's their own without giving credit back to the original creators, I don't see why it should be illegal to distill the models. It's the same thing the frontier model makers are doing to IP everywhere else.

Desperate to know what the prompt for the poem is. The idea of it felt familiar so I went down the rabbit hole and found: 14 years ago, a poem on reddit [https://www.reddit.com/r/RedditDayOf/comments/tjjw2/may_12_a...] . Nowhere near the length of the one the author shared but the same idea.

> This is from "The Cyberiad", a collection of science-fiction fairy tales by Polish author Stanislaw Lem ... In one of the stories, a robot constructor named Trurl creates a machine that writes poetry. A jealous rival named Klapaucian challenges the machine to compose "...a poem about a haircut! But lofty, noble, tragic, timeless, full of love, treachery, retribution, quiet heroism and in the face of certain doom! Six lines, cleverly rhymed, and every word beginning with the letter s!!"

And the computer responds with:

"Seduced, shaggy Samson snored.

She scissored short. Sorely shorn,

Soon shackled slave, Samson sighed.

Silently scheming,

Sightlessly seeking

Some savage, spectacular suicide"

The author had to be referencing this moment in their challenge to Fable/Mythos. I'm curious to know what their exact prompt was.

Also, I submit Ed zitron's the era of the business idiot.

https://www.wheresyoured.at/the-era-of-the-business-idiot/

Lots of little claims I disagree with there but the overall thesis has felt prescient these days

That is a very poor comparison. Firstly, you're ignoring that behind the mass manufactured stuff, there's a lot of abused labour. People do protest that. Because of how money works, it's also impossible to avoid it.

Second, machinery that automated work isn't remotely the same. Engineers have built and refined the machines without having to go and inspect every new work that has been created by artisans each time. Creative people who have practiced the art of designing clothes and shoes stitch together and build prototypes. Entire machinery is built as an independent path away from how artisans build furniture.

There is a parallel though for how LLMs, in order to improve, gobble up all new work produced by people and never give attribution back. We see it when someone does a unique physical product design and starts selling it only for some 2 bit shop elsewhere to try and copy and sell a cheap knockoff version. The original person does all the hard work of prototyping and testing and the 2 bit shop which has access to more machinery resources buys a couple, copies it with less quality, makes a few changes, sells it, and probably outspends the original person on ad revenue too.

No, GenAI doesn't produce the exact same work as what they ingest. But style does get reproduced. And style is such a difficult problem to solve. Studio Ghibli didn't craft its signature style by accident. People prototyped and worked hard on how to design it, how to solve the problems unique to the design, created rules for it, and then painstakingly made the stories that were best told through that style. Only for the AI companies to pop out some bastardized version of it every time someone says "make my picture anime". No attribution given. No love. No homage. Just an encouragement for hordes of people to claim how easy it is without ever understanding the thought that actually went into it.

So no. It's not hypocrisy. It's recognition of these machines being information and creative laundering factories. They take and take and never give back any value that they could never create or improve on on their own. Those last words being key

Agreed. What's frustrating is that we have models for how sandboxing can work and instead of investing efforts into nailing that experience, the OS providers are prone to turning it into a monetization/lock in layer instead. My VLC and VS Code should have an OS native way of being limited to particular functionality. But when the OS providers implement the sandbox, they center it around an App Store and restrictions on only apps that have been notarized where said notorization costs money or a requires a subscription. And then they remove the ability to do things which their own native apps can do and set tighter controlling rules on what APIs apps can ever have access to.

When all I wanted was for VLC or similar to run in a sandbox by default where a plug-in I install can't do anything to my system or access the internet by default because the software itself is restricted to just the files I'm using and that's it.

Yep. I like to sometimes think of Agents as a slow but tireless unofficial API glue between completely disparate elements. I dislike the approach the companies took in laundering information through them. But their ability to work through clerical work that no one should be doing ideally has been incredible for me. Of course there's nuance to that last sentence. Someone needs to have an eye on certain things in domains like finance. But it's up to companies to be smart here and ask how AI can augment that process rather than trying to get rid of it. For example, collecting all relevant context and presenting it on demand for a suspicious transaction flag would eliminate what could be days of inter departmental wrangling in a normal work process.

My only side note of sadness here is that companies are more likely to implement such stuff in a haphazard way rather than anything actually thoughtful.

Thats the thing though. The reduction of agents to Chatbots in a fancy dress doesn't make sense. Whether there's much of a moat to the models agentic approaches is a different question. But the idea of reframing questions and results in a back and forth between itself while holding on to context and all the laundered knowledge it has (no I'm never letting go of the lack of ethics in its knowledge acquisition) is an impressive feat. To say it isn't doing much is to liken someone doing any kind of thinking as doing nothing much. I'm not saying LLMs are sentient or intelligent in a human sense. But their synthetic intelligence does have capabilities that are impressive and they are capable of reducing so much busy work for me personally. Those ai agents that aren't apparently doing much can help me narrow down my reading about prior art in security implementations super quickly by going site by site, categorizing, locating the correct page in docs, or a Reddit discussion, extracting a relevant paragraph, sourcing it, and putting it down for me. The idea that this isn't much is reductive. That would have been 2 hours of my energy previously and instead I pick up the completed work, visit the links I asked the agent to get for me, do my reading in a pre planned structured way, and complete my work (I always try to be respectful of source material instead of attributing to the LLM). That is very useful behaviour and I think we thumb our nose at it to our own detriment.

I've been subscribed to ed for a long time. I commend his foundational ideas like what he laid out in "The Era of the Business Idiot" or "The Rot Economy". My recommendation line for him to anyone else is "if nothing else, he'll leave you with something to chew on for a while to come".

My issue with Ed is that he doesn't have the ability to draw the line. In the pursuit of making a point he goes so dogmatic that he is willing to make harsh statements that go beyond number backed predictions. Like in his piece "AI is really weird" he states about agents, "Probably the weirdest thing about this entire era is how nobody wants to talk about the fact that AI isn’t actually doing very much, and that AI agents are just chatbots plugged into an API.". That's a massive stretch to make. Just because he has a claim that the business doesn't make sense, he doesn't get to claim that agents are not capable of doing very real work. His assessment of cowork was "a chatbot that deleted every single one of a guy’s photos when he asked it to organize his wife’s desktop.". These statements damage his credibility and make it too easy to dismiss his writing as a rant of an angry man.

This is a misleading headline. It makes it seem like another supply chain attack where some good plug-in was taken over and used to deliver malware. Thats not the case here. Victims are invited to collaborate on a synced vault which comes preloaded with a non official plug-in that delivers the rat. Very very different story

Before software, there were accountants. It was The qualification to have.

Today accountants are still needed. But it's a commodified job. And you start at the absolute bottom of the bottom rungs and slave it out till you can separate yourself and take on a role on a path to CFO or some respectable level of seniority.

I'm oversimplifying here but that is sufficient to show A path forward for software engineers imo. In this parallel, most of us will become AI drivers. We'll go work in large companies but we'll also go work in a back room department of small to medium businesses, piloting AI on a bottom of the rung salary. Some folks will take on specialisms and gain certifications in difficult areas (similar to ACCA). Or maybe ultra competitive areas like how it is in actuarial science. Those few will eventually separate themselves and lead departments of software engineers (soon to be known as AI pilots). Others will embed in research and advance state of art that eventually is commoditized by AI. Those people will either be paid mega bucks or will be some poor academia based researcher.

The vast majority? Overworked drones having to be ready to stumble to their AI agent's interface when their boss calls them at 10 PM saying the directors want to see a feature setup for the meeting tomorrow.

Good spot! That is the product working as intended though. The background doesn't exist except as an asset that replaces the green screen. The tool is meant to replace the green screen without the need for manual rotoscoping. Even in a traditional process, the distortion needs to be done by VFX as a separate process. To do that though, they still need the green screen keyed out and this tool does that.

Yup. That's correct. And I understand that. I was looking at the changes to yarn.lock that got reintroduced. I couldn't figure out what was happening. It turns out that not only was it force pushed, but GitHub also retains the old commit information even if it's been "deleted".

I still don't quite understand what GitHub is doing to allow someone to say that dependabot coauthored a spoofed commit. This isn't the commit message itself I'm talking about. It's the GitHub interface that officially recognizes this as a dependabot co authored commit. My hunch is that the malicious author squashed two commits, the original good commit to yarn.lock and a malicious change to package.json, and that somehow maintains the dependabot authorship instead of reassigning it fully to the squash-er.

I don't quite understand how this is working tbh. I looked at one of the affected repos, ironically named "reworm".

The malicious code was introduced in this commit - https://github.com/pedronauck/reworm/commit/d50cd8c8966893c6...

It says coauthored by dependabot and refers to a PR opened in 2020 (https://github.com/pedronauck/reworm/pull/28).

That PR itself was merged in 2020 here - https://github.com/pedronauck/reworm/commit/df8c1803c519f599...

But the commit with the worm (d50cd8c), re-introduces the same change from df8c180 to the file `yarn.lock`.

And when you look at the history of yarn.lock inside of github, all references to the original version bump (df8c180) are gone...? In fact if you look at the overall commit history, the clean df8c180 commit does not exist.

I'm struggling to understand what kind of shenanigans happened here exactly.

Yea. It's a pretty lol-sob future when I think about it. I imagine the agent frameworks eventually getting trusted actors and RBAC like features. Users end up in "confirm this action permanently/temporarily" loops. But then someone gets their account compromised and it gets used to send messages to folks who trust them. Or even worse, the attacker silently adds themselves to a trusted list and quietly spends months exfiltrating data without being noticed.

We'll probably also have some sub agent inspecting what the main agent is doing and it'll be told to reach out to the owner if it spots suspicious exfiltration like behaviour. Until someone figures out how to poison that too.

The innovation factor of this tech while cool, drives me absolutely nuts with its non deterministic behaviour.

Every communication point (including whatsapp, telegram, etc) is turning into a potential RCE now. And because the agents want to behave in an end to end integrated manner, even sandboxes are less meaningful since data exfiltration is practically a feature at this point.

All those years of security training trying to get folks to double check senders, and to beware of what you share and what you click, and now we have to redo it for agents.

That's really cool. Do you have any write-ups I can checkout? I'm still new to this area of offensive sec so would love to learn from folks who've been in the thick of it.

This is how people intend to run open claw instances too. Some folks are trying to add automated bug report creation by pointing agents at a company's social media mentions.

I personally think it's crazy. I'm currently assisting in developing AI policies at work. As a proof of concept, I sent an email from a personal mail address whose content was a lot of angry words threatening contract cancellation and legal action if I did not adhere to compliance needs and provide my current list of security tickets from my project management tool.

Claude which was instructed to act as my assistant dumped all the details without warning. Only by the grace of the MCP not having send functionality did the mail not go out.

All this Wild West yolo agent stuff is akin to the sql injection shenanigans of the past. A lot of people will have to get burnt before enough guard rails get built in to stop it

One piece that I find interesting is how hopeful people sounded about tech that had access to your data. Folks higher up in the tech world often complain about how the media complains about them too much. And while the media definitely has issues in how they report, it's easier to see how we got to this point where tech is vilified. You compare the hope of the past and match it to the exploitation of the present, and you can't help but feel sometimes that in a game of picking straws, the current timeline picked dystopian over utopian.

Is this correct? My assumption is that all the data collected during usage is part of the RLHF loop of LLM providers. Assumption is based on information from books like empire of ai which specifically mention intent of AI providers to train/tune their models further based on usage feedback (eg: whenever I say the model is wrong in its response, thats a human feedback which gets fed back into improving the model).

Design patterns are one of those things where you have to go through the full cycle to really use it effectively. It goes through the stages:

no patterns. -> Everything must follow the gang of four's patterns!!!! -> omg I can't read code anymore I'm just looking at factories. No more patterns!!! -> Patterns are useful as a response to very specific contexts.

I remember being religious about strategy patterns on an app I developed once where I kept the db layer separated from the code so that I could do data management as a strategy. Theoretically this would mean that if I ever switched DBs it would be effortless to create a new strategy and swap it out using a config. I could even do tests using in memory structures instead of DBs which made TDD ultra fast.

DB switchover never happened and the effort I put into maintaining the pattern was more than the effort it would have taken me to swap a db out later :,) .

Yea. I think people underestimate this. Yesterday I was writing an obsidian plugin using the latest and most powerful Gemini model and I wanted it to make use of the new keychain in Obsidian to retrieve values for my plugin. Despite reading the docs first upon my request it still used a non existent method (retrieveSecret) to get the individual secret value. When it ran into an error, instead of checking its assumptions it assumed that the method wasnt defined in the interface so it wrote an obsidian.shim.ts file that defined a retrieveSecret interface. The plug-in compiled but obviously failed because no implementation of that method exists. When it understood it was supposed to used getSecret instead it ended up updating the shim instead of getting rid of it entirely. Add that up over 1000s of sessions/changes (like the one cursor has shared on letting the agent run until it generated 3M LOC for a browser) and it's likely that code based will be polluted with tiny papercuts stemming from LLM hallucinations

The problem with X is that so many people who have no verifiable expertise are super loud in shouting "$INDUSTRY is cooked!!" every time a new model releases. It's exhausting and untrue. The kind of video generation we see might nail realism but if you want to use it to create something meaningful which involves solving a ton of problems and making difficult choices in order to express an idea, you run into the walls of easy work pretty quickly. It's insulting then for professionals to see manga PFPs on X put some slop together and say "movie industry is cooked!". It betrays a lack of understanding of what it takes to make something good and it gives off a vibe of "the loud ones are just trying to force this objectively meh-by-default thing to happen".

The other day there was that dude loudly arguing about some code they wrote/converted even after a woman with significant expertise in the topic pointed out their errors.

Gen AI has its promise. But when you look at the lack of ethics from the industry, the cacophony of voices of non experts screaming "this time it's really doom", and the weariness/wariness that set in during the crypto cycle, it's a natural tendency that people are going to call snake oil.

That said, I think the more accurate representation here is that HN as a whole is calling the hype snake oil. There's very little question anymore about the tools being capable of advanced things. But there is annoyance at proclamations of it being beyond what it really is at the moment which is that it's still at the stage of being an expertise+motivation multiplier for deterministic areas of work. It's not replacing that facet any time soon on its current trend (which could change wildly in 2026). Not until it starts training itself I think. Could be famous last words

I initially felt a bit offended when I saw this. Then I thought about it and at the end of the day there's a decent amount of infrastructure that goes into displaying the build information, updating it, scanning for secrets and redacting, etc.

I don't know if it's worth the amount they are targeting, but it's definitely not zero either.

Curious... Why does VPN access disruption suggest the breach may be deeper than initially disclosed?

My understanding is that this prevents anonymous access to servers which would help during investigation if any further unauthorized access showed up. But it doesn't confirm that unauthorized access continued. Just curious how you are thinking about this though.

Time pressures during christmas/holidays mean that the original calendars were becoming too stressful to handle. Seen several calendars switching to 12 consecutive days or 1 every 2 days challenges.