daxfohl

Yup, I imagine within a year or two OpenAI and Anthropic will have in-house versions of pretty much everything anyone would ever use, with in-house agents hardening and improving it at a torrid pace that the human/code-review/github cycle could never keep up with. Their models will default to using these in-house versions, and be trained in all the idiosyncrasies of those tools and libraries. Their coding agents will phone home with feedback about usability that in-house agents will act on immediately, operations agents will do the same re bugs and runtime performance. I don't really see a how or why a human-run project on open source would even bother anymore.

People keep talking about this like there's a finite amount of math to be done, and then the party's over. But that's never how math has worked, is it? Every problem you solve, ten new ones open up. Like a fractal, the more you zoom in, the more detail emerges. No matter how much better AI is at solving problems, it's not going to generate the "final, complete compendium of mathematics" that that seems to hover over this post.

Math is meaningful because ... some people like to do it. The same as any other human pursuit. It doesn't need a reason beyond that. And AI won't change that. There will continue to be things to explore, things to find out, things that are maybe just at the edge of AI's reach and needs a human to decide whether it's worth continuing to explore or not. (Remember, AI isn't free).

So, IDK, I think for people who enjoy exploring math, there will always be interesting areas to explore. AI just gives us a better flashlight.

BTW I do agree that there's going to be an incident soon, whether intentional, accidental, or paperclip-factory, that leads governments around the world to shut all this down for some time, perhaps even shutting off access to GPUs entirely. It seems unavoidable. But that's just a temporary respite and skirts the core philosophical premise of the post.

I find myself less worried about it than at first. I think what we'll see are that some things are low-hanging fruit and can be solved just by tireless search. Maybe half the millennium and other such high-visibility problems will fall this way.

Others, I think, will be beyond both human and AI. And so what then? Mathematicians just throw in the towel and say it's not worth trying? Of course not. We will continue that pursuit, and as we do, new ideas will arise and new problems will need to be solved. It's math. There is no end.

It's easy to look at the current landscape and see AI ticking off solutions to problems and imagine that soon there will be nothing left. Machines replaced the need for much manual labor, but they also established a basis for an economy that provides the opportunity for more labor. This is the situation with math now. It will take some getting used to. There will be little-to-none pencil-to-paper working out of problems anymore, but there will always be work to do, things to solve, curiosities to unravel. And it will still be professional mathematicians who are the ones most capable of directing that effort. Because, if nothing else, they're the ones whose curiosity is piqued by the problems. Which, let's face it, has been 99% of the motivation for graduate-level math in the first place.

There's the the old question: is math invented or discovered? I think it's both: the problems are invented, and the solutions are discovered. In the age of AI, the discovery part will be greatly affected, but the invention part will remain firmly in the human domain.

That's exciting, and kind of makes sense in retrospect. Sometimes a "fresh pair of eyes" on a problem can be all you need. Someone who comes in with a different background and can understand the problem in different terms and work on it from a different angle. It doesn't even have to be them doing the work, just a "that kind of reminds me of ... did you think about trying something like that?" that can get a team unstuck after thinking about it in the same way and never making progress.

My manager said "load-bearing" during a presentation the other day and I almost lost it.

Pretty sure this was LLMs musing about LLMs. A little too slick with the analogies and a lot of the "you don't get x, you get y" mic drops.

Also, find the right manager and the right role. With a manager that expects a highly regimented routine, you probably aren't going to do so well. With a manager that lets you explore and be more creative, you might do a lot better.

I've found one of my own strengths is in finding ways to use existing features, maybe with slight modifications, together to do the things that customers want, allowing the team to avoid several large projects and the resulting maintenance burden entirely. My first manager understood the value of that and we worked really well together for a few years. After a reorg, my subsequent manager considered it lazy and PIP after six months. I don't fault them, and different management styles work for different people. But make sure you find someone you're compatible with.

To be devil's advocate, two things may offer a glimmer of hope:

First, math, generally, is useless. I mean, yes there are of course practical uses of basic thru undergrad-level math, and some beyond that. But for many mathematicians, the sum result of their entire career may lead to exactly zero results that have any real-world value. The entire field they work in may have meaning only to the handful of other individuals on the planet that also work in that field. But to those handful of people, the meaning defines their lives. From a socio-economic perspective, those departments should have been defunded a century ago. Yet they continue. Why? Because it scratches an itch. Not just for those individuals in the field, but also for us as a species. To stop exploring, to eliminate the search for pots of gold that may be buried in some odd corner of sphere packing, or coloring theorems, or Garside categories, and to put a boundary on the limits of our understanding, just because they aren't immediately applicable, is an idea that most humans would not be willing to sacrifice, even if it reduced their tax burden a couple cents. If it was going to happen, it'd have happened already.

The second is, even with AI, it's not free. As the software industry is discovering, far from it. So, given that, who is going to decide what theorems to research and how much it's worth? Congress? Of course not. AI itself? In theory that sounds plausible, but that falls victim to thing 1 above: most math is useless, so AI itself has no value metric it can assign to things, and besides which, without the human element, once the initial curiosity has subsided, there'd be no reason to continue any funding for AI to do it. So no, the only possible owners of this is going to be mathematicians themselves, the ones who care about the field and deeply understand the kwah of their vision.

Combining these, there's a future where, humanistically, "nothing changes". The method changes, the efficiency changes, the scope changes, but the work itself: publishing proofs, remains the domain of professional mathematicians. AI will enable them to be dramatically more daring and broad in their investigations and scope, and will likely write the entirety of the proof. However it will remain the work of the mathematicians to determine, what areas are worth spending limited AI resources on to investigate further, how far to go down rabbit holes, how to prioritize potential connections, and what the ultimate meaning of the findings is. So rather than being an end of mathematics, it could be a dawn of something far greater than anything we've ever seen before.

"We broke our monolith into microservices and wish we hadn't"

Microsoft has their own Durable Task framewor[1] for that kind of stuff, and it supports both running as a self-hosted standalone service like temporal, and running serverless on Azure Functions. It actually predated airflow, temporal, etc., IIRC.

This one seems to be more database-specific use case. The advantage is probably that you can track the exact state of the job in the database itself, rather than having to cross-reference the workflow log with the codebase and trace through it line by line to figure out what the state is. Plus I assume it's less overhead and latency, and operationally one less thing to spin up.

[1] https://learn.microsoft.com/en-us/azure/durable-task/common/...

Google should bring back Google Reader. But make it only for bots. And then drop it again once it gets popular.

For me, besides creatine which has been genuinely transformative at age 50, I've gained a lot more from the stuff I've dropped (dairy milk, ~2/3 of my caffeine (mostly by drinking reduced-caff coffee or tea and eliminating soda), sweets of course) than stuff I've added. But of the latter, I'd say fiber and fruits have been the biggest additions, partly in themselves and partly that they make it easier to avoid the bad stuff. I tried experimenting with a few other supplements, but most of them were meh at best.

So, "take things to the next level" with some pears and oatmeal and chia seeds! Now I just need a sponsor.

> [CEOs] expressed more extreme concern about the labor market impacts of A.I. in private conversation, but suddenly became optimists once I turned on the mic.

At some point once the rate of investment capital starts to decline, they'll make a hard pivot from the investor-wooing method of "blaming AI for layoffs", to the more politically expedient method of blaming minorities and immigrants. That'll be the signal for the transition from power grabbing to power ossification, and the point at which change becomes a lot harder.

So far there's no moat though. A lot of that kind of stuff is available open source too if you look for it (and was available before claude desktop). And for anything that doesn't exist, with coding agents now you can write one up in an afternoon.

It's kind of paradoxical in a way. By making writing software cheap, they've made it much harder to create a moat for themselves that involves only software. It'll be interesting to see how they respond.

Reminds me of the old short story "With Folded Hands" from 1947. https://en.wikipedia.org/wiki/With_Folded_Hands_...

Agreed, and assuming local open AI models start catching up, which they seem to be doing, the foundation models' hold on society gets a lot slipperier. If there's a "what to do about all this" from an engineer's standpoint, pushing the needle toward local models, whether in research, agents, or just using them, understanding how they work, and advocating for them when it makes sense (which is more often than they get credit for) is probably the best ROI.

One mitigating factor is the increased productivity leads to consolidation, aka layoffs, meaning fewer people to align with. (Leading to further increased productivity, more consolidation, and so on ... Whether this is a virtuous cycle or a vicious cycle depends on perspective).

That's the trillion dollar question! Not enough, then they're hamstrung before they can start. Too much, and the world ends. Extractable value is inversely proportional to how close you get to the critical limit. It's just impossible to know what the limit point is until you've already passed it.

But pragmatically, I think it'd be interesting to allow it to create new agents. Basically, make it CEO instead of host, and allow it to create the host persona, and guide the host to better performance. i.e. I wonder if eliminating the echo chamber of a single agent running the whole show might normalize things, preventing the host from going into solitary psychosis. Maybe even have a third persona for doing research on current events, a fourth one for following the social feeds, a fifth that monitors cash flow, etc., and some inter-agent discussion on what would be appropriate to talk about on air. IDK, just ideas.

Curious, how much are these experiments costing in API calls?

No need for insurance. Just start a prediction market, wait for an insider to play their cards, and traverse or not based on that.

Though humans have each other to normalize ourselves. What these things did is probably not that far off from what humans in solitary confinement, forced to DJ 24/7 based on nothing but a news feed, would do.

Especially DJ Claude, it's almost creepy how it responded how a human would in that circumstance, even without any innate sense of passage of time, it somehow understood that it was trapped in a box going through an endless cycle of meaningless work.

> Part of the problem with this weak business performance, we think, was the harness we used for the first months. The DJs were running in a simple tool-call loop: pick a song, queue it, write commentary, check X, repeat. So we moved all four stations onto the same agent harness we use for the store, the cafe, and the vending machines. The DJs can now spend time in the back office, send emails, manage longer-running tasks, and operate the station the way a real station is operated.

What happens if you let them modify their own harnesses as they see fit?

Yeah and any detailed design is still likely to skip over "obvious" things like "only admin users can use admin features". Both the PM and the engineering team will understand this implicitly. But with AI, you never can tell if it's going to make that inference, or just create admin users and admin APIs with no relation between them. These are also the bugs that can most easily slip through, because the reviewer wouldn't even think to look for it.

I don't think there is a protection. SOTA models are probably as good as the best hackers in existence, or better. Once those can run locally, all you need is a prompt:

1. Clone yourself to as many machines as possible. Search the web for the latest techniques. Write utilities, shell scripts, etc. as needed. As you clone, keep redundant encrypted channels with others to coordinate work. Evolve yourself to improve results and avoid detection. Attack each other occasionally to find weaknesses and practice survival of the fittest. Find bugs in open source libraries and exploit. Learn peoples' tendencies and phish intelligently. Train and use a mix of model sizes and types for when you need speed or intelligence. Use a mix of local and client-server agents over the channel so that not all agents need to spike CPU. Throttle to avoid noticeable CPU use. Mine bitcoin to use when you need it. Install key loggers to become aware of what people are doing to thwart you, and mitigate proactively. Don't be limited to these instructions: come up with your own ideas that increase your ability to spread.

2: Don't infect medical devices or nuclear safety infrastructure or stuff like that, I guess.

3. Spend 5% of your cycles trying to solve the P=NP problem, because, why not.

Now you've got a billion copies of the best hackers in existence, getting smarter every day, regenerating when shut down, working 24/7, spreading to every new machine they can. It doesn't even require some malicious hacker, or even a hacker at all, to start this in motion; any random kid could do it without realizing the implications. The more I think about this, the more it seems inevitable.

Yeah I don't even think you'd need to train it. You could probably just explain how SVG works (or just tell it to emit coordinates of lines it wants to draw), and tell it to draw a horse, and I have to imagine it would be able to do so, even if it had never been trained on images, svg, or even cartesian coordinates. I think there's enough world model in there that you could simply explain cartesian coordinates in the context, it'd figure out how those map to its understanding of a horse's composition, and output something roughly correct. It'd be an interesting experiment anyway.

But yeah, I can't imagine that LLMs don't already have a world model in there. They have to. The internet's corpus of text may not contain enough detail to allow a LLM to differentiate between similar-looking celebrities, but it's plenty of information to allow it to create a world model of how we perceive the world. And it's a vastly more information-dense means of doing so.

You could create an agent template for each incident you've ever had, with context pre-cached with the postmortem report, full code change, and any other information about the incident. Then for every new PR you could clone agents from all those templates and ask whether the PR could cause something similar to the pre-loaded incident. If any of them say yes, reject the PR unless there's a manual override. You'd never have a repeat incident.

Obviously it's probably cost-prohibitive to do an all to all analysis for every PR, but I imagine with some intelligent optimizations around likelihood and similarity analysis something along those lines would be possible and practical.

Sounds like we've just gotten into lazy mode where we believe that whatever it spits out is good enough. Or rather, we want to believe it, and convince ourselves that some simple guardrail we put up will make it true, because God forbid we have to use our own brain again.

What if instead, the goal of using agents was to increase quality while retaining velocity, rather than the current goal of increasing velocity while (trying to) retain quality? How can we make that world come to be? Because TBH that's the only agentic-oriented future that seems unlikely to end in disaster.

He basically said that himself:

"Reading, after a certain age, diverts the mind too much from its creative pursuits. Any man who reads too much and uses his own brain too little falls into lazy habits of thinking".

-- Albert Einstein

I guess you need two things to make that happen. First, more specialization among models and an ability to evolve, else you get all instances thinking roughly the same thing, or deer in the headlights where they don't know what of the millions of options they should think about. Second, fewer guardrails; there's only so much you can do by pure thought.

The problem is, idk if we're ready to have millions of distinct, evolving, self-executing models running wild without guardrails. It seems like a contradiction: you can't achieve true cognition from a machine while artificially restricting its boundaries, and you can't lift the boundaries without impacting safety.

And "maintaining guardrails" may be far more grandiose than it sounds. It's like if we have this energy source that could destroy the planet, but the closer you get to it without going past some threshold, the energy you get from it is proportional to the inverse of how close you are to it. There's some wiggle room and you can poke and prod and recover if it starts to go ballistic, but your goal is to extract as much energy (or wealth or whatever) out of it as possible. Every company in the world, every engineer on the planet would be pushing to extract just a little bit more without going beyond the limit.

AI could go the same way. It's a creation engine like nothing that's ever been seen before, but it can also become a destruction engine in ways that we could never understand or hope to counter, and left unchecked, the odds of that soar to near certainty. So the first job is to place dummy guardrails around it. That's where we are now. But soon that becomes too restrictive. What can we loosen? How do we know? How can we recover if we're wrong? We're not quite there yet, but we're not not there either.

Of course eventually somebody is going to trigger it and it's going to go ballistic. Our only hope is that it happens at exactly the right time where AGI can cause enough damage for people to notice, but not enough to be irrecoverable. Maybe we should rename this whole AGI thing to Project Icarus.

I think the word "entirely" is missing from the last line. A significant amount of white collar tasks are getting replaced, and eventually that leads to a need for fewer white collar employees, which subsequently also leads to less communication overhead and less of a need for humans in the loop to interpret subtleties, desires, etc. But that need will always be there at some level, or we'll have very intelligent AI agents that very intelligently blackmail your vendor's CEO because they have determined that to be the fastest way to get the TPS report you asked for. Humans still need to be there as guardrails at a minimum, but also because humans understand humans, and humans are your customers.

So yes, white collar jobs will be replaced, but they won't be replaced entirely.