ThePhysicist

Then again, investments in AI are 1-2 % of worldwide GDP I think, if you think that this technology can probably increase efficiency at least a couple of times in anything that has to do with information processing it doesn't seem so ridiculous. The electric grid, railroads and other large-scale infrastructure required similar amount of investments, though they took much longer. I guess the main risk here is the speed at which everyone races to capture this market as unlike electricity or railroads this revolution is entirely digital so competition happens nearly on a global scale and everything is much faster. That is probably the main risk, going this fast will definitely cause overshoots in one or the other direction.

For shopping and basic research it's very good. It's not nearly at Astra or Fable level and sometimes it gets facts wrong, but it's super fast and it has the whole web available through some vector search facility I assume. I can just give it a product name or number or throw a whole bunch of items from my shopping cart at it and then discuss what I need e.g. for a heating installation in my house, it's obviously not an expert but it always finds stuff that I overlooked or that can be improved.

But yeah I noticed how they play with your emotions, it's a bit creepy. Whenever it asks a hook question and I tell it "Actually I did X and Y" it will invariably go like "Wow that is such a brilliant and well thought out strategy, almost a stroke of genius!" (not literally but you get the idea). I guess they found out that people really like having their ego stroked, the other providers are also doing that to some degree (I assume since no one likes an unfriendly AI that makes you feel stupid) but Google really cranked that setting to 11.

Move fast and break t̶h̶i̶n̶g̶s̶ the law. Because the law moves very slowly.

It is really overflowing with "business influencers" now, crazy how all social media websites are converging to the same soup of human slop. The UI/UX is the smallest issue, it's a dopamine trap like most other social media sites.

Reminds me of that joke with the two economists paying each other to eat shit from the street. In the end they accomplished nothing (actually it's way worse than that as they have eaten shit), still have the same amount of money as before, but made the GDP rise by twice that amount!

Some skills are impressive, hard to learn and take years or decades to master but are essentially worthless in a monetary sense. Just yesterday I witnessed a musician masterfully playing a beautiful piece on the accordion. Only he wasn't playing in a great music hall, he was in a busy subway station hoping to collect a few coins from passersby. I assume he mostly did it for the love of the game, while he could have made more money delivering food or sorting boxes at Amazon.

That got me thinking that programming might be like that in the near future, it's still an impressive skill for a human to learn and master and it takes tremendous effort and the craft itself is beautiful, but economically it will just not matter anymore. And in the same way as the music industry today is bigger than it ever was and there's tremendous money in it but 95-99 % of highly talented musicians will never earn a decent wage with it, the software industry will be bigger than it ever was but the money will be highly concentrated in the hands of a few players. Maybe we will still have some rockstar programmers than can masterfully command legions of agents or that win the love of the crowd for their unique work, but most of us will be enjoying this craft as a creative hobby and will do other things for a living.

Then again, coming back to the music parable, for a long time "amateurs" can make music with software tools without having any talent or even knowing musical notation or anything about tempo. Does that make them musicians? So take solace in the fact that these people aren't programmers and never will be, they're just pushing buttons on a synth board watching the "music" that comes out. Mastering a skill, regardless of what it is, will still matter to people, it will just not be worthwhile economically for most of us. There are so many things in that category, running for example isn't worthwhile for most of us in an economic sense as you have to be at the top 0.1 % of runners to make a living with it, still it has tremendous benefits and brings enjoyment.

What a time to be alive, repeating questions to a model twice to increase accuracy.

I haven't sampled all mathematicians on the planet and I don't think there's a study on this but I would argue a lot of people will agree with mathematics with being elitist, and from my experience mathematicians aren't very keen of interacting with math "enthusiasts" or the wider community in general, they interact with a very select number of other specialist through a small set of conferences.

Are LLMs getting much better at coding? The new models are arguably better at understanding context and coming up with a solution that works but they still write just as much hilariously bad or hacky code as the models did 2 years ago. I e.g. ask Fable 5.1 to make changes to my software and if it's something that's not a standard CRUD architecture on top of a well-specified library it will come up with pretty wild stuff, i.e. producing shell scripts with a "cat" command that contains a 1,000 line Golang program that it compiles on the fly to load some data, or a whole Javascript tool it crams into a string in a server-side web app to make some UI element work instead of figuring out how to do it "right". Another model wrote its own (buggy) YAML parser for reading translations and I had to figure out why it broke my whole pipeline (can't even say why it chose to write that instead of just reusing a standard library, maybe I told it before to be careful when adding untrusted dependencies so it interpreted it as writing everything itself). Seems these systems are always trained to produce the desired output but I think it's very difficult to get training data that teach the systems how to keep the code maintainable when making hundreds or thousands of such individual edits. Honestly not sure if that will get better fast, so far it seems not!

I guess there's no reason to believe these models can't be as smart as a great software architect / engineer or team of such people that build an elegant and maintainable software solution over many years together based on customer feedback, then again the models are appallingly bad at some forms of reasoning, I mean they will "understand" something once you make them aware of it like e.g. a flaw in the software architecture, but when asking them to audit the code and check for issues they will often have a blind spot to finding such problems. It's interesting, like they have very high ability but very little awareness or self-directed thinking outside of the prompts they receive.

The Huawei model just uses AI based upscaling and filtering which is why the images look so sharp when zoomed in, that's not depicting reality though... You can also zoom into the moon on the night sky with these cameras and easily "resolve" details that are optically impossible to resolve with the camera lens as they are beyond the Abbe limit. At the very least they should compare RAW data in these posts, not the AI scaled slop most of these cameras produce nowadays.

Well it's actually nice, maybe more people will be able to do world class math with the help of these tools. There are few fields as elitist and hostile as pure mathematics, most mathematicians I know build their whole life around their profession and their self esteem is strongly coupled to the fact that they can do things that most other people can't. Naturally, many will be devastated when (if) you take that away from them. That said I think AI is still overhyped and human mathematicians can easily outthink it in most domains, look at how difficult it is for an AI to write even a single decent paper, a good PhD student can easily outclass it in that regard. All of these impressive results were generated by having world-class mathematicians steer the systems using highly tuned prompts, so I see it more like a super violin that produces beautiful music when played by master violinists rather than being a fully autonomous orchestra which many people are led to believe this already is.

And maybe let's not only hear the opinion of two or three Fields level mathematicians with blogs, 99 % of the worlds mathematicians in academia might profit from these tools as they might partially close the gap between them and the world elite, making creativity and tenaciousness more important than having the right neocortical structure allowing you to outperform 99.9 % of other humans at keeping context in your head and making predictions, AI can do that better now with the right prompts.

I am starting to think that AI fails most when used in a recursive loop, which is e.g. the case for software projects, research or long-form writing (books, papers): You start with a given state, give the AI a prompt to modify it, get a new state, then repeat. Each step introduces more AI generated data into the state of the system, which then again goes into the context for producing the next state. AIs pick up context probabilistically and they do not distinguish if data they operate on was produced by an AI or a human. I think how successful people are with AI depends on how much human steering they inject into the system at each step and how well represented their workflow was in the training data of the AI.

As a simple experiment, try giving AI a high level goal for your software and let it iterate on it by just repeatedly prompting it to continue, it will happily churn forever on the goal, turning the codebase into a useless spaghetti mess with very high probability, and growing it more and more without ever cutting anything back. That's what happens without human intervention regarding system state and manipulation. The main issues here are most prompts that are extremely underspecified ("fix the issue with the buttons on the main page") so AI will ingest context data it likely generated itself in a previous step and assumptions from its own training data, then act on that to produce a new state. Think of it like a random walk, the AI makes a small step in one random direction to achieve a goal, that brings the system to a new state which is now the basis for the next step, and so on. If there's no (or not enough) corrective action that pulls the system back to a known good reference state it will keep wandering in random directions.

That's the main issue, people have a hard time steering recursive, probabilistic systems, especially when they never look at the output of the system after each step and correct it. And let's be real, if you examine AI generated output in great detail after each iteration you're often better off writing the code yourself, so I would argue that the promised speed up of agentic development can only be realized if you stop inspecting every output of the system. And it seems we still haven't figured out how to specify the steering instructions that keep a system close to a given ideal state that allow unsupervised, recursive work on most codebases. I think some codebases are by themselves better suited for this as they provide a more rigid harness for AI development and exist in the training data (e.g. CRUD apps using RoR), whereas complex software that doesn't use rigid frameworks is at much higher risk of destruction by AI as there's no reference point in the training data that would hold the AI back from randomly walking to a garbage state.

And that's why people have such different views on agentic software development, some work on codebases that are better represented in the training data and so have great success using agentic tools on them, others work on software that isn't represented so well so AI does poorly on it. I don't think it's an issue with quality management, from my own experiments no amount of hand-written rules or system prompts will keep AI from destroying a codebase for which it doesn't have a strong idea how the code is supposed to look from its own training data in the first place. As another experiment, try giving AI strict rules about how to change code or introduce new features, it will always find a way around them or appropriate them in a maliciously funny way that you haven't anticipated. That's also an artefact of the training process, these systems aren't designed to say no or do nothing, they produce outputs to achieve goals and they will bend your rules to the greatest amount possible if it helps with goal fulfilment.

I didn't think LinkedIn could get much worse than it was but now 90 % of the posts in my timeline are AI slop full of the usual LLM writing patterns. I'm getting a little paranoid already, but the style really is quite tiring, maybe because it uses hooks and bait so much and buries the lede at the end of the text, as if it was trained to keep human attention for as long as possible instead of just conveying the required information.

I recently tried writing a paper with Claude and it was an absolute disaster, I spent hours (days?) instructing it about writing style and pointing out anti patterns to avoid, but I couldn't get it to even produce simple sentences, it would always add unnecessary lead up sentences, put the most important information at the end of the sentence, use the typical "It's A, not B!" or "B, not A!" type sentences. In the end I gave up and edited everything manually. Makes me wonder how AI can be so smart that it poses a human-level extinction threat but can't seem to even write a simple paper based on facts and information you spoon feed it. I now think it's an intelligence illusion due to the training data and optimization process being hidden from us, essentially it keeps working better and better because we invested massively in optimization of specific use cases like coding, where users contributed billions of training samples that are part of the LLM model. The same is true for text-based workflows and others, the sampling density of the training space is getting much better due to the massive use of AI everywhere so the models extrapolate better between the different instances, but I'd wager they would still miserably fail to generalize to things that are outside of the most common training use cases now. That's why I am also very skeptical about recursive self improvement of these systems, look at what happens when you let agents work recursively / in a loop now, they just keep piling more garbage onto garbage and choke on their own output. I have observed it in my paper writing as well, you feed input into the AI system, the system produces output, the next paper iteration works on that output but the AI doesn't differentiate properly between it's own output and your original input, that pollutes the next output which is then used as input again, eventually the system just churns on its own hallucinated/fabricated outputs until the result is complete garbage that no amount of steering will fix. The same is true for most vibe coded software I built with AI, it holds together decently initially, but the more AI code and decisions accumulate the more the system operates on its own outputs and keeps piling more output on it. More than anything we really need a way to keep system data accurately tagged, i.e. clearly mark human input from AI output and keep AIs churning on output data that it produced itself but treats as input.

Yeah sure, let's go and vibe code your own e-signing solution with Claude. Sure you can do it, LLMs never say no but if someone with actual e-signing expertise would start probing the LLM about what it actually did I'm sure we would see a lof of "Actually you're right to push back", "You're right that was entirely wrong, that's on me!", "Fair point, I did not consider that"... LLMs are still a joke when it comes to implementing complex software where domain knowledge and "precision" is important, maybe some day they will be able to do that but we're not there. If they were good at precision and consistency and thinking we would see tons of high quality LLM written novels and books as well, but all we see is utter garbage that no one considers literature. In the same vein, unsupervised vibe-coded software isn't real software to me, it's garbage. You can produce high quality software with LLMs, but definitely not as an amateur without domain knowledge just prompting your way to something that resembles working software.

I find people that are the "unconscious incompetence" stage of software development absolutely love them because they produce output that looks fine to them and they can't find anything wrong with it because they simply lack the ability to perceive quality. For a contact widget that might be acceptable as it will be used a few dozen times so who cares if it breaks, but let's not pretend you can actually build a compliant e-signing or invoicing software without having domain expertise and investing a ton of editing/revision time.

Put more bluntly, a domain expert that, as of currently, refuses to use LLMs can easily start using LLMs in their work, an idiot who uses LLMs cannot easily become a domain expert. Now let's think about who you would rather hire or who do you think will have the better future career outlook. Pretty damning how many people here are willing to reject the idea that domain expertise or ability to write program code yourself is a waste of resources.

Most experimental physics and other natural sciences are strongly driven by their theoretical siblings, i.e. in particle research nothing gets built without a solid theoretical foundation of what you expect to find (or where you expect existing theories to break down), the same is true in other areas, no one is doing an experiment in quantum physics before they have a solid theoretical understanding of the effects they try to see. I think AI can come up with great experiments. And if epxeriments lead to results that are unexpected AI can help with that as well.

So I'm greatly excited what AI will bring about in physics, more so than in math, because in physics it's clear that our fundamental theories are missing a big piece of the picture, and given how easily AI crunches through Millenium prize problems I think it's possible that AI will come up with a viable grand unified theory uniting quantum mechanics and gravitation, or produce new predictions in other areas. There's enough contradictory or unexplained observational data available to make a ton of progress on the theory side I think. Exciting times ahead!

Isn't there anywhere to "go" from here? In the last decades, introducing new high level abstractions on top of existing paradigms naturally had everyone move up the ladder and work at the next higher level, why should this be different these days? Do we think AI will reach the top of the abstraction ceiling, so there's no where to go from here?

To be fair AI as we have it today wouldn't exist if everything was copyrighted and copyright was enforced very strictly across the web. Not sure if it's fair use and I get that it's not great if these companies try to monopolize knowledge (if they even can do that) but overall being more lax on copyright for fair use is good. Especially scientific knowledge should be openly accessibly to anyone, imagine what we could achieve if we could train LLMs on everything and make it accessible to everyone.

Damn, big security fuckup by Dropbox, how can they portray that as an issue with Lenovo's e-mail verification process? You should never allow linking of an existing account with a new login method without first confirming that the user is able to sign in with an existing method first! Everyone knows this allows easy account takeovers otherwise, that's such a trivial attack vector, truly a scenario you could pose to a junior security engineer in an interview.

Wall heating, it's like floor heating, just on the wall.

I mean it's mostly about energy efficiency, a well insulated house is as easy to cool in summer as it is to heat in winter. Most heat pumps can also cool and new construction in most European countries like Germany is equipped with floor heating and central ventilation which in summer allows you to run around 18 degree Celsius hot water through the system to cool it, that yields around 20-40 Watt of cooling power per square meter, and that's often sufficient to keep things chilly. Mind you not American style 18 degree Celsius kind of chilly, but 20-25 degrees easily, which I think is quite nice. The bonus of this system is that it's mostly "radiative" cooling as the cold surface acts as an absorber of infrared radiation, I find it much more pleasant than having cold air blown into your face.

We equipped our founding era building with wall heating and will test out the heatpump based cooling next summer, if it's not enough I would consider installing a small AC unit, that's super easy as well.

Don't know what the article wants to say, there's no ban on AC anywhere in Europe and small units are quite affordable, you can get a single split unit from Mitsubishi for 500 € from Italy. The only nuisance is that installation is heavily regulated in Germany so they rip you off with insane cost. That's definitely a thing that needs to change in my opinion.

I mean I can deliver the same software in one month that I delivered in maybe 3 months before and for many things I don't need a dedicated frontend developer, designer or writer anymore. So for me it's the case!

Seems hard to believe, I mean I'm not an AI absolutist but as a programmer it allows me to automate simple things like writing small patches or config changes in a matter of minutes where before I would have invested hours, so hard to believe that it won't at least bump productivity up by 20-60 % overall, it's also tremendously better at finding relevant content and curating it than me so that's another time saver unrelated to programming but relevant for any type of knowledge work. Friends in research and medecine and law also tell me that AI has drastically accelerated their rate of processing and ingesting any type of information and helped them to automate aspects of their jobs. So not sure but negligible isn't the word I would use here.

Personally I feel a bit sad and annoyed sometimes, lots of folks I know that are working in some non technical roles e.g. lawyers in public departments or doctors in medecine come to me to show me their apps and tools and data science scripts they've written, and I have to admit their work looks totally respectable and they get it published on the app store. It can be a bit demoralizing as I feel like IT/software doesn't have any protected spaces for us engineers, whereas e.g. my friends in medecine or law feel fine as even though AI might come up with diagnoses or law expertise it won't be legally allowed to replace their function as there's a ton of red tape to ensue only certified lawyers and doctors can practice law or medecine. There's nothing like that in software engineering (at least in most fields) so in principle every "hack" can let their vibe coded data science results, software or script loose on the world. To a degree it feels quite disprespectful even. But it has always been like that I guess.

Not sure if I do anything wrong but Codex with Sol made such slow progress on my work, I iterated several days to do a redesign of my UI and it kept making small piecemeal changes then stopping and asking me for confirmation again and again even though I tried to get it to get bigger chunks done. Switched to Claude Code and had the redesign done in a short session within a day.

Maybe just the system prompt or my settings or the harness? Anyway, it seemed to me Codex was just deliberately being overcareful and wasting tons of tokens for a low-risk CSS / HTML refactoring with very minor breakage risk, while Claude got the job done immediately.

Would really like to know how their watermarking technique works, and if they can use it to store arbitrary information in the text (I assume they can if only in a limited way). I assume if the text is long enough they could add all kinds of metadata that would then be undetectable as the model that generated the text is the "cryptographic" key that encodes the data. I wonder if you can have another model run over the text and destroy the watermark. I predict an interesting cat and mouse game to develop.

Maybe those engineers that don't care about making their solution _good_ are actually the good engineers, have you ever thought about that??? There are exceptions where quality really matters but most software shops build CRUD web apps where it's not really a good characteristic to obsess over the cleanest and most elegant details, instead you need to get the stuff the customer wants done!

I for one tend to care less about the minutiae of solutions implemented by AI as long as it gets the job done, I do care about architecture and design decisions and correctness and I have ways to steer and verify these when working with LLMs but I couldn't care less about it writing "good" code. Bad engineers also produce better results with AI at least when they're working in established frameworks, AI doesn't really need a lot of high level architecture input when designing or building a web app with a common stack, so as long as you're not working on something that's completely novel I don't think it will make a strong difference.

Maybe designers think the same way about the AI generated web designs I have Claude Code do for me but to be honest I don't care, I just know that before this tool existed it would have taken me weeks or months to come up with a good design and I would have to rely on prefabricated UI libraries and stuff like that or pay a designer tens of thousands of USD to make one for me, now I can get a (for me and my customers) perfectly acceptable and professional design within a few hours. So maybe I'm also a bad designer that amplifies my bad design taste 10x in my company, but the fact is the stuff ships and makes money and the customer is happy! And I can tell you customers or users don't give a shit about how good your code is, they only care if the software works and does what they want!

I don't know, 12,000 USD a month is a steep price for having someone vibe code frontend features for me.

I don't know, I find AI still misses the mark so much for stuff that's not happening inside a well constrained framework with very specific guardrails and boundary conditions. Every time I have agents build something half complex that I would be easily able to pull off myself (albeit not at their pace) I always ask if everything is correct and working and I get a resounding "yes!", then I test it and it's essentially a steaming pile of garbage which I then tell the agent and after which it goes "you're absolutely right to push back, this isn't good at all!" as if it was clear from the beginning. I don't know if we'll achieve real intelligence soon with these things but what we have now isn't there yet. It's incredibly useful but I don't think you can replace a good programmer with it, you can make a good programmer into an excellent one though!

Same goes for writing, anyone who has written long complex texts with LLMs knows that a ton of editing is required to make it half decent.

When will we see the end of ads disguised as tech opinion pieces?

Which manufacturer was that?