ThePhysicist

I don't know, I find AI still misses the mark so much for stuff that's not happening inside a well constrained framework with very specific guardrails and boundary conditions. Every time I have agents build something half complex that I would be easily able to pull off myself (albeit not at their pace) I always ask if everything is correct and working and I get a resounding "yes!", then I test it and it's essentially a steaming pile of garbage which I then tell the agent and after which it goes "you're absolutely right to push back, this isn't good at all!" as if it was clear from the beginning. I don't know if we'll achieve real intelligence soon with these things but what we have now isn't there yet. It's incredibly useful but I don't think you can replace a good programmer with it, you can make a good programmer into an excellent one though!

Same goes for writing, anyone who has written long complex texts with LLMs knows that a ton of editing is required to make it half decent.

When will we see the end of ads disguised as tech opinion pieces?

Which manufacturer was that?

Coming up with solutions isn't so difficult, getting everyone behind one is the hard part. Obviously easier if you're one of the two biggest AI companies in the (Western) world.

I think Valve is just going to kill Xbox in a couple of years. Basically Microsoft is planning to make the next Xbox an "ordinary" Windows computer with a different UI as well as vendor-locked hard- & software. So essentially you'll get a (maybe) slightly discounted small form-factor PC for the living room that only runs games from the Xbox store, which will be more expensive than the same games on Steam.

Now, everyone laughs about the "ridiculous" pricing of the Steam machine but once hardware prices come down they will make a second generation that will be an unlocked PC in the same form factor and probably just as powerful as the next-generation Xbox. They might even pass on some of their supply side bargaining power to the end user. Then you will have the choice between a fully locked down Windows PC that only plays Xbox games for 1,000 USD or a fully unlocked Steam machine that plays everything from the Steam, Epic, GOG store, can install regular games and even lets you do work on it, for probably around the same price. Built by a company that has the highest gamer trust of any company we know. Guess this will be a no-brainer for most folks! They have everything to pull this off as well, the brand, the trust, the gamers, the store, the platform, the money, the market power as well as experience with hardware. My take is that they wanted to pull this move already but the AI craze made that impossible so they are using the first-gen Steam machine to learn about the market, but I predict that once prices come down we'll see an announcement, they can time it to match the new console generation by Microsoft and Sony as well. The market should be in a great state by then as well, lots of folks were essentially priced out of the GPU market and NVIDIA won't even release a new generation of gaming cards anytime soon, so all those people that have the buying power but don't know where to put it would flock to a new Steam machine if it can deliver good performance for a reasonable price. And the thing is Valve will have the power to make sure studios optimize games for the Steam machine as well.

Really don't know what Microsoft is doing here, they turned one of the most profitable and beloved gaming platforms into something that seems to be in a death spiral, people are already speculating they might sell the whole thing off eventually.

There's this great blog post by Joel Spolsky from 2000 [1], where he essentially argues that controlling your environment makes you happy. He writes about his summer job in a bakery and how the dough mixers would be so unpredictable and how frustrating that was. I think AI agents are quite similar to a lot of folks, they change significantly with each major model update and even every day as the vendor tweaks the system prompts and settings, so you never feel "in control", it's more like pushing buttons on some blackbox and hoping the right stuff happens inside. Most people are unhappy about that as it takes away the mastery and craft aspect of software development and makes them managers of unpredictable AI tools. I certainly get this feeling even though I like AI in general, but having days where everything goes so well working with the agent and then days where nothing really seems to work and not knowing why is quite frustrating.

1: https://www.joelonsoftware.com/2000/04/10/controlling-your-e...

Interesting point about "letting go". I hear that quite a lot, which I find surprising. In IT, when new technologies like cloud computing emerge people always need to adapt their workflows, and for some that is challenging e.g. going from manually maintained servers to virtual machines or containers that are just spun up and down on demand. Maybe it is similar with AI, we gain new capabilities along one dimension like speed of development and we lose some capabilities along the way e.g. manual control of quality. And of course there are large financial incentives here as well, maybe they are larger than we have ever seen before and the change also happens faster. Cloud computing took probably multiple decades to be fully adopted (I think AWS became available in 2006 and we still see large corporations migrating to the cloud from their on premise setups today, though the adoption curve is flattening), LLMs have significantly higher adoption after only around 2-3 years of them becoming "production-ready". So naturally people struggle with how to adopt them and we need to figure out where they make sense and where not. That said it's precisely an engineers' job to figure that out, people that just see the upsides of this technology seem quite naive to me.

I can see this struggle it in my organization as well, in the last year there was a big push to adopt AI everywhere and tons of initiatives to automate processes and produce code and text and other artefacts with LLMs. Now it seems the pendulum is swinging back a little as people see that all of the LLM generated stuff shows all of these subtle quality degradations, and people get tired of managing it as they are suddenly confronted with tons of additional information they need to manage.

Given that models are still evolving and becoming better at a rapid pace I think that we will solve most of these issues in the near future, but for now I don't think the age of hand crafted code is over yet.

I have been thinking about that machine code analogy before as well, I don't think it really holds. Machine code is written in an automated way but following mostly deterministic rules that have been crafted through decades of manual optimizations and testing. AI generated code has nowhere near this level of scrutiny, testing and optimization behind itself. The fault rate of compilers and optimizers is incredibly small (I can't find any numbers but it must be on the order of ppm or ppb), AI generated code has fault rates that are even in the best case on the order of 99-99.99 % maybe (i.e. between one error per hundred lines and one error per ten thousand lines in the best case), try building anything complex using such a fault rate without manual correction and review. It's impossible. I have done it, I dabbled with writing a compiler, a database and even a simple web framework from first principles. I didn't get far, even though I had good mental models of these things and I carefully wrote RFCs and documents for the LLM, specified test cases etc... If you're lucky it will regurgitate some existing code or follow documented guidelines, but when you're on new territory these models won't be able to produce anything good. I would really like to see a single example of someone vibe coding a high quality library or tool with LLMs, I haven't found anything and no one can point me to a complex codebase (say 10,000 lines or more) that was generated using high level prompts that looks decent and doesn't have multiple glaring issues that appear when looking at it in detail.

Oh for vibe coding small scripts it's great. Experience of people with AI can vary, I don't see what attitude has to do with this, at this point I have probably spent thousands of hours trying different strategies to leverage LLMs in my workflows both for code and other artefacts, so one cannot say that I'm not enthusiastic or open towards this technology, I want it to work for me. In some cases it works great and capabilities have increased a lot (the models I used two years ago seem so primitive by todays standards, I wonder how I could even have done work with them), but for me the tools still aren't good enough to be left writing products unsupervised. Maybe that is an attitude thing, but reading raw LLM generated text or code surfaces a lot of issues, and it requires heavy editing to make it meet the quality bar that I have. I'm not the only one seeing it like that. I could just throw that overboard and be happy, in the end I always make the LLM produce what I want, at least superficially, so like other people have suggested maybe I should just let go.

I get that people without this mental baggage will adopt LLMs more easily as they do not care about any of these and they are often not able to perceive quality. And maybe quality is something that doesn't even exist and doesn't matter much, in the end people don't care how the sausage is made and what is inside if it tastes good and nourishes them. It surprises me though how fast most "engineers" throw away their engineering principles when using these systems. So I could also say mean things like maybe these people weren't good engineers to begin with. I guess the divide is more between people that care about the underlying principles and quality of their work vs. people that care about being done and having the desired effect on the outside world through their work. For software that's more acceptable than in other engineering disciplines I guess, people won't get harmed (much) if your vibe coded SaaS app crashes or leaks your whole user database. Trust into vibe-coded software seems to be higher than I think is warranted though, which makes sense as the companies pour trillions of USD into marketing and getting people hooked on these systems. Feels a bit like the whole crypto bubble, LLM absolutists aren't that different from crypto bros in 2015 that were certain traditional finance was doomed and in five years our entire life would play out on the blockchain. I don't think it's entirely comparable though as LLMs have tons of real-world use cases. And I think it's not surprising that people have such strong opinions on them, there were similar discussions about Rust or frontend frameworks, seems in tech there are just types of people that discover something new and immediately think it will solve all of their issues and that anyone who has a different experience must be wrong, even though they are only seeing a tiny part of the whole problem space. So, glad people get value out of LLMs and I do too, but I don't get how one can use models and not see these quality issues.

Maybe one point regarding quality, even for small scripts that I write using LLMs e.g. for data analysis I have to be very careful as they will often create subtle mistakes that ruin the whole analysis. Things that are not per se wrong as the script runs and produces the desired output (which the systems are optimized and trained for, i.e. they know how to create a script that works and kind of corresponds to the prompt), but in disciplines like data science where you need to think carefully about every step of an analysis LLMs are quite dangerous as they produce convincing looking results that seem correct at every stage but are completely wrong. The only way to get that right (in my experience) is to go really slowly testing each step individually with known good inputs and outputs, giving the LLM that as a harness to work in.

I have been looking at LLM produced artefacts for a while now especially in data science and there are very cases where the models actually produced something entirely correct, at least when it's more complex than averaging some numbers or other simple things. The same goes for writing, superficially it looks good but there are often subtle inconsistencies that the model just doesn't see and that are hard to spot. And again, some people will just look at that and think "that's fine!" because they just can't see the quality issues or they don't care about them, but that doesn't make these issues disappear...

Think about it statistically, the benchmarks say clearly these models produce correct code in maybe 90-95 % of cases for most complex questions. That sounds high but given that a real-world system can consist of e.g. 100 such individual components then even a 99 % correctness rate at the level of an individual component gives you only a 36 % correctness rate for the entire system! That's also true for papers, presentations and anything else. Again, most people won't even understand this, for them something that looks correct and works is good enough.

Yeah I have a lot of experience writing software. My code is written mainly in Golang, but I had models write different code in Javascript, Rust, Python, Shell and other languages already. I used a variety of frontier models over the last years, always the best available model at the time.

I prompt LLMs by writing design specs and iterate on them first, then let it implement them step by step, checking the results after each step. That works fine for simpler changes where I use the LLM to write code that I have mostly worked out in my head, it always goes wrong once I try to do that with larger features. I have tried a lot of different things like writing extensive RFCs and design docs for the whole codebase, building harnesses and evaluation loops to ensure we stick to specific paradigms in the codebases but the LLMs still deviate from that in sublte ways and spuriously introduce duplication, wrong abstractions or simple hacks. That said my codebases are quite complex, it's not run of the mill CRUD software, I suspect these LLMs would do much better on these. That's probably why other people report large success using AI based development, 90 % of apps out there are just plain RoR or Django backends, React or Next.js frontend or Android apps, and they are already built following strict cookie cutter recipes, LLMs have no trouble following these. My work is e.g. on novel parser generators, graph data persistence layers, format-preserving pseudonymization and personal information detection in unstructured data so there's really nothing that you can base the software design on apart from general principles, I suppose that is why the models struggle so much.

There was a discussion here explaining the attention mechanism of the larger models and why they are not good at using their full context length, that was quite enlightening to me as it explained a lot of the behavior I saw on more complex changes, so I think one mistake I made was to have too long conversations with too much context (even though "on paper" the context length was fine and well within limits of the given model), I guess I need more careful conversation management and in general reduce the level of abstraction I'm working at with an LLM. For me at least they're not yet good enough to work at the business or concept level of abstraction, but they are capable of speeding up delivery of finished architectural designs.

Maybe it's also a perception problem. A lot of people will just look at their AI generated software and check that it does what it's supposed to do on the happy path and they will be fine with that, calling it a day (and to be honest I did that too for projects with tight deadlines, though it feels irresponsible). Especially juniors or people without programming background don't care about how the code looks that the AI wrote, I only see these issues because I have 10+ years of experience working by hand in large codebases and I have developed a "taste" for what good code is supposed to look like for me. That might explain why people are feeling so radically different about LLMs, if you don't have all of that intrinsic baggage that senior level developers have amassed over their careers then AI generated code will always look good to you. And maybe they are right, could be that in 10 years no one looks at any code anymore and we just care about tests and making sure the behaviour is correct. To be honest I never looked at Assembly code in the last 10 years and I don't care how my compiler unrolls my loops (mostly) as it's a solved problem for me, maybe it will be similar with the higher level code, we just move the abstraction that we work at to a higher level. But I still feel that we don't have the right tools for working at this higher level yet.

I'm about to throw away multiple months of LLM generated code for one of my side projects. I was really careful writing design specs and it wasn't even a new code base the LLM worked on, but still after several months of AI changes I feel my code degraded more and more into a subtle mess. Hard to explain, each individual change looked good and logical and on the surface the codebase looks fine, but looking at the whole picture everything is subtly wrong in multiple ways. The same goes for where I used AI for existing commercial code bases. I would love to have AI write production ready software for me, but it's just not there yet, there simply are things that good programmers and architects do that cannot be captured by the training loop of current generation LLMs.

I notice the same pattern when using LLMs to write longer text like reports or scientific papers, individually each section they write makes sense but overall the whole document feels off in a hard to describe way. I think it's where you can see the difference between human intelligence and whatever it is LLMs have, it's not the same thing. We are much slower and less able on the small scale but seems we can do some higher level reasoning that is still impossible for LLMs. That always becomes clear when you point an LLM at an obvious flaw it produced and it goes "You are absolutely right!" as if it's obvious in hindsight but when running multiple "Please look for issues" iterations it would never have spotted the issue by itself.

That said I think it will be absolutely fine writing a simple CRUD app for you e.g. using some popular JS framework, Tailwind for styling and a regular ORM, there's more than enough training data available for these things. But then again such software could be purchased before already e.g. as a SaaS template, I don't think LLMs are so revolutionary here, they just replace the template (but to be honest a good hand-written SaaS boilerplate is probably still better than a vibe coded one).

DigitalOcean simply does this at a later stage and will then just cut you off, Hetzner is doing it from the start. Their obligations aren't so different, they just approach this differently. It's the same thing with Stripe, it's cool that they just let you sign up and process transactions but at some point they will look at your business and if they don't like it they will just cut you off.

Also Hetzner really doesn't care much about individuals as far as I know, they mostly do business with other businesses and that's where the money is, their business is not designed to handle millions of individual users, DigitalOcean might also simply have a different strategy there.

Takes me back to when I was working on Josephson junctions (JJ) as a student, I-V curves were the first thing we did when characterizing a junction. JJ I-V curves were tricky due to the hysteresis they have in the underdamped regime, so we had some pretty involved stepping logic to ensure we get the curve with the required precision.

That said today everything is pretty much digital, you have Acqiris/Agilent 1 GHz ADCs and all the measurements are done in software, but I still remember using my old 20 MHz HMAG oscilloscope in XY mode with a triangle voltage generator to plot IV curves in real-time. Good old times!

Congrats Sonali, Nick, Dave & everyone else! I had an incredible time with you all in NY more than 10 years ago, it was so cool, and I'm still thinking of it very fondly. Spent my days hacking away in the space near Canal Street and the nights and weekends exploring NYC with other Recursers, visiting museums, parks and venturing out to buy cheap dumplings by the dozen. I was quite poor back then but I enjoyed my life so much, had a tiny room at the Kolping house on the Upper East Side which was very run down and tiny but also very cheap, basically just went there for sleeping and spent every other minute in the space and outside. No distractions, no possessions beyond my laptop, no responsibilities, simple but happy times. And then in 2021 I found my dream job at DuckDuckGo through RC, been working there for almost five years now! Thanks for everything and great to see it's still going strong!

Interesting that on one hand the valuation of these AI providers is based on the assumption that all code (and everything else producing digital artefacts) will be written using AI in the near future, on the other hand almost all popular open source projects fight to keep AI contributions out. Hard to reconcile.

Personally I'm also experiencing a bit of AI hangover after using it a lot in my own open-source projects. I find it's a bit like taking drugs (not that I have much experience with that) in the sense that in the moment I'm using these tools I feel great and powerful, writing features in a span of hours that would've taken me weeks to write by hand. But inevitably some time later I will look at the code and notice all the subtle cracks and inconsistencies the tool introduced, and despair a bit at the mess.

I now plan to use these tools less for extensive feature development and more for planning, debugging and narrow refactoring where I can put very strict guardrails on them. I'd still say it accelerates my work but not by a factor of 10, more like 1.5-3 (which is still a lot) given the care you need to ensure what is being built is actually good. For what I really like these tools is that I need less mental focus to do coding, but on the other hand I have this new kind of fatigue of being in a constant chat loop with a machine and trying to get it to do stuff based on natural language, never knowing how it will interpret what I write and wrote before. In that sense, these tools don't feel satisfying, it's like operating a machine where you try to push some buttons to get it to do something but the internal wiring changes all the time so you never know exactly what a given button combination will do and you have to figure it out by watching the machine and constantly adapting.

Corporations will be quite happy with really simple features if they're packaged right. I'm selling software that mostly does things that a good programmer could whip up in a couple of days or weeks, but what cost me most of the development time wasn't the features but all the FUD around them e.g. SSO, multitenancy, audit logs, corporate design support, ... Most enterprise software could be replaced with simple scripts and command line tools if they had this enterprise layer. I'd wager tons of SaaS is just simple open-source software and libraries behind a management layer.

Noooo, my /22 IPv4 subnet allocation is my personal 401k, I need this money to retire.

It's quite easy to write a generic Maybe struct that performs most of the encapsulation that Rust's Maybe does i.e. allow unwrapping of the inner type through a function or handling the nil case through a switch like statement. I've never seen this in the wild which makes me think people don't care about it too much. And of course it's runtime based so no compile time guarantees, and just to preempt the expected replies I know it's not the same what Rust is capable off and Rust is of course a much much much much better language than Go.

Personally I do experiment with these things as it makes code more readable, it just seems adoption for generics and what you can do with them is still quite low in the broader community. That said I do not deal with null pointer exceptions much at all, and when I do it's often relatively simply to spot and fix, so for me it's not a large issue.

I think for mostly search-focused use case like the one presented here AI is great as you don't ask it to build stuff or invent new drugs, you just want to retrieve relevant documents with laser precision, and agents can do that.

I think right now I'm mostly disappointed with agents writing code as they always degrade the quality of the codebase after a while, and the same goes for writing in general which just requires a ton of editing and mostly just sounds good but doesn't have a lot of substance in the end. I think you can really tell that these systems are trained to just produce plausible streams of text, especially in longer artefacts you notice that locally the inner consistency of what they produce is great but globally it really falls apart, it's like seeing the limits of their "intelligence".

For search however I really like AI, it has improved information retrieval so much for me where before I had to think about which keywords to use and combine and which filters to apply, describing what I'm looking for in plain text and then having the AI find it for me feels magical. Recently I wanted to find an artist that I heard in some old episode of the KEXP runcast (a running podcast), and I didn't remember anything except that it was rap with a kind of monotone voice a fast beat and a strong accent. Googles' agent asked a few clarifying questions and after a few rounds it found the artist for me, Genesis Uwusu. That's why I think Google will win in the AI assisted search market, they just have the best integration between fast and reasonably "smart" agents and high quality search data. Claude or ChatGPT are too slow and don't have fast enough data retrieval it seems, using them for search feels quite sluggish in comparison.

Apple RAM prices always had quite a bit of margin though, I think they charged around 4x the going market rate per GB (that said you can't fully compare their RAM to a loose DIMM stick). I was planning to pick up a new Mac Studio this autumn, now I'll have to see if I can afford it, though I have been spending 1,000 USD on LLM subscriptions in some months so I guess even a 10,000 USD Studio Mac amortizes quite fast if it allows me to run coding models locally.

Everyone always wants to charge as much as they possibly can, and if SK Hynix would be the only manufacturer prices would be 10x of what they are today. Especially new incumbents will not ruin the market prices as they have the highest upfront cost and their calculation of entering the market is probably based on the high prices that can be achieved. In the long run, more competition is still good as everyone ramps up production to profit more from the high prices and at some point supply will outpace demand and prices will fall (assuming no cartel / price fixing is involved).

It's the ultimate power user desktop system, in my opinion it even dwarfs MacOS in terms of how you can customize it and how it looks. It left Windows in the dust a long time ago in terms of functionality and usability (not that this would be particularly hard given how Windows has been degrading over the last decade). Everything is super snappy, smooth, built-in apps like Konsole and Dolphin are super polished, Konsole runs circles around the MacOS terminal app.

Of course running Linux on modern hardware is still a bit fraught with errors, though it has been getting much better. I run a current gen Thinkpad X9 Aura and apart from the webcam which has fundamental driver issues on all Linux kernels everything runs really well, power efficiency is also great at around 10W, not as good as a MacBook (which I also use daily) but close enough for me, and I still prefer Linux over MacOS any day.

Are you sure about that? You are saying that differentially private census data couldn't be used for gerrymeandering and advertisement while non differentially private data could? Hard to believe, I'm not an advertisement or gerrymeandering expert but I would assume people running ads or cutting up districts are mostly interested in aggregate statistics i.e. they won't care about single households? And I would assume they can rely on voter files, party databases etc... And to the contrary there are reports [1] that indicate differential privacy actually makes gerrymeandering analysis more difficult or impossible. So, not really an argument for differential privacy, discriminatory action can be equally well taken based on differentially private data as the government cares about groups not individuals and groups aren't protected by differential privacy. It seems people really fundamentally misunderstand what this technique can achieve and what it won't do.

1: https://pmc.ncbi.nlm.nih.gov/articles/PMC8494446/?utm_source...

No, there are dozens of articles discussing the mechanism and explaining the impact it had in different areas e.g. [1,2,3]. And the release mechanism wasn't just "add noise", far from it, you may read the original paper [4] to see how intricate it was, anyone wanting to make real use the resulting data would have needed to understand that approach in detail to work with the resulting data. The report of the national academies [3] is probably the most comprehensive analysis of the mechanism and the complications it introduced, so writing "it has always been inherently inaccurate" is just wrong, this new mechanism was way worse than just introducing unbiased sampling noise.

1: https://www.aeaweb.org/articles?id=10.1257%2Fpandp.20191107&... 2: https://www.science.org/doi/10.1126/sciadv.abk3283?utm_sourc... 3: https://www.nationalacademies.org/read/27150/chapter/14

4: https://hdsr.mitpress.mit.edu/pub/7evz361i/release/2

I think it should be noted that there was a lot of dissatisfaction from users of the census data as far as I know. So it's not been banned just for politicals sake or because they hate privacy... Some people I talked to in the privacy field even called the whole thing a total disaster and weren't shy to put blame on John Abowd who apparently pushed this through despite a lot of internal opposition and concerns. Not sure if that's true, but what is definitely true is that the way the data was released produced serious issues downstream as most researchers and statisticians that ingested the data weren't prepared for receiving noisy data values. Differential privacy was applied in a way such that many invariants that data users cared about weren't preserved, which was expected as it's not possible as you can't preserve all invariants and at the same time add meaningful noise to the data. The thing is, with such a differentially private data release you need to adapt all of the downstream analyses to take into account the exact mechanism the data was altered in. And since the census bureau used a very intricate mechanism that didn't just add Laplace noise to data values but instead relied on a multi-stage process that preserved some invariants but not others it was very difficult to even write routines to account for the changes being made to the data. They essentially asked of every data user to rewrite their whole analysis pipeline based on the exact disclosure mechanism that contained a large number of bespoke choices regarding which data invariants to preserve and basically produced a mix of noisy, synthesized data that was just really hard to reason about. I don't even know if there even would've been a way to do this better, but the fact is that not every small county or school district has top-tier statisticians at hand that can just read a whole monograph on differentially private synthesized census data and then hotpatch their existing analysis systems to work with that data.

I was a big fan of differential privacy but now I think it might be doing more harm than good, as I haven't seen a single case where it was applied successfully in a problem where it actually mattered, and it contributed strongly to discrediting and preventing a lot of work on other anonymization techniques as it was deemed the only way to preserve privacy by the research community, so showing up with enhancements to k-anonymity or any other noise mechanism not rooted in it was a sure way to get ridiculed and ignored. And it's just not a practical mechanism, even when it works for a single disclosure you always end up having to blow up the privacy budget to a ridiculous amount in order to keep disclosing statistics as otherwise you would for almost all real-world data run out of budget after a few publications.

So, for me it's a technique that works in the areas where it doesn't really matter (publishing highly aggregated statistics that pose almost zero privacy risk even without differential privacy) and doesn't work in other areas where it would actually matter (publishing fine-grained data about individuals or small groups). There are some niche use cases but in my view the privacy community has really overblown the importance of differential privacy by portraying it as the only way to reliably anonymize data.

BTW the German census bureau has an interesting approach to anonymization which they use for several decades already and so far I haven't heard of any cases of successful de-anonymization of the data, maybe the US bureau should have a look at that for their own needs.

Can you point to any "great" projects on Lovable that would actually be useful as full blown SaaS software tools? Stuff that has been written/prompted by non software experts?

Yeah Fable 5 is good but feels incremental and overhyped, also burned through my entire Cursor allowance in my Ultra plan in a single day. Ridiculous. They just want to create FOMO and appear mysterious so companies and users will feel so special for being allowed to use this model and pony up more money. After all they have to grow a few order of magnitude to pump their IPO valuation as much as possible, so I think this is just a strategy to justify their increased token pricing which starts to become absolutely insane. 10-20k per month per developer, do companies really think that's a good way to spend their IT budget? I assume 99 % of software shops wrtite run-of-the-mill web/mobile/desktop apps or some legacy backend APIs and CRUD code, you don't need a superintelligence to crank that stuff out. It sounds so ridiculous to have a model that supposedly can design biological weapons and then 99 % of users vibe code spaghetti Javascript with it. But the spice must flow!

Well you can just scale your AI employees up and down as much as you want. Companies already pay a large premium for freelancers just to be able to fire them on a whim, so spending 5-10k a month on something that more than doubles the productivity of a senior developer might be well worth it as you can just adapt spending based on your business needs. If you can deliver a feature that lets you write a 100k invoice with 10-20k of tokens within a month or have a senior dev crunch that out in 6 months instead I think it's clear who wins. It's all about money and the AI companies know that, they have their pricing down exactly to sit in the sweetspot where it hurts just enough that companies can still afford it but not enough that they would look for cheaper alternatives.

I mean even cheap keycaps won't wear out for many years for most people, so I don't think quality is a big factor. I got tons of keycaps from Ali Express which are just as good as the high quality stuff, in fact most of them are made on thre same machines...

So not sure if that was really the issue, people ordered keycaps because they liked the design, e.g. the Dasher MT3 set was super popular due to a similar one being used in the "Severance" show.

Yeah I mean we have a mechanism that can bypass AI models for log lines where we are pretty sure no PII is in there (kind of like smart caching using fuzzy template matching to identify things that we have seen before many times, as logs tend to contain the same stuff over and over with tiny variations e.g. different timestamps), so we only need to pass the lines where we cannot be sure there's nothing to the AI for inspection. And we can of course parallelize. Currently we use a homebrew CFR model with lots of tweaks and it's quite good but an LLM would of course be much better still and capture a lof of cases that would evade the simpler model.

This is really cool! I am trying to find a way to accelerate LLM inference for PII detection purposes, where speed is really necessary as we want to process millions of log lines per minute, I am wondering how fast we could get e.g. llama 3.1 to run on a conventional NVIDIA card? 10k tokens per second would be fantastic but even at 1k this would be very useful.