> I genuinely believe Groq is a fraud. There is no way their private inference cloud has positive gross margins.
> Llama 3.1 405B can currently replace junior engineers
I'd like more exposition on these claims.
> Llama 3.1 405B can currently replace junior engineers
I'd like more exposition on these claims.
I am not a programmer, and I know C and Python at about a 1 day crash course level (not much at all).
However with sonnent I was able to be handheld all the way from downloading android studio to a functional app written in kotlin, that is now being used by employees on the floor.
People can keep telling themselves that LLMs are useless or maybe just helpful for quickly spewing boilerplate code, but I would heed the warning that this tech is only going to improve and already helping people forgo SWE's very seriously. Sears thought the internet was a cute party trick, and that obviously print catalogs were there to stay.
I then wrote a web AudioWorklet for playing PCM in 3 minutes, which complied to the same interface as my Mac/iOS/Android versions, ex. Setting sample rate, feedback callback, etc. I have no idea what an AudioWorklet is.
Two days ago, I stubbed out my implementation of OpenAI's web socket based realtime API, 1400 LOC over 2 days, mostly by hand while grokking and testing the API. In 32 minutes, I had a brand spanking new batch of code, clean, event-based architecture, 86% test coverage. 1.8 KLOC with tests.
In all of these cases, most I needed to do was drop in code files and say, nope wrong a couple times to sonnet, and say "why are you violating my service contract and only providing an example solution" to o1.
Not llama 3.1 405B specifically, I haven't gone to the trouble of running it, but things turned some sort of significant corner over the last 3 months, between o1 and Sonnet 3.5. Mistakes are rare. Believable 405B is on that scale, IIRC it went punch for punch with the original 3.5 Sonnet.
But I find it hard to believe a Google L3, and third of L4s, (read: new hires, or survived 3 years) are that productive and sending code out for review at a 1/5th of that volume, much less on demand.
So insane-sounding? Yes.
Out there? Probably, I work for myself now. I don't have to have a complex negotiation with my boss on what I can use and how. And I only saw this starting ~2 weeks ago, with full o1 release.
Wrong? Shill? Dilletante?
No.
I'm still digesting it myself. But it's real.
It's less to do with junior/senior/etc.. and more to do with the types of problems you are tackling.
It's a real system, not kindergarten "look at the React Claude Artifacts did, the button makes a POST request!"
The 1500 loc websocket / session management code it refactored and tested touches on nearly every part of the system (i.e. persisting messages, placing search requests, placing network requests to run a chat flow)
Also, it's worth saying this bit a bit louder: the "just throwing files in" I mention is key.
With that, the quality you observed being in reverse is the distinction: with o1 thinking, and whatever Sonnet's magic is, there's a higher payoff from working a larger codebase.
For example, here, it knew exactly what to do for the web because it already saw the patterns iOS/Android/macOS shared.
The bend I saw in the curve came from being ultra lazy one night and seeing what would happen if it just had all the darn files.
This is powerful and significant but I think we need to ground ourselves on what a skilled programmer means when he talks about solving problems.
That is, honestly ask: What is the level of skill a programmer requires to build what you've described? Mostly, imo, the difficulties in building it are in platform and API details not in any fundamental engineering problem that has to be solved. What makes web and Android programming so annoying is all the abstractions and frameworks and cruft that you end up having to navigate. Once you've navigated it, you haven't really solved anything, you've just dealt with obstacles other programmers have put in your way. The solutions are mostly boilerplate-like and the code I write is glue.
I think the definition of "junior engineer" or "simple app" will be defined by what LLMs can produce and so, in a way, unfortunately, the goal posts and skill ceiling will keep shifting.
On the other, hand, say we watch a presentation by the Lead Programmer at Naughty Dog, "Parallelizing the Naughty Dog Engine Using Fibers"[^0] and ask the same questions: what level of skill is required to solve the problems he's describing (solutions worth millions of dollars because his product has to sell that much to have good ROI):
"I have a million LOC game engine for which I need to make a scheduler with no memory management for multithreaded job synchronization for the PS4."
A lot of these guys, if you've talked to them, are often frustrated that LLMs simply can't help them make headway with, or debug, these hard problems where novel hardware-constrained solutions are needed.
---
...I guess the first?...
...But not really?
I'm not writing GPU kernels or operating system task schedulers, but I am going to some pretty significant lengths to be running ex. local LLM, embedding model, Whisper, model for voice activity detection, model for speaker counting, syncing state with 3 web sockets. Simultaneously. In this case, Android and iOS are no valhalla of vapid-stackoverflow-copy-pasta-with-no-hardware-constraints, as you might imagine.
And the novelty is, 6 years ago, I would have targeted iOS and prayed. Now I'm on every platform at top-tier speeds. All that boring tedious scribe-like stuff that 90% of us spend 80% of our time on, is gone.
I'm not sure there's very many people at all who get to solve novel hardware-constrained problems these days, I'm quite pleased to brush shoulders with someone who brushes shoulders with them :)
Thus, smacks more of no-true-scotsman than something I can chew on. Productivity gains are productivity gains, and these are no small productivity gain in a highly demanding situation.
I wasn't making a judgement about you or your work, after all I don't know you. I was commenting within the context of an app that you described for which an LLM was useful, relative to the hard problems we'll need help with if we want to advance technology (that is, make computers do more powerful things and do them faster). I have no idea if you're a true Scotsman or not.
Regardless: over the coming years we'll find out who the true Scotsmen were, as they'll be hired to do the stuff LLMs can't.
It's challenging because working to get a stable set of requirements requires a lot of communication with end users, stakeholders, etc. Then, predicting what they actually mean when implementing said requirements. Then, demoing the software and translating their non-technical understanding and comments back into new requirements (rinse and repeat).
If a tool can help program some of those requirements faster, as long as it meets security and functional standards, and is maintainable, it's not a big deal whether a junior dev is working with Stack Exchange or Claude, IMO. But I do want that dev to understand the code being committed, because otherwise security bugs and future maintenance headaches creep in.
I fully believe recursive application of current SOTA LLMs plus some deployment framework can replace most software engineers who work in the cornfields.
There is a point where in some sense it is less effort to just write the thing yourself. This is the argument against micro-libraries as seen in NPM as well, but my point is that the threshold of complexity for "write it yourself" instead of reaching for a premade thing changes over time.
As languages, compilers, tab-complete, refactoring, and AI assistance get better and better, eventually we'll reach a point where the human race as a whole will be spitting out code at an unimaginable rate.
The logical conclusion is that projects will balloon with code pushing LLMs to their limit, and this massive amount is going to contain more bugs and be more costly to maintain.
Anecdotally, supposedly most devs are using some form of AI for writing code, and the software I use isn't magically getting better (I'm not seeing an increased rate of features or less buggy software).
Cranking out yet more code, though, is not difficult (and junior programmers are cheap). LLMs do truly produce code like a (very bad) junior programmer: when trying to make unit tests, it takes the easiest path and makes something that passes but won't catch serious regressions. Sometimes I've simply reprompted it with "Please write that code in a more proper, Pythonic way". When it comes to financial calculations around dates, date intervals, rounding, and so on, it often gets things just ever-so-slightly wrong, which makes it basically useless for financial or payroll type of applications.
It also doesn't help much with the main bulk of my (paid) work these days, which is migrating apps from some old platform like vintage C#-x86 or some vendor thing like a big pile of Google AppScript, Jotform, Xapier, and so on into a more maintainable and testable framework that doesn't depend on subscription cloud services. So far I can't find a way to make LLMs productive at all at that - perhaps that's a good thing, since clients still see fit to pay decently for this work.
My guess is you're thinking I'm writing duplicative code for the hell of it, instead of just using the CLI tool - no. I can't run arbitrary binaries, at all, on at least 4 of the 6 platforms.
Beyond that, that's why we test.
Here you go - https://pastebin.com/8zdMDEnG
I'd also like to point out[^1] --- meaning, I gave it my original code, in toto. So of course you'll see ex. comments. Not sure what else contributed to your analysis, that's where some clarity could help me, help you.
[^1](https://news.ycombinator.com/item?id=42421900) "I stubbed out my implementation of OpenAI's web socket based realtime API, 1400 LOC over 2 days, mostly by hand while grokking and testing the API. In 32 minutes, I had a brand spanking new batch of code, clean, event-based architecture, 86% test coverage. 1.8 KLOC with tests."
[^2](https://news.ycombinator.com/newsguidelines.html) "Be kind...Converse curiously; don't cross-examine."
[^3](https://news.ycombinator.com/newsguidelines.html) "Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith."
i went to a groq event and one of their engineers told me they were running 7 racks!! of compute per (70b?) model. that was last year so my memory could be fuzzy.
iirc, groq used to be making resnet-500? chips? the only way such an impressive setup makes any kind of sense (my guess) would be they bought a bunch of resnet chips way back when and now they are trying to square peg in round hole that sunk cost as part of a fake it till you make it phase. they certainly have enough funding to scrap it all and do better... the question is if they will (and why they haven't been able to yet)
Already through the gapevine I'm hearing that H100s and B100s have to be replaced more often.... than you'd want? I suspect people are mum about it otherwise they might lose sweetheart discounts from nvidia. I can't imagine that cerebras, even with their extreme engineering of their cooling system, have truly solved cooling in a way that isn't a pain in the ass (otherwise they wouldn't have the clause?) and if I were building a datacenter I would be very worried about having to do annoying and capital intensive replacements.