helloplanets

We need more of these kinds of tightly time controlled challenges for LLMs.

Isn't this exactly the point the original post is making?

Honestly thought this was some sort of satire on corporate ass-kissing at first.

Why is this flagged?

For the visual learners, here's a classic intro to how LLMs work: https://bbycroft.net/llm

Even if you turn the temperature down to 0, it's not deterministic. Floating points are messy. If there is even a tiny difference when it comes to the order of operations on the actual GPU that's running the billions of parallelized floating point operations over and over, it's very possible to end up with changing top probability logits.

This guy is literally smirking while talking about this. It's like he's proud of himself for being able to produce doublespeak so effectively.

He wasn't saying that both of the models suck, but that the heuristics for measuring model capability suck

> Let's say your organization wants to add "dark mode" to your site. How does that happen? A site-wide feature usually requires several people to hash out the concerns and explore costs vs. benefits. Does the UI theming support dark mode already? Where will users go to toggle dark mode? What should the default be? If we change the background color we will need to swap the font colors. What about borders and dividers? What about images? What about the company blog, and the FAQ area, which look integrated but run on a different frontend? What about that third-party widget with a static white background?

Only one or two of those questions are actually related to programming. (Even though most developers wear multiple hats.) If an organization has the resources to have a six person meeting for adding dark mode, I'd sure hope at least one of them is a designer and knowledgeable on UX. Because most of those questions are ones that they should bring up and have an answer for.

China is not just one company... And Google has been recently putting out more than any one of them AFAIK.

The tortoise metaphor doesn't really fit for Google's current AI pace. They've been shipping more than any other company in the field for the last year.

And when programming with agentic tools, you need to actively push for the idea to not regress to the most obvious/average version. The amount of effort you need to expend on pushing the idea that deviates from the 'norm' (because it's novel), is actually comparable to the effort it takes to type something out by hand. Just two completely different types of effort.

There's an upside to this sort of effort too, though. You actually need to make it crystal clear what your idea is and what it is not, because of the continuous pushback from the agentic programming tool. The moment you stop pushing back, is the moment the LLM rolls over your project and more than likely destroys what was unique about your thing in the first place.

Synopsis of Shakespeare's Sonnet 4:

> The poet returns to the idea of beauty as treasure that should be invested for profit. Here, the young man’s refusal to beget a child is likened to his spending inherited wealth on himself rather than investing it or sharing it generously.

https://www.folger.edu/explore/shakespeares-works/shakespear...

Agreed. Overengineering and premature optimization are the root of all crud.

So, your point is that programmers identify the unexpected edge cases through the act of taking their time writing the code by hand. From my experience, it takes a proficient developer to actually plan their code around future issues from separate processes.

I think that it's mistaken to think that reasoning while writing the code is at all a good way to truly understand what your code is doing. (Without implying that you shouldn't write it by hand or reason about it.) You need to debug and test it thoroughly either way, and basically be as sceptical of your own output as you'd be of any other person's output.

Thinking that writing the code makes you understand it better can cause more issues than thinking that even if you write the code, you don't really know what it's doing. You are merely typing out the code based on what you think it should be doing, and reasoning against that hypothesis. Of course, you can be better or worse at constructing the correct mental model from the get go, and keep updating it in the right direction while writing the code. But it's a slippery slope, because it can also go the other way around.

A lot of bugs that take unreasonably long for junior-mid level engineers to find, seem to happen because: They trust their own mental model of the code too much without verifying it thoroughly, create a hypothesis for the bug in their own head without verifying it thoroughly, then get lost trying to reason about a made up version of whatever is causing the bug only to come to the conclusion that their original hypothesis was completely wrong.

The writeup is a bit contrived in my opinion. And sort of misrepresenting what users can do with tools like Claude Code.

Most coding assistant tools are flexible to applying these kinds of workflows, and these sorts of workflows are even brought up in Anthropic's own examples on how to use Claude Code. Any experienced dev knows that the act of specifically writing code is a small part of creating a working program.

Haven't LLMs been good at bouncing off ideas since GPT-4? Coming up with feature ideas doesn't seem to have much to do with programming power tools.

But sending off a SotA LLM agent to do research for 30 min+ to gather information, related concepts, products, etc, on any idea is insanely effective for sure. And something that hasn't been possible for all that long, at this level of quality.

I guess it could be generalized to filling up the context window with any token, but just making sure none of the tokens repeat.

An interesting twist could be making sure a specific token is an anagram of the token N tokens back. This could possibly measure how much a model can actually plan forwards.

Why not both?

Yea, there's a bunch of easter eggs. And then there's the table for the taxonomy tree.

For anyone wondering: This is based on basic text parsing and a key-value lookup table, no AI involved whatsoever.

Here's the table: https://rose.systems/animalist/lower_title_to_id.js

An influencer, who could more than likely blow 99% of us out of the water when it comes to programming by hand with absolutely no tooling, at whatever level of abstraction. Same probably applies to Antirez.

So, what's your take on Andy Warhol, or sampling in music?

How big of a Carmack fan are you really, if you don't know one of his most well known takes on programming? (And you definitely don't need to be a fan.) Carmack has been heavily in favor of leveraging power tools since way back.

Direct quote from the man himself:

> I will engage with what I think your gripe is — AI tooling trivializing the skillsets of programmers, artists, and designers.

> My first games involved hand assembling machine code and turning graph paper characters into hex digits. Software progress has made that work as irrelevant as chariot wheel maintenance.

> Building power tools is central to all the progress in computers.

> Game engines have radically expanded the range of people involved in game dev, even as they deemphasized the importance of much of my beloved system engineering.

> AI tools will allow the best to reach even greater heights, while enabling smaller teams to accomplish more, and bring in some completely new creator demographics.

> Yes, we will get to a world where you can get an interactive game (or novel, or movie) out of a prompt, but there will be far better exemplars of the medium still created by dedicated teams of passionate developers.

> The world will be vastly wealthier in terms of the content available at any given cost.

https://x.com/ID_AA_Carmack/status/1909311174845329874

Care to link your sources? At least one of the studies that got attention here was basically done with a bunch of programmers who had no prior experience with the tools.

Shadcn most likely contains a lot of LLM generated code. Isn't it owned by Vercel these days?

I'd say that it's a different type of learning process, where even a good resource doesn't help as much as it would with a traditional programming language. Sort of like you can't get very good at writing by just reading a ton of instructional books about it.

10^15 lines of code is a lot. We would pretty quickly enter the realm of it not having much to do with programming and more about just treating the LOC count as an amount of memory allocated to do X.

How much resemblance does the information in the conditionals need to have with the actual input, or can they immediately be transformed to a completely separate 'language' which simply uses the string object as its conduit? Can the 10^15 lines of code be generated with an external algorithm, or is it assumed that I'd written it by hand given an infinitely long lifespan?

I'd imagine 99% of ChatGPT users see the app as the former. And then the rest know how to turn the memory off manually.

Either way, I think memory can be especially sneakily bad when trying to get creative outputs. If I have had multiple separate chats about a theme I'm exploring, I definitely don't want the model to have any sort of summary from those in context if I want a new angle on the whole thing. The opposite: I'd rather have 'random' topics only tangentially related, in order to add some sort of entropy in the outout.