Hacker News Reader

>I started using Spark 1.2 for development because if you're willing to let Meta train on your data it was dirt cheap

its free on opencode and i use it for personal projects. most of my personal projects are AI generated since its personal projects. nothing important are on them. it is hilarious if Meta is training their AI model with AI generated code.

The useful training data is when you clarify your intent, when you tell the model a different approach would be better, when you consistently refactor towards Y and away from X, and so on. The training data isn’t the code, it’s the session transcript. (Anthropic would call this a “distillation attack” against their model, but in this case the model is you!)

I would imagine your interactions with it are more important than the output.

Funny. I use it through Opencode Go which gives more use than I can use, but didn't realize it was actually free on Zen. Will switch to that I guess

Training on ai generated content is how the models got a big jump in capability

curious about this. how do we know this?

i thought it was because anthropic bought a bunch data from mercor

Every lab trains their models with AI generated code at this point.

Hopefully, 'validated' AI code

What do you think you're doing when you accept an edit, press thumbs up, or don't ask for modifications after an edit.

Thats not exactly 'validated'. Feels very noisy, it is not a good bar for either - does this code do what the user actually asked - is this code actually 'good'

There would be so many examples of coding projects that these models began or attempted to work in, that were abandoned because the models were floundering.

I would imagine the labs have some decent ways to produce novel requirements and then actually validate they are met, without the noisiness of implicit human feedback.

That said, the more I think about it, you are right, there's probably also very good ways to extract signal for all these sessions.

This is exactly what RLVR is, and the reason that models have improved so much at verifiable domains like coding and math while not so much on unverifiable ones like writing and UI design.

Would be cool if there was a benchmark to evaluate the “tool-like” quality of a model - its capability to quickly, cheaply, accurately, do exactly as it is asked.

This is interesting because I have transitioned to where I use SOT models.. but I kind of use them like employees that I can delegate to. I still review code.

However, I now literally say.. "Here is my objective and here is a starting point for documentation. Research this and build up a plan."

This can be very company specific, like migration from one framework to another in house infrastructure framework. I'm spending my time figuring out how the plan should be chopped so I can have confidence in the parts and not overwhelmed. I don't want a tool, I want a model that can stitch resources together into a plan. That type of model is in a whole other ballpark.

I believe that you can still use 5.3 Codex in the eponym CLI tool, the "Spark" fast version. I hope that it will lighten your day! :-)

If it's a mistake, it should course-correct.

I agree that some of the smarter models are actually worse. I hope they take a model that's good enough--there are many--and just try to get it chatjimmy.ai speed.

I have to think that's the future, somehow, and I'm really excited about it.

"If it's a mistake, it should course-correct"

Maybe, or maybe not. The thing is, that "mistake" isn't something that is generally valid. For example, the enshittification of Google Search through the last 15 years seems to be a mistake --- but perhaps not from the money-making point of view of Google Shareholders. Likewise the enshittification of reddit --- we nerdy users see it as a mistake. But for them this intended enshittification probably increased revenue.

It's the money, always the money! PR-speak like "customer satisfaction is our highest goal" is, like most PR-speak, a blatant lie.

And so it can very well be the case that for coders the frontier models get worse, but they get better for other applications --- and that all of this is just driven by "how can we capitalize the most out of it", not satisfaction levels of programmers.

I do think the timeline of web search getting fixed (yes, google is ASS) took a lot longer than I hoped, but seems like it's finally here.

That being said, it's not a fair comparison to talk about 2011 google vs the AI market right now. There's so many labs I can't track them all, neck and neck in the lead. There was one true web search.

And this is a more tangible quality difference, too. It's hard to know what a google search didn't return, especially as a layperson. It's not hard to see the model underperforming.