Right now they want to prevent the US labs from gaining any sort of self-reinforcing oligopoly on the space, and to let the ecosystem in China flourish.
That will all die sooner or later.
17,752 karma · joined June 4, 2016
Right now they want to prevent the US labs from gaining any sort of self-reinforcing oligopoly on the space, and to let the ecosystem in China flourish.
That will all die sooner or later.
Other things don't though.
The per-repo config is in `.git/config`, so that can still not be checked into the repo itself, unless I'm missing something?
So not very useful at all...
I get the security implications, but there could be a checked in `$REPO/.githooks`, and a prompt asking to allow running those hooks , with an approval marker being stored inside `.git/`.
I found it works very well in once-off scenarios, but the specs often drift from the implementation. Even if you let the model update the spec at the end, the next few work items will make parts of it obsolete.
Maybe that's exactly the goal that "codespeak" is trying to solve, but I'm skeptical this will work well without more formal specifications in the mix.
* This isn't a language, it's some tooling to map specs to code and re-generate
* Models aren't deterministic - every time you would try to re-apply you'd likely get different output (without feeding the current code into the re-apply and let it just recommend changes)
* Models are evolving rapidly, this months flavour of Codex/Sonnet/etc would very likely generate different code from last months
* Text specifications are always under-specified, lossy and tend to gloss over a huge amount of details that the code has to make concrete - this is fine in a small example, but in a larger code base?
* Every non-trivial codebase would be made up of of hundreds of specs that interact and influence each other - very hard (and context - heavy) to read all specs that impact functionality and keep it coherent
I do think there are opportunities in this space, but what I'd like to see is:
* write text specifications
* model transforms text into a *formal* specification
* then the formal spec is translated into code which can be verified against the spec
2 and three could be merged into one if there were practical/popular languages that also support verification, in the vain of ADA/Spark.
But you can also get there by generating tests from the formal specification that validate the implementation.
This puts much more work on the compiler development side, but it's a great boon for the ecosystem.
To be fair, zig is pre 1.0, but Zig is also already 8 years old. Rust turned 1.0 at ~ 5 years, I think.
At least 100k context without huge degradation is important for coding tasks. Most "I'm running this locally" reports only cover testing with very small context.
For Tesla it was "electrify transport to end dependence on fossil fuels and save the planet", for SpaceX it was "save humanity by becoming a multi-planetary species".
With how much he talked about it, he did probably actually believe in Mars.
But now both of these ideals have come into conflict with his newfound political affiliations, so they have to be dropped.
It's essentially just an atomic pointer that can be swapped out.
Many things that would only be caught at runtime in other languages are caught at compile time in Rust, making coding agents iterate until things compile and work well.
Rust also has great error messages, which help the agents in fixing compilation errors.
BUT it is not good at all at tool calling and agentic workflows, especially compared to the recent two mini-generations of models (Codex 5.2/5.3, the last two versions of Anthropic models), and also fell behind a bit in reasoning.
I hope they manage to improve things on that front, because then Flash would be great for many tasks.
Codex quite often refuses to do "unsafe/unethical" things that Anthropic models will happily do without question.
Anthropic just raised 30 bn... OpenAI wants to raise 100bn+.
Thinking any of them will actually be restrained by ethics is foolish.
Research and training are the cost sinks.
With AI, the correct approach is to think more like a software architect.
Learning to plan things out in your head upfront without to figure things out while coding requires a mindset shift, but is important to work effectively with the new tools.
To some this comes naturally, for others it is very hard.
That said ... I do think Codex 5.2 was the best coding model for more complex tasks, albeit quite slow.
So very much looking forward to trying out 5.3.
From the post I expected that the tasks were about analysing traces, but all the tasks in the repository are about adding instrumentation to code!
Some of the instructions don't give any guidance how to do it, some specify which libraries to use.
"Use standard OTEL patterns" ... that's about as useful as saying "go write some code". There are a lot of ways to do instrumentation....
I'd be very curious HOW exactly the models fail.
Are the test sets just incredibly specific about what output they except, and you get a lot of failures because of tiny subtle mismatches? Or do they just get the instrumentation categorically wrong?
Also important: do the models have access to a web search tool to read the library docs? Otel libraries are often complicated to use... without reading latest docs or source code this would be quite tricky.
Some models have gotten better at adding dependencies, installing them and then reading the code from the respective directory where dependencies get stored, but many don't do well with this.
All in all, I'm very skeptical that this is very useful as a benchmark as is.
I'd be much more interested in tasks like:
Here are trace/log outputs , here is the source code, find and fix the bug.
So a 100bn deficit out of 800bn total US imports.
The deficit is there, but it's not nearly as lopsided as some reporting would have you believe.
Massive endeavor for a lot of setups.
The existing AI plugins for neovim aren't great.
And their logo is, lo and behold ... a hand!
Most popular programming languages are optimized for human convenience, not for correctness! Even most of the popular typed languages (Java/Kotlin/Go/...) have a wide surface area for misuse that is not caught at compile time.
Case in point: In my experience, LLMs produce correct code way more regularly for Rust than for Js/Ts/Python/... . Rust has a very strict type system. Both the standard library and the whole library ecosystem lean towards strict APIs that enforce correctness, prevent invalid operations, and push towards handling or at least propagating errors.
The AIs will often write code that won't compile initially, but after a few iterations with the compiler the result is often correct. Strong typing also makes it much easier to validate the output when reviewing.
With AIs being able to do more and more of the implementation, the "feel-good" factor of languages will become much less relevant. Iteration speed is not so important when parallel AI agents do the "grunt work". I'd much rather wait 10 minutes for solid output rather than 2 minutes for something fragile.
We can finally move the industry away from wild-west languages like Python/JS and towards more rigorous standards.
Rust is probably the sweet spot at the moment, thanks to it being semi-popular with a reasonably active ecosystem, sadly I don't think the right language exists at the moment.
What we really want is a language with a very strict, comprehensive type system with dependent types, maybe linear types, structured concurrency, and a built-in formal proof system.
Something like ADA/Spark, but more modern.