Mobile rendering of the post has some issues. Tables are overflowing and not responsive for example
1,011 karma · joined December 6, 2022
You can see here that it’s only given write access to specific directories: https://github.com/qwibitai/nanoclaw/blob/8f91d3be576b830081...
If they are running their production product as is, then of course whatever is built into the product is fine.
def make_pass@1_agent(agent, n):
def retry_agent(problem):
for attempt in range(n):
result = agent(problem)
if result.success:
return result
return result
return retry_agentBuilding multiple attempts into your agent is stretching the rules, even if technically it’s acceptable
71.2% puts it at 5th, which is 4 points below the leader (four points is a lot) and just over 1% lower than Anthropic’s own submission for Claude Sonnet 4 - the same model these guys are running.
But the top rated submissions aren’t running production products. They generally have extensive scaffolding or harnesses that were built *specifically for SWE bench*, which kind of defeats the whole purpose of the benchmark.
Take for example Refact which is at #2 with 74.4%, they built a 2k lines of code framework around their agent specifically for SWE bench (https://github.com/smallcloudai/refact-bench/). It’s pretty elaborate, orchestrating multiple agents, with a debug agent that kicks in if the main agent fails. The debug agent analyzes the failure and gives insights to the main agent which tries again, so it’s effectively multiple attempts per problem.
If the results can be reproduced “out-of-the-box” with their coding agent like they claim, it puts it up there as one of the top 2-3 CLI agents available right now.
A good wrapper has deep domain knowledge baked into it, combined with automation and expert use of the LLM.
It maybe isn’t super innovative but it’s a bit of an art form and unlocks the utility of the underlying LLM
It’s a radical statement that effectively denies the rights of millions of people to exist and is especially problematic given the historical context of the establishment of Israel.
The statement gets thrown around so much in certain circles that it’s gotten normalized. You’ve apparently lost sight of or never stopped to think what actually means, to the point where you’re providing it as an example of an innocent statement that got you banned for no reason. Taking this statement out of radical activist circles and into the real world won’t go well.
Take some time to educate yourself and reflect on what it actually means.
I input a bunch of completely made up words (Quastral Syncing, Zarnix Meshing, HIBAX, Bilxer) and used them in a sentence and the model zero-shotted perfect speech recognition!
It’s so counterintuitive for me that this would work. I would have bet that you have to provide at least one audio sample in order for the model to recognize a word it was never trained on.
Providing it to the model in text modality and it being able to recognize it in the audio modality must be an emergent property.
I like the ideas presented in the post but it’s too long and highly repetitive.
AI will happily expand a few information dense bullet points into a lengthy essay. But the real work of a strong writer is distilling complex ideas into few words.
From the PR: unit tests: what are they good for?
Answer: Personal opinion - writing unit testing is not fun. It becomes even less appealing as your codebase grows and maintaining tests becomes a time-consuming chore.
However, the benefits of comprehensive unit tests are real:
Reliability: They create a more reliable codebase where developers can make changes confidently
Speed: Teams can move quickly without fear of breaking existing functionality
Safe Refactoring: Code improvements and restructuring become significantly safer when backed by thorough tests
Living Documentation: Tests serve as clear documentation of your code's behavior:
They show exactly what happens for each input They present changes in a human-readable format: "for this input → expect this output" They run quickly and are easy to execute This immediate feedback loop is beneficial during development
o1 is very slow - like, you can go get a coffee while it generates a single test (if it doesn’t time out in middle).
o1-mini thought worked really well. It generated a good test and wasn’t noticeably slower than the other models.
My feeling is that o1-mini will end up being more useful for coding than o1, except for maybe some specific instances where you need very deep analysis
50% on its own doesn’t make this the current best choice for production. But I imagine this could become the new base model that all of the inference optimizations are applied to.
Wonder if it’s plug and play or if faster-whisper and others would need to reimplement from scratch?
Based on my understanding, only 1:20 passed the automated acceptance criteria (build, run, pass, increase coverage). Of those that made it through to the human review, “over 50% of the diffs submitted were accepted by developers” according to the paper
Saving $6M is key information that makes this story interesting. It’s buried all the way at the bottom of the first blog and is completely missing from the second blog which focuses specifically on the migration
Sounds bizarre until you remember that over 80% of servers are still on prem
There could very well be similar bias and shoddy statistics at play there
If an entry level employee is terrible, it will be obvious in a matter of days. It takes years for bad hires at the VP level and above to be recognized.