HNHacker News
TopNewBestAskShowJobs

cjsaltlake

57 karma · joined April 30, 2018

submissionscomments
cjsaltlake··on GPT-5.5
Normal people definitely will not benefit from this by default.
cjsaltlake··on SWE-bench Verified no longer measures frontier coding capabilities
SWE-bench was created to replace olympiad coding benchmarks. I think past olympiad coding benchmarks were much worse representative of real-world coding than something like SWE-bench, which is derived from real units of labor.

Further, olympiad style benchmarks are arguably easier to contaminate / memorize unless you refresh it regularly; but that goes for SWE-bench too.

cjsaltlake··on SWE-bench Verified no longer measures frontier coding capabilities
But, that's an enormous source of coding productivity, and it's why Anthropic is worth billions... The reason SWE-bench has been so successful and useful for coding is that software engineering has a ton of tradition and infrastructure for making and using automated tests.
cjsaltlake··on SWE-bench Verified no longer measures frontier coding capabilities
code clash I think would be quite hard to game or contaminate unintentionally; considering that models need to compete against one another
cjsaltlake··on SWE-bench Verified no longer measures frontier coding capabilities
I suggest reading the Mythos report's discussion on SWE-bench and contamination. I think it's fairly convincing that you can account for contamination and still trust SWE-bench numbers on models that aren't over-optimized for it.
cjsaltlake··on SWE-bench Verified no longer measures frontier coding capabilities
If you read the mythos report, in which they discuss and account for contamination substantially, it still suggests that performance on SWE-bench verified is meaningful. Benchmarks, including SWE-bench can absolutely be gamed, but if you're not explicitly benchmaxxing, improving on SWE-bench still measures model improvements, at least up to the level of Mythos.
cjsaltlake··on GPT-5.5
Labor replacing devices means nobody works in those fields anymore. If AI can do this for every field, nearly no one will need to work in any field. We'll have a giant fully automated resource-extraction machine.
cjsaltlake··on Top model scores may be skewed by Git history leaks in SWE-bench
We absolutely did not.
cjsaltlake··on Top model scores may be skewed by Git history leaks in SWE-bench
I'm also on the SWE-bench team. This was simply a classic bug. We had code before that we believed was sufficient to hide / remove future GitHub history and it turns out it was not. We've patched it.
cjsaltlake··on Modernity is incompatible with planetary limits: a PLAN for the future
If trends continue, there will be 10 billion Amish in 100 years.
cjsaltlake··on Ask HN: Has the war in Ukraine changed your view on working on Defence projects?
I think one should be consequentialist about all things.
cjsaltlake··on We don't do that here (2017)
The only people I hate more than woke people are anti-woke people.
cjsaltlake··on We don't do that here (2017)
Companies are entities with political interests like any other. The only way you can curtail their influence politically is through regulation. It is reasonable for entities to pursue their goals through any legal means.
cjsaltlake··on .gay Generic Top Level Domain(gTLD.) is now open for public
The community for straight people is too disparate to be meaningful. .bi is interesting though..
cjsaltlake··on Apple supplier Foxconn to invest $1B in India, sources say
India is a single country, with a relatively well established national identity (obviously not perfect!, but enough for people to work and live in the same area), and a relatively stable government. Most African countries are still in flux in a huge way, and are just barely starting to establish stable national identities after decades of civil war.
cjsaltlake··on California sues Cisco alleging discrimination based on India’s caste system
Additionally, there's north/south discrimination, with greater antipathy from southern Indians against northern Indians.
cjsaltlake··on California sues Cisco alleging discrimination based on India’s caste system
It's not at all highly correlated, and skin color certainly wouldn't trump real caste affiliation.
cjsaltlake··on 16-inch MacBook Pro
~19:50