Basilisk has been removed from the Python typing conformance leaderboard

github.com

4 points·pythontyper··2 comments

"Despite our best efforts to enforce accuracy, we relied too heavily on AI to validate the work we did, and we focused too heavily on conformance results instead of thoroughly testing real world scenarios. It looks as though we didn't pick up code that clearly should have only been just a placeholder and that code proliferated. This should have been predictable and I take full responsibility for letting this happen.

I am sorry for breaking trust, especially when the whole aim of Basilisk as a tool is to build trust. We have an absolute commitment to building the trust of the Python community so requesting this removal is a necessary step until we can work through the issues."

Though unfortunate that this happened, it's commendable the Basilisk author proactively identified and raised the problem & took responsibility for missing it. It's really easy to miss problems that AI introduces, and that's something I suspect most programmers are just learning how to deal with -- myself included.

I'm not impressed, it's an AI-generated apology. A quick skim of their code or commit history makes that obvious that this project's development methodology was clearly flawed, and only someone who turned their brain off would have advertised those results in the first place.

Reply on news.ycombinator.com