eeeeeeehio

Is "proof of vulnerability" a marketing term, or do you actually claim that XBOW has a 0% false positive rate? (i.e. "all" reports come with a PoV, and this PoV "proves" there is a vulnerability?)

This doesn't say anything about many false positives they actually have. Yes, you can write other programs (that might even invoke another LLM!) to "check" the findings. That's a very obvious and reasonable thing to do. But all "vulnerability scanners", AI or not, must take steps to avoid FP -- that doesn't tell us how well they actually work.

The glaring omission here is a discussion of how many bugs the XBOW team had to manually review in order to make ~1k "valid" submissions. They state:

> It was a unique privilege to wake up each morning and review creative new exploits.

How much of every morning was spent reviewing exploits? And what % of them turned out to be real bugs? These are the critical questions that are (a) is unanswered by this post, and (b) determine the success of any product in this space imo.

I agree that peer review can be a strong filter, but it's a filter for claims and evidence that sound true. CS papers can and do hide important details in the code (details which, I argue, would get a paper rejected if they were stated in the paper).

Regardless of the strength of the filter, if the filter's inputs are just "the paper", but the claims depend on the details in another artifact (i.e. the code), how can we argue that peer review filters for the truth?

I don't understand why took the time to leave (three?) personal attacks, rather than just provide your perspective? I'm willing to acknowledge that my opinion has limitations (I think it mainly applies to CS-adjacent fields with empirical performance evaluations, and where experiments can easily be independently verified).

I would be interested to hear other perspectives.

I agree with you. But what part of the blog post, or the peer review process in general, do you think ensures that only true ideas get in front of eyeballs?

I can write anything I want in the paper, but at the end of the day my experiments could do something slightly (or completely) different. Where are reviewers going to catch this?

It's not too difficult to state any idea, even a surprising one. But often, papers with surprising ideas (or maybe the right thing to say is surprising results?) turn out to be wrong!

I think it's still the case that there's lots of ideas that (if they worked!) would be surprising. Anyone can state outlandish ideas in a paper -- imo the contribution is proving (e.g. with sound "experiments", interpreted broadly) that they actually work. Unfortunately, I think clarity of writing matters more to reviewers than the soundness of your experiments. I think in CS this could very well change if the reviewers willed it (i.e. require artifact submission with the paper, and allow papers to be rejected for faults in the artifact)

Academics seem to have this fixation on "ideas":

> And it’s not just a pace thing, there’s a threshold of clarity that divides learned nothing from got at least one new idea.

But these days, ideas are quite cheap: in my experience, most researchers have more ideas than students to work on them. Many papers can fit their "core idea" in a tweet or two, and in many cases someone has already tweeted the idea in one form or another. Some ideas are better than others, but there's a lot of "reasonable" ideas out there.

Any of these ideas can be a paper, but what makes it science can't just be the fact that it was communicated clearly. It wouldn't be science unless you perform experiments (that accurately implement the "idea") and faithfully report the results. (Reviewers may add an additional constraint: that the results must look "good".)

So what does science have to do with reviewers' fixation on clarity and presentation? I claim: absolutely nothing. You can pretty much say whatever you want as long as it sounds reasonable and is communicated clearly (and of course the results look good). Even if the over-worked PhD student screws up the evaluation script a bit and the results are in their favor (oops!), the reviewers are not going to notice so long as the ideas are presented clearly.

Clear communication is important, but science cannot just be communicating ideas.

>The real test of a paper is whether it is cited and built upon by other scientists after publication. Many papers are published and then forgotten, or found to be flawed and not used any more.

This does seem true, but this forgets the downstream effects of publishing flawed papers.

Future research in this area is stymied by reviewers who insist that the flawed research already solved the problem and/or undermines the novelty of somewhat similar solutions that actually work.

Reviewers will reject your work and insist that you include the flawed research in your own evaluations, even if you’ve already pointed out the flaws. Then, when you show that the flawed paper underperforms every other system, reviewers will reject your results and ask you why they differ from the flawed paper (no matter how clearly you explain the flaws) :/

Published papers are viewed as canon by reviewers, even if they don’t work at all. It’s very difficult to change this perception.

> No set of a bit different results or missing exact hyperparameter settings really invalidates the value of the aforementioned research.

If this is the case, the paper should not include a performance evaluation at all. If the paper needs a performance evaluation to prove its worth, we have every right to question the way that evaluation was conducted.

> by-and-large catches the most glaring faults.

I did not dispute that peer review acts as a filter. But reviewers are not reviewing the science, they are reviewing the paper. Authors are taking advantage of this distinction.

> if only for the simple reason that putting out incorrect research gives an easy layup for competing groups to write a follow-up paper that exposes the flaw.

You can’t make a career out of exposing flaws in existing research. Finding a flaw and showing that a paper from last year had had cooked results gets you nowhere. There’s nowhere to publish “but actually, this technique doesn’t seem to work” research. There’s no way for me to prove that the ideas will NEVER work —- only that their implementation doesn’t work as well as they claimed. Authors who claim that the value is in the ideas should stick to Twitter, where they can freely dump all of their ideas without any regard for whether they will work or not.

And if you come up with another way of solving the problem that actually works, it’s much harder to convince reviewers that the problem is interesting (because the broken paper already “solved” it!)

> in most cases the core contribution of the research paper is not the code, but some set of ideas that together describe a novel way to approach the tackled problem

And this novel approach is really only useful if it outperforms existing techniques. “We won’t share the code but our technique works really well we promise” is obviously not science. There is a flood of papers with plausible techniques that look reasonable on paper and have good results, but those results do not reproduce. It’s not really possible to prove the technique “wrong”, but the burden should be on the authors to provide proof that their technique works and on reviewers to verify it.

It’s absurd to me that mathematics proofs are usually checked during peer review, but in other fields we just take everyone at their word.

Peer review is not designed for science. Many papers are not rejected because of an issue with the science -- in fact, reviewers seldom have the time to actually check the science! As a CS-centric example: you'll almost never find a reviewer who reads a single line of code (if code is submitted with the paper at all). There is artifact review, but this is never tied to the acceptance of the paper. Reviewers focus on ideas, presentation, and the presented results. (And the current system is a good filter for this! Most accepted papers are well-written and the results always look good on paper.) However, reviewers never take the time to actually verify that the experiment code matches the ideas described in the paper, and that the results reproduce. Ask any CS/engineering PhD student how many papers (in top venues) they've seen with a critical implementation flaw that invalidates the results -- and you might begin to understand the problem.

At least in CS, the system can be fixed, but those in power are unable and unwilling to fix it. Authors don't want to be held accountable ("if we submit the code with the paper -- someone might find a critical bug and reject the paper!"), and reviewers are both unqualified (i.e. haven't written a line of code in 25 years) and unwilling to take on more responsibility ("I don't have the time to make sure their experiment code is fair!"). So we are left with an obviously broken system where junior PhD students review artifacts for "reproducibility" and this evaluation has no bearing whatsoever on whether a paper gets accepted. It's too easy to cook up positive results in almost any field (intentionally, or unintentionally), and we have a system with little accountability.

It's not "the best we have", it's "the best those in power will allow". Those in power do not want consequences for publishing bad research, and also don't want the reviewing load required to keep bad research out.

They needed to be close to a large city center. Chicago is too expensive -- so the option you are proposing is something like Indianapolis?

Even large, damaging tornadoes have quite localized impacts (max of maybe a mile in path width) -- and you don't generally do much more than stay up to building code in order to prepare for one. In contrast, earthquakes devastate entire areas and require substantial changes to building construction in order to protect against them.

The probability of a large, damaging tornado at a particular spot in Ohio is quite small compared with the risk of damaging earthquakes in other locations. https://www.usgs.gov/faqs/what-probability-earthquake-will-o...