> In an evaluation on Reels and Stories products for Instagram, 75% of TestGen-LLM's test cases built correctly, 57% passed reliably, and 25% increased coverage.
That doesn't seem great?
That doesn't seem great?
> 75% of test cases built correctly, 57% passed reliably [implying test cases by context], and 25% increased coverage [same implication]
The actual report talks about test classes, where each class has one or more test cases.
> (1) 75% of test classes had at least one new test case that builds correctly.
> (2) 57% of test classes had at least one test case that builds cor- rectly and passes reliably.
> (3) 25% of test classes had at least one test case that builds cor- rectly, passes and increases line coverage compared to all other test classes that share the same build target.
Those are two very different statements. They even have a footnote acknowledging this:
> For a given attempt to extend a test class, there can be many attempts to generate a test case, so the success rate per test case is typically considerably lower than that per test class.
But then in their conclusion they misrepresent their findings again, like the abstract:
> When we use TestGen-LLM in its experimental mode (free from the confounding factors inherent in deployment), we found that the success rate per test case was 25% (See Section 3.3). However, line coverage is a stringent requirement for success. Were we to relax the requirement to require only that test cases build and pass, then the success rate rises to 57%.
Of course, this doesn't seem like it's going to replace engineers, but it'll help the organization out for relatively low cost.
The paper does not report code coverage improvement and it is probably not 25%. The paper does say this:
> The median number of lines of code added by a TestGen-LLM test in the test-a-thon was 2.5.