Good Code Coverage means bug-free Software Or does it? | Hacker News Reader

Good Code Coverage means bug-free Software Or does it?

symflower.com

1 pointzimmski5 comments

Another issue with Codecov is that it only calculates it based on the test reports uploaded. In other words, if you have a file that has no tests, it doesn't count towards coverage. You could get 100% coverage, yet only be testing 1 of 10 files. There are a lot of projects out there using Codecov and presenting false coverage information, probably without any clue they are.

I recommend using SonarCloud as a much better and comprehensive product.

Codecov will count untested files towards your coverage, as long as you properly run your tests. For coverage sonar cloud's implementation is rather poor. It also doesn't compare reports nor does it have any decent support for monorepos.

Sonar cloud also works with same coverage reports, so if you have that problem in one tool, you're likely to have it in the other.

Lastly, Codecov doesn't access your source code and does not slow down your CI, by at least 2 minutes.

I've seen Codecov misrepresenting, files are completely missing from the report. I've seen projects reporting 95% when in actuality it is less than 10%. How can they know about all files without scanning?

> Sonar cloud also works with same coverage reports, so if you have that problem in one tool, you're likely to have it in the other.

This is untrue, I have done side-by-side comparisons and talked to support. It is a (not well) known issue for Codecov, understandably as it would hurt the brand.

We use SonarCloud for monorepo. They have specific settings and instructions for setting that up.

We see scanning the code as a good thing. Two minutes is not a big deal for the benefits.

Again, I heavily recommend avoiding Codecov due to the misrepresented statistics.

The files are received from the repo listing. If the files aren't in the raw report, that means your tests are running incorrectly. I've used Codecov for many projects and huge teams, no way what you say is true.

Can sonar cloud track coverage based on a specific test suite or language or file path as some repos have hundredth of modules?

I heavily recommend against this advice, because the person can't figure out how to use the tool and claims that source scanning by some third party, is "a good thing", especially after you see the kind of crap sonar agent installs in your CI.

> Can sonar cloud track coverage based on a specific test suite or language or file path as some repos have hundredth of modules?

Absolutely, it's very nice.

Did you create this account to defend Codecov? Telling another they don't know how to run tests is childish as you have no idea what I have done.

Again, I went through this with Codecov support, so I'd take their word over yours.

Reply on news.ycombinator.com