A Pirate's Guide to Accuracy, Precision, Recall, and Other Scores
blog.floydhub.com
blog.floydhub.com
satie f_measure: 82.93 precision 97.62 recall 72.09 accuracy 70.84
The context here is a program that tries to recognize what is being played in an audio file. The precision refers to the chance that any given note in the output was actually in the input, the recall the percentage of notes from the input that was recognized. Depending on your application you might give more or less weight to those two. In this particular application precision is more important than recall because a wrong note sounds terrible but a missed note makes no sound at all...
You can usually adapt a classification problem to give you these figures, it sometimes requires a bit of creativity to determine what exactly constitutes a false positive or a false negative. In this case a false negative is a missed note, a false positive is a note that wasn't there that got recognized anyway.
To be able to easily determine whether or not a given change is an improvement you steer by either f_measure or accuracy.
That'll be a bit of a pain I'd imagine if you aren't separating out subtracks by instrument.
Besides which, wouldn't a time sliced Fourier transform give you the characteristic frequencies of the piece?
From audio file to midi file.
> That'll be a bit of a pain I'd imagine if you aren't separating out subtracks by instrument.
You're on to something there :) For now I've decided to stay single instrument until I can do that with high enough fidelity to be happy about it. The other day someone posted a link here about a neural net that separates out the various instrument tracks, that would be a very nice complement.
> Besides which, wouldn't a time sliced Fourier transform give you the characteristic frequencies of the piece?
If only it were that simple. Fourier transforms are very useful tools but their output is very far removed from something that can be played again.
That confusion matrix is something I have failed to get traction for in the past, but I'd say all market fit will be measured based on the customer's valuation of that matrix vs. the profile of it your product delivers. It doesn't matter what field you are in, this is the metrics framework for every analysis product out there, imo.
e.g. a customer has an expectation of tp/fp tn/fn rates that is valuable for themselves, and your given tech has hard limits on the trade offs between them. The real secret is that the values of the confusion matrix do not apply to whole or most of a customer vertical, but rather to individual customers with similar matrix values across verticals.
I think you can predict when a ML/AI product company will fail because they think they can apply their product to the vertical of their first few customers, making the false assumption that their confusion matrix applies all the way up that vertical - instead of hunting customers with the same matrix values in other verticals.
https://news.ycombinator.com/from?site=floydhub.com
Especially resubmitting the same links over and over again is really not nice.
(I am not associated with Floyd hub, but I have used their product once.)
Recall should be explained exactly as in wiki - how many of the relevant items are correctly identified.
It doesn't make intuitive sense unlike accuracy and precision.. Thanks!
Another place this idea comes up is a search engine index. If the algo doesn't find, for a given query, documents in the index it should have (falsely classified as not matching the query), it will have lower recall.