Hacker News Reader

Comment by dangerlibrary | Hacker News Reader

tl;dr: With a training dataset of over 7000 images, the system is incapable of performing the cucumber sorting task on live data with greater than 70% accuracy - likely do to over fitting on the "small" training set (where it shows 95% accuracy). Also, it is currently incapable of taking into account a number of the existing variables used for sorting.

If one were to leverage the power of a supercomputer trained with tens of thousands more observations - all meticulously hand categorized - it would probably get better.

Truly, we are only limited by our imaginations.

View on HN

I think he can easily extend the system to use the missing variables. Scratch marks, color, texture and prickles are easily enhanced by digital image processing. Maybe use a different network for each feature and use this output to feed into another network to get the correct classification-function.

If you read the article, the majority of the limitations I describe are the result of his running the system on desktop hardware. It takes multiple days to train the system using 7,000 80x80 resolution images. Hence the caveat about the availability of a supercomputer.

Or he could stop training on CPU and use GPU like pretty much everybody in the field...

How does it compare to human sorters?

I understand the rhetorical implications of your question, but I am unclear on how you expect me to answer it.

I'm not sure it's rhetorical, I think the implication is that human sorters are also imperfect so what's there accuracy level? Is it 70% (a tie), 90% (clearly better), or 50% (the computer is better).

<semantic discussion about the use of "rhetorical" in my comment ensues. We land at some kind of an understanding of one another and we both grow a bit as human beings.>

So how does it do versus his mother? Regarding how much their results agree or disagree.

Well given his mother is by definition the training set, surely the machine can only perform as good as or worse than her...?

There's no reason to presume that.

It's not like conveying knowledge in the sense of information transfer (where by definition a student can't know more than their teacher -- without external help).

It's like teaching a skill.

In human terms, a person can get to play better piano than their piano teacher, better poker than the person who taught them poker etc.

As the "training set" she just gives the basic skills for the recognition. The algorithm could discover further patterns she can't see beyond what she knows (e.g. she knows just that greener ones are usually better, but the algorithm notices that greener and taller are usually also better), do the calculations much faster and consider many patterns while the woman does a more shallow examination, etc.

In order to get the necessary data for a machine to bootstrap new criteria, wouldn't you need an independent scoring function for each sample? How are you going to get that data - are you going to auction off individual cucumbers?

Alternatively, you could track each cucumber with an Independent ID to see when it is sold, relative to other cucumbers - on the assumption that people are selecting the best from any given batch? But then you'd need to control for things like the position of the cucumbers in the display. Now you're in pretty deep into someone else's business, because you're a wholesaler, and the data will still be insanely noisey.

>In order to get the necessary data for a machine to bootstrap new criteria, wouldn't you need an independent scoring function for each sample?

Merely having an experienced human score images for them can be enough for the machine to self-discover new criteria based on the pictures and their scores.

This is what I was getting at - from the information at hand, the mother's classification seemed to be the definition i.e. She had 100% accuracy because whatever she classified was the right answer

Yes, but the machine could e.g. learn to see patterns present in her classifications that she can't see or haven't figured out (relying on fewer other patterns) and thus fare better.

Well, I for one am non-rhetorically, legitimately curious about how well human sorters do.

This question is predicated on the assumption that there are objective, codified criteria against which human sorting can be judged. That assumption is false.

> "In Japan, each farm has its own classification standard and there's no industry standard."

Right now the only question is "how closely can the machine approximate human sorting."

Given humans can't be judged it stands to reason that you can just call the machine a human, and bada boom, you just saved a lot of money.

Humans could be judged against each farm's reference human. In this farm's case, there is only the one human sorter. That reference human is the basis for comparison for the machine as well.

The machine sucks at it right now.

Reply on news.ycombinator.com