Identify Plants with an App | Hacker News Reader

Identify Plants with an App

plantsnap.net

I use the flowerchecker app http://www.flowerchecker.com/ where you have to pay a small fee per plant identification. It actually uses real human biologists to identify the plant.

I've been using it a lot this spring to identify all the wild things growing in my yard. Love it.

You mean college students?

I don't know a lot about the service, but two of the three experts are not students. The third is a PhD student which I hold in a higher regard in terms of knowledge than a typical undergrad college student. If they identify accurately I'd think it matters even less. Also, only $1 so you get what you pay for?

College students are in fact real humans....

> College students are in fact real humans....

That is mostly true. The existence of animals with diplomas[1] is the exception to the rule.

1. https://en.wikipedia.org/wiki/List_of_animals_with_fraudulen...

But having a diploma doesn't necessaraly implies the animal studied in college. So your argument is broken.

[citation needed]

Even if they're college students, they'll know way more than I do about plants

Oxford gives this definition of biologist:

"An expert in or student of the branch of science concerning living organisms."

Well - what is wrong with that? And atleast one of them is a Phd

I did this for common plant diseases using Tensorflow on Android when learning how to use TF last year. Found the accuracy to be not that great with real world images that are not in the dataset (with a dataset of 50,000 images from PlantVillage). I think there are just too many visual similarities between various plants (especially when taken from various angles) that the network doesn't focus on the right things. Did a short writeup here:

http://helminen.co/plant-disease

It could work for a subset of plants, or potentially with a much larger training set - but I think NIR spectral/hyperspectral imaging would be the way forward here with more differentiating data points.

>I think there are just too many visual similarities between various plants

I get the impression that you tried to train a single classifier to diagnose disease in any species in the PlantVillage database.

You might get better results by training a separate classifier for each plant species (or starting with just one species, such as tomato, for which PlantVillage has 10 disease categories). A farmer knows what crop they're growing, so can select the correct classifier when they submit a photo.

Great suggestion, that's the approach I've been looking at most recently (manual selection of crop first, then classifier for only that crop's diseases) and it does give better results. Still, many plant diseases can look quite similar - more training data once we get it (and more variety in the photos) should help accuracy

Did you use inputs other than just image for identification? I would have thought that adding location and time of year alone could significantly improve results.

just images. Location and time of the year both great ideas, but that data wasn't available in the training set

Not an expert -- would this be a result of training data doing a much better job isolating the subject vs a real world photo that may have other plants in the scene? Not that this is a solution in all cases, but is it possible your results would have improved with better real world pictures?

Not necessarily improved, because in real world photos you can have what I would baptise as "the voynich effect".

Plants in Voynich manuscript aren't real, can't even be classified in a family, but strangely still look familiar to us because they are "frankenplants". You can have exactly the same problem in photos of wild plants. It only takes the leaves of a climber growing over the flowers of other plant, or different flowers and fruits mixed together; and you'll have a new species. A very tricky one to identify. After scratching the head for a while humans can sense that something is wrong... machines normally can't see the problem. An (in)famous case is the photo of two juxtaposed black people arranged casually in a geometry that was tagged by the machine as 'gorilla'.

Likely but the whole point of 'AI' is that it should be able to identify flowers that don't look like what it's seen before. If not, it's just a huge lookup table that's like a memory repository.

This is, of course, a critical challenge in data science and is definitely not a trivial one to solve.

In some cases these are the goals, but dharma1 said the goal was to identify plant disease. If you can improve your results by taking better pictures then it becomes a tradeoff between training someone to take pictures and training someone to identify plant disease.

I think we have a tendency to treat AI as a silver bullet when we should be treating it as a tool we can use to help augment what we're already doing.

50k images is too few for that many classes

Nice. Will check it out.

Just knowing what questions to ask, what a bight could be would be very helpful to us plebes.

This product already exists. It's called Google Image Search. For instance, I was actually able to identify the second flower he takes a picture of in the video as Camellia Japonica simply from a screenshot of the video. I'd be blown away if they were able to assemble a better training set than Google. Kind of disappointing, as from the headline I was expecting something based on NIR spectroscopy, like the SciO molecular sensor (https://www.consumerphysics.com/)

There are two problems with this focus. First, lots and lots of photos on internet are incorrectly identified. I had seen web pages telling you how to include some edible spice in your dishes, with a stock photo of a similar species (very poisonous) incorrectly tagged as yummy.

Second, is probably a Camellia japonica... but could be also a Camellia x williamsii. And you need to know that there is a Camellia sazanqua also. A trained human can spot the "too big leaves for sazanqua" in miliseconds (or too much shiny, or suspiciously blue, or photoshopsly faked, etc) but this is not so easy for a program. Search image will not spot the differences and just let you with the most common option.

I agree it's a very hard problem to solve. GIS is far from perfect. But expecting this app to be any better is a real long shot.

In fact. There is a better product, Google Goggles - https://play.google.com/store/apps/details?id=com.google.and...

I don't know if the mention was intentional or not, but Goggles is actually dead as a product. They removed it from the iOS app, and the android app is not supported, while staying in the store.

In this respect, as the GP says, image search is currently the best option

Goggles (the concept) isn't dead. They moved the functionality into Google Assistant so you didn't need the app. Just hold down the home button so it takes a picture of the camera view.

Sadly to use the Assistant still requires disabling almost all privacy controls on your Google account.

Huh, I had no idea. An Easter egg, almost, as I've never seen this mentioned by Google.

It works ok, send pretty beta right now. I pointed my camera at a business card and pressed the home button. Google Assistant appeared and I asked it to tell me about what was on my screen. It noticed one piece of the address, and brought up a listing for that general area, but there was no way to further inquire into any of the rest of the card.

Sorry, I should have been more clear. It's moved into the "now on tap" feature, which now links to Google Assistant. You're right though, if you ask what's in your screen it has the same effect.

  Updated

  May 28, 2014

Kinda dead...

Slightly off-topic: am I the only one who has started uninstalling apps and using the mobile browser for most things?

Nope. I use like 3 apps nowadays, and even those are just occasional. Havent had FB or twitter in years.

A great app, collaborative tools and community around plant identification is the french pl@ntnet http://identify.plantnet-project.org/ from Agropolis Fondation, Tela Botanica, INRIA, CIRAD, CNRS, INRA, IRD and Montpellier University

> Fast forward to today - the tech is available and the team is assembled to make PlantSnap a reality. With a beta version completed, and 250,000 images in the database

250,000 images doesn't sound like a large enough training set to be effective on anything but the most common plants.

To help others understand why, the "Deep Learning" book, which summarizes current state-of-the-art DL, advises having at least 5000 samples (images) per class for OK performance (equal to non-DL approaches) and 50k to 100k for state-of-the-art. A class here would be a plant species.

So even with 5k samples, the 250k image corpus would only have 50 species, using this rule of thumb. A good engineer could pick up DL and build a system that performs to this standard, because the tricks are all written down in the literature.

If they do better, they either exploit unpublished methods, or researched those methods themselves, with their researchers.

But there is a lot of transfer learning between classes. For example, the network may learn to detect "broad leaves" or "red stems" for a common plant. It will then be able to use those feature on less common plants, and require much less training data for them. Transfer learning has been shown to work with just a single image of a desired class.

It's also nice that life naturally fits into nested hierarchies because of evolution. So if you can recognize what family it belongs to, then that narrows down the possible sub families it can belong to. That in turn narrows down the possible clades it can belong to, etc, which narrows down the exact species. You couldn't find a more perfect use case for hierarchical softmax!

We did this http://www.luontoportti.com/suomi/en/ a couple of years ago. It is based on an elimination process – you describe what you see and the application narrows down on the options. Works very well in practice, and can be applied to all sorts of indentification purposes in the wild, such as fish, birds and butterflies.

We also gave the image regocnition path a thought but it seemed to be quite a tall order. Hopefully they come up with a novel approach on this!

I used something called "dichotomic keys" in middle school, which are basically search trees ("does the leaf have lobes? If yes go to page 20")

I've looked for an app like that for years, seems yours is, thanks!

Is there something like this for Android? This app seem to be restricted to iOS.

Unfortunately we did not get to the Android App. Maybe some day :)

I am almost certain I once installed it and paid for the birds. Even emailed support suggesting a Swedish version.

When can we get this for identifying plants/animals/rocks/fungus/etc in the wild? Would love this on a phone when hiking.

For fungus you could try https://play.google.com/store/apps/details?id=com.pingou.cha... https://itunes.apple.com/us/app/champignouf/id1227854971

I made it as an experiment, results vary and can be awfully wrong but it's funny to see the confusions the neural network can make. Funny until somebody eat a deadly mushroom because of me, I guess. I already put plenty of warnings, so fingers crossed, but it's hard to be sure it won't be misused.

This is an interesting problem that almost all guide book authors have to contend with, bringing knowledge of a topic may expose people to risk, but I think it's maybe a false conclusion.

It's a given all guides have false information, it then begs the question are guides worse than no guides? I think this can be easily answered more information is better.

You can give people information but you can't understand it for them.

kudos for making the app :)

What do you think your app would identify this as? Carpet mushroom? http://imgur.com/a/Rqc64

Me too.

I volunteered at Audubon for a decade. I've got all the books, charts. I hike and camp. I can't identify rocks, birds, fish, trees, clouds, etc to save my life.

Square peg, round hole.

I've always been incredibly curious about all the life around me but at the age of 35 it finally started to bug me that I had no idea what, for example, the dozens of types of birds around me were called. For the past few months I've started making it a point to identify every single bird that I see around the yard, to spend a few minutes locating a photo of it and listening to its call and reading information about it on Wikipedia. Just in the past few months I've identified over two dozen birds that I grew up seeing and hearing and now I finally know what they're called.

What I'm realizing is that it takes active participation and interest in indentifying things to really learn their names and their histories, not just a passing interest.

The Cornell Lab of Ornithology has a site [1] and an app that is amazing for identifying birds. My favorite feature is how it also lists birds that are similar, which makes it a lot easier to find the specific bird you're looking for. I then compare visual information with the auditory information of the birds calls to make a concrete identification.

1. https://www.allaboutbirds.org/guide/search/

It will likely require internet access which is less likely to be available in remote areas.

this is made with convnets. so just point me to a dataset with a thousand examples per type, and I'll give you a working app in days. of course, finding and preparing that data is the real challenge

I would love to hear the details of what your process is on developing that.

I am not sure I even know enough to ask the right questions on this. But what tools, languages, computing resources, etc do you use?

This [1] is a good start. You'll probably feel like you're drinking from the firehose though.

[1: https://www.tensorflow.org/tutorials/image_recognition]

I think it'd have to allow users to submit images for the first few years, so that it can build up the sample set.

A similar app was released a while ago, I'd love to compare the two. For reference: http://leafsnap.com/

thats the one i remember; worked well enough. not a big flora person but was curious and wanted to try out the image rec.

I use this android app for recognizing plants by picture: http://identify.plantnet-project.org/

It's made by french researchers I think. It doesn't work perfectly but I did identify a lot of plants I don't know with it.

You can snap multiple pictures of the same plant, for example, 2 pictures of the leaves, 1 of a flower and 1 of the bark, and then use the combination to search. You can also submit your observations to have them identified by experts.

It doesn't work very well in Singapore at all. I guess the training set didn't include much from this part of the world.

In the app it does say that

  Pl@ntNet only works for the plants of the French metropolitan flora.

Only slightly related, but I've always wondered about an app that told you what crop was growing in the area you were driving through. Often they're recognisable, but sometimes they're not. No business case for it, just to satisfy curiosity.

This is very cool. I've been thinking about something similar, but only about mushrooms. Some people in my country pick wild mushrooms and sometimes it's very difficult to distinguish between the edible and the poisonous types. Such application, if existed could save many lives. The same concept could be used for other plants that people gather, like herbs and wild berries.

But if the algorithm screws up and gets someone killed you could be sued.

Finally, I can identify mushrooms that will kill me and eat the ones that won't.

I hope, for the author's sake, the app has robust Ts&Cs ...

exactly. something like not resposible for what you do with this information where the info is correct or not.

Ha! An app that could differentiate LBMs (little brown mushrooms) would be impressive, indeed.

There are old mushroom eaters and there are bold mushroom eaters, but there are no old bold mushroom eaters.

I've had good luck with plants.usda.gov identifying plants by geographical location. Just use the advanced search, choose the location you found the plant, e.g. state:county, and as much information as you can deduce, flower color, growth form, etc. It'll usually provide pictures and interesting things like whether Native Americans used it medicinally, etc.

http://www.gardenanswers.com app has the ability to automatically identify plants. If you can't find your plant, because it isn't blooming or is immature, or just isn't distinctive enough for image search, you can ask a horticulturist for a small fee.

Definitely an interesting concept. There's a bit more information on the indiegogo website [1].

Their problems they mention seem to be quite standard for image recognition (scale- and perspective variance), however I could imagine that this could be quite disastrous for plant recognition, given the massive diversity of plants: e.g. a leaf viewed from the side could equally well be a more narrow leaf from a different species.

As they write on [1]: "Our challenge comes from adapting our image recognition platform to recognize different shapes and sizes of the same plants, flowers and trees. We know this is possible, and we are close to cracking it."

[1] https://www.indiegogo.com/projects/plantsnap-identify-plants...

Honestly they don't seem to be that close to cracking it, like at all.

They may be using camera phone images, but they are well framed images of very distinctive features. Trying to recognize a literal tree in a forest is going to be far harder, and recognizing a tree based on a dead leaf (as a photo towards the bottom suggests) can be very very tricky. Also I have to admit to being concerned that adding "any known plant" won't kill accuracy. The more classification endpoints the less likely you are to get a decent result, and already you have to deal with far greater scale differences than are usual (basically it would be like identifying the breed of dog from anything from a full picture to a photo of a single claw).

It's a nice idea, but I have a feeling it's never going to be better than human enthusiasts/experts, who you can get to identify plants from photos for free already[1].

I can immediately think of all kinds of challenges that are really hard to overcome: diseased leaves, different seasons, plus all the usual glare/shading/background issues.

[1]: https://gardening.stackexchange.com/q/9355/3346

Botantical classification is interesting but difficult. Once trained, it's fast enough, but can "deep learning" be anywhere near accurate enough?

I'd guess there'd be a focus on salient botantical features for classification, and perhaps the human can be enlisted to circle them out. There could be a "twenty questions"-type narrowing down, perhaps using images.

Dichotomous keys are the standard in taxonomy for species identification since centuries. It's basically a binary search tree, generated by hand.

https://en.wikipedia.org/wiki/Single-access_key

Dichotomous keys are mostly useless when you deal with photos. For some reason people always find the most useless chamera position when taking photos to unknown plants.

chamera -> camera

I wonder too how much geolocation could pare things down, provided it's not invasive/introduced species.

That's a two-edged sword; I've seen a few birds which, if the range diagrams in my field guides are to be believed, have no business being anywhere near where I am, and were probably up to no good. Granted plants move less quickly, but I'd hesitate to put all that much weight on location all the same.

I want do the same for mushroom, because they are hard to recognize. Google images works quite fine.

I did it, but the results can be pretty disappointing, I'm still working on improving them:

https://play.google.com/store/apps/details?id=com.pingou.cha...

https://itunes.apple.com/us/app/champignouf/id1227854971

ps: any improvement suggestion is welcome!

Nice, will give a try. Hope you didn't stole my spot where I get my mushroom :P

Google Image is really working quite good. I made a lot of test. The only problem is about there's TOS. I can easily write something but I don't want to be flagged because I don't want use there API.

Visual identification of mushrooms can be deadly. Always do a proper spore print and have it analyzed by a mycologist.

wut? I know it can be deadly but it's your responsibility to eat it or not. I think about tell information, like you want for other type of plant too, you want know if it's the right time to cut it, etc. nothing about eating or not. I don't need really an apps for finding some morels.

Some species of morels may actually be poisonous, none are deadly though. (Unless maybe if you eat then raw.)

A pre-order on an app, seriously?

This landing page might just be a way to discover if there is any traction on the market without needing to build the actual product.

well at least they have a proper privacy policy /s

I would imagine a series of questions that you answer about the plant would help increase the accuracy. (Leaf patterns, etc.) Or, at least, whatever answer it comes up with should confirm the defining characteristics.

Having recently become the owner of a garden, this looks very interesting. I've been using the myGardenAnswers app, which purported to do the same thing. It's been singularly useless so far, though.

> It's been singularly useless

I'm bribable for the job... :)

I've been using this (free) plantnet app http://identify.plantnet-project.org/

Glad to see this as it is something I have been interested in for a while. There are some online tools I have tried in the past but nothing that really seemed to work well.

or go get a book on plants and you will be able to identify them pretty easy. The guidebooks ask you questions to narrow it down (how many petals does the flower have? for example). you dont need any knowledge going into using one, and you'll learn to observe and appreciate plants more as well through the process of IDing! newcombs wildflower is the go-to for beginners on the east coast but there are many.

I love the idea of an app to identify plants, I hate the idea of having to pay to use it... Definitely, I'm not in the target market.

I wanted this app for a while, but it didn't seem feasible even with modern ML techniques? Also what do I get out of a paid version?

Weirdly, the site disables the normal scrollbar but adds a new one inset, which doesn't respond to SPACE to scroll down.

I wanted to do something similar for snakes once, but after some thought it didn't seem like the safest idea.

Just a quick note to the site creator- the twitter icon in the footer is linked to instagram, and vice-versa.

/r/whatsthisplant FTW

Now we need to write an app to post images automatically and distribute a bitcoin bounty (set by OP) based on votes.

This is really cool. IF only I could find an app that does this with bugs.

Why do you want an app with bugs?!

But on a more serious note, I would appreciate that as well, but it might be harder to take a good picture of a bug by which it is identifiable?

I'm always impressed by these type of apps though and would love making one myself

I trained my kids, first for mushrooms, then for flowers and trees. They are better then any app :). But definitely would love to have an app to confirm our findings sometimes.

Great , kids: another pre-release implementation.

Train them to find truffle and you got a business.

Ha ha, good one. It is good that they love nature and it is nice if there are good companion apps. Some I saw, are not that great, so there is clearly need for more apps.

I really want this app. That is all.

would you pay for it? I also kinda want it, as I get curious when I'm out on a hike, but I'm not sure I'd pay for it.

Why not if the price is right ? The biggest obstacle for me is the need to register and create an account not the price itself.

$1.99? Sure. (https://wefunder.com/comments/9227)

WTF 150K of his own money in 5 years? I don't think at a business level this is wise under any circumstances for a bootstrapped project. Unless you have a ton of money to waste and this is your caprice. Of course.

Reply on news.ycombinator.com