Candide – Identify Plants with a Photo | Hacker News Reader

Candide – Identify Plants with a Photo

candidegardening.com

187 pointssails98 comments

Nice concept.

Unfortunately, I have tried to identify some of my plants but it could not identify correctly a single one. Just an endless procession of at best similar plants, most of the time completely dissimilar.

I think this is another naive model that just tries to push entire problem to AI. That is unfortunately what I am seeing nowadays, very unimaginative. Just try to have fun with parameters of the network until you find some kind of configuration that seems to be working.

What it would benefit from would be some kind of analysis/classification of basic features of the plants like what's the basic shape of the leaf, trunk, how things are connected, etc.

The classification would benefit from AI (like identify where leaves are, where trunk is, etc.) but then that intel would be passed to a more classification-oriented algorithm.

(disclaimer, I am not an AI developer, it just seems to me like pretty rational way to approach the problem)

Right. This kind of thing is only going to work on very common plants that are easy to ID. I am very into carnivorous plants and even us experts sometimes have trouble identifying plants, often you need very specific morphological details to ID species that a phone picture will never be able to catch.

IMO this will just lead people to wrongly ID their plants more often than not, and that's a really bad thing.

The iNaturalist app does a spectacular job at identifying plants (and insects, fish, birds…), even disambiguating based on your location

You label geotagged images with the AIs suggestion if you agree with it and then other users can either confirm or suggest a different / more specific species.

You got me curious, so I tried that app, and I'm getting "Unknown species" for every single one of my plants... I guess no identification is better than bad identification.

It is an important feature. For plants, users will immediately see that the answer is wrong but for many other things (identifying suspects automatically) it might not be as obvious.

As a bonus if the algorithm can tell it can't identify correctly you can use this feedback to teach the algorithm.

A great example of this is the Akinator app. The developers built up a database of answers by getting the users to fill in the missing data, which produced more accurate results and subsequently attracted more users. That feedback loop for apps like this seems like a very effective model. What I find interesting about this is that the value of the product, from a business perspective, comes from the success of the software in enabling that feedback loop, while from the user’s perspective, the value is in the data. But the data is emergent from the feedback algorithm.

Is it perhaps also a difference in intended use? If iNaturalist is for wildlife and uses location heavily, eg identifying tropical houseplants indoors in a different region might not work.

Could be, I just set location data on it with the original location of the plants, but it still cannot ID them. Granted, they are not very common species.

The GPS helps with a “seen nearby” weight, but it shouldn’t exclude anything. Could be a factor of photography, with some plants it’s easier to get a nice canonical leaf shape than others. Taking a wide angle of a potted plant such that the whole plant is in frame might not provide enough detail, so there’s a learning curve in knowing how to photography for the algorithms benefit, kind of like learning how to phrase search queries for better Google results

Are they garden plants?

No, they're carnivorous plants grown in greenhouse.

Perhaps they are unknown species! Hah, well, sorry my recommendation didn’t come through, I can imagine it performs better on common plants everyone takes pictures of, which is what I’m identifying when I walk around town.

Still, if you post the plants under a more general species other users can identify it mentally, adding to the data set.

You need to click on the "Unknown Species" box in order to get a list of suggestions. Probably not the most intuitive, but it avoids auto-assigning IDs without prior review.

I've used PictureThis a great deal to identify both wild and garden flowers. I've found it really good for the 95% most common plants and also for getting you in the right ballpark so you can then look things up more easily in a field guide. I've just accepted that there are some species that it can't distinguish between - often you need to know exactly the right feature to look at of the plant to differentiate species. It'd be useful though if it flagged this a bit better. Umbellifers are a classic example. It gets them correct more often than not, but I think it'd be good for it to indicate that uncertainty better.

Not sure if this will put your mind at ease or not. But I really like learning plants, I hike a lot and usually the only book I bring is a plant ID book. But I'm not good at knowing where to start looking in the book. I find apps like this (I use Seek) incredibly helpful, I don't just assume the app is correct but it gives me a place to start looking in the book. That said, I also don't assume I'm correct when I find what I think it is in the book. Anyway just wanted to say I find plant I'd apps a helpful learning tool.

I will say though, most of the time I don't have service where I'm IDing plants, so it's not the most convenient tool.

>What it would benefit from would be some kind of analysis/classification of basic features of the plants like what's the basic shape of the leaf, trunk, how things are connected, etc.

There's no reason to think that needs to be done separately, and in fact that's the kind of things that you'd expect a good model to find on its own.

In general, we've already learned that while handcrafted features can help, they are often ultimately worse than learned ones as techniques get better.

AIs are useful for "fuzzy" problems, but are not very good at doing precise things with high reliability.

Every plant has baked in restrictions on how it grows. If you can identify separate features of a leaf and how the leaf grows out of the stem you can basically look it up in a table and tell what kind of tree you are looking at, without need for guessing.

On the other hand AI will develop its own classification method but one that has unknown faults in it.

Maybe it has learned to look at the lighting direction because half of the data set was non-suculents with light from the left and half was succulents with light from the right, because it came from a different facility?

Or maybe different cameras were used to photograph different types of plants?

So now rather than looking at the leaves it uses light direction or photo grain to tell if it is succulent or not?

Face the reality, you are wrong.

The above algorithm returns completely nonsensical results for my searches, plants that have completely different structure and coloration and nobody would ever mistake them.

It relies at you looking at the suggested solution, and then, Hey!, here you have five more, maybe your plant is somewhere on the list?

This is only marginally useful but could have been so much better if it tried to identify structure of the plant.

Would it be possible to train the model to identify the features and then just have it run those through a trie of plant classifications? Trying to bake classifications into the model does seem silly but you still need an element of computer vision if you want to do this.

That's exactly what I thought up without having any experience in the area.

So, basically do what a person would do -- identify simple features visually and then consult the book to go through decision tree to figure out what you are looking at based on these features.

As you go through decision tree, you keep excluding more and more possibilities until you are left with only one match.

You would never mistake oak with a cactus because there are so many occasions on this decision tree to go one or the other way that you would have to make multiple mistakes.

But that's exactly what the algorithm seems to be doing -- mistaking plants that are very far away from each other when it comes to their build.

That’s the same problem you might have with a self driving car driven by an ML algorithm: is that an open lane or someone wearing a black outfit with white stripes?

Thankfully, the plants and the app user are both moving a lot slower.

> There's no reason to think that needs to be done separately, and in fact that's the kind of things that you'd expect a good model to find on its own.

Disagree - learning to classify morphological characteristics is learning the generalizable features. Especially given that some plants are going to have relatively few photos, knowing with high confidence some diagnostic factors and GPS could absolutely outperform the brute force approach.

This isn’t about hand tuned features, it’s about predicting the right thing.

Also, the approaches aren’t mutually exclusive.

To be honest, nothing works better than Plantnet for identifying plants. They do the pre-selection where you choose wich organ you're looking at (trunk, leaf, flower, fruit, ...).

Plantnet is a gem, they don't even want you to "register"/"connect with FB/Google" for "tracking" purposes

I think it’s a research project and not a commercial VC-money-burning startup. I highly recommend PlantNet as well.

Flora Incognita, a free plant identification app for Android and iOS, developed by a German university, does that.

You choose a category first, like "tree", "flower", "grass" or "fern" and it will guide you through the process, trying to identify the plant with as few photos as necessary. Common ones it will identify from a single image, for others, it will e.g. prompt to take a close-up photo of the bark, bloom or the complete plant in its environment. From what I understand, they are aiming for accuracy of the identification and will provide a description of possibly matching plants if there is still ambiguity. Very recommended!

Edit: here is a link if it sounds interesting: https://floraincognita.com

This will bolster my fantasy of living off the land like the savage I want to be.

Thanks for pointing out!

I've been using an app called PlantNet that works similarly.

Upload a photo, tell the app which part of the plant is in the photo (leaf, flower, fruit, bark), and it'll tell you what the plant is.

Highly recommended as well, almost always identifies the plant with just one photo. I'll have to check out flora incognita sometime.

plantnet works very well. I have also used Google Lens (the magic button on Google Photos on Android) but I have the subjective feeling that plantnet works more reliably for plant identification.

I’ve been amazed at how well PlantNet works. It’s identified a number of invasive species in my yard that I’d otherwise have remained ignorant of.

This app is phenomenal! So far it's picked up everything I've thrown at it. Can't say that of the other two plant ID apps I've tried before. I think it uses geocoding to some extent - it asked for my location (which is fine by me) but it required a leaf, fruit, and bark pic to figure out Northern Catalpa, which isn't common around here. Everything else has been leaf and maybe a fruit or whole plant pic.

Funny enough, I only get a 403 Forbidden error when opening that website. And I am located in Germany with a German IP. Though my server at Hetzner (also in Germany) has no problem retrieving contents from that domain.

To much traffic from HN for that small shared webspace :)

One of these things I think of when I do garden things, "I bet there is an app for this" but typically as soon as I have left the garden I forget all about it.

I have installed the app now for the next time I am gardening :)

Thanks

Related to this, I've been working on a side project (https://www.hedira.io/, house plant care advisor) for nearly 2 years now. We haven't gotten time to add species recognition by images, but I've done enough to know it's really hard! Thankfully, being houseplant focused, people normally have an easy time finding out what specific species they have in front of them.

I made some classifiers using coreML to test the idea and as with a lot of ML problems, 90% accuracy is trivial, but it gets difficult really quickly after that. Especially without flowers (since they tend to be more unique).

The simplest way I could find to add detection was to use something like the plantnet API (https://my.plantnet.org/usage) which powers the app of a similar name. There are a couple of other plant recognition APIs worth looking at too.

https://plantnet.org/en/

I have been using plantnet for a few years and have been pretty happy with it. The ability to contribute+validate your identification is great.

It does fail sometimes but it's pretty good, and the UI has improved in recent releases.

I have to agree on PlantNet. It is good enough to come up with a general name of a plant (ragwort), that you can then look up in a local plant guidebook to find out the specific variety (common ragwort).

Hopefully, a privacy-friendly phone (ie Apple) manufacturer acquires them and does deep OS integration with the app.

I'm not understanding why you say that being purchased by a phone manufacturer would be better - wouldn't the optimal solution be to expose any features from the OS that the app could need, while leaving the app developers their independence?

I’m trying to understand the business model that PlantNet has. Right now, it’s obviously some sort of exploitative data harvesting - hoarding geotagged plant photos to sell to academic researchers, highlighting hotspots for nature tourism, etc.

I was alluding to Apple in my comment, as they are the ones that have a model for their mobile devices being some sort of knowledge lens. They are very easily capable to eat the cost of maintaining the database without resorting to some sort of exploitative business model.

You didn't look very hard[1]. It's a research project funded by grants and donations, and as far as I can tell they give away their databases[2].

[1]: https://plantnet.org/ [2]: http://data.plantnet-project.org/datamanager/_design/dataman...

I tried to look for an explanation of the scope of their project from the FAQ. Since it wasn’t openly included, I assumed it was deliberately omitted.

And leaving out those that are using other platform without the app.

Currently PlantNet works on Android, iOS and web.

https://www.inaturalist.org/ also does this kind of thing. iNaturalist works well for lots of different organisms.

It’s apparently much nicer on Android than on iOS. I have only used their other app Seek, and while the IDs are pretty good, the UX ain’t great.

There is also a free service trained on Norwegian plants: https://orakel.artsdatabanken.no/

The final step in the user guide is (roughly translated):

> Now it's your turn to think! A high match percentage is no guarantee that the result is correct. Artsorakelet tries to match your pictures to pictures it has seen previously, but many species have no, few, or bad pictures! It has not been trained on domestic animals, garden flowers or humans, or pictures with restricted access, such as images of large predators

Or just have actual botanists identify the plants for you (while helping gathering data for AI later):

https://www.flowerchecker.com/

You get a confidence factor, a link to its wiki page, and you can even send a special message with the pictures to help clarify things.

Google lens does a pretty good job of identifying plants, along with everything else. I use it a lot in our garden and a majority of the time it correctly identifies.

There is really very little need for 3rd party apps on this. I've yet to stump lens on any Fauna or Flora in my neck of the woods. I suspect it is location aware, which gives it a pretty big edge in filtering out duplicates.

Interesting, I can't use Lens. Installed it, but it just says "Thank you for updating. Lens will be available soon"

My success rate with Google Lens is about 80-90%, sometimes doing additional research to confirm between similar results.

With a couple different dedicated apps I had zero success.

Awesome. It identified my fully grown pepper (capsicum ) plants as spinach. What could go wrong?

At least it didn't identify it as "not hot dog."

I wonder how much funding these guys have, and how much of it I could get by intentionally non-fatally poisoning myself with a misidentified plant? How much indemnity can the whole "WHILE WE ENDEAVOUR TO ENSURE THAT THE INFORMATION ON THE CANDIDE PLATFORM ARE CORRECT, WE DO NOT WARRANT THE ACCURACY AND COMPLETENESS [...]" thing actually provide?

Or you could try smuggling misidentified spinach across the Mexican/USA border and see how far blaming the app gets you :)

Google lens can already do this. It identifies other things too, which is nice. And from my very limited testing lens is more accurate.

I guess this is nice for the "cancel Google" crowd though...

Try Plantnet, it's free and is a research project. Works really well.

https://play.google.com/store/apps/details?id=org.plantnet&h...

My immediate thought is that this could be quite dangerous if used by inexperienced foragers. It’s very easy as a human to misidentify poison hemlock as cow’s parsley, for example, which is a lethal mistake.

It’s a very cool project though, of course.

How does this compare to PlantSnap and pl@ntNet?

I haven't used either, but in the UK this works incredibly well.

Or PictureThis (https://www.picturethisai.com/)?

Or Google Lens? I've only tried Lens on various office plants (so not in the wild) but it was very effective for those

Or seek! https://www.inaturalist.org/pages/seek_app

This is one of the best apps to install on your kids’ phones. Makes them go outside and get interested in their surroundings.

my partner used one called PictureThis and it identified most of all the plants around our property here in New Zealand ( mix of natives and introduced), it got ones that were a complete mystery to us.

I used to use PictureThis until they started requiring a login in order to use it. Then I switched to LeafSnap, which works great.

At least for me on iOS it doesn't require a login.

I too confirm that PictureThis works great.

For me (Android) PictureThis requires a login, and limits you to 1 identification per day, unless you pay for a monthly subscription. It didn't use to in the past.

However, the linked app (Candide) also seems to require a login, so I'm sure it's just a matter of time before they put limits on free accounts.

I have it, and they don't require a login. I use it several times a week.

My main complaint about it, is that it's a battery hog; but it works well.

This is one of those markets where even ignoring open source/etc you never need to pay for it, because at the moment there is a new plant recognition app once a fortnight with a free trial. And none of them have any real lock-in because they're basically stateless.

I started using PictureThis recently and do not need a login or anything. Works pretty well at least in my yard and nearby parks.

Or Seek (iNaturalist)

> This item is not available in your country.

I don't understand why so many apps do this?? It's a free app, why not let me use it?

Thank developer laziness and GDPR

I uploaded it a photo of my wild cucumber and it said it was passion fruit. But that's a completely legit mistake since I literally made the same one until it started flowering.

But then I uploaded a photo of a catnip plant and it thought it was a milkweed plant. In fact, it's 0 of 4 for common plants in my garden this summer. Seems like a great idea with a marginal implementation.

Google's camera app does this in the Lens mode.

I snapped picture of the example picture; it came up with Swiss Cheese Plant.

Don't know how good it is. I've used it maybe seven or eight times in the past for identifying plants; it worked every time.

Most recently, over just this past weekend, it identified an Elm tree from a shot a branch.

Well, it gets easy ones pretty well - misidentified arabica coffee as robusta but did at least get "coffee", and identified a mango tree - but tricky ones (the various weed trees on our farm) are even more hopeless than my own efforts.

To be fair, it's probably for houseplants more than anything. On the other hand, half the stuff growing on a tropical farm is houseplants.

Wish there were more information about their model.

Not precisely on topic, but two professors at U.C. Berkely are involved in helping people identify edible plants in urban settings, particularly Oakland. This excellent piece makes mention of a phone app they are using in conjunction.

https://phys.org/news/2014-11-foragers-bounty-edibles-urban-...

So, what’s the monetization strategy? What data am I giving up if I use this app, and how will they use it?

What ever happened to dichotomous keys?

Were replaced for something worse, more primitive and less accurate. Sign of the times.

Understandably people are realizing that it's now much easier to discover what plants (and animals, and fungus, etc.) exist around you and whether they're special in some way.

But when picking apps to accomplish that task, I suggest selecting for those that respect your privacy (because a lot of these plant observations involve GPS location), are clear about how the machine learning datasets are trained and what's being done with the data you're supplying, and lastly how these apps and companies are funded.

PictureThis is owned by https://www.glority.com

Candide appears to be a UK startup: https://candidegardening.com/GB/about

iNaturalist and their Seek app (https://www.inaturalist.org/pages/seek_app) is a joint venture between the California Academy of Sciences and the National Geographic Society: https://www.inaturalist.org/pages/about

Moreover both iNaturalist (https://github.com/inaturalist/inaturalist) and Seek (https://github.com/inaturalist/SeekReactNative) are open source. The former is a Rails app, and the latter a React Native app with no account registration system making it safe and legal to use for children since all observations are stored in-device by default unless you chose to send them to your iNaturalist account to share with the community.

On functionality alone it's extremely rare for Seek not to recognize an organism (yeah, it's not just plants) and when it is, I simply send the observation to iNaturalist whose network of naturalists both amateur and professional usually manage to definitively identify my observations within a few hours. Here's a recent example: https://www.inaturalist.org/observations/86844469

The iNaturalist community itself is fascinating and quite transparent with detailed site stats (https://www.inaturalist.org/stats) and a clear mission statement ( https://www.inaturalist.org/pages/what+is+it): > iNaturalist is an online social network of people sharing biodiversity information to help each other learn about nature

At a time where biodiversity is more threatened than ever, I welcome any tool to help folks identify and value the organisms around us, but if you're going to pick one, my recommendation is iNat & Seek.

PS: I'm not affiliated to iNaturalist in any way other than as a happy user.

Seek is AWESOME. We discovered it late in 2020 but it raised the bar for all our outdoor excursions with the kids. Even a routine walk around the neighborhood is an enriched experience as the kids careful observe their surrounding looking for a new bug or flower.

The only downside is when we get to a new place I have to remind the kids we can't stop every minute to 'Seek' something otherwise we'll never get back home!

I really want to like Seek, but it has a nag screen and long load time (even on phone unlock) that prevent getting fast photos. Its IDs are pretty great though.

German research project doing same for free: https://floraincognita.com/?noredirect=en_US

Flora Incognita is great. Very recommended!

How is this better than Inaturalist? https://www.inaturalist.org/

I use plant.id for this and it actually seemed to give me correct identifications every time I used it so far.

ObsIdentify identifies both plants and animals. Works way better than I'd have ever imagined! https://play.google.com/store/apps/details?id=org.observatio...

Not available in the US?

"This app is not available in your country or region"

I have an app on my iPhone that works pretty well (PictureThis). Probably does about the same thing as this.

I noticed that the following clause in their Ts & Cs:

> You grant us an irrevocable, perpetual, worldwide and royalty-free right and license to use your Content for the purposes of providing, promoting, developing and trying to improve the Candide Platform and our other services, including new services that we may provide in the future. We will not sell your Content to any third party

does not have punctuation after the last sentence.

I'm not a Grammar Nazi[0], but that tells me that the sentence was changed at the last minute.

[0] http://queenofwands.net/d/20031003.html

I don't work at Candide any more but I was CTO there when this was written. I can assure you you're reading too much into it!

Candide is very well set up to protect users' private data, and has a great set of engineers and engineering practices in place ensuring this, as well as, you know, not ever selling data to a third party!

Thanks for the clarification. I know that lawyers spend a lot of time revising these types of sentences; usually with an eye towards exit plans.

I appreciate the authoritative response.

How does this perform Vs Google lens?

The augmented reality function that seems to allow you to label your plants seems intriguing. I wanted to try this over PicturesThis (I don’t like how PictureThis makes it look you need their subscription for the app to work) But it seems the app is not available in my region maybe because of GDPR, I guess?

So like Google Lens but not cross-platform

I have tried Plantnet; it works as expected. But looking through the website, I might give this one a try and compare it to Plantnet to see who's better.

Reply on news.ycombinator.com