BirdNet – Identify Birds by Sound
birdnet.cornell.edu
birdnet.cornell.edu
So when she read about this a few weeks ago, she literally smacked me for not building it. Now if I tell her it's on the front page of Hacker News AND everyone here loves the idea even more, I'm going to get another, harder smack because she knows how HN is full of others like me!
(yes I'm aware that this community more than others will agree that ideas by themselves are a dime a dozen, but nonetheless, it would've been a really fun project)
Download any of the existing bird apps that help you recognize birds by their sound and you'll see that each bird often has 3-4 distinct sounds, each of them different, and these are just partial examples.
You'd have to be extremely dedicated (and very good at machine learning) to see this idea through to completion.
Because in nature, it's almost never just one bird you hear. You hear it in the context of all the other sounds.
There used to be an an app called Midomi (I think?) that could identify songs by humming or you singing, which was cool, but then I vaguely remember it rebranding itself and being less useful. Does anyone else know of a song-recognition app that is more like BirdNet and less like Shazam?
Unfortunately/fortunately I can't get the visual out of my head of a southern speaking crow looking for trash near my house now.....
She looks like she is delighted btw.
Effort? Not really so much (stand on the shoulders of giants, etc.)
Data? Yes. Lots and lots of data.
https://www.appbrain.com/app/birdnet-bird-sound-identificati...
I think I had someone in my class (about 10 years ago) attempting this as a project.
What if you used the AI hardware in the phone to do the audio recognition?
For efficiency you could even use geolocation and figure out which species are found in a location and download a model just for those. Anything not matched could be uploaded as before.
When the Lab’s researchers conceived of BirdNET, there were no reliable bird sound identification tools. BirdNET was built as a rapid prototype, engaging computer science students to build an app for that users to test the machine learning algorithms. BirdNET proved to be a research breakthrough and by 2020 was performing with far better accuracy than five other apps tested.
That success opened the way to apply computer vision to sound identification in the Lab’s outreach and education app, Merlin.
Merlin offers OFFLINE functionality, and multiple ways to help identify birds, including through a user describing the bird, taking a photo of the bird, and now recording a bird song or call. Merlin Bird ID is integrated with the Lab’s systems and resources, including updated taxonomy, bird information from eBird and Birds of the World, rich media from the Macaulay Library, life list building tools integrated with eBird, and more.
Some birds in my locale mostly repeat themselves, but some seem to have 'vocabularies' of 3-5 different calls, and you can hear pitch and timing inflections within those - might be just random variation in combination with different calls it might yield 30-60 'words'. Sometimes I've been sitting under a tree and heard what seemed to start out as a conversation that degenerated into an argument followed by a physical fight.
Even crows seem to have distinct patterns/variations in their cawing, and given what we know about their tool-using abilities I'm curious to know how they use their voices. I've seen remarkable behaviors like a group of crows harassing a falcon to interfere with its pursuit of a smaller songbird.
The benefit of using hardware accelerated ML built into the phone is that it’s much more lower power. It’s designed for continuous use cases (“Hey Alexa” or Hey Siri). So you don’t have to turn the recording on and off and miss the bird call.
I don’t know if continuous monitoring can be used by third party apps. But having it on all the time, with geolocation would be amazing. You could set alerts etc.
If you could use multiple phones to locate the bird in 3space that would be neat. Then you could tell people where to point their cameras. Maybe a standalone IoT monitoring device could be placed in forests to count each and every bird. This is the future.
This is actually the 'real' research motivation behind the bird classification work: Slap a microphone to the side of a tree, pick it up in a month, and get some accurate picture of what species have been in the area.
Birds are relatively easy to observe, thanks to their vocalizations, which makes them an indicator species. We have a good idea what many species eat, so they end up telling you quite a lot about the surrounding ecosystem.
However, it turns out that the 'soundscape problem' where the microphone is just attached to a tree is a bit more difficult than identifying foreground birds only, using a device that can be pointed in the relevant direction by the user.
We've been encouraging further work on the soundscape problem by hosting the BirdCLEF and Kaggle competitions, and have been seeing steady progress. Improvements in the 'hard' soundscape problem have been driving improvements in the 'consumer' identification algorithms.
https://www.kaggle.com/c/birdclef-2021/overview
[source: I've been working with the BirdNet folks on and off for the last few years, and co-host the Kaggle competitions.]
Yes pointing a directional mic introduces a whole new set of mechanical challenges.
Maybe you could build an irregular grid of omnidirectional microphones and use signal processing to direct the beam digitally (similar to radio beam-forming). Now you’ll need more processing horsepower to do FFTs to do phase shifts. Although if you assume the bird calls only occupy discrete frequencies you might be able to save some computation by just computing those.
Perhaps a machine learning model could be trained that does all of this for you. Then you get the benefit of hardware acceleration. Some ML chips can handle DSP tasks.
The quality of the single-source classifier is the obvious scientific bottleneck, though; improve it, and everything else will work better. (We've also got plenty* of existing training data for this case.) So that's where we've been focusing most of the energy.
* - depending on species, of course. See also: xeno-canto.org
It would be amazing if BirdNet eventually supported offline.
The only thing she'd improve was exactly what you mentioned: offline recognition! (and also keeping the bird sound recordings to export them later)
Listening to bird sounds is now recognized to have positive impact on mental health[1], So how about selecting a particular region on the earth and listing to high quality bird sounds? There some good YT playlists[2] but a separate service could be more functional, Tie up with bird zoos to do it live, share a piece of revenue for conservation and you'll have my subscription.
[1] https://www.nhm.ac.uk/discover/how-listening-to-bird-song-ca...
There are a bunch of really similar apps with $99/week billing that activates after trial automatically.
Unfortunately they don't publish the code of TF version and only a TF-Lite model is available. Probably that doesn't matter for the exports though since the paper and original version are both there.
More interesting thing is that they've been making the dataset available [3] for $20 (even before BirdNet). This can be great source for training your own bird-net like.
- [1] https://github.com/kahst/BirdNET
- [2] https://github.com/kahst/BirdNET-Lite
- [3] https://www.macaulaylibrary.org/product/the-cornell-guide-to...
Just ask a series of questions:
- Is it a plant or animal?
- How large is it?
- Where are you (location permission)
- Does it have green leaves?
- Does it have woody stems?
- Does it have whorled or alternate leaves? (Show images)
- Does it have yellow flowers?
This is how a lot of field guide books work, but they require a lot of flipping back and forth, scanning tables of contents, and memorizing terminology. An app could just be tap-tap-tap-tap-tap, drilling down very quickly and showing images at every step.
The "only" difficulty is getting the database of attributes - maybe it already exists. Maybe an app like this already exists and I haven't been able to find it?
Thing is, the series your questions you show are easy to answer but they don't get you any further than halfway. Problem is in the questions you do not show: there it starts going into details, terminolgy starts to matter (seriously, if you've never read those words it's like a foreign language) and differences become hard to spot. Part of this could perhaps be alleviated with pictures but I doubt it; I've seen websites attempt it but none were really good and they all did a subset of plants. Probably because it's quite the amount of work to do them all, even for a region.
Apart from the usefulness of the app, interface and usability is great too
https://apps.apple.com/us/app/merlin-bird-id-by-cornell-lab/...
Still, nice app as a field guide.
https://apps.apple.com/us/app/merlin-bird-id-by-cornell-lab/...
With BirdNet you make a recording, highlight the interesting section of the sonogram, and upload that section to the BirdNet servers. With Merlin you start recording and the software ids birds in real time, popping up species as it goes.
My assumption is that, because it runs locally on the device, Merlin is going to be less accurate than whatever BirdNet is able to do on its beefy servers. But it is has the advantage of working without a data connection. Merlin can also id from photos and descriptions.
So the one isn't a replacement for the other. It's great to have options.
BirdNET and Merlin Sound ID are such different use cases (real-time classification of a rolling window of audio vs classification of a user selection) that we don't have any comparison metrics between the two, and they are really geared for different purposes (BirdNET’s goals align more with research in bioacoustics, while Merlin’s goals align more with outreach and education.)
source: Merlin dev
What I've noticed is, rather than focus on endless scrolling through Reddit etc., I'm actually very present in listening to the calls, the nuances of them, and getting familiar with the pulse of the nature around me.
This app simply gamifies that experience and let's me play audio pokemon while at the same time tuning out the rest of the world.
I am also convinced cardinals and blue jays get along great with each other but are "not like other birds" types.
If so, any thoughts on how hard it would be? Since I don't use ml a lot and only experimented with recommendation engines with ml.net
( A lot of trees and a lot of birds at my parents place)
Mostly because my dad would love this and it could be a fun project to get a better understanding of ML.
Another interesting product is bird box, that keys you know which birds are feeding now: https://www.kickstarter.com/projects/mybirdbuddy/bird-buddy-...
I doubt that it will work on a Raspberry, but you can run it elsewhere and just send the audio to it.
Since it contains a tensorflow model and tensorflow can be used with ml.net.
It would analyze recordings every 3 seconds. Since that is what the model expects. No?
I use that "Share"-button a lot (it usually looks like a "<") in order to save interesting stuff I later want to look at on my PC.
There is enough free space on the top right for such a button (I think it's called the ActionBar?)
Those who do this the analogue way, often distinguish between bird song and "bird language." The former focuses on identifying species. The latter focuses on understanding the information birds convey to one another. Since a lot of it relates to predators (watch out, a fox!), this might be augmentable to determine the presence of silent animals too.
fun.
The dialectal song differences are readily noticeable, but mapping the dialect boundaries between populations would be really interesting.
Humans understanding bird calls isn't new. We've probably forgotten more than we know. That's not unique to us either. Different species often recognize each others' calls, particularly danger calls.
It also allows sharing "sightings" (or "hearings") with a central service.
Ironically, at the time I discovered Cornell was doing this same thing, but it looks like they finally got to a product by throwing an ML classifier at it. Very cool, ML is perfectly suited for this.
The interesting thing I learned during my study was that there is an entire system of describing bird sounds with nonsense words ("skee-dlees chis chis chis") that goes back over a 100 years.
The official system used Latin and Romantic language words (frequency modulation, intercourse, feces). It’s the jargon found in textbooks and research papers and taught in universities.
The underground system uses Anglo Saxon words and is used by lab techs and people in the field (figuratively). Examples would be (tune, warbling, fuck, shit).
One of these languages is considered respectable. The other is vulgar and suppressed to the point of cultural genocide.
This is the way. It has been this way since feudal times.
I haven’t tried using this particular one. It has its work cut out for it. Bird calls are difficult. A mockingbird or catbird can sound exactly like a sparrow or finch.
I remember, in the 1990s, when everyone had Nokia bricks, that mockingbirds would sometimes copy the ringtones.
Every app listed here, and the OP BirdNet, are so ineffectual. They have no idea what they are listening to (unless location is on) and they cannot discern a genuine tweet from a washing machine squeak. The posters saying the apps are verging on incredible are completely deluded. I have used these apps, all of them, across Europe and North America, and they are inept and fail at a rate of 100%.
Who are you people?
And the top remark from the person that 'had this idea 10 years ago'... is this a joke?
Where is YC going?
Location (as well as date, time, light, air pressure...) is certainly a factor that should be used when classifying something though.
I think the problem is probably related to data collection / data integrity. At least in my case, I'm using an iPhone that has variously placed microphones and speakers. It isn't clear to me how to hold the tool to collect the sound. If it gets wet or grungy, it's not something that is user-serviceable. I don't have an easy way to tell if it even needs to be serviced. If you're relying on the microphone that follows you literally everywhere to remain clean & untarnished - maybe that's the problem?
My phone rides in my pocket, it picks up dust, it gets moist, it gets dropped, etc. Perhaps I'm projecting. Maybe your phone is untarnished and is working fine? Do you have data to back up the 100% fail rate across Europe and North America? It seems you might be projecting as well?
GPS helps to narrow down the the potential bird choices. So over time apps will learn and become more accurate. The open source Spleeter technology, unmixing stereo music, and its ilk, may be helpful run in real time allowing the app to ignore sound that is not birdsong.
I don't know if it would hold up in a court of bird law
Edit: Cool, mockingbird is on the list! This link off of their site has a sound sample for those that haven't experienced this bird
https://www.allaboutbirds.org/guide/Northern_Mockingbird/ove...
hope im not contravening the rules too much by plugging my current project Birda - 'Strava for Birdwatching' https://birda.org/
Thanks to everyone who works on it. We've used the app relentlessly for a couple of years in the UK and when you show it to people they are amazed. People thank us for it and all we did was share it with them. Great work!
To me it would be cool to be able to decode bird noises to their meaning. Am really curious as to what birds are chirping about.
I contacted the authors at the email on the page, asking them to support iOS 12. Maybe email them too if you're also impacted.
Perhaps we could gather more data with the help of Federated Learning and the like.
This should be interesting to the Earth Species Project. https://www.earthspecies.org/