Hacker News Reader

Comment by atonse | Hacker News Reader

I had this idea nearly 10 years ago when my wife and I started dating and she introduced me to (very amateur) birding. I even talked about contacting Cornell for their database. But it was always going to be a hobby project.

So when she read about this a few weeks ago, she literally smacked me for not building it. Now if I tell her it's on the front page of Hacker News AND everyone here loves the idea even more, I'm going to get another, harder smack because she knows how HN is full of others like me!

(yes I'm aware that this community more than others will agree that ideas by themselves are a dime a dozen, but nonetheless, it would've been a really fun project)

View on HN

I had this idea as well, but as you start to build the app you quickly realize that each bird doesn't just have one sound, but many sounds and trying to do this accurately takes much more effort than you're probably expecting.

Download any of the existing bird apps that help you recognize birds by their sound and you'll see that each bird often has 3-4 distinct sounds, each of them different, and these are just partial examples.

You'd have to be extremely dedicated (and very good at machine learning) to see this idea through to completion.

For the 400+ sounds that Merlin (the user-friendly, offline-functional sister app to BirdNET) can now identify with 80-90% accuracy, it took a team of dozens of bird sound ID experts several years to annotate tens of thousands of individual audio spectrograms. On average, they needed about 1,000 recordings per species). Here is a bit more about how they did it: https://www.macaulaylibrary.org/2021/06/22/behind-the-scenes...

Great recommendation, thanks a lot!

I agree. And also I actually realized a large part of the issue here might also be other background noises adding .. ahem.. noise to the samples.

Because in nature, it's almost never just one bird you hear. You hear it in the context of all the other sounds.

True, but Shazam is a testament to what's achievable even without audio source separation. Even going back about 10 years when I first tried it, I was kind of stupefied that it managed to work in a noisy department store.

But in some sense Shazam has it very easy - it is working from a discrete set of specific audio recordings, not, say, the set of artists, or the set of songs. Shazam is useless for live performances, or covers (by different band) of songs (unless those specific recordings are also in Shazam's database). Birdnet is tackling the problem of trying to get some essential properties of one species' song (and each bird is a new live performance, by a different cover artist).

There used to be an an app called Midomi (I think?) that could identify songs by humming or you singing, which was cool, but then I vaguely remember it rebranding itself and being less useful. Does anyone else know of a song-recognition app that is more like BirdNet and less like Shazam?

Google assistant does this. You can hum a song, or sing and it will recognize it with decent accuracy. Though humming camp town races lead me to this gem... https://www.youtube.com/watch?v=QFFykQIrPCk

Never heard of Midomi, but your description reminds me of Soundhound

As I understand it there are even sub-dialects for each of these bird calls/songs. The example shared with me was compared to the English language in the U.S. where you would expect to hear stark differences in the southern states vs the New England area with a greeting like: "hey ya'll".

Unfortunately/fortunately I can't get the visual out of my head of a southern speaking crow looking for trash near my house now.....

There is cultural transmission between animals too. Like here is some whale song spreading from Australia out into the Pacific.

https://youtu.be/P99CR4y-TYw

She looks like she is delighted btw.

Plus, so many birds will improvise their songs, pulling parts of other birds' songs into their songs. And many will even outright imitate other birds (not only parrots, but corvids and some others). Many amateur birders like me often cannot tell imitations from the real ones.

A couple of years ago, my wife had been practising a particular piece of music on the violin for some time in preparation for an exam; after a while, she noticed that the blackbird in our garden began using the opening few bars as part of its song.

I tested the app on one of our more interesting mimics (tooth billed bowerbird, endemic to the area) .. without great results. It is however very successful in identifying much of the morning cacophony here. This is for far north Queensland, Australia where we have a fairly decent array of resident songbirds.

This is actually a pretty difficult problem. I have been trying to learn bird songs for a very long time and it is pretty difficult. Saying that a bird has 3-4 bird sounds is an extreme simplification.

> I had this idea as well, but as you start to build the app you quickly realize that each bird doesn't just have one sound, but many sounds and trying to do this accurately takes much more effort than you're probably expecting.

Effort? Not really so much (stand on the shoulders of giants, etc.)

Data? Yes. Lots and lots of data.

And some birds can mimic voices, for example blackbirds and jays. They often impersonate other birds, or cats.

The Android app is about about 3 years old, so let her know there's 3 years of kicking to be done!

https://www.appbrain.com/app/birdnet-bird-sound-identificati...

To be fair the idea is probably not that uncommon but it's hard to execute right.

I think I had someone in my class (about 10 years ago) attempting this as a project.

I'm sure there have been plenty of attempts at this, but ultimately became Not A Hotdog for birding.

I've told my wife that. It's not a unique idea, I'm sure many have had it. But that's not even 1% of the work in actually building it.

You can build the next generation bird app. This one requires uploading to the cloud to recognize bird calls.

What if you used the AI hardware in the phone to do the audio recognition?

For efficiency you could even use geolocation and figure out which species are found in a location and download a model just for those. Anything not matched could be uploaded as before.

I work at the Cornell Lab, and we also have an app that is more consumer-oriented, and which DOESN'T require uploading the recordings to the cloud -- see Merlin Bird ID app https://merlin.allaboutbirds.org/

When the Lab’s researchers conceived of BirdNET, there were no reliable bird sound identification tools. BirdNET was built as a rapid prototype, engaging computer science students to build an app for that users to test the machine learning algorithms. BirdNET proved to be a research breakthrough and by 2020 was performing with far better accuracy than five other apps tested.

That success opened the way to apply computer vision to sound identification in the Lab’s outreach and education app, Merlin.

Merlin offers OFFLINE functionality, and multiple ways to help identify birds, including through a user describing the bird, taking a photo of the bird, and now recording a bird song or call. Merlin Bird ID is integrated with the Lab’s systems and resources, including updated taxonomy, bird information from eBird and Birds of the World, rich media from the Macaulay Library, life list building tools integrated with eBird, and more.

Thanks for your work! I'm installing right now but won't be able to try it out for a while. How far away are we from distinguishing between different calls from the same species?

Some birds in my locale mostly repeat themselves, but some seem to have 'vocabularies' of 3-5 different calls, and you can hear pitch and timing inflections within those - might be just random variation in combination with different calls it might yield 30-60 'words'. Sometimes I've been sitting under a tree and heard what seemed to start out as a conversation that degenerated into an argument followed by a physical fight.

Even crows seem to have distinct patterns/variations in their cawing, and given what we know about their tool-using abilities I'm curious to know how they use their voices. I've seen remarkable behaviors like a group of crows harassing a falcon to interfere with its pursuit of a smaller songbird.

Oh right, turns out I had it downloaded but hadn’t used it (the bird pack was too big to download over data).

The benefit of using hardware accelerated ML built into the phone is that it’s much more lower power. It’s designed for continuous use cases (“Hey Alexa” or Hey Siri). So you don’t have to turn the recording on and off and miss the bird call.

I don’t know if continuous monitoring can be used by third party apps. But having it on all the time, with geolocation would be amazing. You could set alerts etc.

If you could use multiple phones to locate the bird in 3space that would be neat. Then you could tell people where to point their cameras. Maybe a standalone IoT monitoring device could be placed in forests to count each and every bird. This is the future.

/Maybe a standalone IoT monitoring device could be placed in forests to count each and every bird./

This is actually the 'real' research motivation behind the bird classification work: Slap a microphone to the side of a tree, pick it up in a month, and get some accurate picture of what species have been in the area.

Birds are relatively easy to observe, thanks to their vocalizations, which makes them an indicator species. We have a good idea what many species eat, so they end up telling you quite a lot about the surrounding ecosystem.

However, it turns out that the 'soundscape problem' where the microphone is just attached to a tree is a bit more difficult than identifying foreground birds only, using a device that can be pointed in the relevant direction by the user.

We've been encouraging further work on the soundscape problem by hosting the BirdCLEF and Kaggle competitions, and have been seeing steady progress. Improvements in the 'hard' soundscape problem have been driving improvements in the 'consumer' identification algorithms.

https://www.kaggle.com/c/birdclef-2021/overview

[source: I've been working with the BirdNet folks on and off for the last few years, and co-host the Kaggle competitions.]

/“However, it turns out that the 'soundscape problem' where the microphone is just attached to a tree is a bit more difficult than identifying foreground birds only, using a device that can be pointed in the relevant direction by the user.“/

Yes pointing a directional mic introduces a whole new set of mechanical challenges.

Maybe you could build an irregular grid of omnidirectional microphones and use signal processing to direct the beam digitally (similar to radio beam-forming). Now you’ll need more processing horsepower to do FFTs to do phase shifts. Although if you assume the bird calls only occupy discrete frequencies you might be able to save some computation by just computing those.

Perhaps a machine learning model could be trained that does all of this for you. Then you get the benefit of hardware acceleration. Some ML chips can handle DSP tasks.

Yeah, one has to think of the 'microphone budget.' For microphone arrays, I think it's probably better overall (from the ecosystem management angle) to cover a wider non-overlapping region than getting a more comprehensive picture of a single point...

The quality of the single-source classifier is the obvious scientific bottleneck, though; improve it, and everything else will work better. (We've also got plenty* of existing training data for this case.) So that's where we've been focusing most of the energy.

* - depending on species, of course. See also: xeno-canto.org

Saw a talk a couple years ago about similar techniques being used to identify marine mammals. https://research.redhat.com/wp-content/uploads/2020/02/RRQ-V...

Entirely Off-topic, but I really like this way of quoting. It's surprisingly satisfying to see the quoted text bounded between a starting and an ending delimiter.

Love the app, thanks for your work. But can you comment on why the Facebook SDK is part of it? (I've seen a request in a proxy on the sign-in screen to graph.facebook.com.)

Thank you for all your work on Merlin. I heavily use that app when I go backpacking, and in fact just last night it helped me identify a red crossbill in Deschutes National Forest.

It would be amazing if BirdNet eventually supported offline.

The off-line version of this (Merlin) worked great in the Boundary Waters in June. Detected visually-verified grouse, vireos, woodpeckers, trumpter swan and bald eagles on Nina Moose and Agnes lakes.

Any chance of making the app available via F-Droid or some non siloed vector

For ID by song, does Merlin sacrifice any accuracy by being offline?

I asked a veterinarian friend who specializes in birds (and is a long time bird watcher), and she knew the app and said that it works pretty well (coming from her, that's definitely high praise)

The only thing she'd improve was exactly what you mentioned: offline recognition! (and also keeping the bird sound recordings to export them later)

see above thread for info on Merlin with offline sound ID capabilities...

Yep, that's great news. Thanks!

Oooh you can maybe geolocate users who don't give you location permissions by listening to the birds around you.

How about doing the reverse? Listening to birds from different parts of the world on-demand?

Listening to bird sounds is now recognized to have positive impact on mental health[1], So how about selecting a particular region on the earth and listing to high quality bird sounds? There some good YT playlists[2] but a separate service could be more functional, Tie up with bird zoos to do it live, share a piece of revenue for conservation and you'll have my subscription.

[1] https://www.nhm.ac.uk/discover/how-listening-to-bird-song-ca...

[2] https://www.youtube.com/watch?v=_EaSSgqsQs4

My mom is a master naturalist and has listened to (hours of) frog field recordings to determine which types of frogs are at specific locations for our department of natural resources (IIRC). There is a paper that describes how to calculate minimum adult population from audial surveys. If you still need to scratch that itch, I'm sure there are still some interesting applications along these lines.

I also met a fellow student who was working on this too years ago. Not sure how far e got. to be honest I think he was just to reluctant to graduate and get a real job so kept finding ways to stay at uni.

I hope you realize that The market price of this service is 0, set by Merlin.

That's also what I would say "there's no money to be made here – it would be a fun hobby project and I could learn some machine learning but that's about it"

It's one of the cases where idea is cheap, but the devil is in the details. Executing the idea well is the whole problem.

A couple years before pg wrote about classifying email, I was using the same technique on classifying movies. So close...

No time like the present.

Why does she keep smacking you? You may want to seek help.

Even a mere 7 years ago this was the joke: https://xkcd.com/1425/

You could have been a thousandnaire.

Reply on news.ycombinator.com