eBird: A crowdsourced bird sighting database | Hacker News Reader

eBird: A crowdsourced bird sighting database

ebird.org

390 pointsoftenwrong85 comments

I do a bit of wildlife photography in my spare time (follow me on Insta for pictures of British birds and occasionally my cat - https://instagram.com/onion4k). Recognising birds is surprisingly hard. A lot of them have very subtle differences that can't be spotted easily. For example, telling the difference between a common Thrush[1] and a much less common Redwing[2] is a real challenge. It's a fun and challenging problem to train yourself to do.

[1] https://en.wikipedia.org/wiki/Song_thrush#/media/File:Turdus... [2] https://en.wikipedia.org/wiki/Redwing#/media/File:Redwing_Tu...

Yes, some species are very hard to tell apart by eye. If you learn their songs it can become easier. Other hard-to-tell-apart species are Marsh tit and Willow tit. I am not confident after close to 10 years with birding interest.

Offf, as some one coming from the UK, thrush vs redwing is not even in the difficult category... Try little brown birds of which there are many, they are basically indistinguishable, they love to hide, so notorious they even get their own wikipedia page[1]; or gulls, also many species, as well as several variations of adolescent plumage; or raptors which in the UK you pretty only see as a silhouette while they are hundreds of feet up in the air, and seen so rarely you never learn to distinguish them by their outline.

[1] https://en.wikipedia.org/wiki/Little_brown_bird

Raptors are really interesting from a recognition perspective. Since volunteering at the Hawk Conservation Trust in the UK I've learnt lots of useful tips that apply even when you can just see a little silhouette. For example:

- Falcons have relatively scythe-shaped wings while Hawks have more rounded wings relative to Falcons

- Hawks and Eagles have more pronounced 'fingers' at the ends of their wings relative to Falcons

- Buzzards and Kites: Buzzards are more stubby and have a diamond shaped tail; Kites have longer wings and a forked tail

- There are similar differences with respect to the length of the tail.

Some raptors have diagnostic flight patterns: Kestrels hover in place, Kites perform extremely dynamic acrobatics, especially in groups. Even when you can see the plumage, it doesn't always help - there can be significant differences between juveniles and adult males and females.

Up here in the North East near Sunderland we get a lot of Red Kites so I've learned to recognise them fairly well. Very hard to get a good picture though.

I spotted a Kestrel in a tree a couple of weekends back, that sat for a while and then dived to try to catch a small bird. It missed but it was great to watch. I managed to get a few decent photos - https://www.instagram.com/p/CHDt2HogIZj/?igshid=1n13m6e6a6e1...

This reminds me of a parallel expresssion in mycology:

> "Little brown mushrooms" (or LBMs) refers to a large number of small, dull-coloured agaric species, with few macromorphological uniquely distinguishing characteristics.

> As a result, LBMs typically range from difficult to impossible for mushroom hunters to identify.

> For mycologists, LBMs are the equivalent of LBJs ("little brown jobs") and DYCs ("damned yellow composites") that are the bane of ornithologists and botanists, respectively.

https://en.wikipedia.org/wiki/Mushroom_hunting#Little_brown_...

https://en.wikipedia.org/wiki/Damned_yellow_composite

I have been birding for several years and it is amazing how repeated attempts day after day train your mind to recognize these subtle differences.

But actually it is true that there are a few pairs of species that cannot reliably be distinguished visually and you need to hear their calls.

If you can find a copy of it, read the paper titled "Jizz and the joy of pattern recognition". A great read with some lovely anecdotes about botany and birding. One that I fondly remember is a description of how someone is able to recognize a particular bird when it's just a distant silhouette. "It flies like a sack of potatoes".

Possibly-related, I wrote much of my undergraduate thesis on how experts (or rather, just people with lots of experience in a field) see or hear differences where non-experts see none.

To a regular person, the cloaca of male and female baby chicks look identical, but to a chicken sexer who sorts hundreds of baby chicks a day, they look totally different, even if the person can't describe exactly how. To expert radiologists, a shadow in an x-ray is almost certainly cancer, while a non-expert can't see any shadow at all.

Similarly, and more mundanely, in languages where there isn't a strong meaningful distinction between two sounds, speakers have trouble discerning them. To a monolingual native Japanese person, /r/ and /l/ sound almost identical, while a native English speaker can't understand how they don't hear the difference. To a French person, the "oo" sounds in au dessus (on top) and au dessous (underneath) sound completely different, yet to an English person they are almost indistinguishable.

These are really the same phenomena, and why an expert bird watcher would have no trouble recognizing the "sack of potatoes" flight of a bird.

I never knew of this, totally going to sign up.

Once upon a time, in the 1990's, I was in the 5th grade and the principal asked us (the class) to pull out our favorite book from our backpacks (she told us the day before to bring our favorite book) and wanted to hear the whole class reading out loud all at once.

I pulled out my "Field guide to the birds of North America" [0] and started reading along with everyone else. Most everyone else in the class either brought a comic book or some Dr. Seuss stuff. For what ever reason she noticed me and came to my desk while I was reading and stopped me and asked if what I was reading was "too hard". I don't know what I said, but I was super confused and scared thinking she'd take way my field guide and make me read dumb comics.

[0]: https://i.imgur.com/y3LPxAX.jpg

I used to keep mine in the kitchen where I could see my birdfeeder out the window when I was a kid. being in the SE US, we get a lot of migration crossing through and the seasonal variety is awesome.

I started using eBird when my daughter got interested in birds. But I fumble with it too much and went out to Amazon and got a new field guide just for quick access.

I could totally see her in the same scenario you were in :). Mostly now she gets strange looks when she refers to a female red shouldered hawk that hangs around the area as "Emma".

Kids should be given more credit. When I was no older than that I would read encyclopedic books about guns and fighter jets. Not birds, but I don’t see how reading non-fiction early is anything but healthy.

Same here, though I never really paid attention to birds until my uncle bought me the field guide for some random reason when I was like 7.

From just thumbing through it a few times, I started recognizing birds outside which made me thumb through it more. I didn't have TV or video games back then so most of my after school time turned into thumbing through the book. I had alot of it memorized within a year it seemed.

From then on, any family trip to a new territory, I could name any birds that I hadn't seen in person before, or if nothing else, could quickly look it up.

http://fallibleideas.com/taking-children-seriously

Probably because some kids will grab the most complex book they can find and read it without being able to understand anything.

Sometimes I'll come across a feather when I'm out hiking and wonder what kind of bird it is from.

The U.S. Fish and Wildlife service has a really nice visual database of scanned bird feathers to help identify them.

https://www.fws.gov/lab/featheratlas/

That's awesome, I had no idea that existed. I feel like there's a feather-ID app waiting to happen there.

Not hotdog.

"Feather, not feather"

It only does feathers...?!

Similar: https://www.xeno-canto.org/, a crowdsourced bird song database (over 600,000 recordings)

Xeno Canto is great.

There is also crowd sourced photo database, in Flickr, "Field Guide: Birds of the World", where people have put in lots of effort to associate species tags to photos. Unfortunately, the copyright and license varies from photo to photo. Majority is "All rights reserved". So the only way to use it is basically to explore it through Flickr's website. It would be good if at least it had creative commons licenses, like Xeno Canto has.

https://www.flickr.com/groups/birdguide/pool/

Would it be too difficult to apply transfer learning ( or create from scratch ), an ML model to train a bird-song recogniser?

Edit: There are sonograms too ( https://www.xeno-canto.org/sounds/uploaded/ZWAQHOJFLZ/ffts/X... ). Could be done via image classification too.

Interesting enough to do on a boring weekend, perhaps?

This is one of those Kaggle contests. They have a database there that you can try your hand it.

> Interesting enough to do on a boring weekend, perhaps?

A weekend, or a PhD!

The Machine Listening group at QMUL covers bird audio - see Dan Stowell's work for example: http://c4dm.eecs.qmul.ac.uk/people/dans.html

Here's a Bird Audio Detection Challenge from 2018: https://besjournals.onlinelibrary.wiley.com/doi/full/10.1111... http://dcase.community/challenge2018/task-bird-audio-detecti...

I don't know a great deal about bird song myself, but I gather the subject has lots of interesting subtleties, especially when applied to live audio or real-world recordings where it isn't necessarily known where the bird soounds appear or how many birds there are.

I've had a bit of a go at this, (although I didn't spend all that much time on it).

One thing that challenged my expectations was that it seems you need a lot more data than you might think. Lots of species only have a handful of samples on Xeno-Canto, and the field recordings need aggressive processing in order to isolate the relevant parts, which is hard to do that in an automated way without inadvertently destroying lots of your valuable training data.

This has been done. But I can't say how easy it was. Check out the Android app BirdNet

It's very cool. Here's a demo of it on a (desktop) size screen: https://birdnet.cornell.edu/live/

eBird has songs also. But Xeno-canto seems to be the place where most people still record.

I still prefer to upload my recordings to xeno-canto; it's more focused on recordists than birders per se, it's easier to search for particular vocalization types, and I control the licensing for my own recordings. When you upload to eBird, your recordings are still yours but they are licensed through the Macaulay Library. That said, ML is doing phenomenal work so I don't mind contributing my recordings there as well.

eBird collects an interesting and important dataset of species observations, but it is far from the only one. There is usually a national dataset for each country that contains more records than eBird. It can be good to know that there is a website gbif.org that collects many of these kinds of datasets.

The eBird dataset is available from gbif here: https://www.gbif.org/dataset/4fa7b334-ce0d-4e88-aaae-2e0c138...

It contains 705 million occurences. Can be compared with a Swedish dataset, Artportalen, that contains about a tenth of that, but that is just for Sweden

https://www.gbif.org/dataset/38b4c89f-584c-41bb-bd8f-cd1def3...

In Sweden, majority of all birders report to Artportalen. I don't think eBird get much data at all, in comparison, for Sweden. EDIT: According to gbif, eBird has about 1 million occurences for Sweden, compared with 78 million for Sweden in Artportalen

In the Netherlands we have waarneming.nl which currently has 76 million observations. This also includes other animals, but it’s mostly about birds.

- https://mushroomobserver.org - crowdsourced database for mushrooms.

- https://www.inaturalist.org/observations - plants and animals.

https://bugguide.net - this one is awesome. has photo submissions for identifications

Sadly, bugguide is less useful to me here in Singapore. As stated in the header for the site:

> Identification, Images, & Information

> For Insects, Spiders & Their Kin

> For the United States & Canada

I love iNaturalist and am sometimes blown away by the automatic suggestions. I use it for birds, plants, bugs, mushrooms, trees, everything. Some time ago they added audio clips as well. It would be great to see more of these open communities inter-connected and sharing their databases.

Most of these platforms contribute their GBIF which aggregates it together. I contribute my sightings to iNaturaliast, but when I need the data for occurrences pull GBIF. I think this is perfect, because each site can compete on the best reporting interface, while not fragmenting data for data consumers.

I worked on the original eBird team back in the early 2000's. Seeing this post brought back so many memories of working very closely together on a small person team. At the time eBird was only a handful of people. The Lab of Ornithology continues to do amazing work. I'm happy they are thriving.

It's already been mentioned in at least one thread, but the accumulation of data (sightings, photos, sounds, etc.) has enabled the creation of some incredible tools. Probably the most useful for the average person is Merlin[0], which uses machine learning to identify birds based on a photo or descriptive characteristics (Size, color, habitat, etc.). I was skeptical when it was first introduced, but it's surprisingly good with most bird species.

eBird data is tremendously helpful for generating the ID because location and date information can be tremendously important for identifying very similar looking species. To use an example local to me, a smallish brown bird with a brown back and spotted chest on September 10th in my location is probably likely to be a Swainson's Thrush, but could be Hermit Thrush. Three weeks later that same bird is 99% likely to be a Hermit Thrush.

[0] https://merlin.allaboutbirds.org/

Is it open data? (meaning open licensed)

https://ebird.org/science/download-ebird-data-products

  "eBird provides open data access in several formats" - sounds good!
  "to logged-in users" - less open but maybe fair enough if they're worried about hosting a heavy download.
  "after completion of a data request form" - starting to sound quite restrictive.

Anyone know what actual license & copyrights are applied to the data? Anyone filled out the form and received the data?

The world of bird sighting is interesting. I have a friend who runs a cruise expedition company in New Zealand. They run trips in the south seas (Antarctica) as well as the North (Russia, Japan) but they have to reposition their ships between seasons. Those repositioning trips are 3-4 weeks at sea with little-to-no land in sight. So what they did to make money was to sell heavily discounted seats to bird watchers because there are some species of birds who are only at sea during this time and this is the only way to see them. Pretty fascinating way to make up some revenue during an otherwise dead period.

The night sky must be great too! Too watch the stars. Too bad it will move with the waves

When you get your sea legs, or a scopolamine patch the stars stop spinning.

For cameras or telescopes, there has to be some kind of stabilization bases out there.

Any info of your friend. Quite interested.

https://www.heritage-expeditions.com/

I’ve used daily eBird alerts in my county to spot some neat visitors. I haven’t had the time to go look, ID, and contribute, but that’s the beauty of crowdsourcing!

if you like this you will probably also enjoy birdnet:

https://birdnet.cornell.edu/

Birdnet is great! It is an app on your phone which lets you record birdsong and analyse it to tell you which bird you can hear.

I've never been good at identifying birds by their songs, but thanks to this app I'm getting better at recognizing the ones in our local areal.

I have a technical question related to this post. On the front page of the site there's a prominent announcement that the site will be offline for 48 hours for a database migration. Is this fairly common, and is this reasonable?

I've made some small sites but nothing that's ever gained real traction. I've always told myself that if I build something that starts to take off I should learn to do maintenance without interrupting availability. But I also remind myself that for anything I'm likely to build, it would probably be perfectly reasonable to take the site offline for an hour or two to take care of maintenance tasks and updates. I like seeing this approach, and I really like the communication style in the section "What can I do while eBird is down?"

Can anyone with experience maintaining a project of this scale comment on how reasonable a 48-hour downtime period is?

https://ebird.org/news/database-update-november-17-19-mainte...

It would not be reasonable for a commercial operation that makes money every minute around the clock. But they also need to build their systems differently to avoid downtime for reasons such as these. For a non-commercial organization, it can be different. It all depends on the value and cost of uptime/downtime.

There are even commercial operations who can have scheduled downtime.

Last time I wanted to check on my savings account, the bank was doing maintenance all Sunday.

Yes where I work (large ecommerce site), there is scheduled downtime once in a while. Not ideal if course, but continuous unlimited uptime at scale is not trivial.

I can't speak much about your technical question, but I can provide some context as someone who's used eBird heavily for almost a decade. eBird does go down for maintenance (both planned and unplanned) every few months, but to my memory this is by far the longest maintenance window they've ever done and it has obviously been telegraphed months in advance and almost certainly planned for a time when the site saw lower usage (Mid November is the closest thing you'll likely get to a worldwide low point in bird migration).

They haven't made any announcements about what they're doing during this window, but I would expect it's a pretty monumental change or several large changes they've been holding off on for a long time. It's also worth noting that this is a fairly small non-profit operation. It might be possible to do what they're doing without taking the site offline, but perhaps it would require taking their devs off other projects for several months to make it happen?

I've been using eBird for several years and this is the first time this has ever happened. eBird has great uptime usually. By the way if you are interested, the complete database in text format is around 300GB.

Does anyone else struggle with counting individual birds? I talk myself out of submitting a checklist to the "Great Backyard Bird Count" like every year because I worry I'm double counting birds and will mess up their data. When you have a bunch of feeders and live halfway in the woods it becomes an issue of: are there 20 mourning doves or 30? 40 finches or 50?

I thought they had articles and videos on how to count the birds. Like with the purple martins. How do you count hundreds of thousands of birds? Or bees?

I think you should not worry too much about that. There are usually special methods for those cases when it is important to get this right - such as, you need to observe for a set amount of time, and you need to report also the species that you know about but that you don't observe. For the popular "backyard counts", the scientists working with the data knows that it has been collected using a method where these details are not as accurate. Your data can still be valuable, but not to be used for the same purposes.

As a long-time eBirder this is something that can be stressful initially, but goes away when you understand more about the math. eBird has excellent documentation of what they hope for [0], but to simplify it I'll just say you should make your best conservative estimate. See one flock of 20 Mourning Doves fly over and another flock of 30 later? Maybe it's completely different birds in the second flock or maybe the initial 20 just picked up some friends? Truthfully the difference in data between entering 30 and entering 50 seems huge, but statistically it's not. In all likelihood you are almost always undercounting common birds and there are probably many more than 50 around if you're seeing a big flock like that.

Don't let feat of uncertainty get you down. A good faith effort to put in estimated data is actually useful, but not entering any data adds nothing.

0: https://ebird.org/news/counting-101/ https://ebird.org/news/counting-201/

Live bird migration maps using active radar:

https://birdcast.info/migration-tools/live-migration-maps/

There is a huge amount of excellent research that has been enabled by the eBird data. These studies are is part of a really exciting general trend that ecology is following, which is to integrate continental-scale data sources to test ecological hypotheses and theories at scales that weren't possible before, due to limitations of field work.

Here's a great selection of some eBird-adjacent publications: https://birdcast.info/about/publications/

This looks so cool. I am casually interested in bird watching, especially being able to identify them via their songs. However the moment I click to "explore" it requires a login. Yet-another login to manage before I am get to sample the data... window closed. I am not their target demo anyway.

> Yet-another login to manage before I am get to sample the data... window closed. I am not their target demo anyway.

My understanding is that eBird is only for recording sightings, and not for doing identifications. For identifications you want the Merlin app/page[0]. I don't think eBird/Merlin distinguish that clearly enough. But I think what you're looking for at this point is Merlin, and not eBird.

[0] https://www.allaboutbirds.org/guide/merlin/id

Merlin is a great tool as well. But I encourage anyone who is casually interested in birds to signup, its a great website with much to learn and I would say state-of-the-art data visualization.

I find it frustrating that Merlin and eBird don't integrate as well as one would expect

eBird is actually very useful for identifications, because you can check out all photos anyone has ever submitted for that species and pick up subtle variations.

Also, you listed the incorrect URL. The correct one is [1], which is an Android and iOS App. The page you linked to is actually the page for the species Merlin, which is a bird in the Falcon family.

1. https://merlin.allaboutbirds.org/

What a great tool it is! I just discovered it thanks to you, not the best season to see that but as soon as i'll hear one I'll use it for sure!

> Also, you listed the incorrect URL. The correct one is [1], which is an Android and iOS App.

Oops, thanks for correcting me on that!

If you're interested in learning to ID birds by ear, check out Larkwire[1][2]. (Full disclosure: it's my app.) I can attest that knowing what you're hearing is really a delightful way to expand one's connection to nature!

[1] iOS app: https://apps.apple.com/us/app/larkwire-learn-bird-songs/id54... [2] web app: https://www.larkwire.com

just search for ebird + your city/area on Google and go to the relevant page. Look at what's been cited recently to match against your own experiences.

eBird is a great resource for the casual birder looking to learn more. When I started, I used it to find out where to go to look for birds and what birds to expect once I got there. It also is an excellent listing tool, and listing in eBird helps you identify by knowing what's likely to be seen. As you get more into it, you'll be really glad you kept early lists in eBird. Push past the registration, it's worth it!

eBird powers the annual "Great Backyard Bird Count"

Mark your calendars for the next event on February 12-15, 2021

https://gbbc.birdcount.org/

For Swedish wildlife watchers there's https://www.artportalen.se/ which has been around for about 12 years or so. It has a dedicated section just for ornithology/birdwatching, but also offers reporting of any species. The site is backed by Naturvårdsverket, the Swedish environmental protection agency.

The ebird app is handy. You can use it to get a crowdsourced rarity estimate on an unidentified bird when you're in the field - handy when you're trying to decide between two species (the more common one is a safer bet).

Birds aren't real. "Bird sighting", "UFO sighting", etc...

Is the quarantine really getting to you that much? Go outside and get some sunlight. If I had a downvote option, this would be the first time I’d use it.

Pretty sure that comment was made in jest...

For those wondering what this is referring to: https://old.reddit.com/r/BirdsArentReal/

(It's a joke subreddit, not an actual conspiracy subreddit).

https://observation.org/ has existed for a few years. It's been used for all kinds of observations ranging from plants to birds.

I suspect more apps like this will pop up. While iNaturalist is great for general organism identification, the specialized communities will fill the gap where generalist AI is less capable.

Looks cool! Wonder if they'd do a collaboration with iNaturalist?

This is pretty cool. Several restoration groups near where I grew up do birding outings and record their results, but I have no idea if those results are aggregated.

Shouldn't that be "crowed-sourced"? I feel like they really missed an opportunity there and maybe even a Schitt's Creek tie in.

In where I live I heard a lot of bird songs or voices. Well any sound db with AI to id them ?

Cornell is the most definitive source for a lot of this kind of thing.

Good show!

Everybody knows the word....

More like crow-sourced amirite

Reply on news.ycombinator.com