The underlying classifier, BirdNET, is a traditional neural network and not an LLM:
It always seems kind of silly to me to throw everything at an LLM. I know they’re huge and can automatically handle a huge number of tasks but something in me finds it wasteful when we could be creating easily trainable, cheap to run bespoke models for a lot of stuff
From my testing of open weights LLMs with audio support, they basically are only trained to recognize audio as an alternative to text input, they treat audio as basically equivalent to a transcript, and can't recognize or distinguish things like music, accents, background sounds, etc.
So they're only really good for transcribing or summarizing or using audio input in place of text input for prompts, but not anything that requires distinguishing any information about the audio that would not be present in a transcript.
It can be tempting to try to use an LLM for a variety of tasks; kind of the whole thing about an LLM is that you don't have to do a separate complex training run for every task, but can just provide instructions in natural language. But it only works as far as what the training data covers, if the training basically always treated audio and a text transcript as equivalent, the model has nothing causing it to learn other relevant features of the audio. If there's enough bird call identification in the training data of an LLM, it might be able to do that, but I think multimodal training data tends to be much more limited than the text training corpus
My current approach, not yet validated, is trying to generate training data from masterclass recordings on Youtube, and then fine tuning MOSS-audio on a bunch of those. But I'm interested if there are better models, or large training sets I don't know about.
People will often reach for "easily trainable, cheap" solutions when they can; the reason people reach for Transformers and LLMs is because when you throw more data at them, they get better.
Determining which tool to use should be a lightweight operation but I’m not expert enough to understand exactly how much lighter than a full LLM call just to recognize it needs a different tool or model.
For instance, OCR is something that can be done locally with no access to a GPU but people (including me) still often use cloud hosted multi-modal large language models for it.
Many systems now use LLMs in conjunction with specialized models.
That's my bro-science understanding of it, anyway.
def get_species(): return 'Canis familiaris'