Show HN: Music Audio Search Engine Using OpenAI's Embeddings on GPT Descriptions

Hi, Freesound developer here, this is great! We're a small research team at a university[1], and we're also working on auto-captioning tasks that include trying to describe sound effects based on the audio content, although we're not doing anything with OpenAI or summarisation of metadata at the moment.

A reminder, we also have acoustic similarity using signal processing analysis and nearest neighbour lookup for all sounds[2]

Could you update the sound displays to include a link to the sound page on Freesound, for citation purposes? You can use https://freesound.org/s/[soundid] or use our embedded player. See the "Embed" option in the share box on the right -hand-side toolbar of any sound page.

Are you interested in integrating this into FS? Talk to us![3]

[1] https://www.upf.edu/web/mtg/

[2] https://freesound.org/people/lebaston100/sounds/243627/simil...

[3] https://github.com/mtg/freesound

Thanks for sharing!

Some initial feedback:

1. Can you provide some example searches that you think or demonstrative of good matches?

2. On mobile safari, the search results overflow and the scrollbar flashes as the results are painted: https://i.postimg.cc/85RMrJxQ/81-BA5335-6-AD9-4277-93-CD-D50...

3. Add some links so people can reach out to you for feedback/bugs

Thanks for the great feedback, I really appreciate it!

1. I'll definitely add some good examples, I think that will help with getting better sounds. 2. Working on improving the UI right now 3. Going to add this in the next push as well. For now reach out at jonathanmir115@gmail.com

This is great! Some suggestions:

  - Add download links with download attribute:
    https://developer.mozilla.org/en-US/docs/Web/HTML/Element/a#attr-download
  - Make the grid responsive with CSS minmax (and media queries):
    https://developer.mozilla.org/en-US/docs/Web/CSS/minmax
  - Display a (visual?) score for closeness of match
  - Add link to similar sounds (by the vector database closeness)
  - Add query parameter so the results can be bookmarked
  - Use HTML5 History API
  - Show tags from freesound.org (and allow to filter by tags from search results)
  - Generate visual previews for the sounds (like in Soundcloud)

I don't know how to use it. I put in "boom boom boom" expecting thumping base sound and I got a completely unexpected beat structure instead. How the heck are you supposed to describe sounds? Is there a secret language?

This was fun to play with. I was initially expecting to be able to search for single notes or instrument sounds, e.g. I tried "warm synth tone single note C4" but afterwards I realised its library has mostly fixed tempo loops.

Like the idea, would be great if you linked back to freesound for each sample so I could explore the author's other sounds.

With my web audio hat on I'm imagining an interface that lets you mix, edit and add effects to the sounds you've found to help create new ones.

Thank you for sharing!

Thanks!

Editing the audio on the web would be really interesting. I'll look into this some more.

Update:

- Redoing UI

- Adding good prompt examples

- Planning on cleaning up code and open sourcing. If anyone is interested in contributing feel free to reach out at jonathanmir115@gmail.com

- Going to try to embed the audio files to create similarity clusters.

- Might add folder upload so that producers can upload large loop libraries and have their sounds clustered or allow them to search their library through natural language.

- If you have any ideas for cool things I could add or improve, feel free to leave it in a comment or reach out through my email!

Cool! Could you implement a download button so if a user hears a sound they like, they can just download it, or save it?

you can download by clicking on the three dots on the audio element, at least on Chrome.

On Firefox you can right click and then choose "Save audio as..."

WOW, it gave me a great match on the first try. Here are some examples to test if you are not into music production:

"dark atmospheric pad"

"crispy sharp dark bass with a beat"

"groovy hi hat"

But!

"cowbells" I gotta have more cowbells!

I think you may need more than 3,000 grooves. I like the project idea.

I asked for "sad folorn trumpet sound over gentle spanish acoustic guitars"

I got: piano synth over drums weird beeping noises drums

Right now it only has grooves in the database. Going to add more sounds soon

We would love to help promote this? Ping me Joe dot Drumgoole at mongodb.com as we can get you some time on our podcast.

Nicely done.

On iOS mobile, the three per row causes overlap of the sound playback bars. (First per row's playback button is also off-screen, so cannot be used)

Easiest solution would likely just be one per row if the screen is small enough.

Nice project and great to see more stuff built with the OP stack (OpenAI + Pinecone). I'm from Pinecone and if there's anything I can do to support, send me an email: greg@pinecone.io

Thanks Greg, Pinecone has been great!

Really like this project. Curious to learn more about the embedding process. Can you describe the process of generating the unique description a bit more?

I used the Freesound.org api (https://freesound.org/help/developers/) to download a bunch of sounds to MongoDB. Specifically, I selected to include the ID, Name, Tags, Description, Previews, Images, and Analysis in the response. I then looped through all these sounds and combined the response info into a prompt for GPT Davinci, instructing it to create a descriptive paragraph about the sound. I combined Davinci's description with the Freesound.org descriptions and embedded it using Ada. I inserted the embeddings into Pinecone with the audio preview url and ID as metadata. Then I just embed the search query and compare it with the sound embeddings (Pinecone allows you to select what type of similarity, I used cosine) and return the 25 most similar sounds.

Fascinating.

What if instead of reading a sound library, you had a music generator on the backend? What might an integration look like if it used the existing samples for description, as input for a "more like this" kind of feature

I tried this with wonderful results – where is my Reaper now? cyberpunk pad "amiga game" "without beats"

Thanks!

What do you mean?

I mean, I am inspired to create some music with the Reaper DAW software. :-D

You could add visualization, thumbnails for sound. Because it's very hard to navigate.

This cannot be used without a government ID card in some countries:

- You require a Google account to use it.

- To create a Google account, the user needs a phone number.

- SIM cards are in some countries only allowed to be sold if you show your ID card. Landline phones require a bank account, and those also require an ID.

Now, call me old fashioned, but even considering that I have nothing to hide about my music taste, I am appalled at needing government approval to search for music :)

And yes, I know I am a party pooper here, but this pattern has already creeped all over all Internet services, so at which point in time should we start to point it out everywhere if not now?

Anyway, to be constructive: If you want me to be human I will happily solve dozens of captchas or have my CPU run hot for a while to generate hashcash :) No need to involve daddy Google.

Removing google sign in now. Any feedback in the comments would be great if you are able to use it!

Why not show people the product before creating a login?

Sorry about that, it is fixed now. No more login and 25 sounds

Stop being overdramatic. You don't need a phone for a Google account. You can even use a fake name, birthday, gender, etc when setting up your account.

source: I have made several Google accounts, but have only tied my primary account to my phone.

Depends on the region you're creating the account from. If it's a VPN, google definitely tries to get a phone number.

Then don't do that.

What country do you live in?

US

Google sign-in required for what reason?