Mozilla Overhauls Speech-To-Text Contribution Interface | Hacker News Reader

Mozilla Overhauls Speech-To-Text Contribution Interface

voice.mozilla.org

237 pointsblendergeek42 comments

This is a nicely designed interface. Well done, Mozilla. Validating sentences is quite fun, listening to different accents from around the world. Try it out if you haven't already: https://voice.mozilla.org/en/listen

It's awesome that the dataset is offered with a CC-0 license: https://voice.mozilla.org/en/data, does anyone know if it includes the answers from the survey? I have a limited bandwidth internet, so I haven't checked it out yet. In particular, I'm wondering if there's user data on whether they picked yes or no, to implement "troll detection" - people who click one option all the time.

Trolling of crowdsourced data isn't unheard of. (NSFW language: https://www.reddit.com/r/pics/comments/cygfx/4chan_is_using_...)

Trolling of crowdsourced data isn't unheard of.

...in that case it was specifically done in retaliation to Google trying to get free mental labour from ReCAPTCHA users in return for being able to post to 4chan. Quite a different situation.

> It's awesome that the dataset is offered with a CC-0 license: https://voice.mozilla.org/en/data, does anyone know if it includes the answers from the survey?

I'm downloading it now, I'll have an answer in a half hour. Does anyone know if there is a torrent for it?

Well, download took longer than expected :).

Anyhow, here's a sample from the csv file:

  filename,text,up_votes,down_votes,age,gender,accent,duration
  cv-valid-test/sample-001224.mp3,but i felt miserable watching him wither away like a shriveled dandelion,1,0,thirties,male,england,

Not sure how some of these are being populated, but yeah; there's several additional folders including invalid mp3, a splintered train set (not sure how it was selected) and a test set folder.

Here's the README.txt. Looks cool! Have happy hacky fun! :)

https://gist.github.com/cwgreene/f7f4df4ddcd9da017b9f4694b3f...

Interestingly; many of the 'invalid' mp3's are actually (mostly) correct. Listening to them is interesting to guess as to why they were downvoted.

We also keep the README in the repo: https://github.com/mozilla/voice-web/blob/master/docs/corpus...

Thanks! Couldn't find the source of the Readme in the zipfile. Can you talk about what the update process for this file is? How often is it updated? Is there a way to just download the new files? Is there a tarball script for this in the repo somewhere?

I see that you have instructions for s3, are the files actually backed in s3? Is it possible to download them with s3 (possibly using requester pays)?

We have no plans to allow users to download the "raw" data from s3 (ie. before we perform the train/dev/test split). But we want to eventually build some tools to automate this. See here for some background:

https://discourse.mozilla.org/t/the-mozilla-guarantee-publis...

Can't wait to try it either.

We used some of the research around Mechanical Turk to find best practices for limiting trolling (e.g. [1]). Our approach thus far has been the two-thirds rule: if two out of three people say the clip is good/bad, we trust that. Also note, that we have seen remarkable low trolling numbers, and that most of the invalid are pronunciation mistakes or saying the wrong word.

1.) https://groups.csail.mit.edu/sls/publications/2010/McGraw_LR...

This is wonderful and addictive. One thing that comes to mind is that the UI allows for very little metadata - for example in some cases the audio has a slight mispronunciation even though the intended word was clear - wouldn't it be helpful to mark "difficult" cases like this? In other cases the volume is just super low or there is background noise.

The other thing is that it's very cool to see the "you helped us reach out x% goal" thing but it locks up all the previous / next shortcuts which means I have to switch back to the mouse after 5 entries.

> for example in some cases the audio has a slight mispronunciation even though the intended word was clear

had similar issue/concern. ideally if enough people mark something as correct, the variations and slight differences will get merged together. it did still bother me a bit, as being able to add a bit more extra data would probably be helpful. but... maybe they can add some geo-ip data - respondents from various areas would probably mark more stuff 'correct' from their own region. ???

Being able to mark something 'close', or rate it (1-5, maybe) would help. Just heard an indian accent reading "It's such an unfair world, innit?" The words are... correct, but 'innit' is somewhat idiomatic (especially spelled out that way - seems more UK-oriented text). The pronunciation was "correct" but "awkward".

Also... (too lazy to check right now) - if I create an account, can I see the 'yes/no' ratings of my own submissions?

> Also... (too lazy to check right now) - if I create an account, can I see the 'yes/no' ratings of my own submissions?

Not yet, but this is something in the works. You can explore our new experience with the evergreen link: http://bit.ly/cv-desktop-ux

Accents would be an issue for some, especially those who live in the other part of the world. Choices among accent depending on the region of the world would be useful.

We do have a issue filed to allow users to tag recordings with certain metadata, like noisy or male/female voice. https://github.com/mozilla/voice-web/issues/814

It is something we are still working on.

> The other thing is that it's very cool to see the "you helped us reach out x% goal" thing but it locks up all the previous / next shortcuts which means I have to switch back to the mouse after 5 entries.

That's a bug! Would you mind filing one here: https://github.com/mozilla/voice-web/issues

We have a workaround in place, btw: https://github.com/mozilla/voice-web/issues/1179#issuecommen...

Am using Brave on Android (basically chrome with afblocker). I accidentally mislabeled some (probably) correct samples because the waves move when you click the play button, even if the audio hasn't loaded yet and I thought it was just a blank recording.

Would you mind filing an issue? https://github.com/mozilla/voice-web/issues

On my 6th try, just to get an idea of what to expect, the sentence: "Birds feed their offspring with spiders, worms, slugs and bugs", was presented to me. Upon clicking play I heard "Fuck fuck fuck, shit shit shit, fuck fuck shit shit fuck".

No I shit you not.

Who was it? Miss, present thyself.

My biggest issue with this project is that more than half of the contributions are from people who have failed to record correctly, or who are not fluent in English.

I guess that's why they have a way for people to contribute by checking contributions: https://voice.mozilla.org/en/listen

That's what the second pass is for, right, to screen out actually unintelligible or misrecorded entries.

The English (in)fluency is more of a feature, though, than a bug. The goal isn't to produce a speech-to-text system that can recognize a perfectly miked BBC announcer. It's to be able to recognize a wide variety of people speaking fairly naturally in imperfect conditions, using whatever accent they use for casual speech.

> "The goal isn't to produce a speech-to-text system that can recognize a perfectly miked BBC announcer."

Wait what? The headline is about text-to-speech aka speech synthesis, not speech recognition (speech-to-text.) Are they trying to do both? It seems to me that you'd train both using different sorts of datasets. If you wanted TTS to be intelligible to the most number of people, training to to speak like a 'perfectly miked BBC announcer' is probably exactly what you'd want to do.

Train it to recognize many regional accents, but train it to speak with the most prevalent and universally understood accent you can find. So either BBC English or Californian/Hollywood English.

Although traditionally TTS engines have shipped with numerous voices, such that you can select either a British or an America accent for the English voice. It may be worthwhile to have other English accents too, maybe one for India (125 million speakers.) But if you trained a TTS engine to have a computer amalgamation of all possible English accents I really doubt the result will be considered high quality by anybody.

Oops. My bad. I accidentally got that wrong. Mozilla's goal here is speech recognition.

> But if you trained a TTS engine to have a computer amalgamation of all possible English accents I really doubt the result will be considered high quality by anybody.

For what it's worth, they do ask you to create a profile after your fifth sample, and that profile includes an "accent" section.

The headline was inaccurate (now fixed) — the Mozilla Voice project is about speech recognition aka STT not TTS.

It would be kinda interesting to have a TTS system learn from a neighboring STT system so that it gradually adopts your accent, though. I'm not sure if that would be more usable but it would be an interesting experience.

You were quite right that in the way Mozilla are using this, it's suited to speech to text, but running it locally is actually a reasonable way to record a single speaker set of data for use in TTS training.

That's exactly what I've been doing recently, and using it with https://github.com/r9y9/deepvoice3_pytorch/blob/master/READM... is providing reasonably good results - it definitely has my intonation (if somewhat crossed with a Dalek!!)

I'm more interested in seeing how a bucket as big as 'english' turns out when you put all accents from around the world together. It would be another interesting project to separate them. I home they're at least saving the browser locale or GeoIP info to help categorize it.

I find the contributions where the person doesn’t speak the recording at all or appears to be intentionally not reading the corpus to be more frustrating than those where the speaker is not perfectly fluent.

Really nice app. It would be a nice feature to add volume normalization as some microphones/speakers are very soft, and I can't hear what they are saying, while others are much too loud.

There has been some discussion around this, but no real movement yet: https://github.com/mozilla/voice-web/issues/336

From the description:

Common Voice is a project to help make voice recognition open to everyone. Now you can donate your voice to help us build an open-source voice database that anyone can use to make innovative apps for devices and the web.

I'll be the first to note that here's another piece of personally identifying information you just "donated"...

I won't contribute because speech interfaces are imo terrible and against the proliferation of Echo and Duplex like services, but the data collected is listed here

https://voice.mozilla.org/en/privacy

Speech interfaces have some limited usecases where they are much better than alternatives. For example in situations where you need your hands for something else, like in a car.

speech interfaces are developed by large corporations and there is nothing you can do about that. With Common Voice you are helping to create open-source alternative, which is a positive thing, I guess

Thank you for bringing this up.

Indeed Common Voice is not for everyone. We try to make it clear in our Privacy Policy [1] what pieces of data we collect and why. We do not publish email address or names with the data, and we even strip speaker identification info (so that a speaker's recordings are not grouped but instead everyone's recordings go into one giant bucket). That said, if this still makes you feel uncomfortable, we understand. And if you would like to contribute without donating your voice, you can always validate the recordings of others.

1.) https://voice.mozilla.org/en/privacy

People seem to have fun with things like this, so I can see this drastically increasing the contribution volume.

Even though is Mozilla I don't like the fact that they force you to receive emails if you one to add another language.

Edit: My bad, is only for the unavailable languages.

Just to note, we will never require your email address to contribute. There will always be an anonymous contribution workflow.

But adding new languages to Common Voice is a bit complicated at the moment, and we haven't built a way to do this through the website yet. So for now, we are doing this through a very manual process, and we plan to use email addresses to communicate.

Did they consider to use MTurk for this?

If you think about it, this is a Mechanical Turk approach, just done for free with volunteers not with the financial impetus of MTurk. That's why they've given particular attention to the second pass over the data.

If you want to read more about it, the GitHub repo and in particular the issues cover a lot of the obvious questions like this. They're here: https://github.com/mozilla/voice-web/issues

In the early days of this project, before we shipped the website (ie. ~March of 2017), we did some explorations around Mechanical Turk. The problem with the Mech Turk approach is that for recording your voices you need a lot of different people speaking (ie. 10s of thousands). But for languages other than English, Mech Turk simply doesn't have these kind of numbers. And indeed English is not that interesting to us, since there exists public data already in English (see LibriSpeech). There are of course other micro-task platforms popular in other countries (for instance, there's a myriad in Indonesia), but we didn't have the time to manage jobs on all these different platforms.

However, Mech Turk is better for things like validation, since you only need a handful of people doing the majority of work.

In any case, I have some very hacky tools we used for this exploration, if you are interested: https://github.com/mikehenrty/mech-turk/

Reply on news.ycombinator.com