Hacker News Reader

Thank you for looking at XRAI Glass!

1. For multiple simultaneous speakers of comparable volume, it’s only as good as the underlying speech-to-text engines we’ve implemented/integrated, which is currently not very good. It’s active area of research and engineering for us and we believe we’ll make strides to improve things; but, as you rightly point out, solving the crosstalk problem is very difficult. For the more general so-called "cocktail party problem", we can do a good job of filtering out more distant/lower volume voices and other environmental noise. Choosing the right microphone can improve things further, for example by pairing a noise canceling Bluetooth lapel mic.

2. We allow one to project the subtitles at varying depth, within the capabilities of glasses. We're seeing an effective focal depth range for fixed apparent size of about 0.5m to 3m. If one also allows change in apparent size, to simulate perspective scaling, the range is higher.

Seems like the multiple microphone beamforming source separation algorithms are getting pretty good these days, maybe just adding a lot more mics would help?

Could you have an AI model that extracts some characteristics of the speaker's voice for each individual word, then translates that to color and font?

If the model was not confident about a word it could show slightly blurred, if it was loud it could be bold, perhaps(Although there's some stereotype issues) you could use different fonts for different pitches, whispers could be grey, quiet could be transparent.

Maybe there's a language model that can pick up overlapping words if you don't have the constraint of needing to sort them out into who said it, just show all the possibile words that could have been said by anyone stacked together, in a "not sure" color, and maybe the wearer would eventually learn to figure it out without much effort?

You could also try to stay consistent so the same speaker gets the same colors I'd possible, and also not reuse colors for new speakers that have been recently used, to best make use of the limited bits of data in font and color.

Maybe just by showing all the words from every speaker all together like that, the wearer would be able to figure it out even if it made mistakes in the speaker identification?

Regarding additions mics, yes that enabled more advanced spatial processing, especially when arranged in a determined and calibrated geometry. One could imagine glasses with several mics placed at optimal locations on the frames. Such multichannel audio could then be processed into multichannel spatial audio streams.

As for the rest, thank you for wonderful ideas! Everything you propose is technically possible. The difficulties arise first in assessing the increased benefit to users versus the increased complexity of user experience, and second in prioritizing the work versus other features. Over time we do hope to add additional selectable “skins”, which is to say different UI designs, that allow users to choose UI’s from a wide range. Everything from the simple to complex layouts, from accessible to exotic color palettes, from professional to playful themes, etc. I could definitely see more advanced visual representations of transcription uncertainty showing up in such optional skins.

Wow! Thanks for the response. This is exciting work, and I’m pleased to see it being iterated on.

Re: #2. I’m assuming the varying depth is manually-controlled? Or is it automated by some method? If it’s manual, can the adjustment be made while transcription is active? In other words, can I change the focal distance to match the speaker without interrupting the speaker?

All in all, cool stuff! Best of luck with the work.

Thank you for the praise and encouragement!

Indeed, as Dan said, one can change the depth on the fly. In fact, one goal of the development team is to make as much functionality as possible changeable on the fly. For example, you can currently change subtitle depth, pinning, and size on the fly, spoken language, subtitle language, microphones, and audio settings on the fly, etc.

I’d love for every setting and feature to support on the fly changes. That said, some things are currently fixed for a session, such as recording audio, and some third party software we utilize is less dynamic and forgiving of changes on the fly. For better or worse, in our software world of today, the sage advice of The IT Crowd “Have you tried turning on and off again?” still seems to hold with pragmatic force. And it still holds with XRAI ... sometimes ;-)

The depth is set manually but can be changed at any point, even mid conversation

I imagine it’d substantially increase the compute load, but I’d be curious if you could use multiple microphones and beam forming to separate out the streams of speech and feed them to the TTS algorithm independently.

Single-mic source separation is possible in an unsupervised manner today that could probably work better than beamforming both compute-wise and with regard to implementation difficulty (you'd just need a lot of recordings to represent the space of sounds you want to separate).

Do you have any references for this or a link to a commercial service? I'm currently in the process or trying to extract some background voices in a video (an interview where the faint background conversation is in English and the loud overdub is in Bulgarian). I tried Melodyne but it seems to only separate on pitch, not volume, and the pitchs are too similar (mono, three voices, all female) and words are made of lots of short phonemes which each create a "note" and makes editing impossible. I looked into Izotope RX as well and it does not seem capable of doing this either. There are services that can automatically add subtitles using speech-to-text translators but they are expensive, and I'd prefer to have the background voices rather than the Bulgarian interpretation of them translated back into English. It seems possible to do, in Peter Jackson's The Beatles:Get Back they were able to separate voices from loud foreground musical sounds and other ambient talking, but that technology was custom and doesn't seem to be publicly available yet.

https://arxiv.org/abs/2110.10739

I haven't seen it provided as a commercial service or free model yet, but there is open source code for Mixit that lets you train using the open source / canned FUSS dataset.

Thanks! That led me to this which looks like a good place to start:

https://github.com/google-research/sound-separation/blob/mas...

Maybe a combination? Even simple beamforming/stereo would be helpful to help display the speaker's location. For example, the "speaker 1" tag could appear on the left, center, or right of the display to give a spatial clue where they are located.

You could only display the speaker's location if you had some way to associate streams with the individuals. So you'd have to train an audio-visual association model.

The thing you could do is train a localizer on the separated audio. Phase is estimated by the source separation process, so you can actually train an ML model provided you have some ground truth (e.g. estimated human locations from camera detections)

Do you think a ChatGPT real time processing engine will soon be able to parse the cross talk for words that are detected and given a probability score that then can then be reordered/reassembled back into a pretty accurate sounding conversation displayed back to the glasses wearer?

This would be such a cool use case for the latest ChatGPT tech when it gets faster in the near future.

> For multiple simultaneous speakers of comparable volume

If you had more microphones placed in multiple spots on the glasses, such as up to 5 microphones - 1 in the center, 2 on the frames/end pieces, and perhaps a final 2 on the arms/temples. Then that would be able to catch conversation coming at a person from behind them, the sides, or directly in front, etc.

This is a classic curbcut in the sense that it will help those with heading as much (if not more than) the D/deaf community. Still very excited for it - agree with all your questions and concerns.

As usual, this marketing seems most directed at normate ideas about what disabled people want / need, but the tech seems very cool and there does seem to be potential. Without looking into the product deeply it seems like there are D/deaf people on the team, which gives me hope.

I do wish that we would just embrace the idea that using machines to make information available in many mediums is something all people can use and appreciate.

To make the comment easier to read for others, it looks like the commenter may have made a slight typo and meant to write "hearing" instead of heading, to convey:

> "This is a classic curbcut in the sense that it will help those with [edit: hearing] as much (if not more than)..."

yes, exactly, thank you

Is multiple simultaneous speakers at the same volume/distance actually an important problem to solve? I already can't have a conversation if that's happening and my hearing is fine.

Let’s say you’re in a meeting with colleagues who are having a side discussion about something, which you can contribute to. In this current setup, that conversation would not be displayed, while as a hearing person, you could context switch on a dime.

It’s a decently-common occurrence. There’s a significant amount of literature pointing to the fact that people who are DHH avoid group conversations because they can’t follow along (see references). This sort of falls hand-in-hand with speech localization, which is another unsolved problem (and likely is unsolvable without true AR).

https://journals.sagepub.com/doi/10.1177/1084713807301322

https://doi.org/10.1080/02674649366780051

https://doi.org/10.1111/j.1473-4192.1998.tb00121.x

I was about to make the same comment: my brain works the same way. A one-on-one conversation with low ambient noise, no problem. A bunch of loud conversations all around me? Forget it, I can't understand anything. I don't see why an aid for hearing-impaired people needs to handle this at all.

This is an extremely common situation at many restaurants and is a reason I don't really enjoy going out to eat with friends without prior investigation.

Is low latency wireless transmission feasible now?

The last time I worked on anything here, there were a number of problems with transmitting highly compressed, low resolution video data. The consumer devices could not handle just sending the packets. In my project we were annotating real time video and only sending back the annotations, but even that would cause devices to overheat and the applications to fail in really interesting ways.

Yes we demonstrated XRAI running on the Qualcomm Reference Wireless Viewer at MWC

I have a different use case, involving wearing them in the house: listen to what my girlfriend says, use some ML to analyze whether I need to know it and if so, put it on the display.

She has a different "speech mode" than I: she speaks while I'm reading or washing the dishes or whatever, and sometimes it's to herself, sometimes to Siri, and sometimes it's something she wants me to know.

There is a simple wetware solution, which is that she learns to say your name before saying something she wants you to know.

That would still require him to listen to his girlfriend talk. With all the technological innovation available today, nobody should have to listen to their girlfriends talk.

Do you have any idea how hard it is to train partners to use your name once they fall out of practice? It's like there is a negative reinforcement stimulus applied everytime they do use your name.

I noticed that my wife stopped using my name when it became more ambiguous as to who she was talking to because our kids are now older and conversations (really instructions and queries) are at the "adult content" level rather then "child content" level.

Retraining her will be challenging.

The Android voice recording app can already write a transcript in real time that indicates different speakers.

But not in AR

It would be nice if it put the captions over the speaker (in a speech bubble?)

You pin the captions next to the person. We did try speech bubbles, but it didn't look great, so we went back to simple subtitles.

There’s a significant amount of evidence pointing to the fact that “anchored” captions (a la speech bubbles, like a comic book) is a preferred way to render captions, assuming the bubbles are at the same focal distance as the speaker. This is solvable with “true” AR, but getting that compute into the lightweight form factor of the original video is an unsolved problem, and is a ways off.

We have true AR with 3D placement now on these glasses

>3. As mentioned by another commenter, this is a useful idea for people who lose their hearing later in life. That said, this is less (although certainly still) useful for people who have congenital hearing loss and primarily communicate via ASL.>

Someone primarily communicates with ASL and then there's me that doesn't know ASL. I can speak to them, and they can read what I've spoken. That works pretty well. They communicate with me via text to speech, or (I guess in the near future) ASL to speech - however that will work.

I mean, that's awesome.

ASL is heavily inspired by English, so it's usually very easy for people to become basically conversant in it rather quickly. For signs you don't know, there's literal "finger spelling" that's part of the language, so conversational learning is greatly aided by this.

Which.. aside from that, you could do this anyways. When I first started living with a deaf person, I just wrote things down on paper, and they wrote back. The glasses have some benefits over paper, but significant drawbacks as well, and many deaf people I've met have been trained to read lips.

This isn't adding much.

I would disagree! It takes time and resources to learn ASL or even fingerspelling. This would allow people without the time/resources to communicate effortlessly, which I would say is a marked improvement. That said, you could argue it would be a one-way conversation, which is a fair point.

>This isn't adding much.

We can communicate without seeing each other.

ASL isn't inspired by English.