The Rise of Audio AR (2024)
dbreunig.com
dbreunig.com
People working on these projects should study pre-digital forms of non-written communication. Commodities exchange pit floor hand signals. Pilot-ATC comms. Military radio. Etc.
If I want Audio AR, I want it to be like a co-pilot flying a jet with me. Crisp, concise, zero fluff.
Make it a slider for people who want to get chatted up vs those that don't.
Fear the Greeks when they give you presents.
But i fear it is too late.
To get good context, you need video data. To get good video data, you need cameras, lots of them, on your head.
To get them on your head you need them to be small and light
To get them to run you need batteries. But the maximum battery size you can get, without wires is about 1-1.5 watt hour.
Then, without making a custom wireless protocol, the maximum bandwith you can reliabily expect to user (with an iphone) is 1megabit.
That means you need to compress the world around you to 1megatbit a second.
Now, with a bit or work, like using eye tracking to segment what you are actually looking at(usually a 64x64 pixel image at 15-20 frames a second), and occasional wide angle view when the scene changed, you can build really good context, even reading a book.
But, you also need good location data, along with room classfication to get context. You can't really use GPS because they aren't accurate enough, and don't work all that well indoors. So you use visual odometry.
Once you have all that, you then now need to teach the machine to understand that 1mbit stream to work out if it knows the answer you need. oh and that has to work in a device that has ~10-15 watthours. or offload to a bit boy machine over a patching network.
At no point did I mention video output. If you _read_ the aria website, you'll see that those do not have a display, because they cost too much money, eat a shit load of power, and require a huge amount of packaging space.
There is a reason why meta's AR glasses system is 6 years late. (Orion was meant to be released as a product in q4 2020, then an SDK, and its pathetic state its in now)
> You would also know that video is absolutely not required for anything in an AR
Ok lets walk through a sample scenario:
I am visually impaired & I walk into a kitchen, I walk towards the electric hob which someone else has only just turned off. This is not my kitchen is I am only vaguely familiar with the layout. My hand goes out to feel for the side of the counter, however its getting dangerously close to hob. Using just RF, how will you get the pose of the hand, identify the hob's location, and identify that the hob is hot?
Oh course you can get an AR experience with just a phone and rough location, but its not that good or compelling. its similar to having notifications read out.
Sure you can have speaker diarisation so you can go full "x said y to z", so you can ask your AR assistant "what did x think about b" but again, you are asking the device, rather that the device knowing enough to provide valuable insight at the right time. The point about AR is that you are being given the right information at the right time without having to ask for it. Again location really helps, as does room classification (ie no recording in the toilet)
One of the best AR demos I have experienced was from the sound team in the sister org. They made a navigation system that used spatial audio and my teams SLAM stack to help you navigate though a forest. Instead of using TTS to say "move x steps forward" it used the sound of a dog. The dog panting was placed on the path, and if you strayed from the path it would get more and more upset until it started whining. You just followed the dog noised and you could walk about with your eyes shut.
Again none of that has a display, but it used cameras to provide precise location.
> refinement past VR led to an almost entirely monochrome interface
The monochrome is interesting, because there are a bunch of reasons why its that way. Most of it is price, colour waveguides are really view dependent, prone to sparkles is another reason. The colour reproduction of the early versions were terrible. Think projector screen in full sun. Finally, they are bigger because they need more power to run and more optics to massage the colours into the right place. Making waveguides is really fucking hard, if you want them curved, even more hard. They also tend to need exotic materials like silicon carbide. Silicon Carbide is a bastard because its almost as hard as diamond and is fairly difficult to etch chemically.
Finally different wavelength LEDs have different efficiency, its much cheaper to have single wavelength microprojectors, because getting all the wavelength at the same output, is expensive.
> I'm guessing your AR glasses were a horrible unfinished
lol if I wanted a display, we'd just use an oculus/pico. It also has a much bigger battery and a massive processor with a boatload of ram. The glasses are real and if you're at the right university, you can use them for your projects.
> that u self funded by being a complete idiot aka substandard engineer.
Haha nice. No, my team was paid to do this, and it made the CTO very happy. The algorithms that we developed are state of the art and in a whole bunch of products.