'A mouse for your mouth': New device allows users to scroll with their tongues | Hacker News Reader

'A mouse for your mouth': New device allows users to scroll with their tongues

nbcnews.com

147 pointscannibalXxx70 comments

Co-founder of Augmental here. I really appreciate your kind comments. It's been a four-year journey to get here, and we're thankful for the amazing people who've helped us along the way. Feel free to shoot any questions you have my way!

I don’t have any physical disabilities, but I am still interested in something like this as may help be smoke less and up my productivity.

I am curious what this would be like pairing it with an eye tracker, so you could look at a particular area then as soon as you move your tongue slightly that’s when the cursor appears where you are looking at. So that the cursor never feels in the way or something you are dragging, and even less movement with the tongue. I’d give a regular mouse up for that in a heartbeat.

But yeah, congrats in seeing your idea out - impressive stuff :)

Thanks for your message and interest in the MouthPad^! In the future, we plan to integrate various health-tracking capabilities, including gas sensing. This could help monitor smoking habits and might assist in efforts to smoke less.

Your use case sounds entirely feasible and promising. We're also looking at the possibility of coupling eye tracking for cursor movement with the MouthPad^ trackpad, functioning as a gestural trackpad. This setup would allow for left clicks, right clicks, swipes, click-and-drag actions, and other gestures. It's an exciting prospect that could significantly improve the eye-tracking experience by overcoming the typically slow response of dwell-based clicking.

Stay tuned for these developments, and thanks for recognizing our work – it’s greatly appreciated!

This very cool and looks like it will help a lot of people.

Have you given any thought to the upcoming Vision Pro? I’m guessing it may depend on how it’s received by the public, if it’s worth putting the time into, but with its eye tracking and simple finger tap input, I’m thinking replacing the finger tap with a tongue tap would make for a pretty great experience.

Thanks for your enthusiasm and insight! We're in talks with Apple about integrating the MouthPad^ with the Vision Pro. The goal is to pair our tech with their eye-tracking system, offering a hands-free alternative for clicks and gestures currently done by hand.

Also, as more AR headsets become common, hands-free solutions will be crucial. Traditional hand tracking for clicking isn't always reliable, as hands might not be in the camera's view. The MouthPad^ could provide a consistent way to interact with these technologies, making computing more inclusive.

I cannot wait to see what colab people come up with to integrate your device with a head-tracking mouse, e.g. TrackerPro by AbleNet?

Haven't looked in to your product at all, but am trying to help an eighty year old genius succumbing to Parkinsonism.

That's a great idea! We're currently beta testing head tracking on the MouthPad^, which is aimed at improving upon traditional head tracking methods. This feature lets users recenter the tracker by simply placing their tongue on the trackpad, making navigation more intuitive and less fatiguing by reducing neck movements.

We're excited about the potential of the MouthPad^ to help people with various motor abilities. Its broad range of functionalities could be particularly helpful for individuals with conditions like Parkinson's.

If I wanted to support this in a user interface, what would the inputs look like?

The MouthPad^ connects via standard Bluetooth pairing and is compatible with most common operating systems, functioning like a Bluetooth mouse under standard Bluetooth HID protocols. In the future, we will introduce an SDK to enable direct mappings for more tailored applications.

This makes me very excited. I have Spinal Muscular Atrophy type 2 [1] and I have lost most of my physical capabilities save for 2-3 fingers and of course speech. Although I am now on Nusinersen [2] treatments I am still becoming weaker over time, albeit extremely slowly.

One day I would love to use this device because for some reason my tongue has not experienced any degradation in all of my life so it would be nice to know I have a fallback, should I need one.

Note that specialists are saying that another promising drug from Scholar Rock [3] would probably prevent any further weakening if used in conjunction with my current treatment. Unfortunately, the FDA takes a long time to approve new medications and I have heard this one is particularly special because there is potential for abuse by athletes.

[1]: https://en.wikipedia.org/wiki/Spinal_muscular_atrophy

[2]: https://en.wikipedia.org/wiki/Nusinersen

[3]: https://scholarrock.com/our-pipeline/spinal-muscular-atrophy...

I hope it works out for you!

Thanks for sharing more information on spinal muscular atrophy. I’m sorry you have to deal with this and hope a cure is found. I’m inspired by people like you who have to overcome challenges like yours.

Ever thought about going to the grey/black market? If the athletes can get it why can’t you? I could understand the reasons why you may not want to. I have chronic fatigue and if it wasn’t for black market medications I would still be stuck in bed until I ran out of money by which time the only state help I could expect is assisted suicide - which made the choice for me. Myostatin inhibitors peptides like follistatin 344 are pretty pedestrian compared some of the stuff I’m on. I would die long before the meds I’m on would get FDA approval.

Dallas Buyers Club vibes

How does one go about doing this? Are these RCs or more black then that?

Follistatin 344 is a research chemical / peptide, most of what I take now is gray market. In some places there are new laws specifically targeting research peptides which would make it black market.

It’s getting easier and easier to synthesize peptides with things like desktop chemistry and recombinant synthesis. And there are things like DNA/RNA aptamers that can be printed out at scale with Oligo sequencers so if you know the recipe quite often you can find a jurisdiction that’ll make it for you in bulk for a reasonable price.

Quite curious what research chemicals you've used against chronic fatigue (I thankfully don't have it or need medication)

The ipa/modGRF/Frag/BPC157/TB500 stack is pretty common and does help. T3 and Test can help but are not research peptides. Then there are the Khavinson signaling peptides which are coded in human DNA and known bioactive, but can be a bit dangerous. Not for the faint of heart. Most of the benefit I have gained is from psychopharmacology which is near impossible to get scripts for so I get gray market supply instead; low dose naltrexone, low dose modafinil (better than low dose abilify in my opinion), and amitriptyline. It’s not just what you take it’s also how you take them. To avoid dysautonomia you have to intentionally swing between sympathetic and parasympathetic states. I think there is a subclinical insulin resistance so meds like metformin and semaglutide (ozempic) at really low doses can help. TUDCA is good but over the counter. Im mostly back to normal but still get rough patches on occasion. As someone who is vax injured I’m interested in ACE-2 ligands which are supposed to help reverse the damage done by the spike protein. I’m also interested in off label use of copaxone. It’s massively complex and many people with this are readily willing to take deadly risks in trying new things so underground patient groups are far ahead of the doctors. We’ve already done bigger trials of some ‘promising’ new meds and already know they don’t work. It’ll take medical science many decades to find out the same thing. I have hEDS by way of multiple TNXB variants which have specific personality side effects (like Sheldon Cooper from Big Bang Theory) so it’s pretty easy to find other people with it as we almost always get ME/CFS and we react differently to specific medications - for some there is a resistance to anesthetics. I think in time the brain/gut axis with microbiota biome/virome being important. So diet and a phage cocktail could help as well as a targeted antibody neutralizing small molecules to fix specific autoimmunity problems. These are whole categories of potential medicines which are largely ignored my mainstream medical science.

Have you registered on the mouthpad website? they may get you one.

i love when my first reaction to a headline is "why on earth would anybody want that" and then the first comment completely answers that question.

Might I ask about your diet?

As a child, I had to wear a horrible orthodontic device called a bionator that forced me to learn to speak with my jaw closed, navigating my tongue around a formidable hunk of plastic and metal to make myself intelligible at all. (It took two weeks to figure out how to swallow and stop drooling on myself.)

Little did I know I was being prepared for the newest cursor control development 30+ years in the future. I'll be one of the only people who can use this thing and carry on a conversation at the same time!

This was done 10 years ago by the guy that runs the YT channel called Applied Science [1]. If my memory serves me well, he worked on R&D at Valve, but I could be wrong.

[1] https://www.youtube.com/watch?v=HyaiszgzIzo

He’s on HN too I think. His mouth controlled sustain pedal was also really really cool.

Easily the most interesting channel on youtube.

William Gibson's novel the Peripheral (2014) and Agency (2020) have characters who control implanted smartphones (like a HUD on your eyes and audio channel built into you ears) that are controlled by using the tip of you tongue on the roof of your mouth. It's possible he was inspired by the video or some other source but yeah it's clear the idea has been around for a while.

Speaker for the Dead (1987) has something similar so the idea is much older than that at least.

IIRC the book(s) mainly referenced "subvocal" controls, based on detecting almost-talking movements within the user's face/throat/voicebox.

I'd argue those are qualitatively different, more like a modern-day system of saying "AssistantBot: Send E-Mail", as opposed to a geometric mapping one could use for, say, painting a picture.

I could actually see myself using this, if it is reasonably precise. A lot of the software that I use is GUI-only, and I really don't like taking my hands off of the keyboard to reach my mouse.

Have you ever tried using a trackpoint?

Not really. The very first computer I used growing up (~2005 dell inspiron) had one but I could barely use a keyboard at that point. Haven't had access to one since. Do you use one? If so, do you think getting a standalone keyboard with a trackpoint is worth it for someone in my situation?

I've always wondered what kind of input device could be used to fix the ultimate first world problem, sore fingers from phone overuse

I'm looking for recommendations on eye tracking technology. My primary concern is with the current solutions' lack of precision and resolution. I'm considering a hybrid approach, where eye tracking is combined with mouse movements for improved accuracy. The idea is to use eye tracking to set an initial anchor point on the screen, and then use the mouse to fine-tune the position from that point. For example, if I look at the 'X' button at the top right of a window, the system sets the focus there. This anchor point remains static as long as my gaze stays within a predefined area (like a 3-inch box around the anchor). The anchor point updates only when I shift my gaze to a different area of the screen. Any insights or recommendations on this approach or other effective eye tracking solutions would be greatly appreciated.

I use a wheelchair and have partially disabled hands and arms from a spinal cord injury (quadriplegia/hemiplegia,whatever).

I saw this and got super excited to try to use it as a means to signal software to help me actuate the intensity of my TIG torch welder, especially after the video on their site showing such esoteric uses as a couple using it to control their sex vibrator and other kind of 'weird' lifestyle uses.

I emailed them to be a part of the beta, they said essentially "Eh, no, that reason isn't that great -- but we'll put you on the wait list for 1000 bucks for commercial units."

Eh, no. So I bought a cheap sip/puff switch for use with a power wheelchair, use it as a binary switch along with an arduino, and the welder itself is set to pulse/ramp itself to take care of the gradual application.

It works, but I sure wish I could have had the more finite control -- but i'll wait until someone doesn't gouge me for the work. I'm not into the whole medical premium thing.

If anyone else is in that situation I suggest they look at low voltage membrane switches with an isolated carrier or any of the various sip/puff devices out there that have dwindled in cost due to volume and time. No, that doesn't offer the same capability -- but maybe you don't need it to.

Ben over at Applied Science did a great video of his quest to build an analog bite sensor for a musician. It would be easy to adapt it to work as a TIG controller.

https://www.youtube.com/watch?v=ag_p8gayrTs

Oh, very cool. My undergrad thesis project was a $20-of-materials device that turned EMG readings into mouse cursor movements, it's always interesting seeing new frontiers in computer input methods.

The source code was sadly lost in the great GitHub Christmas wipe of 2021, but it was really interesting to test it out on myself. Since EMG readings are 1-dimensional but cursor movement is 2-dimensional, I decided to make it a beat based system, like a rhythm game - a small flex moved the cursor proportionally, a large flex rotated the movement of the next cursor movement 90 degrees counterclockwise.

If the source code was public, then it might be in the Software Heritage archive:

https://archive.softwareheritage.org/

There seems to be no information available online about a GitHub wipe in 2021. I haven't noticed anything missing personally. Care to share any information on that?

I thought that the teledildonics industry would have figured this out first.

Glad to see that it can help disabled folks though.

There's some patent trolls sitting on patents related to 'teledildonics' and it's largely stagnated things.

That's such a great idea and I'm sure it could help a lot of people!... as well as a surge of cunnilingus training apps.

Special thanks to cannibalXxx for sharing this today!

You gotta stick your tongue out of your mouth for that.

I used to believe that the TrackPoint was the ultimate mouse input, but that belief has been shaken by this concept.

I had a week mouse that had a track point instead of a wheel. It was great and I wonder sometimes why they don't exist anymmore.

It'd be interesting to find solutions to tongue position detection that didn't require contact with a specific surface.

Maybe we send small electrical pulses out from a under tongue transducer & then wear sensors that like a touchscreen triangulate where the activation site is.

What if we could make a tooth a small low res camera & just watch?

What's here feels like a really small specific range here. It certainly has uses. I imagine more fast-task use if there were a sensor that detects more natural tongue gestures around the mouth.

That's an interesting idea! Currently, our MouthPad^ employs a capacitive trackpad, which we're working on to enhance sensitivity for 3D tongue tracking. It's somewhat akin to a low-resolution camera – it's effective, but there's room for improvement.

Using that same technology, we're also trying to unlock whispering speech and eventually moving towards fully silent speech capabilities. This will expand the range of intuitive interactions possible with the MouthPad^.

We totally agree — smart teeth are the future. We're looking into developing a 'smart tooth' that could use radar technology for precise tongue tracking. It's a futuristic idea, but that's the direction we're aiming for!

In your mouth, you can now have regular teeth... and Bluetooth.

This is what you get when you think of HCI holistically.

This would be incredible if coupled with smartglasses

Almost like that scene from the movie Idiocracy

I've been told of all the muscles in your body, your tongue is the most resistant to developing repetitive strain injury (RSI).

When I was in school I prototyped a simple tongue switch using a flex sensor (idea is it would be coated in a biocompatible, foodsafe material) to replace the mouse button.

I'm looking toward to CES 2030 when this device will be offered as a permanent implant :)

Just ask Elon, he can help you today

>> 'A mouse for your mouth': New device allows users to scroll with their tongues

I imagine some VR applications for PornHub. How 'bout scrolling with other body parts? Focus on something from the rear :))

Something similar was created in the field of teledildonics a while ago.

The return of the joystick

More comments from a few months ago: https://news.ycombinator.com/item?id=35684828

>Professionals who use their hands a lot, like mechanics or surgeons, have shown interest in the device. The company has also talked with astronauts who would like to try it out at the International Space Station.

I'd be that guy using it public transit during winter because I don't want to take my gloves off.

I remember "mentally designing" something like this when I was a teen. I had a weird mouth guard with a flat surface that covered back of upper and lower teeth that I would glide my tongue on, pretending to be a cursor, and then click on my canine tips together for left and right click because I thought that would be more precise. Now I occasionally think about eye tracking to move cursor and teeth clicking to tap.

In case you're not joking about public transit and gloves, you can get gloves with pads on the fingertips that allow you to use touch screens. And if your gloves don't have it, there are pads you can stick on or liquids you can apply to make existing gloves touch screen compatible.

Not joking. Touchscreen friendly gloves mistaps too frequently, especially thumbs in one hand mode - essential when standing in transit. My solution is pair of thin, very well form fitting glove liners underneath gloves. Or using a stylus. Both of which requires two hands. Hence very specific day dreaming about controlling phone with eyes and mouth.

I will give the liquid a try though. I have some nice fit old leather gloves that aren't touch screen compatible.

That's a rooslessly non-native speaker intolerant naming! How am I supposed to buy one of seese?

You can already do face gestures in Mac which shouldn't be hard to send scroll commands with.

I can see one particular use for this device that reminds me of FufMe. Would go well with VR.

Do you want The This by Adam Roberts? Because this is how you get The This.

All I can think of is Yondo from Guardians of the Galaxy.

Cleanliness is a obvious concern with something like this, and I wonder how well people with dexterity problems let alone paralysis will be able to keep something like this clean. The one shown in the article has a lot of groves and valleys even on the bottom (tongue facing) side which look like they could trap bacteria.

Presumably they clean it the same way people clean dentures. The electronic parts make it more difficult, but with wireless charging and Bluetooth it seems like it'd be possible.

Ultrasonic jewelry cleaner should work

Paywalled so if the article already mentions this I wouldn't know, but just going off "scroll" the title, I wonder if this would pair well with Dasher as an input method to replace typing with a keyboard.

And yet somehow the idea that a third installment of the half life series is even possible is a distant fantasy.

Reply on news.ycombinator.com