Hacker News Reader

Comment by shabbyrobe | Hacker News Reader

> In the AR and VR future we won’t be typing.

Oh please god no full stop imagine how hard it would be to even write a simple hello world program example ellipsis linebreak linebreak three backticks no not that delete word delete word backtick backtick backtick c linebreak void main open bracket int argc comma char asterisk argv close bracket space open curly bracket line break tab print-EFF open bracket double-quote percent d backslash n double-quote comma argc close bracket semi-colon line break backtick backtick backtick

Chance of success? Zero. Time taken? Three minutes.

View on HN

Whether you think it's a good idea or not, there are those of us who can't function without voice programming/input.

This video demonstrates full coding proficiency with my Talon project, which still needs a lot of work (e.g. no auto-spacing yet) - https://youtu.be/ddFI63dgpaI?t=30 - this demo took me around 9min vs a 90wpm typist doing it in a little over 6min. I've also benchmarked inputting over 280 commands per minute with no false recognitions.

This is my project, with which I aim to change minds about voice input (even making it viable for real-time gaming): https://talonvoice.com/

The Perl video is embarrassing, but that voice engine is only designed for dictating english text. voicecode.io has some major architectural limitations (no dynamic grammars so it requires a Dragon restart to change commands or word lists, adds commands by adding them to the english vocabulary instead of a strict command tree so recognition seriously suffers), aenea, caster, dragonfly are built on a 90's Python project called natlink and have their own issues. "Continuous command recognition" is pretty much a must, which Tavis Rudd's video lacks (notice the pausing between words).

That looks great! Both the speech recognition and the eye tracking look really impressive. You should make this into a commercial project or something. Is there any popular application that already does this stuff and is as good (I am not familiar with this type of software)?

I had no idea eye tracking could work this well.

As far as I'm concerned, there's nothing remotely in Talon's ballpark for voice, and these demos are just a sort of raw baseline. Every part of Talon will see significant improvements as I complete more of the upcoming features. Tech like Siri/Alexa/Google Assistant is too natural language processing focused to be reflective of the performance I've already demonstrated from a precise control scheme.

To my knowledge I am the only available high performance eye mousing project to not require a separate head tracker (I only require a $150 Tobii 4C eye tracker, and I do not use their integrated webcam head tracking system as it is not responsive enough).

I'm additionally planning to remove the ~$50-150 Dragon requirement for Mac users in the mid future.

I'm very committed to keeping the baseline software not just free, but a net negative cost over using other free solutions. As far as making it commercial, my current plan is to eventually charge a small subscription fee for extremely well integrated video game control systems, and a moderate subscription for professional input schemes like CADD, A/V products, and entire language models built around natural language programming that would ideally enable people who become injured to continue to work without much of any downtime.

These are some of the competition:

Voice:

- http://voicecode.io/

- https://github.com/dictation-toolbox/aenea

- https://github.com/synkarius/caster

- https://www.voicebot.net/

Eye/Head Tracking:

- http://precisiongazemouse.com/

- http://iris.xcessity.at/

- https://github.com/trishume/PolyMouse

- https://github.com/trishume/FusionMouse

(huge shout out to trishume, who was very helpful and a big inspiration for my tracking system)

One of my upcoming goals is to get a hands-free world-record game speedrun into the Awesome Games Done Quick stream.

Your speed and accuracy are impressive, but that looks very unpleasant to recite. Human speech tends to have rhythm and cadence, but that sounded to be like reading a phone book. Very precisely spoken, very choppy, no comfortable flow.

First, it doesn't feel unpleasant as I do it. There are a few reasons why it might look that way.

I have very little practice at this point, especially considering I'm still changing my grammars regularly, so the editing video there is more a technical demo of the software's performance in spite of my own awkwardness while learning. That was my first and only take. I'm basically learning to type again, with cache misses as I try to remember commands and keep track of my position (like trying to remember where keys are, or the next note in a song as you pluck it out).

It's also possible the most accurate form of voice input does not sound like natural language. Finally, it's a different mental exercise to copy text that is already written versus writing and editing my own code, so I'm triply out of my element when recording these early videos.

Compare to Tavis Rudd's video from 2013, where he pauses between every command: https://youtu.be/8SkdfdXWYaI?t=1050

I've recorded my own version of his demo today for comparison: https://youtu.be/wt4PR5j7vBE

A voice interface to programming would either necessitate different programming languages, or macros - maybe gestural or vocal - of some sort to take care of the various ordering syntax that languages like { that languages require. As well as absolutely perfect voice recognition, including adjusting to when you have a sore throat or other problems.

I foresee a problem with maintaining code that has been written with gestures or voice.

How would that work? Would your IDE play back the voice or display animations of the gestures to you.

It just feels as if text is the simplest solution, but I'm open to suggestions.

I agree, just trying to think of how it would have to work if it was ever to exist.

I would suppose that voice/gesture written code would be stored as text, when reading you could edit via keyboard or your voice/gesture interface. This comes back to languages written for voice - probably if this became feasible a lot of these languages would be compilers from voice to another traditional text target.

But like many people here I have a hard time actually believing this would ever be useful.

In seriousness though, while voice input would not be how you do complex programming, you could argue that there will always be more specific and specialized tools for that.

For people who spend most of their time clicking with a mouse or pointing at pictures on a phone screen, I could see voice working with things like IFTTT to create simple "programs" or actions. That's essentially what happens when I say "hey Google, turn on living room lights at 6pm".

Someone needed to program the framework for that to happen, but someone also needed to develop the Arduino IDE that I use when I engage in the very limited hobbyist programming that I do. I get to use a simpler interface because someone more specialized and experienced created it with lower level tools.

In a sense, that "if time=18:00 then set livingroom_lights=on" command is a very simple program entered via voice command.

This one is old, but I don't think things have improved much since then: https://www.youtube.com/watch?v=MzJ0CytAsec

It's worth a watch. Hilarious, despite the bad sound quality.

There are improvements on it :) https://voicecode.io/

and improvements on that: https://talonvoice.com/

I found this to be very ironic based on the time it probably took you to type all this out haha!

Hehe yep but the major difference here is that I was chuckling the whole time I was typing. Can you imagine how bad chuckling would make the end result if you were speaking it? Chance of success: -1 billion.

I won't be switching any time soon, but it's certainly possible: https://www.youtube.com/watch?v=8SkdfdXWYaI

That's why only brain interfaces can replace keyboards

That's why you use Python.

While you were yucking it up, someone already did it. I'd say it takes about a minute:

https://youtu.be/YRyYIIFKsdU?t=9m9s

Of course, there's plenty of room for improvement. Why not have the assistant understand keywords and perform high-level tasks?

"import"

"create function named main"

"return"

"Add two int arguments"

Reply on news.ycombinator.com