Show HN: CodeSpeak, the beginnings of a web-based speech-to-code tool
github.com
github.com
A group of folks using mostly Dragon and plugins https://groups.yahoo.com/neo/groups/VoiceCoder/info
A great pycon talk by Travis Rudd a couple years ago https://www.youtube.com/watch?v=7F4ylvA0Dh0
Best wishes for a successful project!
For example, say you want to loop from 1 to 10, print each number, and then print "done". The spoken-code equivalent might be something like "from i equals 1 to 10, print i, then print 'done'". When you translate that into code, it could reasonably end up as either of the two following snippets:
for(i = 0; i < 10; i++) { print(i) } print("done")
or
for(i = 0; i < 10; i++) { print(i) print("done") }
The worst part about designing a grammar for spoken language is figuring out intuitive natural language to substitute for the "silent" parts of code -- braces, semicolons, whitespace, caret position, etc. Obviously Python doesn't have those symbols specifically, but the concepts still exist -- if you start defining a function (with "create function [function_name]" in your code), what extra language do you have/need to say "okay, we're done defining this function and ready to go back to writing code wherever we were before defining this function".
Say you line up "create function foo", "set variable x 5", and "set variable y 8" voice commands. Will y get set to 8 within that function you created? If so, how do you signify the end of the function? How do you know where your caret jumps back to after finishing a function?
This is a neat project and I think it has a lot of potential for use if done correctly, but it's also insanely difficult to get something both powerful and intuitive when you're lossily translating spoken code to executable code because you're toeing a balance between saying something natural and saying literally just every character you would otherwise write.
In case it's helpful, here's my now-seven-years-old attempt and proof-of-concept in Javascript: https://github.com/drusepth/voice2code
It sounds like gibberish:
SENSLAM tiny test DOT add PREN BING CAMEL example test that fails SWIPE function PREN test
but it renders as
Tinytest.add('exampleTestThatFails', function(test))
due to where the cursor is placed between punctuation. There's also commands for cursor movement and selection modification (expand to containing bracket/quotes, deselect the last word)
There's presumably some underlying scheme to what they're using, but it's not entirely obvious. It's also not obvious how available this is, with a "send an email for further info" link. :(
Would be interesting to get a bunch of people to read out loud various programs in various languages and in pseudocode and hammer down what might be the most natural way to do spoken programming.
1. Try to mimic english grammar. For this, Attempto controlled english (http://attempto.ifi.uzh.ch/site/) is something to review.
2. Go simpler and reduce grammar as much as possible. Prolog would be a good fit, as a program is just a set of Horn clauses.
But you still do need to replace all sorts of brackets, like any language. At the moment I've got:
"bracket" == (
"brace" == {
"index" == [
(And "end $x" for the other side of it.)
Clearly not ideal, but works well enough for now, certainly a lot less painful than trying to maintain indentation in Python with voice only. (com select 20 lines upwards, reverse tab... Followed by swearing as it tabs instead of untabs.)
( parentheses (parens), alt. round bracket
{ brace, alt. curly bracket
[ bracket, alt. square bracket
The biggest confusion is between bracket and parentheses, in that people will use bracket to mean both. If I think there might be confusion I use the alternatives as they seem to remove all ambiguity.
Do you think an intermediate language is necessary? For example, instead of trying to implement speech to code for JavaScript, implement it for "VoiceScript", some purpose built JS variant which is intended to be unambiguous to process speech, and which transpiles to Javascript
So my plan was the first start with something using computer vision. So writing draft code and taking a picture of it having it insert into and editor. I figured I could have something really useful that I'd use early on versus working with voice.
I've got so many intermediary things before I can even start on that. I'm glad I'm not the only one who's heading in that direction.
In either case if you are reading this please comment I have some ideas and resources on this that you'll find interesting for sure.
The error recovery process on a failure in the system is simply to bad. It takes quite a long time to correct an error in speec recognition. So the idea is this that a high error rate is actually acceptable if the recoery delay is small. So while typing on a keyboard the error rate is quite high yet the user doesn't notice since to fix the error is not that painstaking. The same can not be said for speech recognition.
As such i feel that speech recognition would need a better recovery process. How and why exactly is beyond me however the Dasher project have some ideas on this by using a hybrid system. For the fully blind however this is a non option.
The next point that I'd like to make is that the most commmon source of errors seems to come from where the prediction algorithm have two or three probable alternatives for a word. In the case of predicting data in a narrow scope such as a programming language this is greatly mitigated.
To keep these ambiguity errors to their bare minimum the speech recognizer ought to be fed the alternatives that are possible. This can be quite easily be taken from the AST and is stricly defined as the syntax is much more restricted than commonly spoken languages.
I am currently trying to compile a list of alternative input methods. As such I find your work really interestingg. I intened to do a chapter on speech recognition soon and would love to have your feedback on it.
In conjunction with this i want to make a resource of boilerplate code for controlling various operating system and to have a signle resource for state of the art prediction models.
If you are interested please don't hesitate to contact me at: trbefr@protonmail.com
The reason why I am asking is that I'm doing some experiments on making them usable for productive purposes. While practicing I noticed that my posture improved automatically. See you are sort of automatically drawn towards the keyboard giving you a Mr Burns like posture. As if there where a gravitational pull in the screen.
As I was discussing this with another person online I realized that it might be of use for people with wrist pains as well.
My reasoning is this, bear in mind that I am by no means well read on biology but I think it might be given some thought. While using a keyboard you are forced to a certain plane. With a gamepad however you can move your hands, arms more freely.
Along with that I am looking into chorded keyboard methods and eyetracking.
Please respond to the comment if you think this sounds interesting!
For example, I do searches of the codebase constantly while coding, so I imagine I'd be reaching for the keyboard a lot anyway since there's no way a gamepad could compete on efficiency when doing something text-heavy like a search.
Think of it thiis way. You type with 10 fingers right and each can give an on or off signal. With a PS/3 controller you loose 4 fingers for holding the device. Still the thumbs are tied to 14 bits per second movement sensors + a button. Then you also have movement sensors and so forth and two fingers or more have analog input. So the bandwidth is there really. Also what you gain from more buttons you loose in movement time and errors due to Fitt's law.
And notice how I haven't gotten into Valve's "knuckle controllers" or glove solutions yet.
If you are interested I'd love to work with your particular use case and problems to get a working solution that suits your particular needs.
http://tbf-rnd.life/ or mail me at trbefr@protonmail.com
To fully give the wrists freedom of movement why not look at the knuckle controllers that Valve are using for VR at the moment.
Or why not gloves that can measure finger positions.
Bandwidth wise it should be possible. As a glove ought to have a 1:1 mapping with the data output of the hand..
Typing has become agony over the past few years.
This also helps me practice spell native English better as a English learner.