Cursorless – A spoken language for structural code editing
github.com
github.com
I think able-bodied programmers will probably get annoyed setting it up, setting talon up, learning the phonetic alphabet and basic commands, but like.. if your hands don't work, this can literally save your career. this is a disability aid for creating things, it's vital and empowering!
the hard part IMO is getting back into the coding headspace you had when you could type. it's different speaking code, formatting it in your head before you say command phrases. everything's a little awkward, you can't get into the zone the same way.
For one, it needs to be programmed to do anything. This gets you all the options, which is wonderful, but also is a bit of a problem when you're looking into because you can't, ya know, program. And it's not a simple thing to interact with. Lots of moving parts and various behind-the-scenes interactions.
For another, there's virtually zero documentation. Like, officially none, and unofficially not much more than a few paragraphs.
And to put the cherry on top, it's a closed source mess of a program. It's not easy to get much in the way of useful information out of it. You can poke around some in a Python REPL, but that'll basically just get you the interface. And somehow, it's one of the most useful sources of information, as at least you've got something to go on. And as mentioned, Python, so you don't have much in the way of language-level guarantees to try to rely on to figure stuff out either.
Yes, there is the popular user config. To be clear, I'm not trying to be unkind to it. It managed to overcome a lot, and is very useful to lots of people for many of the reasons I stated above. But. It's also clearly got it's roots as one person's personal config that got popular. There is so much in there. It's basically it's whole own thing to learn on top of talon.
In terms of learning from it, well, it doesn't seem trivial. It's basically it's whole own thing, but also how it all interacts is all talon. You're basically learning two separate yet intertwined systems at once and have to figure out what bits belong to what. It's also got various bits from multiple iterations of talon's interface, so the same thing may be accomplished multiple ways. It's actually quite well commented in some places about this fact, which is really nice. But it does make you wonder what isn't well commented...
Ultimately I just gave up. It seems like a potentially hugely useful thing, but actually trying to use it required learning a bunch of stuff I wasn't otherwise interested in, figuring out the minutia of a complex unstable interface the hardest ways possible, and then spending a ton of time writing my own program/config.
Aside from the time involved, the amount of typing required was prohibitive.
So yeah. My feelings on talon... mixed, to say the least. Or most, judging by the above.
I have a clear vision for easier customization and adaptation in the long term.
Maybe check back every now and then, it's a labor of love and I plan to continue improving it for a long time.
As for Slack, I don't really consider that documentation. It's not really usable in the same way. It's more akin to IRC. It's potentially very helpful, even when the documentation isn't helping, but it's not the same thing. Maybe I'm just old, even though I'm not that old... I think...
Anyway, I hope this doesn't come off too negatively overall. I really like the idea of Talon and I'm comforted by it's existence, in case my hands start acting up again. From my attempts to investigate it I got a few glimpses into just how deep you've gone, even down into the speech model itself, to make it what it is. And I know any interface to something as complex as this is itself a large series of extremely ambiguous trade-offs that's just painful to make. And of course documentation itself takes time to create, and then needs to be kept up to date, and all that. I get it.
If it didn't seem to hold much promise, I wouldn't be so sour about failing to figure out how to use it effectively.
One thing that might be worth considering is moving to a source-available model, if full on Open Source doesn't suit your goals. Having the option to try to figure things out by looking at the code could be very helpful.
There's a lot more than a few paragraphs on the wiki (which is linked right after the setup instructions in the official docs), including ~30 pages about the scripting system alone: https://talon.wiki/unofficial_talon_docs/
> As for Slack, I don't really consider that documentation
I think you should try asking questions on this particular Slack before you write it off and recommend that I do something else. It is my official support channel and I am extremely active and helpful there.
> Having the option to try to figure things out by looking at the code could be very helpful
One thing to try - Talon has good Python type annotation coverage and the type information is shipped with Talon. If you point VSCode at Talon's Python interpreter (at `~/.talon/bin/python` on Linux/Mac or `%appdata%\talon\.venv\scripts\python` on Windows), the language tooling in your editor should then know about Talon's API and types.
You're right, I wasn't very fair to the wiki. I would have failed out way quicker without it. It wasn't intentional, it's just been a bit since I was looking at all this. My memory had dropped the details and left me with just my overall impressions.
That said, the wiki still was a bit short on some of the deeper details. I remember a fair number of places I'd feel like the wiki had just enough to tantalize the possibilities but not provide some of the important pieces to put it together into something useful.
And being community driven, I was a little wary of taking it as gospel. Even popular and stable software often has some niggling issues with community documentation.
> I think you should try asking questions on this particular Slack before you write it off
I'm not writing it off. I have no doubt it - and you - would be very helpful. I just don't consider it documentation, in the same way I don't consider a knowledgeable friend documentation.
Part of the reason I didn't pop in was that I was stuck doing a slow one-finger hunt-and-peck on a blank keyboard at the time. Searching the web was more than frustrating enough to make it real tempting to ignoring the cost of typing, never mind Slack. And partly I admit I just don't like asking questions in that format unless it seems like a reasonably complex/detailed/weird question.
> point VSCode at Talon's Python interpreter
Yeah, that probably would have helped. My setup at the time didn't have much for Python, so I was mostly using the REPL and basic searching.
I'm not trying to steal the spotlight from Talon/Cursorless though, they are amazing and super useful. But it is definitely a much more fragmented ecosystem which requires more work.
But I've recently revisted VS Code and found it improved enough to be usable for me. My hands have also improved enough that I can do a reasonable amount of typing now too, so perhaps adding language support would be an option. Hm...
Thanks for the suggestion!
The speech recognition engine built for it is great and the whole thing is free of charge, though I do recommend donating on Patreon [2].
Cursorless is a nice rethinking-from-the-ground kind of project. However, you can just as well keep using your current tools.
I've used Talon on and off with Jetbrains IDEs when I had some wrist pain and the overall experience was great, with my coding speed being fast enough without much practice. When voice typing, autocomplete is also very useful, so the introduction of Copilot was an additional huge improvemet.
But overall, the design of Talon, its config files, and the way this leads to very easy customizability, is really great. A bit like making your whole OS easily automatable and programmable (even if your OS isn't emacs /s).
But after discovering it and getting through the learning curve (and there is one, that's probably the biggest reason why more people don't use it), it's something I look forward to using for the rest of my life. Even as my wrists start to gradually recover, being able to use voice/noise/eye tracking in tandem to the keyboard, allowing you to use the best for each purpose, is just so much more productive (not to mention, cool! fun!) that I doubt I'll ever go back.
Reminds me of a talk I once saw where a developer developed arthritis and couldn't program effectively anymore, so he developed an almost "verbal vim" plugin for vim (or was it emacs?) that allowed him to write/edit code quickly using verbal utterances. I forget what that project was, but I wonder how these two compare.
[1] https://www.youtube.com/watch?v=8SkdfdXWYaI (demo starts 9 minutes in)
That's perhaps an extreme example of what is possible but it's incredible what people can adapt to.
The reliance on mono/skiasharp makes this a far less portable application, and to say there is Linux support seems disingenuous. Binaries only exist for Ubuntu, the binaries are said to work for non-ubuntu systems but nuget fails to install them.
Building from scratch leads to several undocumented dependencies and I'm still receiving increasingly inscrutable build failures out of the box. Feels like 2005... My patience ran out.
If anyone on an RPM-based system gets everything up and running, drop me a line.
(fwiw it's not skiasharp itself, it's just the mono/skia repo, which is C++)
I could certainly have broken run.sh open in vim (and should have), I could have looked into /lib and found the missing dependencies, but given that there was a binary literally called talon, the general expectation is this is the binary to be linked to my PATH and run directly. Using a run.sh is not necessary when the same could be achieved by making the talon file a shell script and renaming the binary to something else.
I contribute quite a bit to open source as part of my work on Talon, including the speech engine, the tools I use for model training and dataset creation, and I've upstreamed security fixes and other improvements to core projects like CPython and LibUSB.
> except when you contribute code to them
That clause isn't for code contributed to the Talon app, it's so other users in the community can use ad hoc scripts posted to Slack. Talon itself does not use any code under that clause and I still recommend users explicitly license their community code.
I've had the open source conversation plenty of times, but basically it boils down to: it feels like the bare minimum to protect myself from hostile forks given I have been working on it full time for over five years now, give it away completely for free, and provide high quality support.
It's also segmented so all of the primary functionality (e.g. voice commands and their behaviors) are implemented in open source user controlled code, and I open source plenty of the components of Talon.
You might read the handsfreecoding review [1] which covers this a bit.
[1] https://handsfreecoding.org/2021/12/12/talon-in-depth-review...
That's good and admirable! It doesn't change the fact that Talon is not open source, which makes it risky for users to come to depend on it.
I'm not trying to change your mind on whether Talon is open source. You can do whatever you want. The only reason I posted the comment was that I landed on your documentation site at first[0] and I mistook it for an open source project, only to find out later that I was mistaken.
- "very much not open source": Talon has a lot of open-source code, including the users' entire custom voice user interface necessary to replicate their setup on a different app (if a compatible app were to exist).
- "except when you contribute code to them": This is simply not the case, users are sharing the code in question with each other as customizations in the community chat. None of that user code ends up "in Talon".
---
Please read the handsfreecoding post. I have mitigated the continuity risks (e.g. bus factor) of being closed source, and I am fundamentally against selling out. It is extremely important to me ideologically that Talon is free and will continue to be free.
Talon is the best option for a lot of people to use their computer without pain. I would not have quit my job and spent years full-time developing Talon under an open source model. To my knowledge I am the only person currently working full-time on this problem.
Please understand that if you convince someone to not use Talon for this reason, their alternative may not be "use some other software", it may be "computers continue to cause them literal pain".
Sorry for the misunderstandings I posted in my first comment, they came from a quick reading of the EULA - unfortunately I can't edit that comment now.
Re your deleted comment about "hostile forks": allowing users to run whatever software they want is precisely the point of open source. It's not "hostile" to have different opinions to you.
The project looks cool nonetheless, but it is still true that if the users don't control the software, the software controls the users.
This is a business, not a pet project. I work on it full time and it is my meager source of income. I'm not convinced it is possible to make much of a living on open source.
I do mean hostile when I say hostile. I have a particular project of mine in mind, where the fork renamed, split the user base, and lagged behind my changes. They later just reset to my HEAD (as I was making way more progress than the fork), then removed attribution in the docs and solicited donations for the work I had done. I burned out on that project shortly after.
> if the users don't control the software, the software controls the users
Talon is free, runs offline, cares about user privacy and security, and gives an incredible amount of control to the user. The entire voice interface and behavior is defined in the user's code. There are several open source projects in this space and none of them are anywhere close to Talon on making it easy to build and customize your own voice / hands free interface.
Maybe that would allow you to keep making a living from it as you currently are, as well as allow people to see the code at some point. (Note that I'm saying "see the code", as you could just open source it without accepting contributions.)
Personally I'd be more willing to pay for a product if I knew that my payments helped open source it.
Not saying you should, just wanted to mention it as a possible solution.
Especially for Vimium (chrome extension, not Vim), I consider it a superpower, because it feels a bit like telekinesis. I can just look at things and make them happen with my keys, no need to navigate toward them.
[0]: https://github.com/cursorless-dev/cursorless/issues/710
[1]: https://github.com/nvim-treesitter/nvim-treesitter-textobjec...
This is at the core of why vim/others are still so popular. The micro-gains in typing efficiency translate to much more efficient problem solving because you don’t “lose your place” within your mental process. I think the same would extend to cursorless.
I guess it's a bit like navigating using only the map in the war room vs navigating on the ground without a map. The ideal route comes through a combination of both, and typing allows for the latter point of view.
Though after typing for a while, I'll take a step back and think about the overall solution again, going over it in my mind. Then back to typing.
That said, I gotta have a responsive editor.
In terms of just typing, I could probably produce something like 50,000 lines of code in a 40 hour work week in a moderately verbose programming language. In practice I'll probably crank out somewhere between 1000-5000.
It's the thinking that is slow. It's made even slower by adding additional tasks and context switching.
> Good question. For efficiency, we try to keep commonly used terms such as letters as short as possible. Note that the words we use for letters all one syllable, whereas the NATO phonetic alphabet are mostly two syllables. The words we use for spelling are also chosen so that they can be chained together easily and quickly, eg “harp each look look odd”. Easy to say quickly without slurring
Air, Jury, and Quench in particular strike me as ambiguous examples NATO would never go for. Krunch is just baffling. Plex breaks the scheme followed by every other (that it's the first letter in the word).
You can remap it to whatever you want, and a few users choose either NATO or a custom alphabet. But most stick with Talon's with minor changes (e.g. they might change a specific word if it's harder to say clearly in their accent, or if their mic or environment makes it more difficult to pick up specific sounds).
You can also use the English alphabet for spelling anywhere you can say a word, e.g. "say hello" and "say H E L L O" both work the same.
This isn't a voice assistant where you belt out a command once in a while then move on with your day, and this isn't a low fidelity radio link. These words may be said tens of thousands of times a day. Using three-syllable NATO words is slow and exhausting. It's common for voice programming systems to define their own alphabet based on a truncated NATO alphabet. I went further because I wanted to be able to use vim and play DCSS at very high speeds with my key commands instead of binding a bunch of app specific custom commands.
[1] https://twitter.com/lunixbochs/status/1378159234861264896
Notice that some users even start using Talon's noise support. I even changed "scroll up" to "suh" because I said it 300 times per day.
I wonder if I have it backed up or could track down what it was again. I was not much of a developer then, but these days it would be simple to wire up simulated mouse events.
My experience with eye trackers was similar. The clicking thing is partially solvable with eye tracker + foot pedal combination, but the biggest turn off with eye tracker was poor support for wider monitors or multiple monitor setup. Eye trackers also require you to be in a certain distance from it which was affecting my posture.
These were some of the reasons why I've built Cursorly https://cursorly.app/ It's still in the early stage but I'd love to hear some feedback. There is a free trial for a few days, but let me know if you want to extended it, I'd be happy to do so.
> If you have tmj/vocal cords/neck problems you won't be able to use eye tracker as a mouse replacement.
I disagree with these assertions. There are several options in Talon for eye tracking. Optikey is a nice suite of functionality, but it's Windows-only, while Talon is cross platform. I also have a few users who use the Optikey UI but prefer to use Talon's eye tracking to control it. I've certainly recommended Optikey to users.
The updated direct control mouse mode in Talon 0.3 can be used without any head tracking at all as long as your targets aren't too small.
The zoom mode explicitly uses no head tracking as well.
To click or trigger the zoom mouse, you can make a pop noise, use voice commands, or a physical mouse button, or keyboard, or any keyboard emulating input device (such as a foot pedal), or use one of the dwell options.
It would also be interesting to test brain interfaces on this language some day.
I am thinking about alternative input method for my project Glicol (https://glicol.org/) quite some time. This work is definitely inspiring. I am also thinking about how can we better support the blind with some feedback. If you have some suggestions, please let me know here: