Hands-Free Coding (2020) | Hacker News Reader

Hands-Free Coding (2020)

joshwcomeau.com

141 pointsjandeboevrie56 comments

As someone with RSI who writes code using speech recognition, before copilot, something like Talon was a pretty good solution. Awkward, kinda rough on the throat, not something you would use in the long term but better than the other options. I've always thought that a editor based on the abstract syntax tree in the language was a far better solution but that's a hard to solve problem and needs to be redone with every language.

I now use copilot almost exclusively when writing code. I've learned how to give it prompts so that writes code the way I would've written code. Occasionally it surprises me but when I dig into the solution, I realize that's a good one and start using it intentionally.

That said, navigation still a pain (sometimes literally). I mostly use the arrow keys and page up/page down to get to where I want to go. It is usually not worth the hassle to build special macros for editor specific navigation. I believe having meat macros like I developed with Emacs is no longer a win. Maybe if I had abstract syntax tree driven navigation, it would be worthwhile.

It's not beneficial to believe that you have "RSI". That's like saying you have "cancer" and are treating it with chemotherapy. What if you just had Thyroid Cancer, in which case you merely needed to take a special iodine pill. In other words, both cancer and "RSI" do not have a single treatment. RSI doesn't have to be this chronic, debilitating condition... but in my experience, you can't put too much faith in orthopedic docs/surgeons, either.

I find orthobullets.com to be a very useful source of information on orthopedic conditions. Give it a look and see if your symptoms really do align... https://www.orthobullets.com/hand/6021/cubital-tunnel-syndro...

I've been living with this problem since 1994. Some of the symptoms match up with cubital tunnel syndrome in that my hands feel split along the middle finger ring finger line. At the same time, the palms hands feel like they are burning, and my shoulders are caused me low-level discomfort. If I overdo it, the pain migrates up into my neck. sometimes my forearms and fingers swell. When the finger swelling happens, I lose mobility in my fingers.

I've also lost fine motor control and can no longer do things like write. One of the drugs they put me on gave me tremors that have not gone away so now I can no longer write or use a soldering iron. Sometimes tremors are bad enough typing is literally hit or miss. I miss juggling and cartooning.

For the first 10 years I chased doctor after doctor hand surgeon orthopedic surgeon etc. looking for a cure. Nobody gave me consistent diagnosis and even some of the recommended surgical interventions for some of my symptoms were not recommended because I didn't present in a way that indicated the surgery would be successful.

I've tried numerous things acupuncture, physical therapy, trigger point massage, and drugs like gabapentin and tricyclic antidepressants. The only thing that gave me any relief whatsoever was lidocaine injections in trigger points around my shoulders. The release lasted only 48 hours but it was so wonderful.

I've given up that there's any medical intervention that will help and I just have to live with it the rest of my life.

I went to a few MDs as well, for what presented like de quervain's tenosynovitis. It was only in my right hand, thankfully, so I could still mouse with my left. I also could use nothing but a split keyboard, which made it tough to use a laptop. And I could no longer use my phone with my right hand either. But after a while, I started getting symptoms in my left as well.

Obviously, no where near as bad as your case, but there nonetheless, for years. MD after MD told me there was nothing wrong with my hand, so nothing they could do.

Finally, after no longer being able to trust my right hand at all, because it would just randomly let go of whatever I was holding, no matter what size object, I finally went to a DO who takes a wholistic approach. After a few sessions of soft tissue therapy, involving massage, scraping and cupping, combined with stretching and strength training exercises, I'm pretty much back to normal. He worked from my neck down, to get the nerves to "floss" correctly through my system, down to my hand.

I've now moved past the treatment part, but as long as I maintain the daily stretches and strength training, both hands are pretty much back to normal. I do have minimal pain in the right hand if I have a particularly grueling work week, but I'm back to using normal keyboards and even laptop keyboards. And I can mouse and use my phone with my right hand again. I can even trust my right hand to not drop things anymore.

Of course, YMMV, but traditional doctors couldn't get the job done for me. I'm so grateful it didn't discourage me after a years long journey, and I'm grateful for the wholistic approach that worked.

My experience has been that orthopedic surgeons are BAD at diagnosis. This is at least partially because they are always in such a hurry (in the US anyway). You were smart to persist and I'm glad you found a solution. Your solution is essentially one that has not only helped me tremendously, but I think would help so many others too. So many who blindly trust what the docs tell them. I don't blame people for trusting in medical professionals, but I do blame them for believing what these docs tell them when it goes against what they believe (e.g. doc says hands are fine but they hurt constantly). At that point you MUST take responsibility for your diagnosis and treatment, and it sucks, and it's not ideal, but if I hadn't done it, I'd be unemployed and on disability by now.

Not saying this is related to your issues at all, but in case it ends up being helpful for someone - my uncle was a postman and lived with RSI for decades, early on they even gave him carpal tunnel surgery for it which just made things worse. After that, I think he gave up on doctors to a large extent but as he got older and was having more health difficulties, his daughter "made" him go see the doctor about them. Turns out he has multiple sclerosis and they now think the rsi may have been an early symptom of it.

Editing to add: I have dealt with RSI as a software developer myself. It started during a months long flare up of new digestive problems (which I didn't connect it to until later). I had to quit programming for years after being unsuccessful at finding long term relief, fortunately if I wasn't doing things that exacerbated it too much I would mostly feel fine and every once in awhile I would test things out to see if I had recovered. Close to ten years later, I found that I could begin working on the computer again and after some accidental experiments I realized that what was keeping the rsi under control was taking probiotics that I had begun taking for other reasons (and had kept taking because of the notably positive effect they had on my mood). When I stopped taking the probiotics for a few days the nerve and joint pain would start up again and recede when I re-started them. Eventually I had an antibiotic (which was not the first time in that time period I had taken antibiotics) and afterward don't seem to need the probiotic anymore <shrug>.

Carpal Tunnel Release is the most-common hand surgery in the United States, and it's estimated to have a success rate of around 90%. I have always thought the remaining 10% or so were misdiagnoses. I was misdiagnosed with CTS myself. It was a misdiagnosis because the condition hasn't worsened despite my continued keyboard and mouse use, and the symptoms do not worsen when I bend my wrists (really the doc should have seen that and not diganosed me with CTS, it's a pretty reliable indicator).

I don't want do downplay what you have been through and all the effort you have put in to trying to get an accurate diagnosis, and I know how discouraging navigating the US medical system is, but there have been a lot of advancements/changes in the area of orthopedic disorders since the 90s. If lidocaine in your shoulders is the only thing that has given you relief, you most likely have some form of Thoracic Outlet Syndrome. Lidocaine injections in shoulder muscles force those muscles to relax, releasing the brachial plexus (nerves) and thus relieving the symptoms of nerve impingement. Physical therapy is the primary treatment for TOS (e.g. building and stretching the muscles in the shoulder girdle), but surgery is an option as well. Ultimately they can just cut the ligament/muscle that is impinging on the nerve and viola, your symptoms are gone. In some cases, the nerves are impinged by something like an extra first rib that some people are born with. This too can be surgically removed. I encourage you to research TOS and come to your own conclusions since you know yourself better than I do, but that's what it sounds like to me.

Thoracic outlet was one of the proposed diagnoses but back in the 90s they had no idea how to treat it except very low success rate surgery. I went to physical therapy route and it didn't do anything for me. The PT person found a couple places on my spine and my right shoulder blade were frozen. I put up with a lot of pain when they tried to get it to loosen up.

I'm sorry that PT was not fruitful. Frozen Shoulder is rough. From what I hear, the surgery route is a lot more successful, but it seems like the main roadblock is convincing a surgeon to perform the surgery. They want you to exhaust all other options (e.g. PT) and run a myriad of tests and scans to be as sure as possible that TOS is what you have. I have also heard often that finding a good PT (PT meaning physical therapist OR personal trainer, I technically have a personal trainer) makes all the difference. All the best.

Have you tried float therapy? I did it twice a week for a while and it helped me a ton. Also talk Therapy and a lot of cardio seem to be key for me too. after years of inconsistent answers and shrugging from docs I was frustrated.

After reading about chronic pain and psychosomatic disorders I think it’s ridiculous how many providers don’t know much about that or are afraid to even mention it. Even after little talks discussing psychological causes of chronic pain with providers I experienced pain relief for days.

Haven't tried float therapy but most things like chiropractic, acupuncture and various forms of massage really didn't do anything except cost me thousands of dollars. Practicing Buddhism has help me let go of attachments to how my body had worked in the past which has helped as I've aged as other things fail.

Talking about the chronic pain in therapy or not, reminds me of how I got injured and how I was treated by my former employer. The more I talk about these feelings and these events, it triggers significant rage some 30 years later. The only way I found to deal with it is by practicing nonattachment. The easiest way to explain is that it's not that I don't feel the anger but it's an attenuated emotion. It passes by without sticking (much) and with some breathing, fades into nothingness till the next time.

> It is usually not worth the hassle to build special macros for editor specific navigation.

Why not? Wouldn't macros be more useful to customize your editing experience now that you use speech recognition, given that it's pretty unusual? In general, I'd expect customizations to have more value for people who use tools/languages in an abnormal way, not less.

I think I know why many injured developers turn to macros to try and solve their problem. When you acquire disability as an adult, it really fucks with your head. Every time you turn around, you're confronted with what you can't do anymore. Even a mild disability like mine I've had days where I could barely feed myself because it hurts too much to whole silverware. Thankfully those days are gone. Now things are better although I still need to hold the bottom as well as the sides of the glass to keep from dropping it when my hand spasms open.

Creating macros is a way someone can take control back from the disability. If I write some macros to do some of my work, I've taken back some of what I've lost. I used to be very critical of all these not fully useful solutions that disabled developers can come up with and I've become much kinder about it because it's more about the process of learning how to live with a disability and take back some of your agency that it is about creating any sort of accessibility tool.

It's hard to convey just how different the user experience is in your hands don't work and you try new speech recognition. One way to think of the relationship between the disabled person and UI customization tools is that customization tools are my agent or proxy for my hands. The macro solution send you down the path of having to memorize context specific commands and say a whole lot of words to get a small effect. A good adaptation looks inside the application context both in terms of what can be done and what data has to be operated on and then interprets natural language to generate what the system thinks is the desired result. That's the beauty of things like chatGPT. It's interpretation was right is usually pretty damn close.

With coding, this is why I like copilot. It takes my natural language and context of the code generated and generates code that is pretty darn close to correct. I speak a little bit and get a big result. I'll never have a force multiplier like chatGPT or copilot with macros.

btw, small pushback. The way we use tools and programs is not abnormal. Your comment shows just how much judgment people have when a user violates a designers UI vision or perceived physical norms. They think that speech interface is mediated by some form of artificial stupid should be the normalize standard but, whom I to judge. :-)

If a user can't make use of the UI provided, it's not the users fault, that belong squarely on the UI developer.

> The macro solution send you down the path of having to memorize context specific commands and say a whole lot of words to get a small effect.

The macros that I'm talking about are exactly the opposite - you say a small number of words to produce a large effect (or, however much effect you programmed into them), e.g. when you define an Emacs function. What macros are you talking about?

> btw, small pushback. The way we use tools and programs is not abnormal.

The vast majority of people do not use speech interfaces to tools and programs, so this use is abnormal by definition - abnormal means "outside of the norm" and this is very much outside of the norm.

> Your comment shows just how much judgment people have when a user violates a designers UI vision or perceived physical norms.

No, it does not. "Abnormal" is not a judgement or a negative term - it literally just means "several standard deviations from the mean" (for some value of "several") - you can describe someone having "abnormal intelligence" (and the connotation with that phrase is almost universally positive). Don't read negative meaning into other peoples' comments where it doesn't exist.

> If a user can't make use of the UI provided, it's not the users fault, that belong squarely on the UI developer.

I'm not assigning fault to the user. You're assuming things here also.

I would highly recommend checking out https://www.cursorless.org/, an editor for voice built on Talon (what’s being used here by Josh) and a VSCode extension. If for nothing else than to watch the dev live code with it.

Looks awesome. Is anyone using it in their everyday workflow (I guess those who WFH)? This seems a bit better than the OP post, with some structure and shortcurs of navigating between code. I think writing comments and later structuring them could be useful in cases where you need to think quite a bit before writing code. Or jumping between search results, opening tabs (tiling on windows through voice doesn't sound bad).

Pity there's no Jetbrains support.

> If for nothing else than to watch the dev live code with it.

YouTube channel https://www.youtube.com/@PokeyRuleJams/videos

As someone who may slowly be working his way toward RSI, I'd like to try to mitigate it before I get to the point of the author.

To that end, I'd like to try to build a maximally ergonomic setup (for the back and neck in addition to the hands) that doesn't necessarily have to conform to the standard chair+desk setup (e.g. a custom setup where one lies on their back with their monitor suspended above their face and their hands lying to their sides with a split keyboard).

In order to do that, I need the principles of ergonomics for the hands/wrists/arms, neck, and back - something that's been very challenging to find on the internet, as most of my search results are either trivial knowledge, blogspam, or just regurgitate the standard advice about various joints being at right angles without revealing the principles that drive that advice.

Where can I find those principles?

My understanding is that any position for too long becomes un-ergonomic, and really our bodies crave variability. I do my most productive coding on my couch in a weird slouching position, but I try not to do it for too long. I'll frequently switch back to my desk, and then try and take as many calls as possibly walking around the neighborhood. I can't focus on code at a standing desk or sitting on a yoga ball or whatever other people prescribe, but if you can throw that into your mix go for it, but sitting on a yoga ball all day is probably as deleterious as doing anything else all day.

There are definitely different ways to use your body with different effects, but I think the problem is over use of any specific form.

I appreciate your attempt to help, but what works for your body may not work for others - I'm looking for general ergonomic principles backed up by sound scientific evidence.

Have you started here? https://scholar.google.com/scholar?q=anatomy+ergonomics

I can't provide you with the scientific basic for correct elbow angles and back erectitude, but I will echo the GP's meta-suggestion of 'variety'. I've heard you're supposed to change position every 20 minutes, and stay no more than 2 hours in any position. Walk around, swing your arms. This is the first-order bit. Most other techniques and tools seem to be band-aids to help you tolerate longer stretches of time in a single position. Which, do what you gotta do. But changing it up is the most important thing.

I've got too much experience of this although undoubtedly not the same as you have, everybody's body being different.

What specific type of RSI are you talking about – in the wrists, back problems, neck, something else? RSI isn't just one thing. Where do you think it's going to hit you hardest?

My advice would be speak to a physio, take a break for 5 minutes every half an hour or more often, and above all get a voice recognition system if you're worried about your wrists/fingers being damaged from typing. Also, don't spend a ton of your life in front of the computer.

i'd recommend experimenting a bit, what works best for you. eg. for me lying flat on my back watching a monitor above me works for maybe 20min, then my eyes get dry, very uncomfortable. also check out http://www.earth360.com/ergonomic.html

Developed RSI and overcame it through learning stenography with Plover. I've used it exclusively (only using the keyboard to login to the computer) for 2 years. I'm a full time developer and it works great. There's a learning curve for a few months before being able to switch to stenography completely. Practice was running drills with a metronome for 15 minutes or so a day, progressing to the next lesson after hitting my target of 70 wpm at 95% accuracy. Comments (generally on this topic) often make it seem harder than it is or raise scares about impracticality. Practice was boring. However, I found that 30 wpm is plenty sufficient to do programming on the job, as thinking takes far more time than putting characters on the screen. Very rarely, at least for me, am I writing continuously for more than a moment. Stenography is a far superior input method than 1:1 character-wise entry. Given my condition, I didn't really have a choice. If keyboard entry works for you, great. If you have RSI or are developing it, learning steno is 1000% worth it. Prejudice against invisible disabilities exists. People dismiss them, judge you, and take advantage of your situation. Stenography is a way out of that. Also, see an occupational therapist (a professional who specializes in repetitive, work-related injuries). Good luck to anyone going through RSI. You're not alone and you will get through it.

Link to Plover to save you (potentially) painful typing: https://www.openstenoproject.org/

I recently started learning Plover!

Did you have any trouble using it for coding? It feels largely geared towards natural langauge and seems like there could be challenges using it for code. Though I haven't practiced enough to try yet.

Before settling on steno had you tried alternative keyboard layouts? Tarmak has relieved a lot of the stress for me but as time goes on I feel I may need a more drastic measure.

Hey do you mind sharing which keyboard you have used for stenography? (Im looking for something that has super low activation force switches)

The demo is incredible. Thinking of adding Talon to my daily workflow simply for the speed improvement. It would be awesome to speak out commands that are slow to type.

And honestly, with the custom python scripts support, this seems like a perfect use case for some GPT enabled programming. Something like "command: gpt add react button component".

I don't want to have to speak out anything, but it could be useful as an option. E.g. the author has an RSI.

If every developer at least pretends to embrace verbal programming, they'll have to get rid of open floor plans.

Good riddance! ;)

I can't currently see myself dictating code or code intentions into an editor, but to a chat system, where I can say, "with the code I just copied over to you, ..." or "with the code which is selected in the editor, ..." and then discuss the problem verbally. Because it gets somewhat tiresome to write out all those sentences.

The demo looks primitive, "go up second and go left third"? What is this, 2020? When now, in 2023, you can say "add parameters". Programming will only become higher and higher in abstraction. Why even say "parameters", just say "handle this use case".

The article is dated 2020 in the title.

Yes, that was the joke. Only 2-3 years have passed and yet we have advanced so far past that point, a sign of the accelerating things that are yet to come up to full abstraction over the machine interaction: just say what you want to achieve, let the machine handle the details, from what dependencies to use to the file structure of the repository.

> When someone says “I want a programming language in which I need only say what I wish done,” give him a lollipop.

(Alan Perlis)

;-)

Mr. Perlis died in 1990 unfortunately [1] [2], could just as well been in the prehistoric times. Of course, LLMs will not be able to give alone such a programming language, call it JUST, just say what you wish, but 5 or 6 more quantum leaps at the same level as the transformer architecture? I just can't see how it doesn't go that way.

AlexNet [3] was just 10 years, 9 months and 17 days ago, September 2012. The same time from now it would be the year 2034, February. Can anyone really dare to say we won't have the JUST language by 2034? I'll raise you even one further: in 30-40, hopefully 20, years we will have a JUST anatomical compiler and we will say programatically to the cells of various bodies: just build an eye here [4].

[1] https://en.wikipedia.org/wiki/Alan_Perlis

[2] The lolliquote is from 1982, Epigrams on Programming, epigram 93, http://www.cs.yale.edu/homes/perlis-alan/quotes.html

[3] https://en.wikipedia.org/wiki/AlexNet some prior work was 2011, Dan C. Cireșan et al., "High-Performance Neural Networks for Visual Object Classification", also quoted in the AlexNet paper. Both papers relied on "Convolutional Neural Networks designs introduced by Yann LeCun et al. (1989)". Curious if Mr. Perlis knew about Mr. LeCun's work. Here is Jürgen Schmidhuber giving a talk about the paper in October 2011, https://www.youtube.com/watch?v=rkCNbi26Hds The 10-year-old YouTube comment "I was still under the impression neural networks were rarely used any longer." is particularly funny.

[4] "The Collective Intelligence of Cells During Morphogenesis with Dr. Michael Levin", https://youtu.be/p4Fm7jLNrpg?t=3898 from 1:04:58 to 1:07:30, but the entire presentation is showing the future that is to come, in biology, in programming, in bio-programming.

I still appreciate the precision of being able to explicitly describe what I want to type. I also appreciate that this is all running on your machine.

It's awesome to realize that lot's of products and services are surrounded by the thought of how can most people, or how can the general public best use it etc. But in reality solving a problem for specific use cases can later on be extended to general purpose products and become a good business.

Lot's of successful companies started of just by someone trying to solve his own, or some colleague/friend/relatives specific problem

I feel the nerve related situation is getting worse because of tiny modern keyboards which focus on aesthetics and sound.

I have noticed split keyboard users in my office have better posture throughout the day whereas those tiny mechanical keyboard users droop their shoulders close and end up sliding down the chair putting their spine in a very awkward position promoting more severe medical conditions.

It could be just that split keyboard users care more about their posture but from my observation it seems everyone starts with a good posture in the morning but tiny keyboard users end up with a very bad posture within a few hours.

Would be interesting to know what others are seeing

I'm not a doctor, but I'm skeptical that the author has/had Cubital Tunnel Syndrome. Similar to Carpal Tunnel Syndrome, it isn't a condition that significantly improves, much less resolves with rest. Both conditions warrant surgery to remove the nerve compression.

I would guess that the author has relapsed by now, unless he made a lifestyle change not mentioned in the article (e.g. more ergonomic desktop, or started regularly exercising and stretching).

I will add too that I have yet to find a study that supports the notion that Carpal Tunnel Syndrome is caused by computer use. Vibrating tools are the main culprit, along with genetics.

Even if you were a doctor, making a diagnosis upon reading a post is not good practice.

It is easy to find subjects who did nothing else but the suspected activity develop RSI, so it is a weird claim to suspect it might not exist. For computers in particular, the "emacs pinky" is common knowledge, unless you'd think people suffer from collective hallucination (maybe a side-effect of emacs?).

It is definitely a condition that improves with rest, even if not totally and over a long period (months rather than days). There is also some level of evidence that suggest that psychological factors have an influence (there are a few obvious potential links that would explain it, but I'm not familiar enough with the research).

> For computers in particular, the "emacs pinky" is common knowledge, unless you'd think people suffer from collective hallucination (maybe a side-effect of emacs?)

I really like the name, considering emacs as "Escame Meta Alt Control Shift" abbreviation. All of that keys typically pressed by pinky.

Eh, read up on Cubital Tunnel Syndrome... it's illogical to expect "ergonomics" to resolve a muscular impingement of the ulnar nerve.

"RSI" isn't a thing. It refers to an umbrella of conditions and therefore doesn't have a single "cure".

Well, I am anecdata in that I was diagnosed with cubital tunnel, recommended surgery and 99% cured it with extreme ergonomics.

I was working on a large library with a verbose coding style and a clear plan. It was a year long sprint typing more than ever before. Started feeling pain down the side of my forearm and random needle stabs in my hand. Doc said I required surgery.

Instead I took a many-pronged approach to ergonomics:

1. 2 keyboards and 2 mice. Switched to a Microsoft ergonomic on my desk and a flat keyboard on my lap. A mouse on the desk and another on the other side on a pedestal by my chair. Switched between them all frequently.

2. Stretches and massage before/during/after typing. Many times per day. https://m.youtube.com/watch?v=TSrfB7JIzxY was a good help.

3. Held my wrists straight and fingers completely loose when typing. Imagine the classic zombie pose with their arms straight out and hands loose. That was me. Moved my whole arms to type rather than move my wrist or fingers at all. Same with the mice.

4. Stopped wearing any rings or a wristwatch.

Took months of this for the pain to go away. And, had to continue it through the project. But, since then I’ve never needed to type nearly that much and the random pain never came back. If I get on another typing bender I can feel a reminder that my hands don’t like it. And, if I ignored that I’d probably get right back into trouble. But, for the past couple decades I’ve been staring at code for 100x more time than I spend actually typing.

I would want to know in which fingers you were experiencing the tingling/numbness/pain. Unless it was only in your pinky and ring fingers, it wasn't Cubital Tunnel Syndrome (or rather, not JUST Cubital Tunnel Syndrome - it is possible, though uncommon, to have say Cubital Tunnel Syndrome and Carpal Tunnel Syndrome at the same time causing tingling and numbness in all five digits, however in such a case I would suspect nerve impingement in the shoulder or neck, before the ulnar and median nerves split from the brachial plexus).

Talon is such an exceptional piece of software. I’ve donated to the project for a few years now. I’m so grateful software like this exists, that it’s cross platform, and that it’s extremely customizable.

Relatedly, are there any accessibility tools to reduce eye usage while coding? I frequently get dry eyes and wonder in what other ways I can use them less. I follow the 20 20 20 exercises for eyes; it has been helpful.

I don't want to shut them off completely, but I want to reduce them as much as possible.

e.g. I am actively shutting down my monitor and using screenreaders for any long-form articles.

Okay, now I want to use voice commands to iteratively code using AI. You'd start out very high level, like, "Write a react component that does X thing." and then you'd modify it more granularly, like "Re-write the last 5 lines so it doesn't use variable X." I could probably do 90% of my coding without typing.

Integrate that with the IDE and let the integration modify and save files, commit and push, etc., then make it speak back to you with the voice of Majel Barrett and you got a winner.

- "Computer, run unit tests on this codebase."

- "Unable to comply."

Related: https://youtu.be/8SkdfdXWYaI (2013)

macOS has built in head tracking & control with expressions or additional hardware switches

Reply on news.ycombinator.com