Semaphore: A Full-Body Keyboard
github.com
github.com
Please let me know if you want to join forces! You can find my contact information at: https://github.com/mristin.
I figured that I don't mind my kids playing computer games as long as they move. The first stage for me is to find a workable approach to DOS games. In particular, I thought about adapting ski-leu to race games such as Outrun & ilks.
These could also be of interest for inspiration as well - basic idea this Youtuber is doing is to use console gamer joystick re-mapper hardware to convert joycon acceleration readings to button presses/button combos.
Can you Lose weight by playing Breath of the Wild? - Ringfit Controller Mod Explained Version
https://www.youtube.com/watch?v=HYn3pWXMPF8
- Today we talk about the Ringfit Adventure controller mod for Zelda: Breath of the Wild. With this mod you can Exercise and play BotW at the same time.
I made a Ring Fit Adventure Mod for Mario Kart - Controller Bending
https://www.youtube.com/watch?v=CNZSfunw85A
- Today on Controller Bending we mod the Ring fit controller to play Mario Kart 8 Deluxe with. With this Ring-con mod you have to run to drive. Literally. Squeezing the ring-con activates items!
I'd imagine a lot of similar mods could be done via computer vision instead of using joycon accel readings.
I think the key would be to show up to a gaming console as a PS4/XBox controller instead of just a keyboard. I think there are Python libraries to do this [1] or just stick to Steam games that just need WASD controls. This allows you to tap into already huge library of high quality games and just focus on the OpenCV movement-> joystick button remapping part.
[1] Something like this - although it might be Windows only https://pypi.org/project/vgamepad/
I implemented a couple of mini-games using a dance mat as well:
https://github.com/mristin/dance-a-mole-desktop
https://github.com/mristin/judo-dance-desktop/
https://github.com/mristin/dance-cat-to-mouse
https://github.com/mristin/burpee-frog-desktop
https://github.com/mristin/dance-runaway-desktop
Dance mat could be even easier to integrate as it acts just as a normal joystick. The problem, though, is that many games expect reaction times achievable with a keyboard & mouse, but leg movements are a bit slower.
The problem with the dance mat is that you can not use it in a school class as you need extra equipment. It worked well in a dojo for my judo classes, though.
https://github.com/mristin/ski-leu-desktop/releases/download...
https://github.com/mristin/pop-that-balloon-desktop/releases...
If you want to run the code manually, install it in the virtual environment. Then you can simply run commands `ski-leu` or `pop-that-balloon`.
Btw., I use pop-that-balloon for basketball dribbling training, and a friend uses it for boxing. Have fun & please let me know if you use it for yet another sport :).
Edit: removed bullet points
Instead of MediaPipe try training a simple NN with a quickstart asking new players to assume the positions for fine tuning? :P
I'm not sure how well it will work yet, but for a game controller I might try to have users record their own custom motions - as in, "show me a few of your chosen [high punch] motions to match against in-game."
Rather than type every letter, a more efficient system is to represent sounds with keys. This is how stenography works. Each stoke of the keys is mapped to a series of characters; a letter, a grouping (such as 'sh' or 'ing'), or a complete phrase. Using such a system would allow one motion, for example for 'b', to map to a whole word, such as 'be'. Something as simple as that would reduce your risk of injury by something like 50%.
A major difference between stenography and typing is that steno is chorded: the signal is sent when the keys are released, not when they are depressed. I'm not sure how that would work with the full body.
Check out the OpenStenoProject for more about steno. Their community would probably enjoy discussing a full body steno theory (if they haven't already) .
Take care and thanks for all the effort that went into producing the video! It was a lot of fun :)
The Charachorder One is one example
The best part was that the speech was also signed for the deaf (that part was normal) and subtitled by a ..., well, ... stenography trainee.
He or she had all wrong, could not keep the speed, was mixing words and skipping whole parts. At some point they seem to have said to themselves "oh fuck it" and were typing more or less random words.
The whole country was with them, hoping that they will go on typing until the end (which they did). That was a memorable event and people actually learned about stenography and the fact that they apparently use a special keyboard in the form of a butterfly.
>>Downward Dog Driven Development
Imagine teaching some ML your personal body movements, and then you could have it transcribe text into Semaphore-output video (or live rendering)... At least well enough that Semaphore-input could read it?
You could probably combine semaphore keyboard with some sort of stenography software for faster input (e.g. Plover https://youtu.be/KZGuBV1xe64)
Here’s hoping this ends up in a Vulf music video.
A dance revolution style tutorial would be fun
Sure, here’s a possible game plan for the idea you proposed:
Gameplay: The player takes on the role of a survivor on a post-apocalyptic Earth, who has discovered the last remaining Fatline capacitator capable of sending signals to alien life forms. However, the capacitator can only be operated through dance inputs. The player must navigate through various levels, avoiding obstacles and enemies, while performing dance moves to activate the capacitator and send the rescue signal.
Strategy:
The game could have different difficulty levels, with more complex dance moves required for higher levels. The player could earn points for successfully executing dance moves, and lose points for mistakes or taking damage from enemies. The player could also collect power-ups along the way, which could improve their dance moves or provide temporary invincibility.
Plan:
1. Develop a basic game engine with the ability to detect and interpret dance inputs from the full body keyboard.
2. Create a post-apocalyptic world with different levels and obstacles/enemies.
3. Design a variety of dance moves that the player must execute to activate the capacitator.
4. Implement a scoring system that rewards successful dance moves and penalizes mistakes or damage taken.
5. Add power-ups and other collectibles to the game.
6. Test and refine the game mechanics to ensure a challenging and enjoyable gameplay experience.
Mechanics
The game mechanics for the game could include:
1. Movement: The player would use the full body keyboard to control the character’s movement through the game world. For example, the player could move forward by performing a specific dance move, or jump by executing another dance move.
2. Dance inputs: The player would need to perform various dance moves to activate the Fatline capacitator and send the rescue signal. The dance moves could become more complex as the game progresses, requiring the player to execute longer and more intricate sequences.
3. Obstacles and enemies: The game would feature obstacles such as pits, spikes, and other hazards that the player would need to avoid. Enemies such as robots or mutated creatures could also be present, which the player would need to defeat or evade.
4. Power-ups: The player could collect power-ups throughout the game, which would provide temporary boosts to their dance moves or other abilities. For example, a power-up could make the player temporarily invincible to enemy attacks.
5. Scoring: The game would keep track of the player’s score, which would increase as they successfully execute dance moves and defeat enemies. The score could be used to unlock new levels or other rewards. Overall, the game mechanics would revolve around using the full body keyboard to control the character’s movement and perform dance inputs, while avoiding obstacles and enemies and collecting power-ups to increase the player’s chances of success.
https://www.youtube.com/watch?v=6HJexftddVw
EDIT: I just realized that it's the same guy behind this gem, that I randomly ran across a few months back!
https://www.youtube.com/watch?v=5q-BQ2Q_pqI
(I Made a Remote Lie Detector to Test Zuckerberg's Pulse)
Definitely a brand new sentence.
> Semaphore (lit. 'apparatus for signalling'; from Ancient Greek σῆμα (sêma) 'mark, sign, token', and Greek -φόρος (-phóros) 'bearer, carrier')[1] is the use of an apparatus to create a visual signal transmitted over distance.
https://en.m.wikipedia.org/wiki/Semaphore
I was only familiar with:
> [A semaphore] is a variable or abstract data type used to control access to a common resource by multiple threads and avoid critical section problems in a concurrent system.
I always interpreted the programming meaning as that - traffic lights.
https://en.m.wikipedia.org/wiki/Flag_semaphore
Or if your Monty python restaging whuthering heights
Smoke signals are literally the oldest known semaphore, and fire is recorded to have been used to telegraph signals over 700km per hour in the Byzantine Empire.
Interesting that Byzantine light seems to have been slower than modern light.
Ah but it says, "The term originated as a whimsical irregular form of the verb think." but without citation. Hmm, I dimly recall some hacker wag saying something like, "it's the sound of a continuation hitting the stack"...
Maybe "plonk" counts? https://en.everybodywiki.com/Plonk_(Usenet)
I would think computer register comes from the noun, not the verb — think cash register, not to register a car.
I cannot think of a way that "semaphore" would come up in non-nerd conversation at all.
"Token ring" is something I learned in networking, and the Wikipedia article doesn't mention railroads at all. I assume that's where the term comes from, but I only heard about railway signalling tokens decades later. (I think they may have appeared in the Thomas the Tank Engine stories on Shining Time Station, but I was quite young at the time.)
> In geometry, a disk (also spelled disc) is the region in a plane bounded by a circle. (wikipedia)
So you wouldn't find those in an SSD, it would have to be a spinning disk sort of deal.
The separation isn't completely clear, but take a semaphore like a basic tool that produces signals with a strong predefined signification, like a red light in a highway traffic light. The optical telegraph have a basic signification on their symbols, like letters or numbers, but a wide range of meanings, like the message transmission.
Take a look to the wikipedia information: https://en.wikipedia.org/wiki/Optical_telegraph
Earlier in life I used to lament the lack of uniqueness to words and the existence of synonyms but later began appreciate the overlap because having the ability to talk about something with a variety of phonemes seemed to help create some conceptual distance in a conversation that was otherwise obscured by close sounding words.
But this almost lazy reuse of existing words, I'm still not sure how I feel about it now that I'm aware of its costs.
But there is a thread to pull on here in terms of tangible user interfaces and the replacement of keyboards as a traditional input system. Meta's CTRL-Labs (ex Myo Armband) are working on EMG where just signaling to your muscles can trigger keyboard strokes and the bandwidth is faster than fingers or so they say. There are several other R&D efforts in that direction from BCIs to free space gesture detection through different flavors of computer vision and sensor fusion. Mr Everything Is Hacked is doing it here just for the sheer joy of the journey but there are serious reasons to play around with qwerty alternatives.
There's Tap Strap, and not-yet-released TapXR, though they are more chording keyboards than "keyboard in front of me".
Or augmentation of keyboards: wand/stylus; 3D finger tracking; multitouch tablet. Extending rather than dumpstering existing refined comfortable power. My laptop used to have three extra fold-out cameras for TFA-like mediapipe games. Though tech resets are an opportunity to escape trap deadends, to "do better this time" - that thinkpad keyboard was a crippled and crippling 2-key rollover. And sensor fusion with diverse latencies (keypress vs pose-tracked video) is nontrivial - complex event processing with backtrackable app state. LLMs with UI/HID "languages" will be such fun!
https://web.archive.org/web/20151025130300/http://www.google...
Where you type email with your full body.
https://stackoverflow.blog/2017/03/30/stack-overflow-unveils...
[edit] I originally put U+262E in my comment, but apparently HN strips some unicode characters?
A good way to play interactive fiction games: X, I, U, D, N, S, W, E, Z, OPEN, CLOSE, GIVE, ASK/TELL ABOUT https://www.microheaven.com/ifguide/step3.html
How far away from effortless mastery are you?
I find your project incredibly cool. And your video is awesome. Well done!
command: lift left leg to ~horizontal thigh
control: lift right leg to ~horizontal thigh
The lifeguards at the beach I visit use it to communicate to each other from between lifeguard stands, and I'm always curious what they saying.
Plus, I thought it would be fun to be able to communicate with them in their own secret(ish) language.
...Actually they just happened to be the first cheap bands that came up on Amazon.
Then you could have a computer in the pocket or a bag and a HUD! That could be very cool!
Still I don't feel this would be really comfortable to type on.
It worked well enough.
Perhaps DDR-esque foot placement?
Love the project idea, the outfit worn for the demo video AND the fact that the "!" sign seems to be a "shaking fingers" power pose :D
The next version will be more customizable, so you can create whatever gestures you'd prefer with whatever means at your disposal!
Oh yeah, watch me