Show HN: Weylus – Use your tablet as graphic tablet/touch screen on Linux
github.com
github.com
It is written mostly in Rust, some C and a tiny bit Typescript and works by serving a webpage that uses the browser API to capture pen, touch and mouse events from the tablet. Additionally the desktop or a selected window is captured and encoded to a video stream via ffmpeg. This stream is played in the browser using the Media Source Extensions API.
This should also run on macOS and Windows but with very limited features.
Unfortunately the ability to playback videos via MSE API with low latency seems to vary quite a bit from device to device. You could also try an older release (v5.*). Those work by repeatedly sending PNGs and updating an image tag, actually the gif in the Readme is from that version. If this works better for you I will probably add it back as fallback in case video playback is to slow.
I wonder if you could integrate this HTML5 VNC client: https://github.com/novnc/noVNC , since the complicated stuff above will be taken care of by the VNC server.
I feel like the boss in Office Space saying "if you could just go ahead and...".
This is a possibility but encoding is already pretty optimized (ffmpeg and libx264 with ultrafast as preset), which is faster than my previous approach of just streaming PNGs. And streaming PNGs was not that slow either, even on a rather old laptop. That's why I suspect there's something wrong on the tablet.
Regarding noVNC: What latencies do they achieve? I am not a webdev so I guess they might have some more tricks up their sleeves, I will probably have a look. As far as I can tell noVNC does not support things like multitouch and pressure sensitivity so that rules out just using it (at least for my usecase).
If someone wants to build a mobile app I am more than happy to provide other means of connection to Weylus than websockets. This also might bring the latency down as udp streaming and more control over video decoding and playing is possible then.
It looks like this is more of a mirroring setup where you can use the tablet for touch-enabled input.
Regardless, this project looks incredibly cool. The cross-OS support is a killer feature.
Anyway, got a Luna Display and couldn't be happier.
As far as I can tell this is not possible: https://tools.ietf.org/html/rfc6143#section-7.5.5
Also selecting a specific window, which is possible in Weylus, does not seem to work with vnc.
This is awesome.
This is important enough I'd donate right now, if (1) there were a donate button (2) it'd fund further development to the point where this was production-ready.
Bringing this from proof-of-concept to mass-market would be a huge amount of work:
1) From the video, latency would need to go way down.
2) Pressure support is important, if not critical.
3) Packaging. To be impactful, this would want to be distributed in Apple's store, and in Debian/Ubuntu repositories (or at least a .deb). Paid on the Apple side is fine (and not contrary to open source -- I'm paying for convenience, and that's a payment I'd gladly make).
Next step would be to go from mirroring to second display. This is less bad than it might seem on the surface; it doesn't involve a whole new X driver, so long as outputting from the current one to a phantom device, and mirroring the phantom display (same thing as if I unplugged one of my monitors; HDMI keeps going out, and I could continue to mirror it).
One possible way to get there is to get the architecture to a place where others can do that work (e.g. leave appropriate places for people to plug in, and appropriate breadcrumbs where others can finish the work).
Disclaimers: Of course, one donation isn't enough to fund further development. I'm not rich, so we're not talking massive donation. Aside from that, by the time you set something up, I'll likely be long gone from this discussion (you don't have my contact info to follow up). But that's also not an abstract comment; I donated to Digimend before, and this would address part of the same major gap. Take the offer more as an indication of the massive value this would provide to at least one person.
> 1) From the video, latency would need to go way down.
On the encoding side the only thing I can think of to improve latency is to try hardware encoding if available. Something like NVENC might bring it down. That leaves optimization for the network protocol to use, something based on udp is likely better suited than websockets. And finally mobile apps probably leave more room for improving playback latency (and enable other protocols than using websockets).
> 2) Pressure support is important, if not critical.
It is pretty straightforward to add new input methods, I will add some instructions to the Readme, maybe someone will step forward and add support for macOS and Windows.
> 3) Packaging.
There already is a .deb, although I still need to fix some dependency issue [1].
> Next step would be to go from mirroring to second display.
Apparently this is almost possible right now (at least with X11), see [2].
> breadcrumbs
I think this is the way to go, I will add some more info on how to contribute soon.
[1]: https://github.com/H-M-H/Weylus/issues/6 [2]: https://github.com/H-M-H/Weylus/issues/5
On latency, the thing other tools used to really bring it down is a wired USB connection.
And yes: This is something in which there will be an almost desperate amount of interest in some communities. I think the bottleneck on how much attention it receives is, quite frankly, whether individuals in those communities find out about it.
I'm committed at about 160% capacity for the rest of the fiscal year, but if this were a more normal time, I'd probably hack on this myself.
Just install the Steam Link app, and then minimize the Steam app. Now you have access to your computer from your tablet.
Also there are many devices (including some smart phones) where the web will work but that app might not.
But it is by far the easiest solution to install at this moment, with predictable results.
- have a qrcode when running the server instead of displaying the url, much more practical
- have a "relative cursor mode" to behave like a usual trackpad, where you move the cursor from its current position. I am not interested in seeing my display on the phone, just use it as a trackpad while looking at my display.
I've researched the possibilities for something similar a few months back but my idea was far far simpler, just avoiding the need for an app to scribble things down in a web app. Also, unlike you I didn't finish.
Hats off to you, for me this is one of those twice a year moments on HN where I see something utterly brilliant that no one else though about!
This seems almost perfect for such a setup. You can even get a graphics tablet for little money, since those 8-10" intel atom windows tablets from 2013 or so are super cheap now! Lag doesn't look so bad. Wayland support will be a huge headache, but is not a huge problem yet as long as you can run many apps via XWayland.
In an ideal world, Xournal++ would have a modern web-based front-end as opposed to GTK (which currently means it's de-facto Linux-only until GTK resolves issues with Windows pen/inking APIs).
Weirdly, GTK can do web UI, although i don't know what gotchas may exist. https://developer.gnome.org/gtk3/stable/gtk-broadway.html
Thinking about it, the only advantage would be if one anticipates a multi-head scenario—like drawing to a common canvas from a mobile OS and a desktop OS.
Would this look any different if we could a desktop OS like gnu/lInux on all of our computing devices?
As a Linux user, the iPad pro writting feedback is probably the only thing I envy from Apple's users. What they did is black magic.
It does as well as pencil and paper.
It's not as good as a fancy pen and nice paper, mind you, but it's good enough.
My major problem is straight lines. I haven't got that down yet (albeit with just a few hours practice). But writing has no lag, and comes out fine.
Need to see if this works with PostmarketOS and off to buy one old tablet.