Hacker News Reader

Comment by HMH | Hacker News Reader

Hi HN, I created this program out of my personal need to be able to do some handwriting on my Linux system. I know, I could just have bought a graphic tablet but as I already have a tablet with the necessary hardware for note taking around I was unwilling to do so. And after quite some fruitless search for a solution to this problem Weylus (= Web + Stylus) was born.

It is written mostly in Rust, some C and a tiny bit Typescript and works by serving a webpage that uses the browser API to capture pen, touch and mouse events from the tablet. Additionally the desktop or a selected window is captured and encoded to a video stream via ffmpeg. This stream is played in the browser using the Media Source Extensions API.

This should also run on macOS and Windows but with very limited features.

View on HN

I have to say it is quite cool how easy it is got it set up. However, I for me the screen updates on my tablet were too laggy. I wonder how much it would improve if the stream had a lower resolution and limited colors.

Thanks for giving it a try! If you are on Linux you could try to select a window and check if making it smaller still results in lags.

Unfortunately the ability to playback videos via MSE API with low latency seems to vary quite a bit from device to device. You could also try an older release (v5.*). Those work by repeatedly sending PNGs and updating an image tag, actually the gif in the Readme is from that version. If this works better for you I will probably add it back as fallback in case video playback is to slow.

I'm guessing it's slow because capturing the display and encoding it takes a while?

I wonder if you could integrate this HTML5 VNC client: https://github.com/novnc/noVNC , since the complicated stuff above will be taken care of by the VNC server.

I feel like the boss in Office Space saying "if you could just go ahead and...".

> I'm guessing it's slow because capturing the display and encoding it takes a while?

This is a possibility but encoding is already pretty optimized (ffmpeg and libx264 with ultrafast as preset), which is faster than my previous approach of just streaming PNGs. And streaming PNGs was not that slow either, even on a rather old laptop. That's why I suspect there's something wrong on the tablet.

Regarding noVNC: What latencies do they achieve? I am not a webdev so I guess they might have some more tricks up their sleeves, I will probably have a look. As far as I can tell noVNC does not support things like multitouch and pressure sensitivity so that rules out just using it (at least for my usecase).

Have you thought about having a companion mobile app instead of using the browser? If so, what made you go for a web based solution?

Yes I have. I have decided to go with using a browser for now as I simply do not have the time to support multiple mobile apps and I am not too inclined to pay Apple 99$ for a Developer account. Personally I need this to work on my iPad, so you could consider using the browser as my way to break free of Apples walled garden :)

If someone wants to build a mobile app I am more than happy to provide other means of connection to Weylus than websockets. This also might bring the latency down as udp streaming and more control over video decoding and playing is possible then.

Hi, -H-M-H- . Totally newbie here. I tried installing the deb package. I set up the uinput, but I still can't make it work. I'm using xubuntu along with an IPAD. Do I need to change the "bind address" on weylus? What could I be missing here? Thx

If you're on macOS and iPad this is exactly what the 'sidecar' feature does. Needs recent hardware though since it uses the video encoder built in recent Intel CPUs to make it less laggy.

I've used Sidecar, and I thought it was more of extending your screen to the iPad, with some basic input controls on the device as well.

It looks like this is more of a mirroring setup where you can use the tablet for touch-enabled input.

Sidecar works great with touch-enabled input as well—I personally use it for Photoshop/Illustrator projects, and it has completely replaced a traditional tablet (one less thing to carry).

Regardless, this project looks incredibly cool. The cross-OS support is a killer feature.

You need to sign in with the same Apple ID, which is such a joke. I have an iPad that I created a unique Apple ID for and I can't use it with my maxed out MacBook Pro without losing the purchases. Truly infuriating

Why did you do that?

Because my personal devices are personal devices, and I got the iPad in order to do research for a business I'm building on the side. I didn't think I wouldn't be able to use both seamlessly just because of an arbitrary decision on Apple corporate's side

I'm so mad about Sidecar. Works great, unless you use the wrong VPN (in my case, required for work). Why having a VPN affects a connection over USB, I have no idea.

Anyway, got a Luna Display and couldn't be happier.

Did you know about GfxTablet* (limited to Linux)? Any thoughts?

* https://github.com/rfc2822/GfxTablet

Yes I do! That is where I got the idea to use uinput from. But as this is no longer maintained and does not mirror the screen I decided to create Weylus. Something that is different in GfxTablet is that inputs are not directly mapped to screen coordinates but it works like a regular touch pad.

This is a pretty damn cool idea. Do you only use this at home? I'm thinking about how I would use this when I'm not home, I guess the wifi hotspot on my tablet would suffice?

I already used it the other way around with a hotspot created on my laptop. As long as you can connect your browser with decent speed it should work just fine.

VNC should be the ideal solution to this on a fast enough network ...and if vnc client can send a pen input to the host and host can see it as a proper HID device.

> ...and if vnc client can send a pen input to the host and host can see it as a proper HID device.

As far as I can tell this is not possible: https://tools.ietf.org/html/rfc6143#section-7.5.5

Also selecting a specific window, which is possible in Weylus, does not seem to work with vnc.

Reply on news.ycombinator.com