borzunov

You can switch the chat bot (http://chat.petals.ml) to the "few-shot mode" and provide a couple of "task description & code" examples. Then you can add a new task description and it'll respond with code.

The underlying LM, BLOOM, had a few programming languages in its dataset, so it works at least with Python and C++.

A Petals dev here. Indeed, the public swarm should not be used for any kind of sensitive data (we have warnings about that in the instructions). If someone wants to process such data, we recommend to set up a private swarm among the orgs they trust (e.g., a couple of labs/small companies who don't have many GPUs themselves may set up a private swarm and collaborate to process their datasets).

Regarding homomorphic encryption (HE), I'm afraid the current methods to run neural networks in the HE fashion involve 10-100x slowdown, since they are mostly not designed for floating-point operations. We'd love to find a way to do it faster though, since privacy is obviously an important issue for many tasks.

A Petals dev here. It is not real-time, but we think the speed of ~1 token/sec may be enough for some interactive apps such as chat bots (especially, if you show tokens to a user once they are generated). You can try one at http://chat.petals.ml (heads-up: it may be laggy right now due to lots of HN users trying out the system).

Of course, you could do better if you have enough high-end GPUs to host the entire model yourself (3x A100 or 8x 3090). But if you don't, 1 token/sec is much faster than what you get with other existing methods.

Offloading is another popular method for running large LMs when you don't have the GPU memory to fit the entire model. Imagine you have an A100 GPU with 80 GB memory and want to generate text with BLOOM, a 70-block transformer model with ~2.5 GB of weights per block. For each token, offloading will load the first 1/3 of the model (~27 blocks) from RAM/SSD to your GPU memory, run a forward pass through them, then free the memory and load the next 2/3, and so on.

It turns out, Petals is faster than offloading even though it communicates over the Internet (possible, with servers far away from you). That's because Petals only sends NN activations between servers (a small amount of data), while offloading copies hundreds of GB of NN weights to GPU VRAM to generate each new token.

See technical details in the paper: https://arxiv.org/abs/2209.01188

I'm not sure it's a problem since newer versions of CPython usually introduce only minor updates to bytecode ops.

A while ago, I made a similar tool for Pocketbook eInk readers specifically: https://github.com/borzunov/remoteink

Yeah, I've remembered this episode as well!

I do the same in my module that enables increment operators in Python by patching the bytecode for two unary plus/minus ops: https://github.com/borzunov/plusplus

Hi, I'm the author of remoteink. Recently, the idea of using an e-ink device for programming/writing texts/showing dashboards/etc turned out to be popular among HN users [1, 2, 3], so I've decided to share my tool for this that works on the Pocketbook e-ink readers. I hope someone will find it useful.

[1] https://news.ycombinator.com/item?id=28432921

[2] https://news.ycombinator.com/item?id=28426741

[3] https://news.ycombinator.com/item?id=25939042

Years ago, I've made a program turning Pocketbook e-ink readers into computer monitors with some extra features: https://github.com/borzunov/remoteink

The update speed is quite good (comfortable for writing texts and browsing most websites) even when connecting via Wi-Fi.

Reminded me of the tool turning Pocketbook E-Ink readers into computer monitors, which I made several years ago:

https://github.com/borzunov/remoteink

This could also be used for the dashboards if connected to a remote server or a Raspberry Pi (no screen updates happen if the content doesn't change).

Besides that, it allows to zoom in/out, switch between the opened windows, and move around the screen with keyboard shortcuts. No jailbreak is needed for the reader.