Show HN: Offline sketch to image geneartor in a whiteboard
drawing.pics
In this post, I'll share all the technical details behind it.
So, what exactly is DrawingPics? It's an image-to-image and sketch-to-image generator that runs in a whiteboard, allowing you to draw your draft directly on the canvas and easily generate the desired image.
How does it work from a technical perspective? TLDR: Miniconda + Diffusers + Electron + Excalidraw
One of the most important aspects of creating this tool was finding a stable diffusion locally inference solution. There are many solutions available, such as https://github.com/leejet/stable-diffusion.cpp and https://github.com/apple/ml-stable-diffusion. I tested the C++ version, but the inference speed was very slow, and Metal GPU support still had issues (you can find relevant issues in their repo). Ultimately, I decided to use python to run it because PyTorch is mature and MPS support is well-established. And I chose Miniconda, because it can create a small, isolated Python environment to run the program.
The AI model should run in the background so that we can continuously produce images while you draw. We need to find an RPC method to enable communication between the Python process and Electron's Node.js process. The easiest way is to run a Python HTTP server, but the memory usage is too high. We need a more lightweight solution, so I used xmlrpc for memory efficiency, although there might be better alternatives that I'm unaware of. The AI inference part is handled by diffusers, which is great, but I had to apply some custom patches to make it work in this situation. This can be a bit challenging if you're not familiar with Python.
For the frontend, I initially used a low-level canvas library and tried to implement a drawing pad from scratch. However, it had too many details, so I chose a more mature option: Excalidraw. It's fantastic, with the only shortcoming being limited API support.
Finally, I combined all these technologies in Electron, ensuring they work smoothly on both the main process and the renderer process.
Ok, Is DrawingPics free to use? 80% of the project is free to use. I only charge a lifetime license for the effort I've put in, as I've spent 4 months on it and made numerous improvements based on user feedback. Currently, only Mac is supported. Windows support will be added later.