Show HN: Booste – Run apps in EC2, synced to your own text editor
booste.io
booste.io
Sounds easy to implement, and many companies have attempted to do so. Generally, the solutions you've suggested are what they turn to, with mixed results. Rsync bogs down, bare ssh doesn't let you use gui editors, and the users I've talked to so far consider docker to be opaque.
Folks here on HN are strong engineers. They're happy working with current tools, because they've already gone through the ringer to learn them.
My assertion is that this skillset shouldn't need to be learned. When I write code, I don't need to choose which CPU core is used, and I'm grateful for that abstraction. I'd like Booste to abstract cloud ops in the same way.
The point about GUI editors not supporting remote development is basically false. VSCode, GVim, IntelliJ, Eclipse, Visual Studio, Sublime Text all support remote dev.
* https://code.visualstudio.com/docs/remote/ssh
* https://www.jetbrains.com/help/idea/2016.3/creating-a-remote-server-configuration.html
* https://code.visualstudio.com/docs/remote/remote-tutorials
* https://docs.microsoft.com/en-us/cpp/linux/connect-to-your-remote-linux-computer?view=vs-2019
As far as Docker's opacity, I guess that's a fair point, although in my opinion it's kind of unacceptable these days to not understand Docker or, if not Docker, then containerization in general, as a software engineer.Edit: list formatting
The Booste implementation of rsync is definitely not indestructible, especially during this beta. Been a pain in my butt, but making progress.
Main improvements over rsync: - .boosteignore. Just like a .gitignore, a .boosteignore helps filesync filter past bulky files that may not need to be monitored and uploaded every time. - end-to-end encryption
Admitted flaws of our current algorithm: It's built pretty ground-up in python. Haven't speed tested against rsync, but using it so far, I've never managed to edit code and run it through booste before the sync made it through to remote. IE the ssh tunnel doesn't beat rsync, mostly because files sync asynchronously. That said, I plan to do port it to a compiled language to cut time.
but thats besides the point, i only wanted to nitpick a little:
> Main improvements over rsync: - .boosteignore. Just like a .gitignore, a .boosteignore helps filesync filter past bulky files that may not need to be monitored and uploaded every time. - end-to-end encryption
thats a verly little improvement over rsync, considering it already has a file-based blacklist feature as well and is usually used over ssh, which is also encrypted ;)
though it (to my knowledge) doesnt behave like a gitignore, so whitelisting files inside of a blacklist is afaik not possible.
alias rsy(){
if [ -f ~/.rsync_ignore ]; then
rsync --exclude-from="~/.rsync_ignore" $*
fi
rsync $*
}I'll definitely play around with this alias and see how it performs relative to .boosteignore
This is a tall claim, given rsync uses ssh for encryption of connection.
Diving deeper, it seems rsync doesn't encrypt itself, but SSH obviously does. You got me on that one :)
Still experimenting with how to pitch it. Dropbox for devs makes sense, knowing the details, but would be super vague for anyone who doesn't already know anything about booste.
Other pitches that fall into that "short but vague" pitch.
Filesync + SSH
Docker + Heroku
*Cloud IDE without the IDE
(edit: list formatting)
Is there any site that ties open-source programs to their equivalent SaaS product? That would be very interesting to look at, to see what went into building these various SaaS products.
I do have the "my laptop's hardware is not enough" problem, and my current approach is simply spinning up an EC2 instance with the desired resources and ssh into it and then cloning the git repo I'm working on.
Normally, I set up the instance once (I have to add the ssh keys to my github account), and (if it's an expensive instance) start/stop it so I don't leave it running over night. Not a big pain point as of how I currently work.
I could see myself using Booste if it allowed me to, with a single command, spin up and sync bigger instances that would run for a short period of time (eg, for ML model training) and then intelligently stopping/terminating it. Also, if you integrate more than one cloud provider, I could switch to the cloud provider best fit for my needs (cost, spin up time, latency, cpu/gpu, etc).
A couple thoughts on the current implementation (you might be already addressing): - I generally have more than one ongoing project at a time. Having everything in ~/Booste sync'ed with every box doesn't work for me (aside from the fact I'd use a different location) - I sometimes do need to work with big data files, and even if I don't want them syncing all the time, I do want a way to retrieve them easily. - I also often work with jupyter notebooks, so I'd need a way to tunnel (like I do with ssh)
Hope this helps and good luck!
re: ~/Booste directory, just pushed and update yesterday that allows you to move your filesync directory around to wherever you want. It's not quite perfect yet, as I haven't built a way to clean old project files out of the codebox. But your new repo will overwrite anything that'd get int the way, so functionally, you're good to go there.
There's a "booste upload bulk" command, which takes a .boosteignored "bulk/" directory, for hefting code up to the codebox without bogging down filesync. Haven't get built a "booste download bulk" command, since customer pull recently has been more web-dev oriented, but I'll get on that once more ML folks play with Booste.
Codeboxes have auto-spindown. Default four hours live, but could make a command to monitor processes and spindown once done. Like a flag "booste --stop-when-done python3 train.py"
Nothing for Jupiter notebooks yet, and it's frankly going to be a tough technical nut to crack. Interested in exploring though.
(Edit: added below)
As for cloud flexibility, I'm sure more enterprise scale companies would absolutely require to host Booste on their own infrastructure. Like a bring-you-own-server model. Bring-you-own-server and multi-cloud not in the works in the near future, but perhaps mid-long term.
For years the ssh type lag made a remote editor too painful. Now we have things like mosh (with local echo) or syncthing or even native vim scp url support. What makes this better? I can’t tell from the website.
Booste allows GUI code editors like VS Code and Sublime.
I’m not saying your idea isn’t viable, but there are multiple players moving in this direction.
To me, the main thing you’ll have to convince me of is that this is better than editing code locally, but running the code in a local Docker container. For me, this solves the issue of “it runs on my machine”.
From my perspective, your main value add is trying to make executing and testing code on a remote server frictionless (it’s not quite there, but pretty good so far). And that you can edit on a low-resource machine, but effectively “develop” on a more powerful remote system. But these might be a tough sell if you’re targeting the HN crowd (which I’m not sure you are).
The docker point is fair too. Docker containers solve the machine problem, and are incredibly helpful teck. There are perks of cloud hosting (visibility by team, potential to monitor prod and have the codebox reflect it, performant hardware) that we can layer on top of docker container tech. Just hypotheses, to be clear. But to me, we're doing to Docker what Google Docs did to MS Word.
Coupled along with something like docker-machine for instantiating those remote instances, what does this setup not offer that yours does?
Booste is a command-line interface (CLI) that takes code written on your local editor/IDE and executes it in EC2. It runs remotely, in a pre-configured environment, but works with your local tools thanks to a background filesync process.
I want to fix the “it works on my machine” issue by bypassing personal machines entirely. As if Docker and Heroku had a lovechild.
How I came to building Booste:
Being a mechanical engineer turned software dev, there were countless times where my laptop (glorified tablet) couldn’t handle the workload needed, so I made Booste v1: a tool to host full desktop apps on AWS and stream it over RDP (remote desktop protocol) to clients. Some may recall the Show HN of Booste v1 six months ago.
Booste v1 sucked. Unusable latency, crap unit economics, and an infrequent pain point.
Through it all, the most common customer request was for a hosted VS Code. Prospective users wanted the perks of a cloud IDE (namely no-setup environments) but wanted to use their own tools. Diving deeper, it seems that most large companies build out their own (usually buggy) version of remote “devboxes”. The need seemed consistent and clear.
After the first Show HN and a failed W20 interview, I pivoted. I gutted the graphical desktop component, zoomed in on developers, and rebuilt as a CLI: Booste v2.
Running any shell command proceeded by the world ‘booste’ will trigger the CLI. The CLI tunnels that command through SSH to EC2, executes it there, and streams the stdout back down to your command line. Filesync makes sure EC2 is ready with your code.
Where “npm start” would run an app on localhost, “booste npm start” runs the same app in the cloud, visible to your team by URL. You can edit the code locally (VS Code, Sublime, Vim) and the save is synced and hot-swapped on the live URL within a second.
How the environment sharing works:
- The creator of a “Codebox” (EC2 instance) gets a barebones Ubuntu VM, on which they can set environment variables, install interpreters and packages, or set up databases. (pre-built env templates coming soon).
- Team members join the codebox by ID and password with a single “booste join” command.
- Codebox members get repos placed next to each other on the directory tree, so code doesn’t clash.
- There is “booste clone” feature in the works, where you can clone someone’s config without actually entering and messing with their codebox.
Under the hood:
-The CLI is built in Python using the Click library
-Codebox infrastructure is automated with the AWS SDK.
---Currently, they are EC2 instances, for the sake of getting the beta live. I plan to migrate to a more flexible containerized setup in the near future. Containers make much more sense.
-I built a Dropbox-like filesync, inspired by the rsync.
---It monitors a chosen directory on your local machine and syncs it with the codebox.
---When files in that directory are added, modified, or removed, the codebox reflects this within a second. Only differentials are sent.
---All communication is encrypted with frequently cycled keys, unlike rsync.
---Bulky files (ML data, dependency packages), which would otherwise slow rsync to a crawl are ignored by a customizable .boosteignore file.
-Since files sync asynchronously (on edit), the only latency is from ssh. There’s a 0.5 - 1 second delay on booste commands.
I’ve moving from closed to open beta. Fueled by covid and the now mandatory remote development teams, I made a free tier for you. Give it a try! Let me know what you like. Let me know what you hate.
- not having to install things like Docker and Postgres on my own machine, save space, not having to get a computer with as much space in the first place
- not having the heaving lifting on my machine means that I won't get the whirring fan and heat from my laptop
- not having to deal with managing different versions of node on my machine for different projects
- developer onboarding could be faster. There were a lot of little details to get up and running. Getting the npmrc token on my machine so I can install the private packages etc. etc. These could just live in the cloned cloud box that every new developer gets.
- lower the barrier of entry for new developers to just start doing something fun and useful. First day of bootcamp was this install-fest where instructor had to babysit everyone to make sure they got everything configured right. It's probably good for people to learn that stuff eventually but it would help to be able to just jump right in.
One downside: if the wifi is down you won't get able to get anything done at all.
My go-to cloud IDE so far has been Stackblitz. I use it to do little proof of concept stuff, share the link with people. But I prefer working in my own text editor. And you can't do anything more than a simple throwaway project in stackblitz.
I have yet to see what the usage will be, between having one living "shared" environment, or multiple "cloned" environments.
Wifi is an unfortunate issue. Probably a mid-term feature, but one of my early beta users suggested an "offline" version of booste where we clone images downward to the laptop, for use during internet downtimes. Could be a move.
Did you know rsync supports providing exclusion filters in files? See `--exclude-from` and `--cvs-exclude`. The latter uses .cvsignore files just like git uses .gitignore (and, I assume, you use .boosteignore).