Hacker News Reader

Comment by mnutt | Hacker News Reader

This, to me, is probably the most interesting thing about Sandstorm as it relates to open source web apps. Today, I can write an open source web app and post it to github, but it takes someone with operational knowledge to deploy and run it. (even if that's just "create a heroku account, create a new app, git push" you still need to know something about development) But it feels like open source web app distribution has been stagnant and possibly even a lull since the wordpress/drupal/etc php heyday.

My hope is that Sandstorm rejuvenates open source and indie web app development by providing a channel where end users can easily and safely run random web apps they find.

View on HN

Since you give the example of GitHub... Sandstorm itself doesn't use their own Gitlab app to host and collaborate. How can they convince others? Doesn't make me confident of their platform. Centralized platforms like GitHub exist for a purpose.

When the Sandstorm project started, we obviously couldn't host it on Sandstorm. But, we are gradually moving towards more dogfooding. For example:

- We use Etherpad on Sandstorm for all documents we write, e.g. design docs or project plans.

- We use Wekan on Sandstorm for project / task planning.

- We use Rocket.Chat on Sandstorm for internal team chat.

- We commonly share and publish files using Davros.

- We host docs.sandstorm.io -- and the analytics for docs.sandstorm.io -- directly from Sandstorm.

- We host the Sandstorm app index (back-end for apps.sandstorm.io and for the automatic app update pipeline) as a Sandstorm app.

- We host the purchase flow for Sandstorm for Work as a Sandstorm app.

- We sometimes use private Gitlab or Gitweb grains on Sandstorm for non-public code, or for working on security fixes before they are ready to be disclosed.

That said, we still need to:

- Switch the main Sandstorm source repo to Gitlab or similar. This will obviously be a disruptive change, so we haven't tackled it yet.

- Host our e-mail on Sandstorm. Sandstorm's e-mail support admittedly needs a lot of work -- it's a much more complicated problem than most of the other things one does on Sandstorm, and we haven't put much effort into it yet. We'll get there eventually.

- Host our CI server on Sandstorm. This requires packaging Jenkins as a Sandstorm app, and also has a bunch of other complications relating to the fact that some of our tests need to run in VMs or use privileged syscalls, so it might take a while.

- Host our main web site on Sandstorm. Currently it's hosted as a plain-old static file server that we maintain. We could switch but there wouldn't be a ton of benefit in doing so, other than to say that we did.

We'll get to all these eventually.

If we can help to your use GitLab CI to test your app we would be happy to. Since GitLab already runs on Sandstorm I think we can do it relatively easily compared to Jenkins. Of course the privileged syscalls still are an issue. But maybe we can host the GitLab Runner on a non-Sandstorm box to work around that.

But the browser itself is an application delivery platform, and it includes a sandbox. So I'm curious why one would need something like Sandstorm on top of that to run apps?

Many (most?) apps need server-side logic. Apps that have multiple permissions levels need to enforce those permissions somewhere. Apps that support real-time collaboration need to maintain the authoritative copy and coordinate the OT stream. Etc.

Online services are never going to be truly free-of-charge. (If you're not paying for the product...) In many cases the business plan is the reason that server interaction is necessary. However, there exists F/OSS that is truly free-of-charge for the typical user, so it seems feasible that many browser apps could be like that too, which would often obviate the server. For the example you cite, every client needs to agree eventually, so they could all keep their own copy.

Your talking by phone is "free of charge", you pay for air time, and buy a phone. Equally, running a personal cloud can be free of charge, provided that you pay for connectivity and processing power.

Perhaps I was unclear. By "online services" I meant third parties that provide processing power and half of the connectivity. That is, they run servers. With a peer model, each collaborator pays for a fraction of connectivity and processing, but we wouldn't say any one of them is a service provider.

Sorry, but you're massively underestimating the problem here. There's a whole lot more to it than "they could all keep their own copy."

Reliable distributed consensus without a "master" node is one of the hardest problems in computer science even when all the computers are always-on servers located on the same network. When we're talking about random laptops and phones all over the world that flicker in and out of connectivity -- and you want real-time collaborative responsiveness, and you have users who are not allowed to see the whole data set -- you're deep in unsolved problem territory.

If and when someone builds a real system that can handle this, performs well in real-world network scenarios, and is easy enough for typical application developers to use... then we can start the arduous process of rewriting all of our applications to use this model.

Instead of all this, Sandstorm aims to make it easy for everyone to have their own general-purpose server (maybe by using their own physical machine, maybe by paying a hosting service). This way we can keep using our existing code and existing application development practices.

All true, except you're really just hiding the problem behind an abstraction. If you want to have a reliable server instead of just a server, then you want more than one server. And now you're back to your massively complicated problem, if you insist that all servers are equal.

Of course, you don't have to make that assumption, neither on the server side nor in the peer model. Just say that the peer who created the document is the "server" (or use any other reasonable way to determine this), and when other peers have a copy that disagrees, then they update their copy. Now it's just like talking to a real server, except it happens to be another peer. Go ahead and use If-Match, 412, etc. (...or the equivalents in whatever protocol makes sense, of course)

Such an immediate jump to a massive-scale solution for a few to at most a dozen hosts talking amongst themselves is a classic YAGNI.

You are equating the reliability of a server machine with the reliability of a user's client machine, when in reality these are not remotely similar. Laptops are regularly suspended and move between networks, losing connectivity. Phones even more so. Consumer internet is unreliable, usually with poor outgoing bandwidth. Mobile networks even more so. Etc.

When I share a document with five coworkers then close my laptop, the others need to be able to keep working. Under your model, that means they'd need to choose a new server. Guess what? This means you now need a consensus protocol. One where partitions happen 1000x more often, it is regularly necessary to operate without a quorum, latency is high and totally unpredictable, and all kinds of other properties that make the problem exponentially harder.

Entirely disagree with you here:

-Nowhere was it insisted that all servers are equal.

-Most people would prefer a hosted solution to hosting one locally(not constantly using their local machines resources)

-For many apps the performance and uptime of a server is of enough importance that this simply wouldn't work.

-A hosted solution provides a stable controlled and relatively unchanging environment, depending on what else you use that box for.

-Unless there's guaranteed uptime(which there isn't in this case) you have even more merging headaches. (read 4 slave-peers(not really even 'peers' in this context) all have different diffs after a week of the master being offline)

-Putting an app on a box somewhere isn't a 'massive scale solution'.

>If you want to have a reliable server instead of just a server, then you want more than one server.

No. Not for a personal or small app you don't. You can have 'reliable hosting' (say one of the many VM providers backed by solid hardware and a RAID tolerant to multiple drive failures). Reliable doesn't mean completely immune to failure, and you're underestimating the reliability if you think this is a regular occurrence. Your average hosting provider offers orders of magnitude more reliability than any users local PC or mobile(even if we're talking a constantly online desktop with wired ethernet).

It's just a ridiculously complicated thing your asking for here while simultaneously dismissing the parent comment as 'hiding the problem behind an abstraction'.

Hahaha thanks everyone there's lots here for me to consider. The peer model is probably better suited to ephemeral stuff like gaming than to document collaboration.

What I'm wondering now is, where is sandstorm going to run? If it's on AWS like everything else, I guess I misunderstood the point...

Sandstorm runs wherever you want it to. You can run it on your own physical machine -- even your desktop, if it's always-on. A lot of people run Sandstorm on Digital Ocean. The company also provides Sandstorm Oasis (oasis.sandstorm.io) which is provided as a service.

Wherever you run Sandstorm (including Oasis), you as a user can upload and install any package, including ones you built yourself. Thus it accomplishes the goals in the blog post, even if physically the servers are "centralized".

> What I'm wondering now is, where is sandstorm going to run?

It already runs and has been running in production for months, both as a hosted server and in various basements etc.

Source: small-time backer, ran it on my desktop almost without a hitch for a few weeks or months from the start until I could get a hosted account.

> It already runs and has been running in production for months

Actually 2.5 years now. :) (And Oasis has been up for a little over a year.) Time flies...

I'm in the process of trying to setup some things I'd like to seriously try in Docker containers (GNU Social and iodine for example). Now I want to try to get Sandstorm working in a container now as well. :-P

Honestly though I was talking about this exact concept that Sandstorm has implemented, a few years back. I was telling a buddy, "We have general purpose desktop OSes where we install apps. Why isn't there a building-block type server OS we do the same thing?"

Of course I'm sure like a million other programmers had that same idea, and it's am incredibly difficult one. Sandstorm seems to have a C++ base with Javascript frameworks on top of it. I'll totally put it on my list of things to try out.

The future is distributed.

> Why isn't there a building-block type server OS we do the same thing?

Because everything is built on Unix as an OS abstraction. Even Windows (since the early 2000s).

Consider: byte streams and file systems are less than ideal fundamental abstractions in a distributed world.

IMO we won't have what you're seeking until we rethink what an OS should be in an Internet-everywhere world. We're still using OS designs that predate the Internet entirely.

It would be nice to see a future where all applications have some concept of peer-to-peer networking and would be able to talk to each other. My hope is that this leads to blurring the line between having separate architectures for server, desktop and mobile apps, to the point where the only differences are reflected in the physical limitations of the device.

It's interesting to see how modern public cloud businesses seem to have borrowed a lot of their business models from old timesharing systems of the 70s. From there it's easy to analogize timesharing systems being killed off by personal computers to cloud computing being killed off by personal cloud.

Have you seen unikernel approaches like MirageOS (https://mirage.io), which challenge the POSIX basics of operating systems? If not, take a look :)

I can help create a very lightweight container image for this if you want?

Reply on news.ycombinator.com