Sandstorm, Tempest, and the Future | Hacker News Reader

Sandstorm, Tempest, and the Future

sandstorm.org

150 pointsocdtrekkie41 comments

Just in case anyone is wondering, sandstorm.org represents the continuation of the Sandstorm community project, as opposed to sandstorm.io which was the startup that I co-founded. The company has been dissolved, and the community project runs under Open Source Collective. We felt it was important to make this change in continuity clear via a change in domain name (and release signing keys) as well. I plan to update the old sandstorm.io domain with a blog post and a top banner marking it as a historical site soon, and linking to sandstorm.org.

For my part, I am no longer heavily involved in the community project -- mostly because I just don't have any free time. :/ But I trust Jake and co.

It's cool that you handed over the project to the open source community when you decided to fold the startup. I know absolutely nothing about Sandstorm and my interest has now skyrocketed in both it, and you.

Thanks. FWIW Sandstorm was always an open source project, even as a startup, so technically my blessing wasn't actually required here. :)

The call for donation, for this very very excellent project, are at the end of the post.

> Unfortunately, as we hope to continue to support Sandstorm long-term, we’ve had an increasingly challenging problem with the legacy MongoDB database underneath, as Meteor has dropped support for the version we use. Ensuring a seamless migration on end-user servers has proven to be a challenging problem. Dependency upgrades are not fun and this has become a fairly unpleasant project to undertake for unpaid open source contributors.

> Over the past few weeks, we have spoken to a number of potential outside parties about taking on this project and building a migration so that we can update the databases of existing Sandstorm servers and allow us to continue moving forward.

> We are looking to raise the funding for this project through our OpenCollective.

Please take a look at the long list of supported apps [1], which are basically one click install.

[1] https://apps.sandstorm.io/

Really sorry to hear about Ian.

Sandstorm is an excellent project. I sort of consider it the platonic ideal of self-hosting software. It does everything the "right" way, especially with respect to security, but ultimately isn't compatible enough with the broader ecosystem for general adoption.

I've been wondering if a lot of self-hosting platforms are targeting the wrong level of scale. On one extreme you have single user instances (SUI hosting). The other extreme is centralized platforms like Google. I used to think the ideal for most people is the "nerdy cousin" approach, where each family has someone capable of running something like Sandstorm or Nextcloud. The problem is that I would much rather trust Google with my private data than anyone close to me. There's so much more incentive for them to go looking through my stuff. This makes me think SUI might be the way to go. Or maybe large federated providers, similar to what we have for email.

The biggest thing we have to accomplish for single user hosting is what Sandstorm excels at: Removing security footguns. Self-hosting can't just be for folks who can sysadmin. Ultimately, it'd be nice to have Sandstorm available preloaded on a box someone can just plug into their home router and set up from a mobile app... but we're a long way from there yet.

I don't see why we are a long way away. At the sandstorm end, we need to get to the point, where all updates (of both sandstorm and the apps) on the user machine are automatic. Much like they are automatic on various OSes (mobile OSes in particular but also MacOS/Windows). This is not impossible if a single OS like Debian-testing is targeted.

Mailinabox [1] almost does it. They target Ubuntu stable, and upgrades are mostly silent. But they needed user intervention when Ubuntu had to be upgraded from 18.10 to 22.04, which was quite challenging.

If sandstorm can get there, then putting it into a RPi box is simple. What's difficult is helping the general public understand the importance of protecting their data, so such a product has a reasonable market.

[1] https://mailinabox.email/

How is the average person supposed to install it? It needs to be as simple as installing an app on Windows or your phone.

GP suggested "Sandstorm available preloaded on a box", which I think is possible because that makes things a lot easier for the average user. As long as the box takes care of updates and backup without user intervention.

But for that to work Sandstorm itself has to be as easy to install as an app on your phone, and there needs to be way more apps.

IMO Sandstorm should be preinstalled like the OS on your phone.

What particular killer app do you think is missing?

Can Sandstorm host a Matrix server? Or a Mastodon server?

This is definitely something we'd like to do with Tempest. The primary issue is that Sandstorm makes it intentionally very hard to have a well-known URL that can run app code, and services like Matrix and ActivityPub generally expect a reasonable address that represents that server. Since Sandstorm servers can have many users who do not interact, you do not want one of them able to install Mastodon and define it as the server's Mastodon... probably.

There are probably a couple interesting approaches. One might be to have users point some other domain at a Sandstorm API endpoint, end-running around Sandstorm's own domain, which might work in scenarios remote servers are only fetching. But another idea would be to sort of "special-case" an ActivityPub server deeply integrated into Tempest, and instead of letting a user install "Mastodon", they are installing a custom frontend that creates a user account on the server's ActivityPub.

Because Sandstorm was intended to work as a large scale hosting service, it avoided pretty much any design which could let one user know about another user on the server, but Tempest may end up making choices more suited to smaller deployments where we assume a Tempest server hosts either one organization or one family or one group of friends, etc.

It's really not meant for public facing servers. Now people will say it can but sandstorm like any other software can be hacked into doing whatever. It's meant to host "documents", think Google docs style hosting. You can make documents public which is quite different from a public server.

What you really want is something e2e encrypted so you don't need to trust the server.

There are ups and downs to that approach: If we're hoping for a solution that works for the average person, e2e encrypted solutions often lead to people who do not fully understand the risk of losing their keys to lose all of their data. Even without e2e, you'd be surprised how many people I've personally interacted with who have gotten permanently locked out of a Gmail account and unable to convince Google to give them access again.

One idea I think is really neat though is decentralized backup solutions, where an e2e scheme would work really well: Your home server could have a key to store encrypted versions on someone else's server (and vice versa for theirs). Losing a key would still be problematic, but only in the case you also have lost your primary server.

That would be a good idea for file sync and documents, but wouldn't be as effective for RSS readers where the server needs to know the list of URLs to fetch, and terminate the TLS connections itself (unless the client does the fetching and pushes up articles in an encrypted format).

Looks like the new version Tempest may address some of your thoughts.

I wish server applications reached the streamlined experience of desktop or phone applications where I can go to some App Store, install application with one click and then have a consistent experience, authentication handled by OS and (in case of mobile operating systems), strong data protection, notification and privacy controls etc. There are several attempts hoping to get there (Umbrel, CasaOS, Yunohost, Synology etc), but in my opinion, Sandstorm is (was?) the most consistent, daring, and well-engineered stab at it. What did not work for me was the per-document security model which I think is an audacious and advanced idea, but was a bit too much friction.

The per document model is actually super easy to write apps for outright, just unfortunately fairly painful to package existing apps into. Ideally if everyone was writing apps specifically for Sandstorm it would be a non-issue. :) It's also extremely hard to get a consistent experience across apps without it: When every app has it's own concept of file management, it's impossible to build a unified experience.

I think Sandstorm as a startup was, if anything, just too early. (I remember people questioning why anyone wouldn't trust Google back when it started!) I suspect in several years everything comparable that isn't Sandstorm will look a lot more like Sandstorm than Docker and such.

Don't forget networking. Having to deal with DNS, NAT, port mapping, TLS, firewalls, etc etc is a complete non starter for basically everyone.

IMO we need a nice simple OAuth2-based protocol for setting up something like Cloudflare tunnels.

Does the new version Tempest share that model?

Have you tried cloudron? It's pretty close to phone apps.

I think they should also target enterprise users as well as hobbyists. I've evaluated it for my company (I'm the CTO of a small Fintech of 100 people) and I was kinda lost on how to share data between a team and not just individuals. Felt like a big gap to me, even if I invested on contributing what I needed.

With the support from a business case, things could get easier for the hobbyists. This is just constructive criticism. I sincerely think this is the coolest open source project around.

Targeting enterprise users was indeed the startup's focus! Unfortunately I think at the time a lot of businesses were perfectly comfortable giving all their data to Google, and some of the core apps weren't really good enough to justify the switch. I think a business building Sandstorm would do better today, but right now our nonprofit is focused around the self-hosting community first and foremost. I've done a lot of thinking about how to prep Sandstorm for different business verticals though and what that would look like.

The Collections app is currently the best way to share "a group of permissions", which you can in-turn share with a bunch of users, but there's definitely a missing feature there in terms of being able to share to groups of users or especially share some grains by default to new users.

For Tempest, Ian and I had discussed doing a lot more with "keyrings" of capabilities, and I absolutely would like to be able to do that someday.

From the website

   Sandstorm is a self-hostable web productivity suite.

Sandstorm has some nice features, especially in regards to single-sign-on, automatically setting up a reverse proxy, automatically handling SSL certificates, etc.

But I really don't understand what the sandstorm runtime is bringing over just having a docker container. It's trying to reinvent docker, which seems weirdly out of scope for the problems it's trying to solve. Or even if docker isn't suitable you can still use OCI-compatible container images instead of a custom format?

Sandstorm and Docker are really totally different things.

https://sandstorm.io/news/2014-08-19-why-not-run-docker-apps

>We do often use Docker’s build tools in the process of building Sandstorm packages, simply because they provide a structured and reproducible way to gather an app and its dependencies into a chroot environment, which we can then package up.

It sounds like "we do use OCI-compatible images, but we wrap them in our own filetype". Which I suppose is fine.

Does anyone remember a hw+SW company from the late 90s that had an application server that was sort of similar to this? I thought it was RackOS but I'm having zero luck. You could install/add different applications to the box and make them available on your local network.

Cobalt Qube and RaQ. https://en.wikipedia.org/wiki/Cobalt_Qube

Thank you! I tried every rack-like word I could think of.

ELI5: Sandstorm is an open source project to make it easy to run open source web applications.

Source: https://sandstorm.org/

these upgrade challenges are a very damning verdict for meteor and mongodb.

To be fair to Mongo, I've seen an enterprise application with an embedded Postgres backend choke on itself every single time it's upgraded the database. Of course, I'm a sysadmin and we had an enterprise support contract so that was not a big deal.

MongoDB is perfectly fine at what it's designed for: Enterprise data infrastructure. But while needing a sysadmin or a support call may be no big deal for an enterprise platform, for an open source project without a customer support team or any product telemetry, the risk of shipping a less than perfect database upgrade is extreme.

Tempest, or course, uses SQLite, which is far more suited to the role here with ten years of hindsight.

Having seen MongoDB in use in an enterprise context, I may have to disagree with you about its fitness for purpose there. :-/

I've been following this project for a few years now. Always thought it was a great idea.

I found this helpful in getting started with Sandstorm:

https://youtu.be/y6120QOlsfU

As a project that's been around for a decade, this joke has gotten very old.

This old: https://github.com/sandstorm-io/sandstorm-website/pull/19

And here I was feeling cheeky.

In all seriousness, I just learned about Sandstorm and think it could be a great fit for a local co-op I’m building. It’s a shame about the startup, but I hope we’ll get a chance to support the open collective.

Reply on news.ycombinator.com