An Open Source, Self-Hosted Heroku
bitmatica.com
bitmatica.com
Keeping up with new containerization tech is starting to feel like keeping up with new JS frameworks/tooling...
Has anyone had success with sticking to Dokku for small side projects on DigitalOcean/Linode/etc?
Nice to see that others are finding Flynn useful though, there is plenty of space for PaaS offerings, and I truly hope they are financially successful.
[1] My experience in this area is that it's hard to get peple to pay for something that is free, which dis-incentivizes work/releases. I'm not surprised that many use the free version of Flynn, or that there were a ton of Heroku users that left once they stopped getting free resources 24/7. Dokku does take donations - https://opencollective.com/dokku - but to be quite honest, it's almost certainly nowhere near even 1% the amount of money we've saved our users.
This wasn't a good place to be so I decided to clean it up and instead of running with Puppet again, I decided to check out PaaS solutions and I couldn't be more pleased with Dokku, I'm currently running JIRA, Confluence, Rails, Node.js, Go, PHP and several static (with middleman/jekyll/etc) sites on a modest VPS at Digital Ocean and I love that I don't have to think about overhead anymore when spinning up new projects. Just `dokku apps:create` and `git push` and it's done. The killer features for the migration for me have been the storage feature for specifying mounts for apps that aren't 12f friendly and the biggest one of all is the Letsencrypt plugin. It's so easy to setup SSL/TLS for new apps that it just feels wrong.
Dokku is amazing, and I'm looking forward to sending some funding to the maintainers.
Edit: forgot to mention in addition to all those apps, I'm also running the necessary backing services, MySQL, Postgres, Redis, and Elasticsearch and Kibana.
Why not wait until you encounter a problem that Flynn solves and Dokku doesn't?
I'm really sorry to hear that you switched off of Flynn. We're aware of this issue, and are in the process of fixing several things that can cause it (it only happens when not using an external blobstore backend like S3). Hopefully you'll try Flynn again at some point in the future!
Well, if what you have currently is working fine, you don't need to change anything. You don't actually have to migrate your hobby website to flynn.
For better or worse, we're going the full kubernetes route though, and still working out what deployment etc looks like. So far a lot of un-DRY looking YAML files have inspired a spike at a tool to help manage creating and maintaining kubernetes yaml[1], which others reading this are welcome to try and I'd love to hear better alternatives to or get feedback/pull requests/etc. I may push up a branch I'm working on later that partially automates creating deployments from a Procfile, but we discovered having seperate Procfiles may make applying env-variable only changes harder, curious to hear if others have any guidelines on that, it's so easy to edit an ENV var on Heroku we took it for granted.
Convox is a great pick if you want to use AWS-specific services for your entire stack. Instead of using portable open source components, Convox uses AWS services wherever possible, and acts as a lightweight coordinator to combine them into a platform.
Flynn has no external dependencies on cloud features, so you can run it anywhere, whether that's on your laptop, AWS, Google Cloud, a VM somewhere, or bare metal in a colo or private datacenter. We also include RDS-like highly available database appliances so that your whole stack is portable and you are not locked into a single hosting provider.
edited to add: We didn't know about this post until Bitmatica posted it, this is just a great post from a happy user, not planned Flynn marketing!
If that's not far off the mark, have you looked at using an object-oriented k8s client? I've been playing with a pattern where I create classes which know how to render themselves into k8s API objects, and that gives the ability to use language-level looping to create, for example, lists of env variable definitions in the resultant spec.
I'm writing python apps so PyKube[1] is the base client that I've used, but there are clients in lots of languages at this point.
I also tried out Helm, which looks like it has a way of generating k8s yaml files using Go templates, and given a set of config parameters[2]. Unfortunately my use case was a bit too dynamic to fit in that mould, but this seems like it could be useful for many things.
[1]: https://github.com/kelproject/pykube/ [2]: https://github.com/kubernetes/helm/blob/master/docs/examples...
ky is definitely still just in the "maybe useful/maybe practical" territory, but now that the basic concept of compiling from a Procfile seems to work, I suspect adding some configuration and more environment aware and extensible/overrideable templating could make it pretty practical, but I also had/have my eye on Helm. There may be some practical/logistical complications in using Helm, and I think if we went that way we'd probably go full hog and use Deis Workflow (https://deis.com/workflow/), which also deserved mention in this thread's discussion, but the k8s clients like the python one you point to or the ruby one (https://github.com/abonas/kubeclient) may be worth bringing into the mix depending on how this all works out... Thanks for the links and for having a look!
You want to use an external blob-store, by default it uses its own postgres to store them. However, it also doesn't do any garbage collection, so we ran out of HD space on our cluster, causing postgres to go into read-only mode. This prevented us from pushing updates / changing any app-config settings ( ENV, Scaling, etc ). Thankfully we were hosting our own DB off-cluster or that too would have been read only.
The default limits on memory is 1gb, and for our app, when it runs out, it just crashes. Leaving no obvious error messages to be found.
Obviously RTFM as well, (especially this part https://flynn.io/docs/production) we had some other issues that could have easily been prevented.
Huge shout-out to the team that hangs out in their IRC channel, they were a massive help in solving the problems we've came across.
Using it when it works is lovely, we switched off of heroku recently and its been a very similar experience.
The lack of garbage collection by default will be fixed in next week's stable release, we're really sorry that it caused you pain!
> The default limits on memory is 1gb, and for our app, when it runs out, it just crashes. Leaving no obvious error messages to be found.
Indeed, we're working on documenting this and exposing out-of-memory events in the app logs so that this is obvious.
By garbage collection are you referring to the binary blobs for built slugs?
Garbage collection of old slugs is supported though it isn't on by default (it will be in the next stable release).
Just in case someone is feeling the same, deis[1] is providing exactly this. I did not try their software but I've read a lot of positive about them in the last time.
There are still a couple of rough edges, specifically around upgrading and persistence across power failures, which is unlikely in a datacenter, but aside from those things, all in all excellent.
In particular, though, the flynn.io team are particularly excellent in terms of support.
> There are still a couple of rough edges, specifically around upgrading and persistence across power failure
Indeed, we have been finding and fixing bugs in these two areas lately, and things have been improving. Next week's stable release will have a bunch of fixes for issues we found. We also recently added a bunch of new test coverage for backup/restore, which should ensure that future upgrades go smoothly.
As always, if you see any issues please let us know so that we can fix them!
Seconding this. I've had many (like 10 or 20) very positive experiences with them, where I went from "OMG WORLD IS ON FIRE" to "oh, sweet, I just delete this row in postgres?"
[1] Globo.com is the internet arm of Globo Group, the biggest broadcast television in Brazil and second in the world [2] https://github.com/tsuru/tsuru [3] https://github.com/tsuru/tsuru-bootstrap [4] https://github.com/tsuru/now [5] https://docs.tsuru.io/stable/experimental/installer.html
I work on Cloud Foundry buildpacks, several of which are downstream from Heroku's. So I'm professionally interested in seeing another perspective on how to drive them.
So far no one is using Flynn in a fully disconnected environment, and we're aware of this limitation. My take on this is that we need a replacement for buildpacks that allows for repeatable/functional builds (think Nix for app deployment).
One gotcha: if you are snapshotting images after building, your images are not truly reproducible. Heroku occasionally swap out the binaries. This surprised once or twice. This is one reason we wound up building all our own runtime binaries[1].
When you get to disconnected environments, look us up. We built a whole tooling, compile-extensions[2], to make it pretty close to seamless. We intercept URLs and transform their either into our URLs or local copies of the files.
(edit: actually, you could just switch to the Cloud Foundry buildpacks, since they have an identical detect/compile/release cycle)
I agree that buildpacks are not the super long term solution. One area being explored by engineers at Pivotal and IBM is to give a buildpack-like experience to OCI layers[3]. I'm sure they'd be happy to work with you on this as well.
[1] https://github.com/cloudfoundry/binary-builder [2] https://github.com/cloudfoundry/compile-extensions [3] https://www.youtube.com/watch?v=DSTT0przx4g&list=PLhuMOCWn4P...
For example, I wanted to get off Google Analytics and try out Piwik, which uses PHP and requires MySQL, Postgres isn't supported. It would have been really difficult to write out a bunch of Ansible config for PHP and MySQL, both technologies I'm not familiar with and don't know their best practices. So, I'm still using Google analytics. A similar story happened when I wanted to host a Tor relay.
The allure of just pointing Dokku/Flynn to a pre-made Dockerfile for Piwik/Tor/ownCloud/etc. is pretty compelling. It let's me mix and match technologies without feeling like I'm polluting my server with a bunch of random services and tech. Containerization provides just enough separation to make me happy.
I'm not sure if Dokku or Flynn is the golden goose I'm after, but it seems like a step in the right direction.
I still use Ansible to do basic provisioning like users and SSH keys, setting up Fail2ban, and installing Dokku.
We designed Flynn to solve a bunch of the common problems around running apps in production.
This means that we're working full-time just on the automation and stability of your production environment. As you grow and scale, Flynn will scale with you and provide integrated features without any additional effort.
Here's an incomplete list of things that Flynn provides out of the box:
- Automatic high availability. Everything in Flynn is highly available, as long as you are running three or more servers. Additionally, it's designed to fail gracefully, so even if there's a network partition or some hosts fail, everything will keep working if at all possible.
- git push deployment, after installing Flynn you can run `flynn create && git push flynn master` and your app will be automatically built and deployed.
- Polyglot apps. Want to deploy a Phoenix app written in Elixir? You can do that just by specifying the appropriate buildpack. Everything else works exactly the same.
- Zero-downtime deployment with release management and easy rollbacks. If a new version or configuration variable causes the app to crash at boot, Flynn will automatically detect this and stop the deploy before any traffic hits it. Deploying new versions of the application does not cause user-visible downtime.
- Easy app configuration. `flynn env set FOO=bar` rolls out a new immutable release with the configuration change. You don't have to edit any files, and you can roll it back later with `flynn release rollback`.
- Built-in HA databases. `flynn resource add postgres` configures your app with a PostgreSQL database in a cluster that is highly available and already configured with replication and safe automatic failovers. You can do the same to get a MySQL or MongoDB database.
- Automatic load balancing and TLS. Flynn automatically load balances HTTP and TCP traffic, and supports HTTP/2 out of the box. You can add a TLS certificate with a single command, and all changes to the routing configuration happen immediately without restarts or downtime. Want to send a subpath to a different app? You can do that in a single command.
I could go on and on like this, but you probably get the idea. Of course, it's probably possible to build every feature of Flynn that you want using your configuration management tool of choice, but that would be a huge amount of time not spent developing the actual applications that you want to deploy with it.
I agree with the thrust of your comment here: rolling your own PaaS is hard. Just plain old hard. There's so much, so damn much that you wind up having to do.
Before I worked on CF I worked in Pivotal Labs. I got to see various custom home-grown PaaSes. Some were brilliant. Some were terrible.
Every single one of them was a millstone.
A lot of people don't realise it yet, but the Linux of our time has been written -- in the sense that nobody writes an operating system if they're not in the OS business.
I don't know if it's going to be Cloud Foundry, or Flynn, or OpenShift or some mix of these, but we have already passed the point at which it makes rational engineering sense for the 99% of engineers to build their own.
Disclosure: I work for Pivotal, the majority donor of engineering effort to Cloud Foundry.
I'm not talking about the deployment specifically, but rather isolating the code once it is deployed.
Am I missing something there?
Due to the security posture of the Linux kernel, we won't recommend running untrusted code side-by-side on the same hosts as more sensitive workloads, but we plan to harden everything to the maximum extent possible.
> Jim Hacker: People can wait in the lobby. Or in the state rooms.
> Sir Humphrey Appleby: Some people. But some people must wait where other people cannot see the people who are waiting. And people who arrive before other people must wait where they cannot see the people who arrive after them being admitted before them. And people who come in from outside must wait where they cannot see the people from inside coming in to tell you what the people from outside have come to see you about. And people who arrive when you are with people they are not supposed to know you have seen must wait somewhere until the people who are not supposed to have seen you have seen you.
This is one of the priority engineering efforts for Cloud Foundry at the moment. People want it.
Disclosure: I work for Pivotal, the majority donor of engineering to Cloud Foundry. I guess that makes us competitors to Flynn.
We are using powerful dedicated server to host multiple sites/apps like the way shared hosting providers do. Maintenance and upgrade is so painful, I am trying to move to something else.
I looked into Docker but seems things are still not stable. Another approach I am looking into is KVM virtualization.
Aactually, all I want is something like Heroku (but self hosted or German providers cause our clients are German and that's important) where we can host each app on it's own container.
So far, this meets all the requirements.
When it comes to dev UX, I'd agree that Flynn's is better, though both are easy to use. But I'd point to Convox for great dev UX in something that deploys only to AWS, and that can use a dockerfile/binary artifact instead of git push (I've been a minor contributor to Dokku and Convox, using both in prod on a few different projects).
I mention k8s because "self-hosted" is mentioned in the title.
Personally, the main reason I've been doing personal projects in k8s over the past 6 months or so is that 1) it offers out-of-the-box container networking and dns-based discovery, and my side projects are all distributed software so it makes it easy to get nodes talking, and 2) minikube runs a cluster where I can actually scale my apps up and down and test out distributed software on my laptop similarly to how it will be deployed.
(I'm investigating using Weave's ECS AMI, so that I can also get container discovery and networking in Convox, because I love the Convox dev UX so very much. But I'm also thinking, more and more, about how I can automate a smoother dev UX for my k8s projects, because I also like being able to easily run a real cluster locally in minikube, and Convox can run locally but with only one instance of each app).
Flynn and deis definitely seem interesting, as does the tooling that coreos and docker themselves are working on.
Setting up could never be easier (read https://glebbahmutov.com/blog/running-multiple-applications-...) and the large number of plugins for databases is a huge plus.
I know a lot of enterprises that are interested in using serverless arch (the framework serverless[0], not the literal definition of not using servers) but don't want to use AWS or other public cloud providers.
The open source Parse server (https://github.com/ParsePlatform/parse-server) already works on Flynn, and we can't wait until someone implements an open source system like Lambda.
[1] https://github.com/awslabs/chalice [2] https://github.com/fabric8io/funktion [3] http://www.open-lambda.org/
I'd be happy to explain this more on IRC if you're interested (#flynn on Freenode).
This aspect is a really big part of Heroku/AppEngine/Elastic Beanstalk and saves me a tonne of money each month due to the auto-scaling that we get (we're on AppEngine). Is there a reason projects like Flynn have not tackled it? Or is it just a question of time?
Like most useful production software, coming up with industrial-grade implementations is harder than it looks. The tricky part is picking the right basic metric.
CPU load is often a bit misleading.
I think old-school SPC techniques will take us a lot of the way.
And even before that, asking people what they already watch is good too.
The standard tier at Heroku is 512MB RAM for $25. Over at Vultr, I can get 2GB RAM for $20. I also get 45GB of SSD, so I don't have to pay for something like Amazon S3 to store uploaded files.
If you're building an actual product, I agree with you, pay Heroku and focus on building your product. However, there's a lot of people looking to host smaller projects that wouldn't break even financially if they used Heroku.
There are lots of reasons why users need a different, or especially an open source, PaaS.
Some users run into scaling problems when their products grow beyond a certain point. Others want to have more control over their infrastructure for compliance, governance, or other administrative reasons. Others have huge deployments and want to save money by using their own cloud accounts.
It varies from customer to customer, but for many Heroku isn't an option or isn't the best option.
This looks like a nice middle ground for me. I should be able to migrate to flynn on Digital Ocean pretty easily (easy deployment of flynn infrastructure, same buildpack system). I'm sure there'll be regular maintenance involved, but it looks like nowhere near as much as a roll your own kubernetes cluster.
- Persistant workloads - Fine grained ACL's for RBAC
They're also working on shipping a consolidated logging and metrics API in Winter of 2017 which will enabled users to get workload plus host-level logs and metrics into almost any log and metrics aggregation solution (in your case, ELK would be easy to ship to).
Best of all it runs on top of production proven scheduling software, Apache Mesos, which has wide community adoption and support.
It's opensource, multi-tenant and widely tested (used) by big companies.
Flynn is designed to be an end-to-end solution for production deployment, and all of our components are created to work together. The whole system is self-bootstrapping and self-hosting, so installation is easy, and the same APIs are used to manage the whole platform as are used to manage apps deployed on it.
In addition to the twelve-factor stateless webapps that Deis Workflow supports, Flynn also includes highly available database appliances with safe, automatic failover (currently PostgreSQL, MySQL, and MongoDB with more coming in the future). We also have a bunch of security features coming over the next few months like Let's Encrypt support and flexible user authentication with 2FA and very granular access control.
If you don't need or want our database appliances and you are comfortable with Kubernetes and happy to install and operate it, then Deis Workflow is a good option. If you don't care about using Kubernetes specifically, Flynn is a good pick as it is easier to get up and running with.
I've taken to referring to this as "Not Invented Yet Syndrome". It's hard to base an architecture on a subsystem that doesn't exist.
You can do HIPAA on AWS yourself or use (expensive vendors). However both of these routes lose you the management/deployment abilities of things like Heroku/Flynn.
Compliance is a really interesting vertical. As we make progress on our security roadmap, Flynn will become a very compelling option for environments like HIPAA, PCI, etc. especially when combined with clouds like AWS that are also compliant.
The docker security sandbox has had a few issues, and probably should not run sensitive data for multiple partners on the same ec2 instance. ymmv though.
BTW Met the Flynn team at 32C3, nice guys indeed.
You can watch a video of my setup here: https://www.youtube.com/watch?v=zxPB2v-tWvQ&list=PLjQo0sojbb...
Distributed communication should work fine, feel free to ping us on IRC or GitHub if you run into any trouble or have any questions!
The install guide mentions Ubuntu. Does it work on FreeBSD? Does it work on other Linux distributions from the Redhat family (fedora, centos, rhel, etc)
The CLI can be installed on any Linux distribution, FreeBSD, Windows and OSX.
Disclaimer: I work on Flynn.
They are working on more lightweight versions. CF is pretty "enterprise," with the good and the bad that entails.
{ tldr : 'https://flynn.io' }I am also curious to find out what is the running cost for maintaining flynn. Is there one dedicated engineer minitoring the infra?
- Small and side projects run well without supervision.
- Existing ops teams can manage Flynn without much additional effort.
- Medium sized teams that don't want an internal ops team pay us for Managed Flynn where we take over all ops-related responsibilities.
We do have a dedicated engineer managing / monitoring our infrastructure. It's not a set-it-and-forget-it kind of thing, so if you don't have somebody to manage your infrastructure, the hosted version might be a better choice.
The big cost savings win was for all of our /other/ engineers who now don't need to know anything about ops to get their apps running. Previously with Chef, everybody was responsible for writing recipes and setting up environments, but now we have a standard set of buildpacks that work with our apps, and when we need to transfer an app to a client (as we're a dev agency), we can set it up on Heroku and it should "just work".
Flynn is entirely open source and BSD-licensed (https://github.com/flynn/flynn). You can run Flynn on any infrastructure without paying us anything.
The price you reference is for our Managed Flynn product, where we act as your ops team, operating a Flynn cluster for you and providing hands-on support for your apps and databases in production.
I know it's probably not as feature-complete as those other PaaS's, but it would be great to get it at a glance.
Many people who choose Flynn are not directly comparing us to hosted platforms, as they want more control over their infrastructure (where they run it, lock-in avoidance, etc).
As far as individual features go, we are closest to Heroku (and maintain buildpack/runtime compatibility). However there are differences including our support for other databases (MySQL and MongoDB in addition to Postgres and Redis) and many other smaller things like HTTP/2 support.