Flynn: open source PaaS
flynn.io
flynn.io
They announced during their 1.0 release that Flynn is now production ready but I have to disagree at this time. I do honestly love the idea of running my own Heroku but it's just not stable enough and doesn't recover well if a node goes offline without manual intervention using the flynn-host fix command (and sometimes that command just chokes up completely).
Next time it happens (when, not if, at this rate) I'll try to set aside some time to gather up the logs to help debug the problem but I'm using Flynn to avoid the sysadmin side of things and just build my product. In my opinion Flynn is tinker-ready, not production-ready. It's infuriating that it has these random problems because like I say I really _really_ love the product and desperately want to be able to use it reliably.
Flynn does self-heal after node reboots, so if that is not working for you then it is a bug we need to fix.
When you do have issues it would be really helpful to us and other users if you could open a GitHub issue or come and chat to us in IRC (#flynn on Freenode) so we can fix any bugs you are encountering. I fully understand you just want to be concentrating on building your product, this is why we are building Flynn!
One of those things where I wish I could say it's something I'm doing that's breaking it but in each of the situations I've had I've not been interacting with it at all when it happens. I know my waffling here doesn't help anyone without logs of some sort, this post just caught me a couple of days after a cluster brick.
I'll set it all up again and will grab the debug info and jump on IRC this time if it all goes to pot.
If you don't have time to interactively debug with us on IRC just open a Github issue with what details you can and the output of the `flynn-host collect-debug-info` command and hopefully we will be able to discern what went wrong.
Thanks for trying out Flynn!
Just to blow away some of my grump cobwebs I'm not just trying out Flynn, I'm still using Flynn! The hiccups were just going to be temporarily patched by hijacking my ServerPilot controlled machine (WordPress on Flynn.. Haven't managed that one yet!) until it felt a bit more stable in the dev/stage environment. :)
Would be cool to learn how you are approaching Wordpress on Flynn. I have poked around at writing an asset backend that talks to Flynn's blobstore because a family member wanted me to host their Wordpress site but haven't quite gotten around to it.
We merged the MariaDB database appliance a little while back which took care of the other major roadblock to getting Wordpress to "just work" on Flynn.
I'm `jpg@freenode and josephglanville on Github, hope to see you around. :)
I imagine my thought process at the time was "Flynn is like Heroku and Heroku is all about 12 factor apps. Bedrock seems to be a 12 factor WordPress, I'll try that!". It also somewhat matched up with the rest of what I do in Laravel so the standard composer, PHP, .env, MariaDB vibe. I'll give it another shot over the weekend, I don't recall what stopped it working.
Major thanks on adding MariaDB support by the way, I know mysql and the likes get their fair share of flak but it's what I've used for the past decade or so, more comfortable with it than postgres :)
if it's coreos it's probably the auto update mechanism.
For us to invest time/effort adopting a PAAS solution requires an amazing amount of trust in your ability to execute now and in years down the line. Unlike alternatives, you are a company and not an industry collaboration / foundation, how open can you be to make us believe you are not the next RethinkDb-like shutdown?
e.g. How many in your team, why can they do this, how much funding do you have, what is your burn, when do you become default-alive?
We, up until recently, had our own (closed source) ansible/terraform/packer immutable AWS-based PaaS sort of thing. We abandoned it in July when Kubernetes hit 1.0.
It's not just if Flynn and Convox and other VC-powered startups whomever can even exist another couple of years - but whether they exist and if they can out-compete Docker and CoreOS and Kubernetes and... etc.
My feeling is that a lot of infrastructure companies died in July. Just like Blackberry after the iPhone - gradually, and sometime in the future, suddenly.
What's the future for these PaaS companies? I dunno. Consulting services? Managed PaaS? These are fine, but... have a finite growth curve. Not everyone wants or needs to be a unicorn - for sure - but if you're tying your SaaS or web business to someone's infrastructure code, you kinda want:
a) the thing everyone else is doing (so you can grow your own team eventually - it's easier to hire a kube developer than a eg. flynn dev)
b) something that will be around for a long time and
c) something that has the best blend of features/power/ergonomics (Kubernetes has almost 1,000 contributors. Flynn: 98. Convox: 52. Bigger isn't always better, but...)
For us, that's kube. We're still iterating on our our business model - perhaps we just stay as a kube consulting shop for a good while - but running an Open Source PaaS business is not where we think the puck is heading.
tldr: we're open to making Flynn a foundation/community project if we could find the right kinds of partners and once the core innovation is complete.
This is something we've thought a lot about over the past few years. Everyone on our team is a hardcore open source believer and contributor and the long term success and sustainability of open source projects and Flynn in particular are something we care deeply about.
When we started working on Flynn (with just a crowdfunding page posted to HN) we had no intentions of commercializing it. Before asking HN to fund the project we reached out the founders of many growing startups (many of which are now unicorns) who we saw as the core users. Our goal was to agree on an architecture and set of APIs, then have each company "own" one component of Flynn, with one engineer at each working on their component full time.
Everyone we talked to said they loved the idea of an open source Heroku that could run anything (not just 12 factor apps) and run anywhere from AWS to their own colos and datacenters, but none had the staff to spare on infrastructure engineering (which is why everyone wanted a PaaS in the first place). Several, like Shopify, contributed to Flynn financially so we could hire a dedicated team to build Flynn for everyone.
So a community/industry collaboration was absolutely our goal from the start, but several things made us question that route. Not only were our first choices for collaborators and partners unable to participate directly, but we spent a lot of time talking to companies in the then-thriving OpenStack community as well as other foundation/industry run infrastructure projects. They shared a number of horror stories of how the most promising new features and capabilities were killed by cross-vendor compatibility concerns and how specs were limited by legacy vendors on committees.
Committees and bureaucracies are great at keeping something alive and maintaining the status quo, but not at innovation.
We also met executives from very large companies in regulated sectors like banking and telecommunications. They were extremely excited about Flynn but had a completely different set of requirements than the younger companies like startups whose needs we knew best, and had experienced ourselves. It was clear that we would have to build a very different project to serve the largest of enterprise customers. Crucially, foundations/industry collaborations on infrastructure attract the largest of vendors who have these mega-companies in their sights as the real customers.
If you want to see what a PaaS built to their (imagined) needs are, look at CloudFoundry built by Pivotal, which is currently owned in part by Dell/EMC, VMWare, GE, Microsoft, Ford or at OpenShift, which was built as a startup but acquired and retooled by RedHat.
While we think Flynn has a lot to offer today and in the future to these large institutional users, they have not been our focus.
Flynn today has the minimum feature set we consider necessary to be useful. Looking towards the future we want it to encompass a lot more. We'll be publishing a comprehensive roadmap/master plan soon, but most of that innovation would be harder or impossible to accomplish if we were locked into a foundation today. Development would be slower, changes would have to work their way through approval to consensus, and many partners would object to new features competing with their own established products.
We've been very careful both to limit external dependencies and to provide end-to-end integration between all our components. As a result we're able to move very quickly when changing APIs, implementation details, even major pieces of architecture, all without impacting customers. This gives us the freedom to stay nimble in a changing landscape, but partners might be more attached to technologies in which they already had a large institutional investment.
Infrastructure companies today are already a world apart from where things were in 2008 when Heroku upended PaaS. The default is now open source, making lock-in (historically the bigger concern) far rarer. There are few companies among our peers battling to become the next Oracle.
Additionally there's a tremendous amount of portability both among unified PaaS's and build-your-own infrastructure because of technologies like buildpacks and containers. As a result most apps that run well on one platform can be moved to another. There are of course features, like Flynn's database appliances, that are not available on most others.
We've made other choices like a small codebase with few external dependencies that make managing Flynn's development and maintenance easier, whether it's our team or yours doing that maintenance.
That being said, we're entirely committed to Flynn's future personally. While we'd love to see it become a huge company, that's not necessary for us personally. We're a team of five, and speaking for at least my cofounder any myself, we're not going anywhere, regardless of the VC market.
We work on Flynn (for several years and many long hours now) because we believe it needs to exist in the world, not because we think it'll make us rich. We're committed to keeping Flynn alive as long as that stays true (basically until it becomes unnecessary or there's something else that does everything we want but better). Since our 1.0 launch we've had huge growth and adoption and are currently fundraising. We're single digit new customers away from default alive with the current team. Whether we become a small, sustainable company to selling to/partnering with a large enterprise vendor, to creating a foundation, everyone on the team is 100% committed to a permanent, sustainable future for the product, with or without venture capital, unicorns, or IPOs.
The headline feature that attracts me is HA postgresql - that has instant relevance. My greatest concern is stability - I'm not seeing many success stories but lots of problems. I want to hear that Flynn is "the one bit of my stack I don't have to worry about". I hope you can publish some case-studies (both successes and post-mortems) including discussion of the attempted architectures.
A fundamental philosophical worry I have is that it is a "wrapper tech". I increasingly stay away from these - the promise is that they make the underlying tech easier/quicker to use but they make assumptions that don't hold for you in the long-term, they serve too many unrelated use cases so become bloated and poorly fitted to your own needs and they reduce the features, flexibility and stability of the tech they wrap. The cost in fixing & working-around the wrapper can quickly exceed the cost/value of building your own glue based on the underlying tech because it also gives you a greater depth of knowledge in the fundamentals and more power.
a) it's not hard to set up flynn b) migrating to another buildpack system is not a lot of work, except setting up some environment variables and installing a cli
We had some very bad bug that happened at the time (like deploying a new version made the dyno crash and we had to deploy a completly new environment).
Other than pretty bad bug, it was working properly, and did what they sell.
Good point is that the dev team is very responsive and we reported a bunch of bug that got fixed in no time. They also have a smooth updater when a new Flynn release came out, side bad note is at this time, it was kinda bugged (because of our specific setup).
For all those reasons, we choose to stick with Heroku at that time (we also tried Doku https://github.com/dokku/dokku, various AWS services such as BeanStalk, and DEIS http://deis.io/), but for sure it was the best candidate out there and I would love to give it a try again.
I highly suspect many of the bugs you encountered have been fixed, and we make it a priority to investigate and fix reported bugs as quickly as possible, so if you end up giving Flynn a try again let us know if you have issues and we'll fix them.
As for updating, the in-place updater still has some kinks but the backup / restore route is fully tested and is reliable (see https://flynn.io/docs/production#updating).
(Originally coded to deploy services on the Raspberry Pi, but now used on "real" machines too)
> The CLI includes a local browser-based installer that can boot and configure a Flynn cluster on Amazon Web Services, Digital Ocean, Azure, and your own servers via SSH. It automatically performs all of the steps required to install Flynn.
Flynn also allows portability between clouds. Since we don't use any cloud-specific features (like, say RDS) you can backup a Flynn cluster (including database appliances) from one cloud and deploy onto a new cluster on an entirely different cloud.
It allows the same distributed system to be defined and run on AWS, GCP, Azure, vSphere, OpenStack, tin, and more.
In this age of it being next to impossible to hire devops folk, the idea that you can have one of them servicing the devops requirements for a sizable fraction of developed applications is somewhat attractive.
I wouldn't put Flynn in place for external-facing and high-traffic applications at the present time based on the experience I've had with it, however. Not ready for that, I think. You have to have a fair tolerance for short outages.
perhaps it's just me, but the devops part isn't that annoying, yet, but i've got an ultra small environment running nginx, golang binaries, and some services like rdb and k-v store.
I think the main motivation compared to IAAS is to run on your own choice of machine e.g. on-premise or dedicated from OVH/Hetzner, or across different cloud providers while not incurring the full admin overhead of baremetal. The reasons can be portability, control, performance, privacy or the potential for 1-2 orders of magnitude cost-savings.
And yes, scale, high availability, self-healing, auto-scaling & load balancing is where the admin pain comes in.
i guess what i don't like about most paas is the fact that you lose some level of control over the environment for the sake of convenience and bliss, but at the end of the day, someone has to fix all those bugs in your paas layer.
For us normal people at more human levels of scale - I agree its wise to be skeptical - serve your users in the simplest manner and don't get sucked into HN hype. Its easy to get an infrastructure inferiority complex :)
A quick glance shows a couple of things:
* Automatic cluster scaling, so your PaaS deployment is as small as possible. Cloud Foundry doesn't have this, and I wish it did
* Flynn has built-in service discovery, in Cloud Foundry you bring your own (if you want)
* Flynn comes bundled with some data services
* Flynn has no BOSH equivalent, so there's nothing there to bring back VMs when they fail, perform rolling upgrades or manage the IaaS
* Flynn doesn't seem to offer a metrics stream, only logs
* Flynn builds apps from source, so not sure how that works for compiled Java/Go, or anything you'd run with a binary buildpack
* I couldn't find anything about role-based access control in Flynn, something Cloud Foundry has (which can be backed by LDAP for enterprises)
* Flynn only appears to route HTTP traffic, so no TCP routing for IoT workloads
* Cloud Foundry is run at massive scale in production for mission-critical workloads by banks, payment processors, IoT data ingesters, manufacturing companies, research companies, governments and everything inbetween. I can't see any evidence of the same being said for Flynn.
* Flynn is owned by one company, Cloud Foundry is owned by a non-profit organisation affiliated with the Linux Foundation and can never be transferred back to a regular company. This prevents the sort of nonsense you've seen in the Docker community recently.
You can deploy docker images, at least that was the case one year ago
* Cloud Foundry is run at massive scale in production for mission-critical workloads by banks, payment processors, IoT data ingesters, manufacturing companies, research companies, governments and everything inbetween. I can't see any evidence of the same being said for Flynn.
I think that's the whole idea behind flynn. Instead of being heavy-weight, complicated to deploy (no BOSH) and resource intensive, you can run a cluster with 3 nodes and deploy a ton of services there. It's not targeted at banks, it's targeted at people who don't want to use heroku or pivotal or GCE. And this makes it a lot faster in many ways, at least last time I checked.
Fair points about BOSH, it's got a steep learning curve (cliff). I'd love to see Cloud Foundry have an install experience that's along the lines of "Hey, here's some IP addresses and an SSH key. Go install yourself there."
Flynn auto-heals after node restarts (and if it doesn't that is first and foremost a bug we need to fix), but also has a cluster monitor service which brings services back online automatically, and as a last gasp there is `flynn-host fix` for manual intervention.
Flynn has node promotion / demotion so you can totally do rolling upgrades (users have reported they do this with a modified CloudFormation stack for example). There is also a built-in updater to update to a more recent version of Flynn.
> Flynn doesn't seem to offer a metrics stream, only logs
Something I think should be built-in, but there are many off the shelf solutions which can be deployed to Flynn as apps.
> Flynn builds apps from source, so not sure how that works for compiled Java/Go, or anything you'd run with a binary buildpack
Flynn has built-in buildpacks but also supports running custom buildpacks (see https://flynn.io/docs/apps#buildpacks). If you just want to deploy binaries, you can either:
- use the inline buildpack: https://github.com/kr/heroku-buildpack-inline - deploy a tarball: https://github.com/flynn/flynn/issues/306#issuecomment-23321... - deploy a Docker image: https://flynn.io/docs/docker
> I couldn't find anything about role-based access control in Flynn, something Cloud Foundry has (which can be backed by LDAP for enterprises)
Flynn does not currently have strict access control, but work is very much on the way, see our security improvements project: https://github.com/flynn/flynn/projects/1.
> Flynn only appears to route HTTP traffic, so no TCP routing for IoT workloads
Flynn supports TCP routes, although it does not seem to be documented very well: https://flynn.io/docs/cli#route
The Flynn auto-heal - does that monitor infrastructure (ie at the VM level) or is that process auto-healing?
seems interesting but please enumerate things I can do that are not delivered by docker compose and docker swarm.
For example, Flynn comes with PostgreSQL, MariaDB and MongoDB appliances which are deployed in a highly available, fault tolerant configuration with sane defaults: https://flynn.io/docs/databases.
Through the use of buildpacks, it is also straight forward to deploy applications written in many popular languages and frameworks without being intimately acquainted with how to containerise those types of applications (e.g deploying a Ruby on Rails app on JRuby: https://flynn.io/docs/languages/ruby).
You also get things like full cluster backups (including all your data), simple git / docker / tarball based deployments, log aggregation, release rollbacks, domain / path based routing, service discovery, and many more!
The Docker-PaaS movement reminds me of content-management systems in the early 2000s. Everyone (myself included) thought it'd be great to spend loads of engineering effort reinventing the wheel, instead of delivering business value.
I have yet to see any other PAAS come close to offering HA postgres built in.
The second bit was dead simple service discovery.
The main issue with deployment/orchestration platforms is that they cannot handle ALL different kinds of stacks/apps (each one has unique scalability requirements).
With Baasil.io, our strategy is to get it 100% working and auto-scalable; one stack at a time.
Flynn is an actual product which, from what I can see, is also a collection of tools but these are written by the Flynn team themselves.
I'm happy to answer any other question you might have about it.
And from the perspective of the developer, does Mantl offer a similar PaaS experience as Heroku or Flynn?
There wasn't any particular magic in it, more that it was a neat package of terraform+ansible tooling for getting mesos and the surrounding ecosystem going. This was about a year ago, so things may have changed since then. I do like the idea of not inventing your own paas, but rather having a consistent framework for deploying whatever kind of paas you need on whatever cloud is available.
Indeed that is a great plus, no need to understand all the intricacies of each Cloud platform.
What's your thought comparing it to Heroku or Flynn from the perspective of the (user-)developer? It seems getting little traction on fora like HN or Cloud conferences. Why would that be?
Disclaimer: Work for Mesosphere
Is it possible to store objects in something similar to S3 (for example using Minio)?
NFS support is not currently implemented, but we have a clear plan and it is on our upcoming roadmap: https://github.com/flynn/flynn/issues/1521
> Is it possible to store objects in something similar to S3 (for example using Minio)?
Flynn has a built-in blobstore for storing files and supports multiple storage backends: PostgreSQL Large Objects (default), Amazon S3, Google Cloud Storage and Azure Storage (to be released into stable next week), see https://flynn.io/docs/production#blobstore-backend.