Hacker News Reader

Comment by ekidd | Hacker News Reader

Over the years, I've deployed applications using various combinations of custom RPMs, Chef, Heroku, Docker, ECS and Kubernetes.

If you can, you should probably deploy to Heroku. (Or a similar service.) It's far cheaper than spending time on devops. Just run "git push" and you're running.

When I've deployed on ECS (or other "simple" orchestrators), I've found that I ultimately wound up re-inventing lots of Kubernetes features.

So if you can't use Heroku, but you do know basic Docker, then it's worth considering somebody's fully-managed Kubernetes. Google's is nice. Amazon's is a considerably more work. I hear Microsoft's is still a bit sketchy. And I'd love to take a look at Digital Ocean's. But do not attempt to host your own Kubernetes if you can possibly avoid it.

If you do try Kubernetes, then read a book like Kubernetes: Up & Running first. Kubernetes is not self-explanatory, but it's pretty straightforward if you're willing to spend a few days reading.

Finally, don't overcomplicate it. Just use the basic stuff for as long as you can before trying to layer all sorts of other tools over it.

View on HN

What pain is Heroku saving you to justify being 5x more expensive than Lightsail/Digital Ocean/Linode?

All websites I maintain/deploy are either built as a Docker image and published by CI on git check-in or deployed locally with a single rsync/supervisord bash script (or right-click Web Deploy for some older IIS/ASP.NET Apps). I probably have over 50 sites I'm currently hosting so using anything that much more expensive wont enter into consideration.

But I'm not seeing what could justify the extra cost? especially as the cost is reoccurring, if it's some kind of effortless/magical scalability I'd rather put that additional cost towards more hardware and buy more headroom.

I agree. You can set up a Heroku like setup pretty easily. Push to GitLab to trigger your pipeline that builds your container. Now your $5 vps with docker compose and watchtower is updated.

Sounds like you should start a Heroku competitor :)

Btw watchtower seems abandoned, anyone know is there a story to it?

Heroku turns a strategical liability ("I have only one employee who understands what rsync is; if he leaves I'm screwed") into a fiscal one. Any company I've ever known will always choose the latter, for anything outside of their core competency.

We use Heroku heavily, and we went from 2 full time devops engineers to 0. Everything is now buttons and sliders. There are no "security patches". The CEO could log in and scale if he needed to; it's a slider. We get monitoring for free (labour free, not cost, i.e. the "good" free). Memory usage, CPU usage, HTTP status codes, logging: it's all there. We spend no time thinking about rsync or devops or any of that: we just solve the problems we're good at.

Of course, everything is a matter of scale. Legend has it, Deliveroo UK only moved off Heroku after they grew so large, Heroku wasn't willing to offer them more dynos on their account. That sounds like a reasonable time to go in house. But any <100 people company.. why bother? focus on what you're good at, and let other people do devops.

IMO the fear is being overstated, everything's written down in either in single deploy script or configured CI, i.e. there's not going to be some loss of know how. It's the same as if the person in charge of Heroku leaves, someone else needs the login credentials and know how to setup Heroku as they would with any CI.

I don't know what Heroku is offering, but Lightsail and ECS instances also have metrics and pretty graphs (tho admittingly I rarely check them myself), maybe it will save me some ssh sessions to manually update security patches, I was recently able to upgrade my Lightsail instance to the latest Ubuntu 18.04 LTS with just:

  $ do-release-upgrade

> But any <100 people company.. why bother?

Because it's 5x more expensive.

> focus on what you're good at, and let other people do devops.

But it already takes hardly any time/effort to keep doing what I'm already doing.

I guess it's for different Companies who see the value-added benefits that justify the cost, but it's being propositioned here that everyone should be using Heroku first, just boggles my mind why most people would do that as the first option when it's so much more expensive. I already think the cloud is too expensive, so there's little chance I'm going to be paying a re-occurring premium for something that's not going to save me any time over what I'm already doing.

If it replaces one engineer, that's like $150K+/year (when including taxes and overhead, that is not at all a high estimate). So it depends on what the 'x' is in '5x more expensive'. And will probably be more reliable than what you'd get paying one (more) engineer to do it in-house too.

If you're hiring a "devops engineer" whose total responsibility is cloud touching, sure. You're right.

But where is that actually the case and your app can run comfortably on Heroku?

At multiple jobs I've been the only person who could credibly claim to understand the entire stack used at the company, from the web frontend to the OS the backend database runs on and the person to whom teams would come to validate their designs for scaling and reliability. I didn't write product code in those roles. But I multiplied the effectiveness of the people who did.

Heroku is a wonderful tool that doesn't get you the actually hard parts of the job req.

at 5x the cost of 10k/year in infrastructure spend, Heroku is significantly less than a dedicated DevOps team. however, If you're product is backups as a service and you'll need 100PB of storage, then Heroku is probably not the best option.

> at 5x the cost of 10k/year in infrastructure spend, Heroku is significantly less than a dedicated DevOps team.

How do these costs scale with 10x or 100x the traffic/load?

At 100x the traffic the business would be at 5 million dollars a year in spend, and would exceed the standard pricing model of Heroku. The business can then

1) Negotiate with Heroku for an enterprise contract 2) Consider migrating to a more cost effective platform 3) Dedicate time home-growing a solution.

Part of the reason Heroku charges so much is because their customers are typically small, but they'd likely rather find a price that keeps you on their platform vs. home-growing a solution.

Upgrading with do-release-update is simple if it works... but what happens when it breaks?

> Upgrading with do-release-update is simple if it works... but what happens when it breaks?

That is what snapshots are for.

So now the non-engineer needs to understand snapshots, ensuring snapshots don't break, ssh, roll back, etc. That's assuming the bad upgrade didn't leave any damage behind (like DB data, etc.). And assuming they didn't lose the post-it they wrote the CLI credentials on since you easily can't reset those unlike Heroku.

Snapshots are typically part of the provider's web management interface and are basically the first thing anybody who makes changes to anything should learn how to use.

Moreover, if you're having someone else manage your systems and they upgrade them, now what do you do when the new version causes problems?

I ran into an issue recently where newer systems default to a newer version of a protocol but the implementation of the newer protocol has a major bug the old one didn't. When that happens on your systems you roll back until you can solve the issue. When it happens on systems managed by someone else, better hope you can identify and solve the issue quickly because in the meantime your system is broken.

This may just relate to the circles I move in, but I'd have a hard time finding a group of what I'd call skilled developers, where not a single one can manage rsync or find their way around a Linux server in general.

It's all well and good at the "we need Kubernetes" scale to say you need specialists, but a team that can't manage a VPS is strange to me.

Wait until your VPS is breached because your OS wasn't patched, or because the firewall wasn't correctly configured. Learning is good, however, at a certain point I think it's fair that you can't minimize DevOps expertise to just "rsync" a jar or a docker image.

Nothing about "just use a PAAS" changes this. Sacking ops and telling developers to they should "just use EKS" or whatever is precisely why we keep seeing open S3 buckets and ES servers.

Any project that still fits Heroku has much cheaper options available with all the same reliability features: Elastic Beanstalk, OpsWorks, ECS, etc (I’m naming AWS options because I’m not knowledgeable enough about other cloud providers to suggest anything else, but I know all the major players have something in this realm.)

It really doesn’t take a devops engineer to run these if your app still fits on Heroku. A little bit of overhead goes into learning the service, much like you’d learn any new API or programming library.

The AWS docs are significantly worse than Heroku's, and in general you'll be likely to get tripped up and waste time figuring something out.

So? Continue to wade through it and you’ll end up with a significantly lower IT bill at the end of the month. The extra work pays off quickly.

> Heroku turns a strategical liability ("I have only one employee who understands what rsync is; if he leaves I'm screwed") into a fiscal one

So, what's your plan in case Heroku shuts down, or gets bought and changed completely? Isn't that also a strategic liability, just a much larger and arguably less likely one?

It's a negligible liability compared to the risk of the devops guy leaving the company within a year, without leaving any documentation or any clue how the app was deployed or run. The same thing will happen next year with the replacement guy, if there ever is a replacement.

Use Google App Engine, Clever Cloud or Scalingo.

Heroku got bought by Salesforce several years ago.

If you have 12-factor apps then you have a fighting chance of moving off it anyway.

Apart from Dokku, I'd say Cloud Foundry is the closest next environment that you can install and operate directly, though it's an 800-pound gorilla by design. But there are fully hosted services for it (eg. Pivotal Web Services, IBM BlueMix, SwissCom Application Cloud). There're also semi-hosted options (Rackspace Managed Cloud Foundry) and IaaS-provided installer kits for AWS, Azure and I think GCP as well. You can also buy commercial distributions from Pivotal, IBM, Atos, SUSE, IBM and SAP.

Disclosure: I work for Pivotal, we sell Cloud Foundry and Kubernetes distributions (PAS and PKS).

> What pain is Heroku saving you to justify being 5x more expensive than Lightsail/Digital Ocean/Linode?

The pain of setting it up, applying security patches, making sure you set it up securely to begin with. The pain of having a mental model more complex than "the server is what I git push to."

> I probably have over 50 sites I'm currently hosting so using anything that much more expensive wont enter into consideration.

No one would argue someone in your position should use heroku. It's for people who are willing to pay to avoid sysadmin work... which is a lot of developers.

How does Heroku and other managed services perform updates that might contain breaking changes? Or do they only perform minor updates or security updates with no breaking changes?

My biggest fear with managed hosting and managed databases is being given too short of a window before they update.

https://devcenter.heroku.com/articles/stack

tl;dr: You get a few choices of Ubuntu LTS releases, which they maintain for a long time (currently they still support 14.04, now nearly 5 years old). Or you can push Docker images, at which point the underlying OS is squarely back in your court — technically they must be applying kernel patches, but Linus is fairly religious about not breaking userland.

1 person company: sure, let's do everything myselve. As cheap as possible.

Two: here I'll teach you, take over DevOps so I have more time.

10 person team, everyone new and your application runs in the cloud. The person who took over, by now, left the company. Extended the infrastructure, didn't inform you and the application has been reworked so it runs on the cloud.

Welcome to the perils of working with people.

With your stack, how do you deploy new versions with zero downtime? Do you have a load balancer and start the new version, switch new traffic to the new version, drain connections to the old version, then stop the old version?

And how do you update the OS and the kernel without downtime? Do you setup a new machine, deploy to it, switch the traffic to the new machine, and decommission the old one?

I'm asking because these are the kind of things Heroku and other PaaS do for you.

Isn't Heroku really expensive though? I looked at it a few weeks ago and the cheapest plan was $25/month (there's also a $7 "Hobby" plan though, but that seems just for, well, hobby stuff?)

I instead got a Linode VPS at $5/month, which gives me more than the $25 Heroku plan? Setting up a VPS is not very hard either – although this may depend a bit on your environment, my app compiles to a static binary – and a lot more flexible.

My devops stack thus far consists of scp and tmux.

Look into Dokku, it is a free Heroku clone that you can run on a cheap VPS. I set it up last week and now all I have to do to deploy is type "git push dokku master".

Thanks! Maybe it would be worth replacing my 100 line nodejs script that does the same thing with this in the future :)

(not sarcasm, if this works the same and adds more value I'd use it)

Second dokku. Its essentially a 100 line node js script if u remember correctly. Nice eco system growing around it. No scaling though

I missed this comment and made a similar one up-thread. I’ll second.

I’m ignorant as to whether large projects have run using it, but for smaller ones it’s useful.

I'll second this. Have had a great experience with dokku, about as simple as Heroku.

i dont get this thinking of how $25/month is considered expensive when heroku has already figured out and abstracted away all the busy work that's required for deploying code

Heroku is great, but that's 5x the cost of Linode/Digital Ocean + Dokku.

Unless you're a pretty large company, 5x the price in server hosting probably isn't worth your time worrying about.

Or working on a handful of personal projects that aren't revenue generating.

Unless the goal is to get more ops experience, wouldn’t those be an especially tempting place to spend more time on the project and less on support toil?

What toil? You put a service up, it stays up.

Unless you’re just dropping a .php file in a shared server account, you have more setup and upgrade work setting up a running service. The first time it fails and you have to manually rebuild it, you’ll learn a valuable lesson about the PaaS value-add, too.

I find that dokku is a REALLY good middle ground... can even grow to have 2-3 servers, with a hosted load balancer and DBaaS, just deploying 3X exactly the same if you need redundancy. And starting off with a single server to experiment and host multiple apps is pretty easy to get started with (comparable to Heroku and similar).

I think the issue is that for hobby projects, the alternative to Heroku isn't devops toil, it's scp.

You can get surprisingly far on a single box. PlentyOfFish, Mailinator, and Hacker News are all services that got to millions of users with one server. StackOverflow and Google are ones that got to millions of users with a handful of servers, mostly for redundancy.

When you have big teams that are all concurrently making changes and writing code to a live site that's mission critical, then things get complicated. But when there's only one dev and your few hundred thousand users don't mind too much if it goes down? You can just spend some one-time effort installing a stock Postgres install, scp over a single binary or tarball, and run it in the background with nohup or screen. When it's time to re-deploy, upload another version, kill it, and restart it.

I’m not arguing that you can’t do a lot on a single server but think about how many separate skills you mentioned even after cutting corners for reliability, data loss, or security.

I’ve gone to a bunch of hackathons where some of the participants got derailed on that kind of stuff and never even got to the part of the project they cared about. My point was that it can be worth a modest amount of money not have that overhead on a small project.

Start:

* Single server on Dokku, with multiple projects

Grow:

* Switch DB to DBaaS on your hosting provider's cloud

Grow More:

* Multiple Dokku instances behind load balancer/proxy as a service

...

You can then scale vertically a LOT before you need K8s style infrastructure.

>>> StackOverflow and Google are ones that got to millions of users with a handful of servers, mostly for redundancy.

Servers that can cost $50k each. Not a good example for affordability.

Setting something up like Dokku, which approximates the deployment of Heroku (git push to deploy), is fairly trivial.

Okay, now you have to support a Docker cluster, backups for persistent data, and deal with things like DNS, proxy, etc. management. It's easier than doing those from scratch but it's still a non-zero amount of work which will at best take time away from that small side project and at worst cause major problems if you never come back to installing security updates, running backups, testing the restore process, etc.

LOL... Dokku on DO is pretty simple... as to backups, it's a checkbox option, as to redundancy, for non-revenue generating projects, it's generally acceptable to have some down time. As to back up again, beyond some potential data loss, it's pretty easy to get CI/CD up and pushing to Dokku.

In any case, it doesn't need to be as completely flushed out as a company with millions in investment capital could do.

+1 on this, love Dokku myself. Great option for self-hosting and can even do some redundancy with multiple servers behind a load balancer.

Dokku isn't much more support toil, so to speak and can operate starting on a single server... $10-20/month on Linode or DO. With DO, can eventually grow a bit with hosted DBaaS, and multiple dokku instances, with a load balancer in front. Start small, some room to grow, long before the likes of K8s is needed.

only consider proportion when the problem is scale. Unless you're deploying many many projects I don't think $25/month is expensive

$5 compared to $25 doesn't seem like much, but over a year that becomes $60 spent vs $300.

For $300/yr you could get multiple Squarespace websites, and they abstract even more.

Squarespace is great for solving lots of problems and if it solves your problem you should definitely use it before heroku. But as devs we're usually brought in after someone has figured out their problem can't be solved with Squarespace.

So, I don't really understand this comment. I'm not a programmer and I don't really understand what kubernetes, heroku, etc do. But I thought they were for applications, not websites.

I've done things like go to the Wikipedia page for kubernetes. It hasn't helped me figure out what such things do.

Anyone care to point me to some kind of 101 explanation to help me follow the conversation?

Sorry to see you're getting downvoted, but it's probably because that's just too long of a story to put in a comment. The parent you're replying to was being sarcastic: he meant to say, "well if we're going to pay other people to do our job, why stop at Heroku? why not pay squarespace even more money to do more work for us?" A reductio ad absurdum.

Ironically, he's right. But, as someone pointed out: Squarespace is probably not relevant in a conversation about Kubernetes, while Heroku is.

What Kubernetes is, is a long story. Suffice to say: if you don't know what it is, count your blessings. :)

So, what I'm hearing is that my confusion is justified because Squarespace doesn't do what kubernetes and heroku do.

Is that accurate enough?

yes, but that's on purpose. "This is about Kubernetes, stop changing the subject to Heroku." or something. it's hard to read sarcasm online.

Thanks.

Right. Squarespace is a hosting provider and website builder that anyone should be able to use to build a website, typically a smaller scale website like a blog or a small to medium online store / e-commerce site. It's similar to Wix, Weebly, WordPress.com, and the website builders that some hosting providers offer.

Kubernetes is a tool large companies use to manage a large number of servers. Google invented it and open-sourced it so now it's free for anyone to use. It's not something people would use for a single website unless it was a huge website that required lots of servers, like FoxNews.com or something.

Thanks. That makes me feel a lot less stupid. It's much more in line with what I thought.

Kubernetes is a tool that lets Squarespace more efficiently host large numbers of websites on a smaller number of machines.

https://www.cncf.io/people/end-user-community/

Squarespace abstracts away code so non-developers can make websites. To the best of my knoweledge, you can't really do anything complicated with Squarespace but it takes care of a lot of the hassles of building a website such as figuring out security, social media integrations, and design layouts.

Web applications (web apps) are essentially websites but with some additional functionality. Web apps can work the same as websites by having urls for different pages but are distinct in that they can do things that a website hosted on Squarespace can't do. This is because to build a web app requires some coding up front which is both an advantage and disadvantage of webapps in comparison to a site hosted on Squarespace.

Heroku is a platform as a service (PaaS) offering that lets developers utilize version control tools to update their webapps. This means the level of complexity for deploying a new version of webapp is incredibly simple.

Imagine that you collaborate in an office that publishes technical documentation and various technical writers can work on multiple parts of the same document. Each new version of the document gets published to a PDF that users can access.

This is essentially what Heroku provides along with abstracting away some of the difficulties of getting a web app hosted such as security, setting it up so your site uses https, and some basic monitoring and logging.

Sometimes web apps look and feel like a singular thing but are actually multiple pieces working together. In this case, you may want to isolate these different pieces.

Doing so can be really difficult, and to the best of my knowledge, Heroku isn't necessarily designed with this level of orchestration in mind. This is where Kubernetes comes in.

I've never used Kubernetes, so I might get a few things wrong here but from what I understand, Kubernetes gives developers/devops people a lot more fine grained control of how the various pieces of a webapp or webapps get deployed and managed. Kubernetes let's you take advantage of containers which are sort of like micro operating systems but only with the dependencies you need installed for a service to run. This means you can write in a file "I want x replicas of this part of my app and y replicas of this other part of my app to be ran across z number of workers".

What this abstracts is how an webapp should be ran without manually setting up each piece your self.

Here is a video with a bit more info on Kubernetes: https://youtu.be/PH-2FfFD2PU

Thank you.

These two online books (made to seem like Children's books) represent a true ELI5 explanation: phippy.io

Thanks. That's perfect.

Try caprover.com, minimalistic orchestration similar to Heroku or Dokku (but with a GUI). Can scale too with Docker Swarm. I'm using it and it's a breeze, really happy with it. Previously I was using Rancher, but didn't like the switch from Cattle to Kubernetes.

Just tried CapRover this weekend for a hobby project and it was even easier to set up than Dokku. The NetData monitoring integration is great too, and just as painless to set up.

For use cases where support plans and SLAs are not an issue (hobby), it's a great option.

A third recommendation for CapRover. It's simple, it's beautiful, and it's open source. I've been enjoying it so far :)

CapRover also features one click SSL generation via Let's Encrypt, similar to what is offered on Heroku.

Look into google app engine. It’s very similar and much cheaper. For hobby projects the free tier is usually sufficient.

It gets much more expensive quite quickly

Not really. I run large enterprise apps on it and it’s cheaper than anything else I’ve used.

Heroku's $7 instances and the $25 instances are not very different. There is also the free instance that sleeps for 6 hours.

You have to factor in that you get free tiers of many things, all managed together like docker would help with, such as: 30 MB Redis memory cache compute instance for free 500mb Mongo database compute instance for free External logging compute instance for free

and a whole marketplace of all these managed services, with the grouping of containers further managed by heroku.

A Linode VPS at $5/month does not give you all that. If you like configuring all the above (and of course, assuming your use case calls for it at all), then the $5 plan with 1 GB RAM would let you put in a bunch of 256MB containers if you really wanted. The $10 plan with 2 GB RAM would let you put in comparable 512MB containers, but then you should have just been paying $7/month for Heroku already.

Hope that helps!

For a long time I struggled with inability to understand my potential costs for server-side projects, and occasionally read horror stories about other devs who got it wrong.

Suggesting that $25/month is expensive illustrates how insanely cost-sensitive this community is.

Don’t come to HN for a representative take of how people in US businesses evaluate vendors and their pricing.

Do you know how much it cost my employer for me to spend an hour reading about Kubernetes?

> Suggesting that $25/month is expensive illustrates how insanely cost-sensitive this community is.

I completely agree but I’m not sure “cost-sensitive” is an adequate term because it’s like a cost on hosting (or an app you’ll use for years) triggers extremely high awareness by people who are extremely blasé about hemorrhaging staff time on support and slipped deadlines.

Sometimes it can be hard to give up sysadmin work because people consider it to be a core competency and to let that go feels like a loss. Kubernetes hits the sweet spot where you get to use a tool to manage sysadmin work that is at least as much work as hiring a sysadmin.

So true - billing rates for consultants run 200+/hr. Even a small $1mm expense budget - $25 is not relevant and in my experience you can quickly be at 75%+ personnel cost (w2/1099 + desks / space etc for them)

Not all businesses pay $200/hour for consultants. There are a lot of small businesses out there.

"A small $1M expense budget" may be "small" for certain companies, but it's more than the yearly revenue for a lot of businesses, never mind people who are trying to start a business and don't have any revenue at all (yet).

Of course you need to be reasonable and not penny-pinch or "spend money to save money", but in general I would say that frugality is a virtue.

Well, $25 is 5 times as much as $5, so if the costs scale at a similar level then your $500 hosting costs will end up being $2,500 which, depending on your business, may or may not be a significant cost.

Back when I worked for a hosting company specializing in RoR we had more than a few customers migrate to us because Heroku costs were getting out of hand (and we weren't all that cheap either!)

In my specific case I'm prototyping what could perhaps be a startup, and with a few small cheap Linode VPS's I can get a lot of bang for my buck.

> Do you know how much it cost my employer for me to spend an hour reading about Kubernetes?

A lot, which is why you shouldn't use it. You can run a VPS without k8s.

Not everyone is thinking in terms of an employer, or looking to build a profitable business. One of the main sticking point for my side projects is related to deployment environments.

Heroku is painless enough for unprofitable hobby projects, but far too expensive. Using AWS, GCP or Azure directly is relatively affordable, but requires a lot of extra work.

I'm aware of various tools that are supposed to make working with the various cloud platforms much easier, but every time I see one of these used in practice (eg at work), people seem to spend an enormous amount of time getting things working properly. It still feels like we're missing a sweet spot for hobbyists who don't want to invest their spare time learning about and wrangling with devops, just to get a simple project up and running.

Has anyone ever recommended Kubernetes to anybody as a production environment for an "unprofitable hobby project"? In the context of the article we’re discussing, the suggestion that $30 is an unreasonable operations overhead for a team of four engineers is thoroughly preposterous.

Tangential, but if anyone wants something Heroku-like in for deployments and monitoring but less [eae-of-use gui] control over scaling I recommend Digital Oceans “one click” Dokku deployment.

I’ve used it at work for sunset of our projects and it’s been pretty good once you learn a few of the gotchas. Once you’re set up it’s pretty painless. There are limitations for sure, but it’s been handy for a few situations where we didn’t want to focus on deployments and keep them as simple as possible.

If you're going for Heroku because of its simplicity, I don't consider the alternative to be "spend an hour reading about Kubernetes" (and presumably many more hours when I have to troubleshoot it next week). I'm not touching Kubernetes or Docker or any of that.

Even on AWS, I can deploy with one short command, and it's much cheaper than Heroku.

Can you run a one-off instance like “heroku run <command>”?

The $7 hobby plan is surprising useful if you’re smart about what you’re deploying.

GKE is head and shoulders above the rest.

EKS is a joke. The only people that use it are those either experimenting, or those that are forced to.

AKS is pretty okay. They're definitely way ahead of EKS. They lack a few things, but they've made some pretty good strides in the last year. Kinda suffers from being part of Azure which likely reflects my personal bias.

Managed k8s or Heroku? I don't understand why people don't try to do AppEngine. It's the oldest "Serverless" platform out there and wickedly mature. If you're worried about vendor lock in (and you should be) I have deployed AppEngine apps on the F/OSS AppScale unmodified.

Why should you be concerned about vendor lock in?

I haven't heard a convincing argument why vendor lock in is a problem regarding the cloud.

It can be a problem, but all solutions result in you avoiding the things you went there for in the first place.

There is exceptions to this obviously but I find most people worried about vendor lock in are no where near big enough to bother running multi cloud.

because you have to agree with every change vendor make. If they double pricing, you pay twice. If they ban service type you host, you have to stop. If they make an incompatible change in some of the hosted parts, you have to adapt. If you grow too much they refuse to allocate you more resources or it is expensive for you to use them at your scale, it is difficult to migrate away. That is vendor lock in -> loss of freedom and choice.

neop1x has a great response. A more concrete example would be a gaming company that started out with AppEngine for its server support for the game. Once the game has a proven revenue stream, the economics of moving from AppEngine to AppScale are a critical next step to maximizing ROI. You have all the code, now you just need to host it somewhere. Can't do that with any other serverless Platform.

It's important to note that AppScale is an aPaaS, and API-platform-as-a-service, where you're guaranteed a consistent API with the ability to plug in different implementations. Something even beyond your typical openness in F/OSS software, too.

> I have deployed AppEngine apps on the F/OSS AppScale unmodified.

How did you manage secrets on Google App Engine?

In the flex environment you can inject secrets at container build time. In App Engine standard I've used a deploy wrapper around ansible vault to do it.

> In the flex environment you can inject secrets at container build time.

Does it mean your secrets are stored in plain text in the container image?

> In App Engine standard I've used a deploy wrapper around ansible vault to do it.

What does the deploy wrapper do? Does it produces an app.yaml file with the secrets injected in it, after having been decrypted by Ansible Vault?

Q: Does it mean your secrets are stored in plain text in the container image?

A: No. It means the secrets are stored as environment variables in the container.

Q: What does the deploy wrapper do?

A: It prompts the developer to input the ansible vault password , decrypts the vault and injects the secrets into the environment.

Generally speaking, I follow the 12-factor approach:

https://12factor.net/

What do you mean by "injecting secrets into the environment"?

You mean a section in your app.yaml like this one:

    env_variables:
      DB_PASSWORD: "this is a secret"

As for as I know, in the Standard and Flex environments, app.yaml is the only way to define environment variables.

Broadly agree with this. Use K8S at LastCo -- was great so long as I didn't have to manage it, but actively managing our own K8S cluster was a nightmare. Using Heroku at CurrentCo and it's a breeze. I highly recommend it. The only thing I think that could unseat it of its ilk would be Zeit, which looks very promising. I'm skeptical of all things serverless right now but Zeit looks promising and most impressively, extremely (perhaps as much as is possible) cost effective.

>When I've deployed on ECS (or other "simple" orchestrators), I've found that I ultimately wound up re-inventing lots of Kubernetes features.

Could you elaborate a little on this part? Our team is looking at ECS/Fargate as a possible container solution and I am curious what you felt was missing from it.

If you're comparing to Kubernetees, "everything".

But I'm using ECS and Fargate right now (orchestrated with Terraform). https://rivethealth.com

Probably the biggest thing I would like is native secrets management. "Security groups" are limited as it depends on having an AWS interface per container.

ANd I don't know how much longer our time on Fargate will last. It's expensive, but more significantly there are limitations in not having access to the host. Having to install an SSH server in every docker container to be able to debug (simple things, like top) is annoying.

We nixed ECS for the same reason, no secrets through ConfigSet.

[Update: See below, many of these features have been added to ECS since the last time I touched this portion of our infrastructure.]

> Could you elaborate a little on this part? Our team is looking at ECS/Fargate as a possible container solution and I am curious what you felt was missing from it.

Some typical examples:

- Kubernetes allows you to run a monitoring container on every single node using something called a "DaemonSet". On ECS, you'll have to build all your monitoring tools into your base image, or use cloud-init to spawn an ECS task on each machine.

- You're probably going to end up writing a bunch of scripts to generate ECS task definition JSON and to update running services, and you'll need to integrate this into your CI system somehow. With Kubernetes, you can get away with "kubectl apply -f" for a fairly long time.

- Kubernetes makes it relatively easy to allocate and manage persistent disk volumes. I wouldn't necessarily use them for a production database, but they're great for smaller things.

- Kubernetes has autoscaling support, plus the ability to control which containers run on which types of servers.

- Kubernetes has basic secret management built-in. It's nothing as nice as Vault, but it's good enough to get started.

- Kubernetes has support for a whole bunch of useful minor things that would typically wind up as Terraform scripts on AWS.

And so on. None of these is very hard individually, but there's a ton of things like this. So we're slowly migrating pieces of ECS infrastructure over to Kubernetes so that we can stop reinventing so many wheels.

Again, for those things which can run on Heroku, either choice is overkill. And I have to admit that ECS is very reliable at the things it does. If you do decide to look at Kubernetes, I highly recommend skimming the O'Reilly books, which provide a solid overview of how it all fits together.

All the things you mentioned are available on ECS:

- https://aws.amazon.com/about-aws/whats-new/2018/06/amazon-ec...

- https://docs.aws.amazon.com/AmazonECS/latest/developerguide/...

- ECS runs on ASG / EC2 so there is auto scalling

- https://aws.amazon.com/blogs/compute/managing-secrets-for-am...

I think you don't understand all the glue between AWS services, off course ECS doesn't have everything that's why there is EKS, but all the thing above exists on ECS.

Nice! I'd been busy with another project and missed a couple of those announcements. The DaemonSet equivalent will allow me to rip up some particularly annoying cruft the next time I need to touch it. Thank you.

I've looked at various guides for setting up cluster autoscaling on ECS, and so far, everything I've found looks far more complicated than Kubernetes cluster autoscaling on Google. Is there a nice guide?

Look no further:

https://github.com/awslabs/ecs-refarch-continuous-deployment

https://github.com/aws-samples/ecs-refarch-cloudformation/bl...

The secrets example isn't very comparable. With Kubernetes, the secrets are injected into the container as environment variables or files, whereas the ECS example requires assigning an IAM role to the task and doing a Parameter Store lookup from within the container. This usually requires a custom docker image with an entrypoint to handle that part. The same is true for ConfigMaps, which I believe ECS lacks.

No, you don’t need to do any of that. ECS can inject secrets into environment variables in much the same way as Kubernetes. This behaviour is already built-in, there’s no need to build your own solution for it.

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/...

That's good to know. The parent linked to an AWS blog post that did work that way. The documentation you linked is a new feature since I last used ECS.

Exactly. You can deploy a highly available, auto scaled, logging enabled, spot instance or Fargate multi-service ECS reference architecture here by deploying a “one click” cloudformation stack: https://github.com/aws-samples/ecs-refarch-cloudformation/bl...

ECS already has all the things you mentioned here. Some, like daemon set deployments, are built in. Others, like secrets management, are provided as integrations with other AWS services. But they are all there.

FWIW did the fargate approach very happily. I know k8s getting the love but found fargate easy to get going on and set and forget once deployed

Why don't more people just use the native machine imaging built into cloud providers? (I would actually like to know this)

Building an AMI on Amazon is not difficult and if you use it you don't have to reinvent so much architecture on top of AWS (or cloud provider of your choice)

For us, it's much more economical to binpack multiple apps onto a VM than to run one service per VM. Also it makes it much easier to scale out particular services once they outgrow your chosen VM size.

If your application is 50 microservices, how would you manage that on ec2 instances?

Why does the number of microservices have anything to do with where they might live? Wouldn't that be much more related to CPU load?

Because you generally want to deploy them independently. If you have 50 microservices and have to run an AMI build every time that's going to significantly slow you down. With containers you generally only deploy the service that has changes to a VM that doesn't change as often.

You want them to be independent, atomic. How they are deployed and the choice of technology you choose to couple them with? That should be driven by load -- and dynamic.

The reason everybody's so hot on microservices to begin with is that we kept coupling everything all together and it became a huge mess to manage. You don't want to repeat that mistake, only at cloud scale.

One autoscaling group, AMI, instance (or more than one) per service.

A nontrivial portion of those microservices could probably run on Lambda.

> Why don't more people just use the native machine imaging built into cloud providers? (I would actually like to know this)

I've done that, and I've done Kubernetes, and Kubernetes is definitely easier once you get past the initial setup. The initial setup is also getting easier over time.

It is also more portable. Kubernetes runs on multiple cloud providers as well as your own hardware and presents the same interface and runs the same containers. Docker containers are more portable than AMIs.

If you plan to write your own system to control deployments, secrets, load balancing, and DNS based on AMIs and other AWS features, you may want to consider that you are reinventing the wheel. You are also locking yourself into AWS to a far greater extent than you would if you used Kubernetes.

Expertise is also a big differentiator. You can hire people who know Kubernetes on day one, but you cannot hire people who already know your custom in-house system.

Was using Heroku and moved to GCP. It's almost as nice but not as dead simple. App Engine Standard is now updated for latest languages, Python 3.7, Node, etc and really good, its also pretty cost effective. Overall would recommend.

Though, there is a middle ground between running vanilla Kubernetes yourself on your own hardware and tying yourself into the specifics of a cloud provider just to get managed Kubernetes. There are really good Enterprise distros like OpenShift or PKS that relieve you from the hassle of running Kubernetes and focus on using it but don't force you onto a particular platform or cloud.

What parts of Kube did you find yourself reinventing with ECS? It’s not the most complex microservice setup, but I run around 60 services over 3 ECS clusters and have found it pretty easy and solid at this point. Only real pain point we’ve had was the limit on Awsvpc mode service per ec2 instance—never have figured out why instances are limited to just a few NICs.

For my side projects I just wrote a simple service that listens for Github events and just checks out the repo, builds it, and ssh's to the target server and runs a "stop and run" script. This way I can even put all of the configuration for the different databases in version control too.

It's super dirty, but I can deploy dozens of services without issue. Monitoring is another problem, but I don't need Prometheus for this stuff.

(I couldn't use Heroku cost effectively because running graph dbs and some custom stuff, but Heroku is AWESOME)

> do not attempt to host your own Kubernetes if you can possibly avoid it.

We use Google's Kubernetes and are pretty happy with it, but overall this statement bothers me.

It's absolutely true. Don't get me wrong. The problem is what it says about Kubernetes as software. It tells me it's ugly and crufty and in some ways immature. Good software should not require that much babysitting.

Yes. I think Kubernetes has a lot of really useful features but it is needlessly difficult to use and administer. Unfortunately, it seems to have become the de facto standard, so everyone will be forced to deal with it the same way they have to deal with ancient Unixisms (or worse, windowsisms).

The statement is bothersome because it says more about the person saying it than anything else. If your job is to manage infrastructure and you have to deal with thousands of snowflake deployments by other teams then k8s is a godsend.

I just hate unnecessary complexity. I am not sure if the complexity in k8s is necessary or not. Mighy be.

The thing is if you keep to the basics Nomad/consul will do the job just fine.

As the fine article points out.

Why is this upvoted? It's just a bunch of rules based on no reasoning or evidence.

I’m surprised no one has mentioned AWS ElasticBeanstalk. It supports Docker (ECS behind the scenes) and is very close to Heroku in terms functionality but with EC2 pricing.

Beanstalk is nice for rapid development. The problem I is that it doesn't scale for nontrivial architiched apps. I've converted failing beanstalk deployments to real orchestarators a few times.

Beanstalk is a joke.

You can launch a DB through EB but the docs basically state it's a bad idea (as it gets taken down when you delete your app, which you might find yourself doing if EB gets in to an unrecoverable state).

So now you have to manage EB + RDS separately, which should be automated so now you need CloudFormation to add the security groups and manage the vars for the DB connection properties in your EB.

Heroku is really a different solution, I don't know anyone at medium scale project that uses Heroku, Heroku is like the Digital Ocean compare to AWS.

You can't really compare DO, Heroku with k8s and AWS, the laters are much much more powerful.

I work for an occasional competitor. Heroku are legit.

"Finally, don't overcomplicate it. Just use the basic stuff for as long as you can before trying to layer all sorts of other tools over it."

This is I think always the best advice for almost anything.

And we all know it. And yet so often, we ... can ... not ... resist.

Reply on news.ycombinator.com