Hacker News Reader

Comment by maslam | Hacker News Reader

HN, I'm a co-founder at Appuri. Happy to answer questions! PS: We LOVE most AWS services like Amazon Redshift. Just not ECS ;)

View on HN

Did you run ECS on a custom AMI, or use the stock one?

We've been running with vanilla EB + ECS for months and haven't seen this at all.

From an outside perspective, it sounds like the primary issue you referenced here (the agent disconnecting) could have been due to a mismatch in configuration between the agent and docker, or maybe just a permissions issue. IIRC, the ECS agent tries to clean up containers every few hours so perhaps not being configured correctly caused it to get stuck?

We used the stock AMI. I would not be surprised if we had a configuration issue, but we spent a lot of time trying to debug it and were never able to find the root cause of the issue.

yep, we were using Amazon ECS-Optimized Amazon Linux AMI, docker and agent should've matched there. It was strange behavior, either all agents were connected or most of them were disconnected at the same time...

How did you first discover the bug? Did your deploys just start failing in an obvious way, or did your instances disappear (from ECS's perspective)?

I'm wondering if Elastic Beanstalk is hiding this from us in some way. Or maybe just has a workaround built in?

Yes. This is pretty much what we saw.

Did you deploy K8S on AWS? If so can you add any details about how? Or are you using K8S elsewhere? I love AWS but planning on spinning up on GCE this weekend to play with K8S.

I am ops at Appuri. We deployed k8s on AWS since we are using other services like Redshift and RDS inside the VPC, also happy with how EC2 works plus of course we have Reserved instances so we didn't look into GCP yet. We're running kube on CentOS 7, we bootstrap nodes using cloud-init (user-data) to setup k8s, which we then use to run everything else. I would love to give you more details, I might write blog post about our kube setup decisions later. Kelsey wrote nice manual for setting up k8s - https://github.com/kelseyhightower/kubernetes-the-hard-way which is definitely on my to-read list this weekend :)

How I wish we had Postgres support in Cloud SQL already...

Depending on how many RIs you have, maybe our custom shapes, much lower prices with per-minute billing and sustained usage discounts, and BigQuery vs RedShift you'd still come out ahead. You could always try your hand at reselling your RIs on the "market".

Would you rather run pgsql by hand or K8s? (And yes, I don't want to make you choose, that's just the current state of the world).

Disclosure: I work on Compute Engine, so of course we want your business.

heh, I would rather run psql as a service which is what we have with RDS :). We are aware of BigQuery advantages, Redshift resizing is tedious job, but we currently don't think that the cost of migrating to GCP would be so beneficial that it is worth the effort. This can of course change in future, but for now we are reasonably happy with our setup on AWS and its reliability.

Perhaps you can take a look at some folks who've done this with great success: https://cloud.google.com/customers/sharethis/ https://youtu.be/6Nv18xmJirs?t=8m7s

I knew it. I'll chalk you up as yet another customer that "might" move if only we had managed Postgres.

The kube-up.sh script can start a complete Kubernetes cluster on AWS in one fell swoop. It's pretty smooth.

We deployed K8S on AWS (I'm a dev at Appuri). Like Bilal mentioned we run pretty much everything on AWS, so it was an easy decision.

Off topic from the article but just wanted to let you know that I love the design of your blog.

Thank you!

Reply on news.ycombinator.com