Hacker News Reader

I recently interviewed a PHP dev who asked me why we even have backend developers. From his perspective, we could just use Parse and it would do all the "magic" of the backend.

Then Parse shut down.

I think the lesson here is, you should never trust your core product to a third party. Also, when you use cookie-cutter services, you really limit the kind of things you can do, and vendor lock-in really, really limits your options in the long run.

I have heard this repeated many times and it's nonsense for two reasons:

First, it isn't actually true: Third-party managed services (say, exotic databases) are a mere network hop away. If you really had to you could call out to your own self-managed services in EC2 or whatnot. Even on Google App Engine, which is one of the most restrictive systems, we use a multitude of services running elsewhere in "the cloud". This is occasionally a mild annoyance, but it doesn't significantly alter the value equation.

Second, it's irrelevant: Migrating our application off of GAE would be tough, sure. You can call this "lock-in" if you want, but it ranks somewhere down a distant thousandth after a long list of concerns about my product and how well it is meeting the needs of my customers. Worrying about lock-in is a premature optimization.

The typical argument about the superiority of cloud services is that it's more hands-off, but what they don't talk about is how shared resources causes all kinds of new headaches I never had before.

I recently migrated from a dedicated DC to AWS because of business reasons (ie. CAPEX vs OPEX).

Before, my PostgreSQL DBs were always perfectly in sync because they were on a dedicated, private network. Now, I often get large replication delays because the network in EC2 isn't of the same quality I had before, so I've had to spend a lot of dev time working around the delay issue with creative caching solutions.

My point is, there's trade-offs and mostly my calculations show that they're not really worth it, if you have knowledgable Ops people. Which, once you grow to a certain size, you should.

I think you're misreading the message - managed services doesn't mean EC2. Managed services are things like DynamoDB, ElasticBeanstalk, SQS (in the AWS world) or Cloud Datastore and App Engine (in the Google world). If you're building on IaaS like EC2 or GCE, you will need ops people. If you build at a higher level, you don't.

If anything my argument holds up because you're putting even more abstraction between you and your systems by using the managed services, which leads to less understanding of how things actually work.

For instance, I went with running PostgreSQL instead of RDS because I have much more control over what's going on in the DB than I would with RDS.

But your point that the managed services don't require ops might be true in the short term, but what happens when something breaks? Or you experience a level of growth that expands past the capabilities of the services you've organized? There's hard practical limits to all the of SaaS offerings - mostly latency and throughput - that doesn't really come up in these discussions.

The analogy about it being like assembly programming might be more apt than I originally thought, given how not doing it yourself generally results in slower and poorly optimized solutions, but then "good enough" is often the mantra of the day at a certain scale.

Further on the "assembly programming" line of thought: there was a long time where compilers just couldn't match the speed of hand-coded assembly...but that time is almost completely behind us, and probably has been for quite some time.

As management tools mature and automatic, systemic optimizations are developed for SaaS systems, I wonder if the analogy will continue and areas where you need to manage it yourself to eke out performance and flexibility will get fewer and farther between over time.

You hear this same sort of argument with the adoption of any new technology.

I imagine that when Flight Navigators started disappearing from flight crews in the 1970s with the advent of GPS and other related electronic navigation systems, there were those saying "I'd never leave the ground without a flight navigator - what if the GPS stops working?!?" Should we keep an old school Flight Navigator on standby "just in case"?

I remember talking to an retired mechanic about 20 years ago, who told me he never did a road trip without taking all his tools with him so he'd be ready to fix anything on the side of the road if need be. I wonder if he is still doing that....I doubt it.

As technology gets better and better, the "Number Needed to Harm"[1] needs to be considered.

[1]https://en.wikipedia.org/wiki/Number_needed_to_harm

edit: missing word

It really limits your ability to react when things DO go wrong. Not having direct access to hosts limits your options a crap ton

It's ultimately a short-sighted choice - you get running really fast but then if you do well, you basically have to re-engineer from scratch.

It's basically the ultimate technical debt.

This is what I'm worried about, PaaS lets you get going very quickly but if you succeed...well I haven't heard great stories about scaling out PaaS.

The situation I'm in is our apps currently do low enough traffic to exist in a PaaS, but they weren't built that way from the ground up and live on bare metal currently. Migrating to say, AWS, would using ElasticBeanstalk be burdening myself with technical debt I don't currently have vs using something like CloudFormation (Terraform) and CodeDeploy?

Well, the question is - does that app have the potential to outgrow the SaaS offerings?

There's plenty of things that won't, for a long time, so you're fine to save the time since you won't need to re-do it later.

But if you can really see the product expanding to a large scale, you need to think a year or more into the future when planning your infrastructure.

I'm not sure that's always the case. If it's e.g. RDS, moving to your own Postgres cluster is doable when necessary

I am reminded of a comment by someone famous:

"The first software companies had thousands of developers. The second wave of software companies had hundreds of developers. The next wave of software companies will have 1 developer."

Can't find the source, might not be the exact quote either, sorry.

Do you have a SLA with Google? Do you have any plans to proceed if they just shut down GAE within of a few months?

This has happened with Parse, and this will happen again.

GAE is subject to Google Cloud's Deprecation Policy, which says you'll have at minimum one year to migrate off it: https://cloud.google.com/terms/

Which is okay if you just use Google’s Cloud as container host.

But if you have to rebuild your whole company, because everything you use runs on GAE, 1 year is almost guaranteed death for your company.

Google has been making significant visible investments in GAE, even with the advent of Compute Engine. There is also a hybridization of the two going on. While I can't claim to expect my code to live unaltered forever, I'm not worried about having an upgrade path. Also: I do have a lot of GAE apps that have been running nearly unaltered for several years. So there is a good track record.

For a point of reference, when Google rolled out the High Replication Datastore, they kept around the Master/Slave datastore for years and even built a pushbutton tool to automatically migrate apps. GAE has been around since 2008; this is not an experiment like Wave.

I absolutely agree. If you want to stay in the AWS world, look at Elastic Beanstalk or Lambda. I've been doing DevOps for years and most apps (internal or external) that I deploy anymore are just one Jenkins job that packages and deploys a Beanstalk version. Give Devs permission to Jenkins or have it auto-poll git and I never need to do anything else.

What about the team(s) that manage the underlying PaaS/SaaS Services?

Does DevOps or SRE's not exist in this realm?