Hacker News Reader

Comment by dr01d | Hacker News Reader

In most cases, very few companies have products that need to scale to extreme load day 1 or even year 1. IMO, instead of reaching for the latest shiny cloud product, try building initially with traditional databases, load balancing, and caching first. You can actually go very far on unsexy old stuff. Overall, this approach will make migration easier in the cloud and you can always evolve parts of your stack based on your actual needs later. Justify switching out to proprietary products like lambdas, etc once your system actually requires it and then weigh your options carefully. Everyone jumping on the bandwagon these days needs to realize: a LOT of huge systems are still rocking PHP and MySQL and chasing new cloud products is a never ending process.

View on HN

In my case switching to an AWS API Gateway + Lambda stack means I have zero-downtime, canary deployments that take less than 5 minutes to deploy from version control. Api Gateway is configured using the same swagger doc that autogenerates my request/response models (go-swagger) and (almost) completely removes routing, request logging, throttling and authentication concerns from the request handlers. Combined with a staticly hosted front-end and SNS/SQS+lambda pub-sub for out-of-process workers I never have to worry about auto-scaling, load-balancing or cache-invalidation and we only pay for what we use. It may not suit every use case, but in our case, we have bursty, relatively low-volume traffic and the hosting bill for our public-facing site/service that comprises most of the main business revenue is the same as a rounding error on our BI service bill.

How do you deploy in less than 5 minutes? Our serverless deployments are running into 20 minutes now, and it’s becoming a bit unfunny.

We use golang lambdas, binaries are built in our CI pipeline. Build stage takes ~10 seconds, tests (integration + unit) take ~30 seconds. We use AWS SAM for generating our CFN templates, we package and deploy using the AWS Cloudformation CLI and this takes the remaining 3-4 minutes.

I didn't include post-deployment end-to-end tests in the 5 minute figure, but technically speaking, we do deploy that quickly

How does it take 20 minutes? Do you deploy everything at once? Most of our APIs take a few minutes (depending on what kind of a deploy it is).

We have a fair number of endpoint, so due to the CloudFormation 200 resource limit per stack, we end up creating about 10 different stacks that frankenstein themselves onto a main API gateway stack.

Obviously not using cloudfront...might be the slowest service I’ve ever seen to deploy

Try deploying changes to google cloud loadbalancers. Updated within a few seconds, but changes will take seversl minutes to be applied. The first time i was scratching my head why my changes don‘t work as expected...

This (pay for what you use, so many fewer scalability issues) is so big it, by itself, can give you a competitive advantage against anyone who isn't doing this, which is almost everyone.

Maybe I'm overstating it, but I don't think I am...

I think you're overstating it. Why do people care so much about scalability issues, anyway? Given that (a) a stateless server plus an SQLite instance is much, much easier to set up than the proprietary, poorly documented mess that is Lambda, (b) that server can easily be horizontally scaled, and the SQLite instance can be swapped out with any other SQL database with some effort, and (c) a single server with SQLite will easily handle up to 100K connections, it doesn't seem like scaling was ever an issue for most websites.

I don't think you've used the tools that can work with lambda or had to actually scale something in production, based on your response...

It's all a lot harder than you make it out to be, but at least with lambda (and something like Zappa) you don't have to figure anything out beyond how you get your first environment up. There's just no second step, and that's huge.

Really? Lambda is poorly documented mess

Using a scripting language like Python or Node it literally is adding one function that takes in a JSON event and a context object as your entry point.

Serverless is also easier to develop for.

With Google Firebase Functions I was able to start writing REST APIs in minutes.

Compare that to setting up a VM somewhere, getting a domain name + certs + express setup + deployment scripts, and then handling login credentials for all of the above.

I had never done any of that (eventually I grew until I had to), so serverless let me get up and running really quickly.

Now I prefer my own express instance, since deployment is much faster and debugging is much easier. But even for the debugging scenario, expecting everyone who wants to Just Write Code to get the horrid mess of JS stuff up and running in order to debug, ugh.

(If it wasn't for HTTPS, Firebase's function emulator would be fine for debugging, as it is, a few nice solutions exist anyway.)

But, to be clear, on day 1 the option for me to write a JS rest endpoint was:

1. Follow a 5-10 minute tutorials on setting up Firebase Functions.

1. Pick a VM host (Digital Ocean rocks) and setup an account

2. Learn how to provision a VM

3. Get a domain

4. Get domain over to my host

5. SSH into machine as root, setup non-root accounts with needed permissions

6. Setup certbot

7. Learn how to setup an Express server

8. Setup an nginx reverse proxy to get HTTPS working on my Express server

9. Write deployment scripts (ok SCP) to copy my working code over to my machine

10. Setup PM2 to watch for script changes

11. Start writing code!

(12. Keep track, in a secure fashion, of all the credentials I just created for the above steps!)

I am experienced in a lot of things, and thankfully I had some experience messing around with VMs and setting up my own servers before, but despite what everyone on HN may think, not every dev in the world also wants to run a bunch of VMs and manage their setup/configuration just to write a few REST endpoints!

So yeah, instead I can type 'firebase deploy' in a folder that has some JS functions exported in an index.js file and a minute later out pops some HTTPS URLs.

If you don't want to learn DevOps why not use a PaaS like Heroku? That way when you want to learn DevOps, you can move your application without rewriting large swathes of it.

It's funny but when I learned to code basically all ISPs provided you with free hosting and a database, and you just needed to drag and drop a PHP file to make it live. It's like we have gone backwards not just in terms of openness but also in terms of complexity.

The last time I had done server side dev, yeah, it was all PHP and FTP drag and drop a file over.

I was a bit shocked at how asinine things had gotten.

This seems a bit exaggerated. You definitely don't need to start with certbot or even with a domain name. Why can't you just start with a regular server on localhost, which can be set up in way less than ten minutes, and publish it only when you're ready? Not to mention that Firebase Functions is way less beginner-friendly than Express.

Mobile app development requires an HTTPS connection.

Every React Native tutorial out there has a section on setting up user auth with Firebase, and then putting a few REST endpoints up.

It is simple enough that a beginner "never touched mobile or web" development tutorial can go through it in under an hour.

Firebase is incredibly simple to get started with.

Another solution is to use one of the HTTPS port forwarding services that takes a localhost server and gives it a public HTTPS endpoint, but that is more work to explain than

    firebase deploy

so the tutorial authors go with Firebase. Auth being super simple is icing on top.

you should check next.js and now deloyment, even faster/easier setup for JS stuff

I'm generally a proponent of DIY, this doesn't make sense. When an org needs to scale, it makes sense to cultivate skill in the underlying infrastructure. On day 1, serverless makes a lot of sense because it encourages development patterns that eventually will scale nicely.

Placing stateless web servers on simple VPS instances with load balancers in front doesn't scale nicely? Or perhaps I'm misunderstanding the parent.

Serverless does force you to think for higher concurrency than bigger server instances which is worth something.

And while we spend all of our energy and money doing that, we aren’t actually creating the product that is suppose to be bringing us revenue....

This is the correct answer. AFAIC, Intro to Economics should be the first CS course any software engineer takes.

I have mixed feelings about this and I don't have enough experience to label on method better than the other.

Lambdas basically require zero maintenance. SQS requires zero maintenance. EC2 load balancer is zero maintenance. And the setup is trivial too and there's no migration time down the line.If you start off with native cloud for everything you can keep your maintenance and setup costs down drastically.

However, a lot can be done with the old school unsexy tech.

So I'm mixed.

Zero maintenance? What about standing up multiple environments for staging, prod? Sharing secrets ,keys, and env vars? Deployments? Logging? No migration time down the line? I'm pretty sure GCP, Azure don't have Lambda, SQS, or EC2 load balancers so you absolutely will have migration time if you have to retool your implementation to switch cloud providers or products.

You make really good points about that. The classic things do migrate providers the easiest of all.

I've just found in my experience maintaining a web server or a database server, keeping security in mind, upgrades, scaling, etc. is alot more work than simply spinning up RDS and a Lambda with API gateway. Or even hosting static sites on s3 or Netlify.

Like I said I don't know enough to say one is better.

Also, is your name Jason?

I think we may have worked together in the past. Lol

Honestly, I've developed the last few projects in Serverless framework and deployed to AWS Lambda. My biggest project is a custom Python application monitoring solution that has 4 separate Lambda function/handlers, populates an RDS PostgresSQL database with monitoring data (runs in a VPC), then reads from that database using complex joins across multiple application metric tables to send time-series data to Cloudwatch metrics.

Then, I configured Cloudwatch alarms to have thresholds for certain metrics that send a web hook to PagerDuty.

The benefit is that my monitoring system never goes down, never needs to be patched manually (AWS even patches my Postgres database, and fails over to the warm standby during patching), and never needs any system administration.

Have you ever worked at a company where you had a serious outage that you didn't detect quickly enough because a monitoring system was down? Having a Serverless monitoring system means this has happened 0 times despite our app running in production for almost a year now.

> In most cases, very few companies have products that need to scale to extreme load day 1 or even year 1.

That wouldn't be a great reason to choose serverless indeed. However, that doesn't mean serverless isn't still the right choice.

We've tried both the traditional approach you describe and serverless, and from experience the latter is 10x less infrastructure code than the former (we compared both Terraform implementations).

If serverless fits your use case, saving time and effort is a very good reason to go for it IMHO.

Reply on news.ycombinator.com