Hacker News Reader

Comment by vs2 | Hacker News Reader

"Due to the inelastic architecture of our AWS system, we needed to have the systems scaled up to handle our peak traffic at 10PM when the daily puzzle is published."

WT... I had to reread this to make sure I didnt misunderstand... why not work on making the current arhictecture elastic?! #cloudPorn

View on HN

The "inelastic" might have been a shot at AWS. When pressed, the AWS people do use phrases like "pre-warming", "over provisioning" and "advance notice" around their ELB/ALB setup and ECS.

Google's cloud salespeople pitch that they don't require any of that.

Curious if the need for pre-warming ELB/ALB still applies. Last time this came up, an AWS employee mentioned it is no longer necessary (https://news.ycombinator.com/item?id=14052079), but would be nice if this was documented.

Possible. It is still in the documentation: https://aws.amazon.com/articles/1636185810492479#pre-warming

The "advance notice" and "over provision" advice is still being given for things that could scale up fairly large. (where fairly large isn't anything that exciting, really)

I don't want to dox myself but about a year ago when my employer forgot to notify AWS about switching our production traffic (about 5K rps at that time) from one ELB to another we failed requests for several minutes before we decided to just switch back to the old ELB then ask them to do a prewarming before we switched again.

It definitely still applies.

The poster was talking about ALBs. I'm not experienced enough with them to know whether the claim is true.

I do know on ELBs though, pre-warming is essential for high throughput.

It's funny they'd mention that, and then later in the same article say they had to use a Cron job to scale up their gcp solution for a daily spike.

Using a cron to scale an AE service to 500 instances 5 minutes before 10PM > bugging the infra team to bring up our 19th MySQL replica.

I think it depends on the nature of your spike.

AppEngine instances can typically start in 30 seconds or so. So if your spike is because your video went viral on facebook and lots of people are looking it it, that's fine.

If your spike is because you have 10 million clients with an app set to do an HTTP request at exactly 10:00:00pm, and they all arrive within a quarter second, thats a problem.

30 seconds is actually a gross overestimate for AE startup. Your description of our problem is very accurate, though. We still need to over-provision just before the spike, which we do with a cron that scales up to several hundred instances 5 minutes before 10pm and then back down to normal levels 5 minutes after. Autoscaling takes care of the rest.

Imo, one really killer bit is the first piece they mentioned:

"Google provides an SDK that enables users to run a suite of services along with an admin interface, database and caching layer with a single command."

I really wish AWS had a decent local dev story, rather than relying on 10 separate half-baked OSS solutions

Have you tried the documentation for google's sdk? I stopped in disgust trying to track down what I needed and then trying to manage through multiple different admin interfaces.

I've been using google bits recently. I agree that the docs for e.g. all the python sdks need a lot of work.

That said, boto3 has thorough docs, but I wouldn't consider them particularly well organized. I can only really navigate AWS sdk docs because I already know what I want to do, and can google the specific terminology

Are you talking about this Python SDK doc? https://googlecloudplatform.github.io/google-cloud-python/la...

Yeah. In particular, I was really looking for something for "here's how to do most of the basics for Cloud Storage".

If you look at https://googlecloudplatform.github.io/google-cloud-python/la..., for example, there's not even a top level navigation index that I can read through to guess what function I might need by name.

(Google cloud support here) The pages you linked to are supposed to serve as client libraries reference only. If you want higher level instructions and examples, always start with the main Google Cloud docs first. On any page that offers instructions, the top of the code windows offers a selection of client library languages/CLI tools/REST API available to do whatever that task is. For cloud storage, start here:

https://cloud.google.com/storage/docs/how-to

Choose a topic, and select "Python" at the top. It should provide instructions and examples using the Python libraries.

Also, we have a repo of demo projects and examples for nearly every GCP product/service, and then some. Great examples to be found here (some might be out of date though):

https://github.com/GoogleCloudPlatform/python-docs-samples

Heya,

Thanks for the link! You're right that this is what I was looking for. Unfortunately, that hadn't shown up in a convenient place while I was googling around. Would be good to add direct links to those from the client lib references, because those pop up for e.g. "google storage python" first. (Unless they're already there and I didn't see them).

Links from the READMEs of the repo might be useful too to help people get there? https://github.com/GoogleCloudPlatform/google-cloud-python

That's not a bad idea, I'll bring it up with our client library maintainers today.

Of course not :)

It's up there right at the top of the README:

https://github.com/GoogleCloudPlatform/google-cloud-python

The main story here is probably this line: "The system is generally at that peak traffic for only a few minutes a day".

AWS charges by hour. Google by minute. If your peak traffic only last a few minutes, hourly billing is inflexible, simple as that.

AWS lambda charges by the 1/10th of a GB/CPU/second.

I'd bet they could have seen significant cost savings on AWS by migrating to lambda, and gotten continuous scaling to boot.

Serverless is probably far too immature for companies like NYT.

I don't disagree with you, but it's also true that you don't have to go all-in. I run services that run server-ful-ly on EC2, while requests to the "front-end" are handled by Lambda. If you've got one or two very hot endpoints and the majority of your server load spike is thanks to the processing for those endpoints, moving just those endpoints to Lambda could give you reassurance that you don't have all of your eggs in one basket (not having to go full-serverless) while also getting the benefit of being able to scale up and down essentially effortlessly.

Of course, there are lots of other consideration that you should be making when going even partially serverless, and it could be that the NYT chose not to experiment with Lambda for other reasons. For instance, you're effectively tied to CloudWatch for logging and monitoring, which could be a deal-breaker. Much of the processing could be happening in the DB, which would make Lambda moot. It may simply be that their estimated usage of Lambda was too costly.

Three devs seems like a bit of a bottleneck... Why maintain two separate architectures for a single problem if there's another option?

Hourly billing is also equally inflexible if you need 61 minutes, or 74, or 110- paying 120 for any of those isn't too rad.

(Disclosure: I work on GCP)

I'm not sure why people are so enamored with ELB -- just terminate SSL at your web boxes using nginx and publish the public ips of all of these machines in your DNS records.

You remove a bunch of ELB per-request costs doing it this way and you can scale it however you see fit.

That works great until a machine starts failing health checks and you need to take it out of rotation ASAP. It's also the case that DNS gets cached and not all users create the same load: one user making many requests will burden one server instead of having the load evenly distributed.

Clients will try another ip if they can't connect -- partial failures may be a problem, but in my experience as long as nginx is alive, you can load balance to a different web backend if the app processes on that machine are wedged.

I've deployed this solution in 50k req/s environments and not seen a single user be a problem like you mention -- any motivated bad actor could cause problems in either scenario I expect.

Clients might not fail to connect. That's even worse. They connect and the server hangs and returns no response (perhaps due to bad configuration changes). Now you're stuck, and your server is too oversaturated to SSH in and fix it.

It out depends on your application and users. Building a website? Probably not much of an issue. Building a low-latency API? YMMV. Keeping your load evenly balanced across your front end cluster also can keep your cost low, since you are able to distribute load more evenly.

That's another point: if you scale your cluster size up and down frequently to accommodate load, doing that with DNS is a nightmare.

> It's also the case that DNS gets cached and not all users create the same load

It's also, also the case that DNS gets cached and propagating the removal of a broken server could take ages.

Reply on news.ycombinator.com