Compute Engine Predictive Autoscaling | Hacker News Reader

Compute Engine Predictive Autoscaling

cloud.google.com

53 pointstzury16 comments

Back when I was working at Massive Entertainment (part of Ubisoft) we actually implemented a predictive autoscaler on top of SaltStacks REST API -- in C++ of course for The Division 2[0]

This would have been quite handy back then, it seems to operate on almost exactly the same principles.

The hardest parts were the fact that it was a 10-20 minute endevour to create a new gameserver instance, and predictions get more difficult the further into the future they are. We didn't use any fancy AI stuff, just linear regression with smoothing across the history. We preferred to overshoot, but the predictions had a tendency to spike sometimes with relatively normal trends. You don't want to start creating a machine with a 20 minute lead time and abort it. It's a wasted hour of compute time. (and an hour is the minimum windows licensing cost).

Windows licensing was 50% of our costs on GCP.

[0]: https://cloud.google.com/customers/massive-entertainment

Sorry if this is a dumb question.

>a new gameserver instance.........Windows licensing was 50% of our costs on GCP.

Are there any reason why the gameServer part is on Windows? And if this is a norm in Games industry. I always assumed the Server part would be a Linux stack.

Everyone has the same question, so it's not a stupid one for sure.

I answered this a bit more in depth here: https://www.gcppodcast.com/post/episode-265-sharkmob-games-w...

but tl;dr: When you tell developers that they can't run VMs and new hardware takes 6-12 months and a lot of bureaucracy to acquire, you make it work on your machine.

And, of course, Gamedevs understand win32 API very well.

And, making high performance networked AsyncIO is a bit nicer on Windows (IOCP vs EPOLL).

I am glad Google is finally trying to catch up with the basic features of AWS infinidash. They have still got a lot to do!

What does it say about AWS branding that even after googling the term and reading the top three hits, I can't tell if "AWS infinidash" is a joke or a real product?

To prevent others from losing time googling, its a joke ;)

https://twitter.com/jna_sh/status/1410178986978775040?s=20

Did produce some quality comedy material!

=> https://twitter.com/TartanLlama/status/1410959645238308866?s...

In AWS, the equivalent is called Predictive Scaling:

https://docs.aws.amazon.com/autoscaling/ec2/userguide/ec2-au...

My experience with GCE's autoscaling has been that the up-scaling is much more aggressive than anticipated. I see with this predictive autoscaling, up-scaling would be even more aggressive.

And the downscaling in my experience takes too long. Sometimes it feels it's done this way on purpose to rake in a few extra hours of compute pennies. It starts adding up... imagine if you have a million customers and you can suddenly charge them an extra few dollars per month.

I have an app that has been running on the old Python 2.7 App Engine platform for a few years. It gets very light traffic and usually costs $1 or $2 a month, mainly for datastore read ops.

Suddenly this month there was an additional $11.65 charge for "frontend instances". I've searched the dashboard, documentation, and my configuration files trying to figure out what this means. I still have no idea where it came from or how to stop it, short of turning off the app.

If you search for "frontend" on the App Engine Standard Pricing Page [1], it seems like your usage might be exceeding the free tier for frontend instances. Did the load on your app suddenly increase?

[1] https://cloud.google.com/appengine/pricing

Yeah, that makes sense. A load increase that tripped the autoscaler to add a second instance that put me over free limits for the first time.

I think I figured out how to stop it from doing this: https://old.reddit.com/r/AppEngine/comments/od11sg

Thanks for the link.

At least with AWS, auto scaling is biased towards aggressive scale-outs and conservative scale-ins, under the general principle that customers would prefer to err on the side of providing service and meeting their SLAs over conserving every last dollar of spend.

It makes sense that GCE would adopt the same attitude.

Does it monitor slashdot to anticipate the slashdot effect?

scalling by adding CPU Cores and RAM would also be useful for certain workloads, I did not see this option.

Maybe that is step 2. I agree that this would be hugely beneficial e.g. for databases where a low number of very fat replicas is preferable.

Reply on news.ycombinator.com