nosefrog

Yup, the hardest part about migrating to Azure was jumping between all the managed services that work everywhere else but are insanely buggy on Azure. We ended up with the most basic architecture you can imagine (other than AKS, which works great as long as you don't use any plugins) and we're still running into issues.

We have a very long list of Azure features and services that we've banned people from using.

Just got off a call with someone at Azure today who told us to setup our own NAT gateway instead of using Azure's because of an outage where we made too many requests and then got our NAT Gateway quota taken away for the next 2 hours.

Are there any good alternatives to gunicorn out there that don't require async? We're not ready to migrate everything to async, but gunicorn's forking model is blowing up on macOS 15.

Very cool! Finding the Kubernetes API docs is such a pain.

Wasn't it always the same tech as app engine? I was there when they first released it.

That's why I said it's an unpopular opinion :P

I worked on the largest gcl deployment at Google (Union). We had multiple outages caused by our config system.

I agree, it's better than helm templating. But it's not better than python!

I got burned so bad by config languages at Google (specifically gcl) that we're generating Kubernetes yamls using the Kubernetes python client now.

My unpopular opinion is that config languages with logic are bad because they're usually very hard to debug. The best config language I've used is Bazel's starlark, which is just python with some additional restrictions.

Same.

Glad these are still happening :)

Nope.

Strange article. Saying "99.99% uptime maintained" and that they had 4 major outages in a week is kind of strange, since 99.99% uptime only allow for 4 minutes of downtime a month...

We moved everything to Azure. Still mad about it because things in general don't work on Azure lol. Especially networking features. Avoid any feature that sounds even a little complicated to develop, because it probably doesn't work. Even basics like the CDN have showstopping bugs that Azure blames on Chrome for some reason (update an Azure blob and access it via a cdn, it will fail ~1% of the time).

I hope the DOJ investigates Azure next, because I can't believe such a garbage platform would get to 2nd place without an abuse of Microsoft's monopoly power. Specifically, using their office product monopoly to create a cloud monopoly by signing 365 discounts with customers that require all cloud services used by those customers to be on Azure.

Whoa, didn’t realize I need this. I’m constantly rebasing stacked branches.

Anyone who has read LangChain's code would know better than to depend on it.

To be fair, I worked on multiple projects removing queues at Google, so it's more than just that.

You prefer Azure? Things on Azure don't work by default.

Oh, I didn't realize that the design also has a database of revocations. In that case, you can just query that directly :P

Most EVs can make the trip today? The most popular EV, the Model Y, has a stated range of 320 miles.

And hope your service hasn't been restarted so it doesn't lose the in-memory revocation list?

Sure, how many 9's of availability does it have again?

Thanks, chatgpt.

Who argues that?

This is crazy. Amazing work!

Is a fair queue worth it vs spinning up more capacity? I've worked on multiple projects where we've ended up ripping out a queue and just spinning up more machines to handle the load synchronously instead.

How did you avoid version hell? At Google, almost everything just shipped from master (except for some things that had more subtle bugs, those did their work on a dev branch and merged into master after testing).

> In addition, many applications within Google have switched to more database-like storage systems such as BigTable and Spanner.

And where do BigTable and Spanner store their data? ;)

Coming from Google, where Spanner is this magical technology that supports infinite horizontal sharding with transactions and has become the standard storage engine for everything at Google (almost every project not using Spanner was moving to Spanner), I'm curious how Figma evaluated Cloud Spanner. Cloud Spanner does have a postgres translation layer, though I don't know how well it works.

It seems like they've (hopefully only temporarily) given up real transactional support with their horizontal postgres scheme?

Eh, I loved my time at Google (left in late 2022 for a startup). Amazing coworkers, amazing tech, and you get to work at a mind-boggling scale.

We had seccomp containers at Dropbox, and I remember Max Serrano helping me set that up with ReactServer :) Talented engineer, though I do remember that the jails were kind of a maintenance nightmare for the security team.