How Snowflake Rebuilt Its AWS Stack on Azure | Hacker News Reader

How Snowflake Rebuilt Its AWS Stack on Azure

snowflake.com

46 pointsthe_child27 comments

>We couldn’t have built this without close coordination with Microsoft. The Snowflake on Azure project included many long phone sessions and meetings in Redmond conference rooms. Several new features were built with Snowflake in mind, including Azure storage soft delete and improvements to virtual machine provisioning. A big thank you goes out to the Azure team for helping us deliver together.

Good to know that you need long phone conversations and meetings and Microsoft to roll out special features to get yourself onto Azure. I'm sure they didn't intend it to be this way but not a good look for Microsoft.

Yeah...that jumped out at me. We're not a huge Azure spender--somewhere around $20K/month, though it's hard to tell--and it is a frustrating place to be coming from both AWS and Google.

I probably would have a much better time of it if I even had a TAM to talk to instead of a CSP (and that's a baked-in problem on our end, but Azure looks and feels worse every day when you have to go through a CSP--Microsoft would do well to jettison the whole thing, it makes their product worse) and I'd have a great time of it if I had an Azure product team invested in fixing the problems I have. Which are legion. Mostly around the bad APIs and stuff like "oh yeah, the AD connector for Ruby is broken and has been for a year, just monkey-patch it like so"...but, funnily enough, also around security, which the blog post kind of just waves off as "oh but you can encrypt". Yeah, sure, they have a KMS, but IAM is a constant and complete tire fire, so even though you can encrypt your data, who knows if it's correctly protected?

MS doesn't really have a role that focuses on the things you're talking about (though I'd really hoped the big DevAdvocate push would include "advocating on behlaf of devs" but doesn't seem that way) and they pretty much implicitly don't care about rewarding or even thanking the people that go out of their way to try to address those things or bring attention to them.

I'll just say it hurts, personally hurts, to be in Azure, bring up feedback, passionately advcate for it for the users sake, be ignored, and watch as the product is shipped ("we have as schedule!"). In at least one instance, they went so far as to basically discourage public use of the feature until theyd redone it exactly the way I'd suggested 18 months earlier. So much of it was just a result of having no idea what users wanted, where cloud computing was headed in terms of containerization and automation, and fundamental open ignorance of how and why GCE/AWS had designed similar features.

Sounds about right; your description matches what I've heard from other people and it's one of the reasons why, last time I was contacted by a recruiter for an Azure team I said no. The products are unfocused and it really feels like there's simultaneously a lack of vision and a feeling of undeserved excellence. (One of my coworkers came to us from Microsoft, and he mentioned that the Ibiza team thinks that they made a good product...)

The inability for the various services to actually work together is befuddling too. We had a meeting the other day to sketch out some stuff about our environment and, being way more familiar with AWS and GCP, I started by sketching out a network only to have it pointed out to me that app services aren't even in private virtual networks in the year of our lord 2018. And I thought AWS was dysfunctional!

Maybe if they actually wanted product vision I'd listen to a job there, but that seems at odds with what Azure is: "we have a cloud too, don't you want more Windows in yours?". OTOH, I want a cloud offering that works like AWS and GCP because that's minimally surprising.

Isn't that to be expected? Not every cloud provider will have every feature that the others do, it seems likely that if they did a migration in the reverse direction (from Azure to AWS), they'd have had to have the same series of meetings.

Cloud Computing is the new RDBMS, you are tightly coupled to your provider

You can take your RDBMS to any server you want.

You can migrate your RDBMS to any other RDBMS you want.

This is like saying using an ORM locks you into using that ORM.

I think they meant that changing RDBMS is hard, even though in theory you should be able to because you're "just using standardised SQL". As it is with cloud providers, in theory they're mostly the same, except in all the places they aren't, which make it hard to move once you're using one.

Just like you can export your data from any cloud database provider and move it anywhere you like.

That migration might be painful if you have a lot of data, but so is an RDBMS migration. You might need to change your code to support the new vendor's query language, but the same is true when moving to a new RDBMS, since there are usually implementation specific extensions that you'll need to port.

Migrating a hot database is non-trivial in many cases. Even same database to a different cloud is not really trivial without downtime. Cross-cloud replicas are difficult, and if you do happen to use a managed DB you may have limited replication options.

I don't get this? What would you prefer, that MS azure is not willing/able/available to those who would want to build something on their platform?

Why would you expect them not to be engaged?

If the idea is that you would prefer them to not be required in order to accomplish such a deployment, then maybe azures tech/offering matured such that the next snowflake like company will not need such close support.

Your comment makes no sense to me.

Sounds like they'd prefer that the Azure platform is sufficiently functional/comprehensive already so you don't need to ask MS to fix/build things you need.

My fear is Azure will build solutions specific for a customer which might never be used by other customers. So that solution is a special case in a homogeneous cloud platform with all its own problems.

For example, I feel I can trust S3 because of its simplicity and the sheer amount of systems depending on the same simple functionality of S3 I am using. If I would get a special feature in S3 just for my use case, there is no guarantee others will use that feature and it will be as battle hardened as the generic features.

Yeah how about accurate documentation, properly followed standards (just lol at Azure AD’s version of “OIDC”) and sensible architecture so that one doesn’t need so much hand holding?

Our experience of Azure has been nothing but delays, misery, vendor lock in, expensive consultancy fees and bewilderment. We started our move to AWS last year and things are 100x better.

The point is that most of us won't get premium Azure support.

This happens with all the vendors. AWS, Google, and other cloud providers as well.

> I'm sure they didn't intend it to be this way but not a good look for Microsoft.

Yes, it's a bad look that they work with and consider their customers needs and help them achieve what they want to. So bad.

Audio ads popping open some helpful cartoon character means I hit the back button and never return.

Three! Disclaimers to hide at bottom of page on mobile! That’s 3 taps on the X! Jeezus.

Does anyone have experience working with Snowflake? I feel like what their promising is too good to be true and their website is extremely lacking in substantive details.

We've been using it for a couple years and pretty happy. Every once in a while there is a bug that takes a while to fix, but really it's been pretty great compared to what we were using before.

How does it compare to bigquery/redshift (other than multi cloud thing).

From my experience, it was pretty speedy even on some (relatively) large datasets. The decoupling of compute and storage makes it appealing compared to Redshift if you have bursty load on your cluster.

It was also cheaper for us than Redshift, of course your mileage may vary.

Used it for several years, beat the pants off Redshift and was comparable to BigQuery for our use cases. They have a ways to go on the maturity/reliability front though.

Did an evaluation, works as advertised.

If you need a HIPAA BAA, talk to them before the evaluation to avoid misunderstandings.

This is a lot of upfront work, and signing up for continuous maintenance, computing costs, dealing with vendor-specific oddities, and a host of other concerns. Whatever is gained from this (being able to sign AWS phobic clients) must be substantial.

There are some large companies that don't want to run on Amazon, so they are probably trying to court that business... but I'm surprised they went with Azure instead of Google.

I wonder if they are trying to get Walmart?

https://www.foxbusiness.com/features/wal-mart-to-vendors-get...

EDIT

I don't have to speculate, that's definitely why they went to Azure:

Retail clients have pushed Snowflake Computing to make its software available on Microsoft’s (NASDAQ:MSFT) Azure cloud.

Snowflake CEO Bob Muglia (a Microsoft vet) says that a number of retail customers “particularly a large one based in Arkansas” have “a fairly strong opinion” about the Azure availability.

The Arkansas reference is likely to Walmart, which prefers not to use competitor Amazon’s cloud.

https://seekingalpha.com/news/3369703-cloud-data-warehouse-c...

Reply on news.ycombinator.com