AWS is not a dumb pipe | Hacker News Reader

AWS is not a dumb pipe

matt-rickard.com

116 pointsantigizmo87 comments

But it can be a dumb pipe. More and more companies prefer that.

Just deploy everything on EKS and use only foundational services such as S3 and PostgreSQL Aurora. There you go: Dumb Pipes.

Edit: When I say dumb, it's more like: Dumb enough so that another public cloud vendor, such as GCP or Azure, can be swapped in.

Except you’ve picked out two of the most complex and deep services as “foundational”.

Yeah, I would be really careful about relying on something like Aurora and planning to migrate to Cloud SQL or another vendor's managed PostgreSQL. Apps couple very tightly to implementation quirks of their database, and even the tiniest difference between implementations is going to cause problems when you migrate. (For example, we used to test our Postgres app with SQLite. We found that each query engine treats "WHERE foo = 't'" and "WHERE foo = true" differently, though an exciting production outage. Postgres does what you'd expect, SQLite does not. That was the last time I'll ever use a different database for production and testing.)

If you are going to be cloud provider independent, then you have to bring your own foundational services that run independently of the cloud provider. Minio instead of S3, CockroachDB instead of managed Postgres, etc. (Honestly, I would feel safe switching between plain Postgres on RDS and some other provider's equivalent. Don't use any extensions, though. S3 I would probably try and get away with as well, since everything provides an S3 compatible API these days. But definitely not Aurora.)

You also need to think about your other cloud provider integrations. Do you programatically change DNS? You'll need something that can handle any provider. IAM? You'll need something that handles both providers. Even at the Kubernetes level, things work differently. Compare Google's L7 load balancer to Amazon's ALB, for example. You might create them both with "kubectl create service loadbalancer ...", but the semantics are going to be different. Persistent volume types also vary between cloud providers; your StorageClass "default" is going to have different IOPS characteristics, for example, and that can easily burn you.

I am willing to bet that anyone using the cloud for something serious will have some hiccups migrating away, even if they're using Kubernetes. It's all the stuff that's not in Kubernetes that will get you!

I hate this practice that some companies do of testing their sql database usage... By using another database. For postgres, if you are using the jvm, you can use the amazing test containers library to test reasonably fast, with a postgres database that you can set up to be the same version as you use in production.

Any database vendor should either support having their database used quickly in test like these, or at least release a simplified in memory version of their database so people can test their code. If your database is not testable, that should be a black mark against choosing it for a project.

Which library are you referring to?

I'm new to jvm programming, and was wondering about this as I'm used to rails where the test db is spun up on your behalf and is usually the same db

Much better than mocking out cursor objects!

For PG you can just initdb in a tmpfs.

Or (and this can be done with pretty much everything) spin up a container

With elixir and ecto it wraps the whole thing in a transaction so it runs Async across multiple tests.

Running your own Kubernetes cluster is complex and deep. Giving a yaml file to EKS and suddenly you have infrastructure is not complex at all. Its actually pretty sweet.

You are confusing simplicity of usage with simplicity of implementation. "dumb pipes" implement simple services for fixed or low per unit costs, the entire appeal of ready to use complex stacks like EKS is simplicity of usage.

It’s dumb in the sense that you can just move clouds without issue. You’re not using the secret special sauce that binds you tightly to AWS, everyone has a K8s and they’re all broadly the same.

Huh? Yes Aurora is highly proprietary, but S3 is a commodity in many use cases.

Commodity in usage, yes, but not on the service side. Doing S3 properly is really hard, having a service with a 'compatible API' doesn't make it the same as S3. Same goes for EKS (Yes, it looks and smells like a normal API server) and RDS.

If you never use anything that ties this stuff together (like IAM), then there is practically no point in using AWS at all, it's way too expensive to use as a 'dumb pipe'.

That said, I'm personally a fan of using things that would technically also work outside of AWS, but not shying away from the AWS-specific optimisations. I'm not going to use Kubernetes on AWS without EKS for example (plain EC2). Or Kubernetes on AWS without the Amazon's CNI, that's not a thing I'd enjoy in production either.

When managed Kafka was announced, that was another great example of "nice, now I don't have to mess with EC2 node contents anymore", but I still can if I want to, including outside of AWS.

The most AWS-native thing (or two things) that make most other offerings not work out all that great is the integrated IAM and Security Groups. Kludging those together with some crappy port to AD or NSX is just a waste of time and sanity.

Doing EB-scale s3 is really hard, throwing a few hundred terrabytes (or even couple petabytes) into s3 compatible storage on top of k8s is not that hard (getting easier every day). There’s numerous options available today even with enterprise support

> The most AWS-native thing (or two things) that make most other offerings not work out all that great is the integrated IAM and Security Groups

SGs are easily covered by kubernetes network policies. What do you need iam for if you’re not using any manager stuff? Sounds like circular argument to me (“you need managed bc you need iam bc you need managed…)

You need SGs because not everything in the world lives inside a K8S cluster or flows through a CNI. And you need IAM because if you have 10 users and 10 resources but not every user is allowed to perform all actions on all resources you need policies. Policies apply (generally) on roles, actions and resources. IAM that only works on some random resources but not on others are pointless, unless you have a very small setup (i.e. less than 100 resources, less than a small office of users), but at such a small scale, you shouldn't even be thinking about AWS (unless you are on track to scale 1000x).

S3 isn't just "I have many files and it needs a lot of disk space". Like I wrote, if you just use base S3 API primitives then no, you're not really using anything AWS-specific, and making a comparison between something that does more than you need seems odd to me. Edge storage, cold storage, cheaper storage based on reductions of redundancy or access times, mirroring, versioning, multi-level IAM (integrated with the rest of your IAM) are all things that make the difference between "look at my virtual USB disk" (most MinIO configurations out there, the unsupported ones that is) and "durable and reliable multi-level object storage, configured for my needs".

I’m sure you can find a feature that minio/ceph/swift don’t support out of the box just as i can probably find something that’s very easy to do do/add in ceph/seaweedfs but will be on the back burner of your cloud tam for a few quarters/years or do it at a cost that can’t be beat by any volume commits. At the end of the day we are payed to solve business problems with set constraints so statements like “you need s3 bc everything else is hard” are kind of myopic

Unless you are in the business of reinventing wheels and building knowledge silos that will never transfer, none of what you wrote applies in general commercial business.

If you never use anything that ties this stuff together (like IAM), then there is practically no point in using AWS at all, it's way too expensive to use as a 'dumb pipe'.

Must cheap providers have crappy networks compared to AWS.

That's true. And sometimes you get an odd mix of good network quality but crappy features, or the other way around and that isn't amazing to work with either.

Honestly, it should be.

AWS is much better at the dumb, commodity services like s3, ec2 and rds (that are actually pretty smart) than they are at something like redshift, sagemaker or similar.

Some of their more complex/bespoke services are very good, like Lambda, CloudWatch, and DynamoDB.

Quality declines steeply once you're off the beaten path.

Define “good”? They are pretty reliable and scalable (if you exclude ebs from this mix) but have considerable cost (especially on networking side)

They're "good enough" for most users that might not care about super high availability, devex etc. As long as its cheap and solves a business problem, it will win.

AWS would be a far superior product if it trended toward being a dumb pipe.

The "all the things" approach that AWS has taken has led to a lot of great ideas being poorly executed, leading to a mess and poor engineering.

E.g., the new (V3) of the JavaScript SDK is such a clusterf*ck I almost can't believe they had the stones to ship it.

SDK design problems not unique to AWS. Companies that startup dedicated language SDK teams eventually find those teams needing to justify their existence so you end with needless new versions. Some better than others

Like a sibling comment I’d also like to know what was bad about the SDK. It seems.. alright? That alone is good enough for me considering its implementation is autogenerated from something else.

It also has tree-shakeable imports which v2 doesn’t have, and a very extensible looking middleware architecture which I’ve yet to need but good to know that it’s there if I do. The auto-generated docs leave something to be desired, but are passable, with a few extra clicks here and there.

Wow... I had not touched the AWS SDK with JavaScript in a long while. This is actually very surprising. It looks significantly different for what seems like a marginal improvement from V2.

> AWS would be a far superior product if it trended toward being a dumb pipe.

"Superior" by which measure? AWS is making money hand over fist. Companies use AWS because they think it's better than alternatives.

What’s not working for you with the V3 JS SDK?

A few irks:

- Auto-generated Typescript documentation that's unintelligible (Lord Jesus help me).

- Weird naming of imported functions (e.g., appending "Command" to the end of imports).

- The need to create instances of requests (commands) and then "send" them to the API vs. just passing an object of options. The irk here is that it breaks the affordances around API design.

For example, using S3 used to look like...

---

import AWS from 'aws-sdk';

const s3 = new AWS.S3({ ... });

s3.putObject({

  Bucket: 'some-bucket',

  Body: 'data buffer',

  Key: 'file-name',

}).promise().then((response) => { // Handle URL here. });

---

Now, it looks like...

import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";

const client = new S3Client(config);

const command = new PutObjectCommand(input);

const response = await client.send(command);

---

This example is taken straight from docs (https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clien...). config and input are not described anywhere near this example, meaning you have to go hunt for them.

---

The gold standard I reference for API design is Stripe. They have just as complex an API as anyone, and yet, they don't go out of their way to overcomplicate things. For example, this is stupid simple and obvious:

const stripe = require('stripe')('sk_test_4eC39HqLyjWDarjtT1zdp7dc');

// `source` is obtained with Stripe.js; see https://stripe.com/docs/payments/accept-a-payment-charges#we...

const charge = await stripe.charges.create({ amount: 2000, currency: 'usd', source: 'tok_amex', description: 'My First Test Charge (created for API docs)', });

---

Not only is the API descriptive of what it's doing by design, they had the wherewithal to explain where the missing piece can be found. Contrast that with Amazon where they give you a half-assed example and you have to go on a goose chase to find the params/options for the request. Why?!

The above changes look like a wank or job protection, not any form of well-considered improvement. Couple that with turning the docs into a maze (as opposed to just saying "here's the function, here are the required/optional params, this is what a response looks like) and you realize that the person responsible for this just greenlit it without any question as to "why are we doing this?"

Yes, I'm being an asshole about it. But this is now going to inform API design for other people as it's representative of "what the big guys do." Rinse and repeat that thinking for a decade or two and everything turns into a f*cking Rube Goldberg machine for absolutely zero reason.

False dichotomy -- AWS is not only a dumb pipe

The value I see in AWS isn't that anything is done particularly well. It's that there's enough of it that I can get most all of what I need in one place.

The value of curation becomes greater when the supply is there. What comes after curation, I wonder?

I recently experienced this with setapp. It’s a $12/mo subscription that offers me a couple hundred apps and utilities to choose from. When I needed more insight into Wi-Fi signal issues, I just went with their rec. same for a pomodoro timer, or clipboard enhancer. Pull up their “App Store” on my Mac, type Wi-Fi into the search, and use what they offer. So nice.

It has become the Kaiser Permanente of the Internet

AWS isn't a single "thing", or perhaps, it's only a bundled brand. There are many teams, sub-companies, divisions, groups all working on their own parts. Most have to follow a similar set of rules and some attempts are made to make it cohesive, but in the end it's not a single minded entity with a single set of parameters it follows for every service offering.

You can run your stuff on AWS without ever touching compute, storage and networking yourself, and you can also run your stuff on AWS by only touching compute, storage and networking and nothing else. You can guess your risk wrong and do everything as a root user in a single account, and you can create an account vending machine in a well-architected organisation with policies in place to prevent the huge amount of footguns lying around everywhere from going off.

There are so many ways you could be using it that trying to put a single label on it seems pointless to me. Just like the only constant you can count on is 'change', the only 'proper' usage or terminology in terms of AWS is what you know, what you need, and what you ended up doing. This includes shooting yourself in the foot with CloudFormation because you accidentally thought CDKs are your friend and JavaScript was the best tasting language for your goal, or thinking that clicking around in the console would be a good way to get your department of developers up and running.

Ok, but by that logic it would be impossible to make any kind of analysis about AWS (or other cloud providers) in the sense that the author attempted to make.

In the end I believe, the question is about lock-in and control. In a true "dumb pipe" offering, you could just take your software elsewhere and run essentially the same code without changes. Meanwhile, if you use various proprietary APIs, this is not the case. I'd argue, even if the APIs have open source implementations that you could theoretically use elsewhere, it's not clear how viable this would be in practice.

I think AWS does not become a "dumb pipe" only because you could theoretically use it like that - at least not if doing so would require you to go "against the grain" of the platform the whole time and would force you to only use a fraction of what you paid for.

You definitely can, but I'm not sure what the point would be. Say you are a commercial entity and you need to not run everything vertically integrated. You go look for external entities that can do some of the work for you, preferably specialised in such a way that it efficiently delivers whatever you need. At that point, this whole 'dumb pipe' concept is just a column in a crappy spreadsheet somewhere that might make a minor impact on the choice being made. Spending 10 million extra each month because an engineer liked the sound of a dumb pipe better with a vague promise of easier migrations isn't realistic.

If someone truly wants to not be dependent, you need to identify to what degree you want that, and then end up paying a lot more to get there, usually ending up multi-could. Even three 'dumb pipes' aren't similar enough to create a 'write once' IaC and application definition. The closest you can get is not using FaaS-type offerings and sticking to OCI containers. You'll still have to work with the IAM and network primitives each vendor requires, and even if you don't run active-active you'll still have to write every deployment to every provider you use to ensure your systems are always ready to deliver.

It's not impossible, but impractical and expensive. For most companies, the ROI just isn't there. Offloading more day-to-day and non-company-specific work is the way, and dumb pipes can't do that.

I think "lock-in" is a terrible term.

You can avoid all lock-in by ignoring all sorts of proprietary features of any vendor, and leave ENDLESS benefits and optimizations on the table for the hypothetical likelyhood that you might one day have to "free yourself".

Conversely, if you embrace those things wholeheartedly, and at some point find yourself wanting to break the relationship and do a full egress migration, even if your costs end up monumental, I would bet anything that over the time you were "locked in" you will have gained more from that cloud than whatever "one-time migration costs" you have to pay.

If your core business proposition is literally something like storage (e.g. Dropbox) where with time you will want to vertically integrate your whole technology stack rather than relying on a 3rd party, yeah avoid lock-in.

If you have a frivolous C-suite that will want to change cloud providers every 24 months depending on what discount they can negotiate on the golf course, yeah avoid lock-in.

Everyone else? Throw that word out from your vocabulary, and embrace the idiomatic capabilities of your cloud vendor.

I'd expand on that a little: 'lock-in' is not always what people think it is. Having all your 'stuff' in a binary database of an unknown format that only the vendor can read/write would be a lock-in, or if it's a known format but you don't have access to it.

If you have access to all the details of your systems you can always 'go elsewhere' but it becomes a matter of cost, knowledge and time. The balance is how much you want to spend on those, and what it gives you in return.

As an anecdote: I and the teams I manage avoid CDKs like the plague, they end up a version of IaC self-lock-in that makes it nearly impossible to integrate between vendors. Using tools like Terraform or perhaps Ansible or SaltStack (or Idem) still means you write your abstractions against a specific provider, but the code and data formats are standard enough that you can retain and re-use your knowledge and skills and depending on how much layering you apply you can reasonably easily migrate wherever you want to (from an infra perspective). Contents (unique data, that is) are always a different issue, even with full access you'll be thinking about egress cost. All of those issues are of course mostly annoying at scale, but that's also one of the reasons you'd go to AWS: for scaling options.

> When there's no product differentiation, distribution wins. In the telco case, products build on top of broadband were 10x improvements over products built into broadband. YouTube, Netflix and other internet content providers could do things that cable and telephone simply couldn't. Now, competitors must differentiate on expertise, community, and developer experience.

I don’t disagree, but I think the important thing to understand is the flip side of this argument: products built on top of broadband were 10X better… and that created demand for the dumb pipe. If you wanted these new products, you had to order broadband. And then cable and telcos had local monopolies/oligopolies, so there was no need to differentiate themselves. Others created the demand and they locked customers into renewing contracts. Why be anything other than a dumb pipe in such a scenario?

I think this over-estimates the intelligence of cloud businesses, and under-estimates the motives of paying customers.

First there's the assumption that Amazon, Microsoft, Google, "get software". This doesn't really mean anything. So you're a tech company and you know how to churn out software; so what? That doesn't mean your particular tech is better than anybody else's. Amazon alone has 70,000 engineers; they didn't snag all the good ones. If you look at their code (or try to package it for a Linux distribution), it's a hot mess. Their tech is not why they're successful. They know that software is just a tool you use to sell.

These companies' major wins were for all kinds of reasons other than their technical acumen: spinning off a new business from an already-profitable one, acquiring and integrating other businesses, or keeping a moat (sorry - "platform") around a particular market or customer base. It doesn't matter if they "get software". Just look at Google Cloud; it'll probably be sunset in 2 years. Who cares if they "get software" if they suck at acquiring customers? Apple is launching their own public cloud soon, as well as a couple other small and large companies. Will they be successful? It won't be because of their technology (they're all using the same code anyway). If you want to steal customers from Amazon, you need to sell it.

Paying customers don't care if you can see the future or are "internet native". They care if you can lower their costs and get them to market quicker, if you provide stellar support, if your product is more popular (more people know how to built with it/maintain it), if you make their life easier, and if your service appears more reliable.

And AWS knows all this. They're a savvy salesman, a reliable contractor, and an industry standard. They know how to keep and grow their business, and it isn't by reducing what they sell or focusing on a narrow market. If you're the incumbent, you sell more integrated services and diversify your business, and keep pushing until a market implodes. Then you tourniquet that part of the business and pivot. But what they'll never do is strategically refocus on some smaller business sector if it means making less money.

> On the other hand, that means that services built over-the-top pay a higher cloud tax. Not sure how this one plays out.

It’s unlikely services will continue to build on top and pay a high cloud tax for two reasons:

1. It’s not safe to bet on any entity remaining at the top forever, applies to corporations being on the top of their game too.

2. Tech tends to be an upward moving pendulum as it converges on an optimal solution. Back in the day, companies owned most of their stack from top down (initial position of pendulum) and now companies own only the top part of their stack by building on top of Azure/AWS/GCP (pendulum swing). There’ll be a pendulum swing back but it’ll be an improvement over the status quo for at least some use cases.

It’s likely new technologies and regulations will make building/owning deep vertical stacks a viable option and it’ll be worth it for some use cases. For example, visual/audio stacks should be deep and owned by an entity, I suspect there’s a large margin of improvement in this area that can be attained with fine-tuned vertical stacks (e.g. https://tonari.no/ is doing some low level stuff to build a better audio/visual experience in meetings). I think companies like https://oxide.computer/ will inadvertently, collectively help make building deep vertical stacks an option on the table.

A difference between telcos and IaaS is IT is 'baked into' AWS. AWS's IT team (people, software, automation) silently becomes an extension of your team. It makes AWS more than a dumb pipe, and makes it difficult to leave AWS (generally you have to be very purposeful to avoid this lock-in).

Interestingly, considering the OP's comparison, many enterprises are also tethered to AWS via telco dumb pipes (e.g. App user -> WAN -> Private DC -> Colo facility -> Direct Connect (MPLS) -> app in VPC).

We are back into the timesharing days, and newer generations are finally getting that cloud platforms have replaced UNIX with hypervisors, language runtimes and cloud vendor specific APIs/services .

Most businesses don't care about dumb pipes, at the end of the day how the accounting comes out is what matters.

So many wasted projects trying to make the whole stack technology neutral and then it never changes during its lifetime, with added maintenance complexity to ease migration to something else that never got used.

To their own failure. It's a dumb pipe with lots of bad microtransactions and traps added.

The best features of AWS are the dumb ones. EC2, ECS, EKS, S3, Lambda.

The truly horrible ones are ones like Athena etc. where you are better of hosting your own solution on the dumb features. No self-hosted data services fails queries because the query has taken too long, and charges you for the privilege.

Nothing is a dumb pipe on today's internet. Remember "the network interprets censorship as damage and routes around it"? Yeah, that went out like JNCO jeans. Today's network interprets information unpleasant to the surveillance-industrial complex as noise and filters it out. And the information it does allow, it is very interested in surveilling and price-tiering. Expecting anything to be a dumb pipe in today's world is a pipe dream.

AWS is the next Oracle

This is like saying "the market will crash". Anyone can predict that eventually the AWS value proposition will turn sour. What's valuable (and hard) is to predict _when_ this will happen.

Big, expensive, boondoggly, stranglehold on legacy workloads, friends with your C-suite, and engineers and ICs hate it?

What is the opposite?

Cloudflare, HashiCorp, Fly.io, Oxide Computer, Vercel

Render.com too

Heroku* and DigitalOcean too. Hell even Hostgator.

*Heroku is expensive though but is nice to use

And DigitalOcean now supports deploying Docker containers, so there is no reason not to migrate to it.

> so there is no reason not to migrate to it.

Sure, "no reason" if literally the only problem you have to solve is deploying docker containers. While I'm not taking anything away from DO -- it's actually not a bad alternative for people who don't want nor need the complexity of enterprise cloud solutions -- it's still a huuuge exaggeration to say "no reason" when Docker containers is just a drop in the ocean (pun intended) of the scale of problems that AWS (attempts to) solve.

Heroku runs on AWS, doesn’t it?

Heroku:

* Big NO

* expensive YES

* boondoggly NO

* stranglehold on legacy workloads NOT SURE

* friends with your C-suite NO

* engineers and ICs hate it NO

It hits many of the pain points, not all.

Heroku is non-leaky in it's use of AWS. If they switch to something else under the hood you wouldn't know and wont need to change code. The only leaky part is the pricing of someone who is buying wholesale and selling retail.

It's a series of tubes!! Not a big truck.

For those who might not know the reference: https://en.wikipedia.org/wiki/Series_of_tubes

"won many others. Kinesis vs. Kafka, DocumentDB vs. MongoDB, MemoryDB vs. Redis, OpenSearch vs. ElasticSearch."

Lol. Wrong on all of those.

I was just about to add a comment about this too.

Also saying they “missed the Snowflake opportunity” doesn’t make sense either: they have Redshift? They don’t need to acquire Snowflake, they’re already the incumbent in the field.

Author here. They missed the Snowflake opportunity by having the wrong architecture for Redshift (decoupled storage and compute). They shifted to the Snowflake model in 2019, but the damage might already be done. For other the other services I listed, the main differentiators are mainly plugins/extensibility and developer experience.

Isn’t this a bit of a moot point now though?

If you’re going to pay someone for a data warehouse, you can pay AWS some money, or you can pay Snowflake a heinous amount of money, for something not significantly better?

Mongo, ES and Redis all come with open source versions, which shifts the appeal back away from the AWS offerings, especially with things like Elastic Cloud for Kubernetes which has made our ES cluster basically a hands free experience.

Snowflake is technically far superior to Redshift. The performance and features are somewhere else. Even if you pay more in operating cost (which is not always the case, especially if you count Redshift DBA salary), you get to do what you actually need to do much faster instead of fighting the platform.

Snowflake is also technically going to always be far superior to Redshift, because AWS is a follower, not a leader. Their strategy is to copy what others are doing and build a moat of "but we can do it too, and you already have other things with us".

And if you really care about operating cost, you should be running Trino+Minio or Clickhouse onprem or something as your warehouse.

> Snowflake is technically far superior to Redshift. The performance and features are somewhere else

From the discussions I’ve had about this before, I think I’m in the minority when I say I’m categorically unimpressed by Snowflakes performance.

Add that to the hideous cost, and the worlds most aggressive sales/account management team and I’ve less than zero desire to ever deal with them again.

> Snowflake is also technically going to always be far superior to Redshift, because AWS is a follower

While this is true, how many businesses actually need or actually utilise the features they’re paying for with things like Snowflake. Sure it’s got separate storage and compute, but how many places have so much data that they need that? I’ve worked with places that were running multi-node data warehouse clusters, that we migrated to a single ClickHouse or Postgres instance and got equivalent or better performance for a fraction of the economic and operational overhead.

> And if you really care about operating cost, you should be running Trino+Minio or Clickhouse onprem or something as your warehouse.

Can’t disagree here, although I’d say it shouldn’t just be relegated to operating cost: CH blows most alternatives out of the water and is available in hosted options now. Haven’t used Trino recently, last time I used it, it was still called Presto, and it was frustratingly slow, know if it’s improved recently?

I think that depends on your target and especially scope.

ClickHouse demonstrates this well - it is incredibly fast and powerful, but also very limited. It has its own dialect of SQL incompatible with anything else. It doesn't even have a traditional query planner so you have to be expert to write fast queries. It has no update, no merge, no CTEs. To get most out of it, you have to think about data sorting, data types (is this LowCardinality String or just String?). And once you reach scale where single node can handle it (which is much more than single node Redshift could handle) you manage a stateful cluster which is a pain compared to storage/compute separated clusters like Presto/Trino over S3.

So ClickHouse is very good if it is in hands of dedicated team that is willing to learn CH in and out and is using it for a particular purpose. But you can't really setup ClickHouse, dump data into it and then send 100 random Data Analysts to "go forth and make reports". The barrier of entry and list of things that are different in ClickHouse is just too long.

Snowflake is exactly at the opposite end of "how much knowledge / how much of a dedicated team do I need to make use of this". Dumping data in and giving other teams access is something you can do easily. Scaling isn't a problem, users interfering with each other isn't a problem (you can give them their own compute that suspends when not in use), all the typical "data features" are there, backup is handled via time travel once they delete something they shouldn't, they don't have to understand data types or what is the best partitioning and sorting for the querying they are planning to do. It even has fancy web UI with charts and CSV upload.

You do pay a pretty penny for that, but there are many companies with a lot of money and many data problems to solve, so paying more for infrastructure and enabling a many of less technically skilled users (who are likely SMEs) is worth it.

Redshift sits somewhere in the middle. You still need DBAs, you still somewhat need to think about data layout, you still need to worry about managing backups and workload management where one using can slow down everyone else, etc. It feels like worst of all worlds.

In onprem world, Trino behaves more like Snowflake than ClickHouse, which has its uses - again if youre looking to build a general platform instead of dedicated application. Might be worth it even if you never get milisecond latency queries out of it.

> ClickHouse demonstrates this well - it is incredibly fast and powerful, but also very limited.

Your examples are not entirely correct. ClickHouse introduced CTEs in early 2021. I use them constantly. ClickHouse SQL is turning into a superset of standard SQL at least for queries. Most SELECT syntax, including window functions, just works. ClickHouse does have updates; but they are asynchronous. This too is being fixed. Synchronous DELETEs will be available this year, UPDATE is next. One big issue for my money is distributed joins. They still require a lot of reasoning about data locality.

You are right that ClickHouse requires attention and skills to get the best performance. However, that includes fixed, low-latency use cases like real-time marketing that Snowflake simply does not handle. ClickHouse and Snowflake aren't interchangeable for these use cases.

Oh wow, I missed that. I guess I heard about the CTE somewhere and understood dbt-clickhouse not having ephemeral materialization as CH still not supporting it.

I am still somewhat salty about not being able to do GROUP BY 1, 2, 3 :)

> You are right that ClickHouse requires attention and skills to get the best performance. However, that includes fixed, low-latency use cases like real-time marketing that Snowflake simply does not handle. ClickHouse and Snowflake aren't interchangeable for these use cases.

Yup, that was my point.

> GROUP BY 1, 2, 3

It is not ANSI SQL.

Nevertheless, it is already available in ClickHouse as well under `enable_positional_arguments` setting and we are considering making it enabled by default.

Ooooh.

Is there a way to turn unquoted column and table names case insensitive too? Thats like the second biggest annoyance.

Not yet. I will add it to the short list.

> Snowflake is also technically going to always be far superior to Redshift, because AWS is a follower, not a leader.

Redshift was the first cloud data warehouse-as-a-service in the Amazon cloud. Every data warehouse since then has built on that concept. Speaking of innovation, Snowflake depends on object storage, which Amazon basically invented.

Yes, and credit where credit's due, with Redshift Amazon changed analytics industry forever and they changed whole IT space forever with S3.

But since then, there's not much innovation in analytics space from them, is there?

Not so sure about that: https://aws.amazon.com/ground-station/. Enabling data collection from the rest of the solar system seems pretty cutting edge to me.

Are you saying AWS does not make money off the infrastructure Snowflake consumes? Snowflake consumes a lot of compute, plus there are lots of other SaaS services that integrate with it and also need compute, networking, etc.

It's my understanding the margins on compute at least are pretty good. So it seems as if AWS wins either way.

I don't know what you are smoking here but let's take MongoDB vs DocumentDB or Elastic Cloud. Mongo built a 30B$ business around Atlas that is growing at a staggering rate. DocumentDB has a tiny share. Same with Elastic.

Also what the hell are you talking about with RedShift. Decoupled Compute and Storage is a far superior architecture. Here is an expert from snowflake.com - " "Snowflake was founded on the belief that tying compute and storage together is not an effective approach for limitless, seamless scaling."

Yeah, I'm not sure the point they are trying to make here. Each of these AWS services is clearly and demonstrably less popular than the alternative it's compared against. Mongo and DocumentDB for example aren't even close: https://db-engines.com/en/ranking_trend/system/Amazon+Docume...

OpenSearch is a massively inferior offering compared to Elasticsearch too. It became outdated the moment it was forked, the documentation is lacking, and since you'll end up looking up ES docs and forgetting to switch to version 7.10, you'll get a nice reminder of everything new that has been added that you can't actually use.

The only thing it has going for it is that it's managed and you're already on AWS, so you don't need to spend months working up a contract with a new vendor and doing the security audit dance.

> so you don't need to spend months working up a contract with a new vendor and doing the security audit dance.

That's a very big moat. Many decision makers are risk-averse w.r.t to infrastructure vendors and don't mind paying (or making someone else pay) a premium for that.

The only thing that changed in the saying "no one was fire for choosing IBM" is the name.

> The only thing that changed in the saying "no one was fire for choosing IBM" is the name.

Yes. IBM, then Oracle, then AWS. Wonder who's next.

> The only thing it has going for it is that it's managed and you're already on AWS, so you don't need to spend months working up a contract with a new vendor and doing the security audit dance.

That's a pretty substantial win for many projects, similar to how popular RDS is while giving up the ability to get updates as fast as if you run your own servers. People pay a lot for stability and reduced staffing requirements, which is a choice — calling it “massively inferior” seems like the wrong call versus recognizing that not everyone has the same needs and resources as you do.

It's functionally inferior, because if you go into it expecting modern elasticsearch you'll be sorely disappointed. It's only now that it's been renamed to OpenSearch that this becomes less of an issue, as the two technologies have completely diverged.

Anything AWS decides to run a managed version of will automatically have an advantage over the non-AWS equivalent but that's not a property of the technology itself really, it's just that you've already done all the paperwork and bureaucracy to use AWS in your org.

I mean, the word we're not using here but ought to is vendor lock-in. I don't really need the patronising remark about recognising other's needs either.

> It's functionally inferior, because if you go into it expecting modern elasticsearch you'll be sorely disappointed.

Or, for a large number of people, won't notice the difference. There are some important questions about how ElasticSearch's license & community works, and whether you're risking lock-in by using a managed service, but detail-free hyperbole contributes noise but not value to that discussion.

For example, what are the features you think no user of ElasticSearch could live without which are in ElasticSearch after 7.10 but not OpenSearch 1.1? What percentage of users depend on those features? How concerned are you about vendor lock-in with ElasticSearch's unilateral control of the project's roadmap? Do you think there's more or less risk from an open source project you can run anywhere which, you fear, will not be updated as frequently or from one which has a history of backwards-incompatible changes requiring you to stay current or fall out of support by popular clients?

These are all questions about the merits of either technology.

We've already identified that the merit of a managed AWS (but really any cloud provider) solution is that you've already done your due diligence for that provider, and that alone far outweighs whatever other merit you might consider.

Some of the other stuff is what I might consider if I had to make a case for approving a new vendor, but this is an overwhelming barrage of rhetorical questions that makes it difficult to have a reasoned conversation about.

I mean, come on... what percentage of users depend on those features? Unilateral control over a product roadmap? Am I supposed to actually know these things when stating my opinion that OpenSearch is lame in comparison to the original ElasticSearch, based on my anecdotal experience of dealing with the two? Do you have those answers yourself?

As to the risk, a fair question. There's currently a threat that ES client libraries will reject a connection to OS (and I recall this has already been done for some languages). Not the end of the world but the only people who have lost out from the licensing spat you've alluded to are the users. The risks of onboarding Elastic as a new vendor or self-hosting are well known and part of the usual discussion of trade-offs one will have.

You were making some absolute assertions, so yes, I would expect you to have some actual examples based on real experience. Asking you what homework you based that on wasn’t rhetoric but simply a question because I know multiple projects using OpenSearch who’ve had no problems other than Elastic sabotaging certain clients. It would be useful to know, for example, if there was a certain class of functionality or performance challenge where the difference is significant.

For big cloud markets, being #2 or #3 is a win:

- #2, #3, even maybe #4/#5 are big revenue. Cloud is unusually big and still growing insanely.

- crazy margins when they don't have to invent the core concept, go through core product/market fit R&D, nor fight for a distribution channel to market+sell it, nor fight middlemen for competitive pricing

- cross-selling & ecosystem lock-in means even revenue/profit don't have to be high or even positive

> Each of these AWS services is clearly and demonstrably less popular than the alternative it's compared against.

Subject to the limitations of the data, which is mostly what they can scrape from open sources with an unspecified weighting algorithm: https://db-engines.com/en/ranking_definition

That's important to pay attention to because there are many areas this can go wrong — for example, it doesn't include AWS support or Amazon's own Q&A forums so you know you're missing a certain fraction of highly-relevant activity and, more importantly, as a bulk data-mining exercise you have a big challenge differentiating breadth and depth. MongoDB was heavily promoted about a decade ago so there are a ton of SO questions from people who fired up a copy and were looking to use it — which is great, but it doesn't tell you how many of them ended up actually using it for something serious or how big their project was. A thousand hobbyists storing 1% of the number of records of a single enterprise customer is not really something you can easily distill down to a single value. It also doesn't tell you how well people are sticking with it — for example, mature development / ops teams tend to ask fewer basic questions (they've been resolved & in-house expertise means they might never hit StackOverflow) but they might post harder questions about scaling. Does that mean that use of the technology is tapering or that the community is maturing?

The other big question is how you account for managed services. For example, if I use Kinesis I'm outsourcing a lot of operations to AWS; if I use Kafka I have to bring that to the table — one of those scenarios is likely going to involve a LOT more questions and open activity which on its own doesn't tell you anything about how many applications or how much data I'm using it for in either case.

Reply on news.ycombinator.com