Hacker News Reader

Comment by esafak | Hacker News Reader

> It genuinely makes sense for incumbent database players to offer vector search, because that eliminates unnecessary data movement to separate vector databases. Co-locating vectors and original documents also reduces latency.

Yet OLAP databases continue to thrive alongside OLTP databases, the nascence of NewSQL hybrid (HTAP) databases notwithstanding. Different needs dictate different design choices for optimality.

View on HN

Different needs dictate different design choices for optimality.

Could not agree more. Even for time series, which could be seen as a subset of OLAP, trade-offs and design choices inherent to time-series data are necessary. As an example of a TSDB that I know well, QuestDB: Data is always ordered by time once it lands on the disk, the data is partitioned by time, and the ingestion protocol is conceived to stream large volumes of data, which can be either continuous or in bursts.

It’s interesting that I never considered why OLTP and OLAP are basically orthogonal technologies. Are there any major players that have an integrated solution for both?

I guess it makes sense because the infra is so different, but I’m not sure whether it need be.

Effecient OLAP queries need a different shape of data - some combination of columnar storage for efficient scanning, and roll-up tables with pre-aggregated measures. Even in an integrated scenario, behind the scenes there will need to be a bunch of copying to transpose and / or refresh roll-ups.

The pre-aggregation is one too many try and skip. People seem to think they can build a single schema to rule all things, and then assume they can quickly calculate any aggregation on demand.

As far as the big players are concerned, Google offers AlloyDB (https://cloud.google.com/alloydb) while Amazon offers Aurora (https://aws.amazon.com/rds/aurora/)

I had understood Aurora as just cloud-native MySQL / PostgreSQL

How does that relate to the OLTP vs OLAP dimension? Are they not both primarily OLTP dbs still?

I think they have, they are just not that well known. E.g. SQL Server - https://learn.microsoft.com/en-us/sql/relational-databases/i... - you can also find quite a lot of papers by microsoft employees on the designs and capabilities (starting around 2016 I believe, so "pretty new"). I have used it with TPC-H and it worked wonders, never got around to using it in a production workload though.

Is anyone considering a new OLAP system these days? If “NewSQL” (which seems to be a fancy buzzword for running analytics in your transactional database) takes off wont it be the final nail in the coffin for OLAP?

I think the opposite concept, "reverse ETL," is actually more popular. You put everything into your data warehouse and then pump whatever you need out from there.

Not if you expect to do a bunch of data transformation before you materialize it into one of these engines.. if the way you want a materialize it is materially different than the way you handle it transactionally you're going to be moving it anyway and if you're moving it why not put it into an optimized context

"NewSQL" is not about analytical performance, it's about multimaster distributed writes and reads. EG, you don't want to run count(*) on spanner.

"HTAP" is the buzzword you're looking for. It's promising, but also complex and nascent. It'll be interesting to see how much traction it gets over time, but things like TiDB and Unistore are pretty early on to call a nail in the coffin for redshift/bigquery/clickhouse etc.

Reply on news.ycombinator.com