Hacker News Reader

Comment by ajd555 | Hacker News Reader

> Postgres and Apache Iceberg are both mature systems

Apache Iceberg as mature? I mean, there's a lot of activity around it, but I remember a year ago the rust library didn't even have write capabilities. And it's not like the library is a client and there's an iceberg server - the library literally is the whole product, interacting with the files in s3

View on HN

I suppose, in fairness, the Java library has been around for much longer

A lot of people will spend dozens of hours and tens of thousands of their company's money to avoid learning Java.

I'm not even sure if I'm joking. :)

This is data engineering, where people spend thousands of dollars of their company's money to avoid learning SQL. The place with no Java is across the street (old Soviet joke, originally for meat/fish stores).

Sad but true. Or they learn "something" about SQL but not about indexes, data types, joins, or even aggregate functions. I've seen some python horror shows that would select * entire tables into lists of dicts, only to do the equivalent of a where clause and a couple of sums.

Ouch. I was really hoping this wasn't so common, but I guess it is. I'm sure they heard somewhere that "joins are expensive" or something along those lines, so they solved it!

Prisma built a whole ORM around that, complete with the client-side joins.

There are also other "lets pretend we have big data" patterns I've seen. Running something like AWS Athena or Hadoop over and over, for less than a couple gigs of data that would fit into memory on a laptop from 15 years ago.

Classic, I remember seeing something like this

SELECT * FROM X, into a C# list, Filter it with LINQ, and then use C# to do calculations

"Why not EF?" "Someone told me it was slower"

I mean RisingWave, the solution mentioned in the article, is a complete startup rewriting things in Rust mostly to avoid the larger Java solutions like Flink and Spark...

Flink and Spark are painful for streaming.

They do work, but they have some sharp and rough edges.

I think 90% of the use cases for streaming data is because managers have nothing better to do than to reload their profits dashboard.

They are rarely supported by real business value.

Save the data as soon as it comes in and transform it in batch mode.

Most tools the data ecosystem support Iceberg as first class citizen, including major analytical query engines. Lots of development in this space has happened over the past year in particular.

Reply on news.ycombinator.com