Databases in 2023: A Year in Review | Hacker News Reader

Databases in 2023: A Year in Review

ottertune.com

> Anything loosely connected to AI + LLMs received the bulk of the attention (rightly so, as it is a new chapter in computing)

Beyond LLMs and vector search, the scope for applying AI / machine learning _within_ databases is enormous: join planning, learned indexes, compression, workload prediction, configuration tuning etc.

Given the current pace of advances perhaps a dedicated "AI in Databases in 2024" review will be on the cards this time next year.

Agreed this is an underexplored topic, which is funny because a few years ago prior to LLMs, if you asked someone how they'd apply ML to a database, you'd get an answer more like this than about any vector indexing or LLM shenanigans.

Likely worth mentioning that Andy (the author) has been organizing an ML⇄DB Seminar Series (Machine Learning for Databases + Databases for Machine Learning) for the past few months (Fall 2023); materials & lectures: https://db.cs.cmu.edu/seminar2023/, https://www.youtube.com/playlist?list=PLSE8ODhjZXjYVdJKka5g3...

I actually just read a paper where some new file format achieved far faster and smaller writes than the Parquet used in all of these modern big datalakehouses, and it was done by guessing like 20 different simple compressions on a sample to figure out what will work best for each chunk of data. Sounds like manual feature detection to me!

Thanks for the writeup; in my opinion WCOJ is the most interesting development in core technology.

I'll have to disagree about database customers becoming more savvy; cloud has brought in a whole new crackpipe for managers to smoke.

A bit random but I wonder if anybody knows what happened to Tigris? It looked like a pretty promising product [1], then one day all activity just halted without any sort of announcement.

1: https://hn.algolia.com/?q=tigrisdata , a FoundationDB-based document DB, alternative to Mongo and DynamoDB.

DuckPGQ is a super interesting project, since it would mean we would now have a good option for OLAP + graph queries in the same package. But afaict at the moment is a research project... I wonder if DuckDB has it in its roadmap to actually ship this feature.

Interesting article, what's with the weird sucking up to Larry Ellison at the end though? It was so weird I thought it was sarcasm but it seemed like it might be genuine?

Definitely sarcasm, it's a running gag - see the 2021 and 2022 posts also :)

Old joke in the RDBMS world is why is Oracle RDBMS so expensive? Because how else is Larry going to buy another yacht?

You have to thank Larry for charging so much that all of the other innovation had a reason to happen!

> it seemed like it might be genuine

It is genuine. Larry and I go way back.

Source: https://youtu.be/_UGUDVeSQSI?t=872

Sorry to hear that.

Ok, you were way too subtle for my first reading or I should have gotten more sleep!

> Basho went bankrupt in 2017, and now Riak is maintained by one person who works for the UK's NHS

Let Riak go...

Is there any progress or news on the main memory based databases?

Reply on news.ycombinator.com