Hacker News Reader

Comment by foota | Hacker News Reader

Reading about databases always makes me sad about the enterprise database I work with.

You can read about the awesome theory behind relational algebra. Gave me a great appreciation for how awesome SQL databases actually are (and why they work the way they do).

Any good books on that?

This java project gives you a glimpse at how it all works. It was extracted out of a now defunkt db vendor (I think?). https://calcite.apache.org/

Yeah, it came out of the Eigenbase project. (http://luciddb.sourceforge.net/ and http://www.eigenbase.org). DB internals are indeed quite fun. (I worked on LucidDB(https://github.com/LucidDB/luciddb) for a time.)

I really liked 'Database in Depth' by C.J. Date.

Transaction Processing: Concepts and Techniques [0] by Jim Gray and Andreas Reuter. Not so much relational algebra, but everything you need to know to implement a classic relational engine.

[0] https://www.amazon.com/Transaction-Processing-Concepts-Techn...

The original paper[0] on relational databases by Edgar Codd is quite accessible.

I also loved the MOOC by Prof. Jennifer Widom[1]

[0] https://www.seas.upenn.edu/~zives/03f/cis550/codd.pdf

[1] http://online.stanford.edu/course/databases-self-paced

anything from C. J. Date

In addition to the great links from siblings, it might be interesting to read this Stonebraker paper on the cyclical nature of databases and how, ultimately, they all reduce to a few core paradigms. [0]

[0] http://pages.cs.wisc.edu/~anhai/courses/764-sp07-anhai/datam...

Databases are one of the most fun computer science topics. And I don't mean _databases_ from the perspective of an admin or even a developer, but a database developer of the internals. I love trying to read the code of complex systems like databases or language compilers.

Do you have some examples of interesting code ? I'm curious now!

I find postgres source code very interesting, e.g

https://github.com/postgres/postgres/tree/master/src/backend...

Sqlite is another good one to read, and play around with.

You may also find this paper interesting

http://wwwlgis.informatik.uni-kl.de/archiv/wwwdvs.informatik...

CPython is an interesting read. The internal type system (PyObject) is pretty cool. It also uses type pruining instead of a discriminated union.

Granted, this is in-memory computing and not intended for persistent storage. There's nothing stopping you from using Spark's SQL drivers (which support ODBC) to load things into memory, unless your database does not support it.

They support JDBC out of the box.

ODBC works via a JDBC bridge which though they are often commercial and vary in quality.

I work with SAP HANA as an enterprise database, and it works quite well with Spark, has code push down, query compilation, all sorts of fun stuff.

Depends on the enterprise database, is my point.

Where do you delineate workloads between HANA and Spark?

(I'm just curious...used to work the internal SAP HANA dev team)

Spark is not a database ;-)

Would you please tell that to the architects I work with? We're reinventing relational databases over the top of Spark.

Post their linkedin profiles, I would be happy to drop in a word :)

But it is a SQL parser and execution engine.

You could easily turn it into a database by managing a set of flat files.

Reply on news.ycombinator.com