So I was wondering, what are recent developments in databases and databases management system that you found interesting? Are there any practical applications you'd recommend looking into, be it niche or broad?
- The "No[t just ]SQL" movement brought us schema-free DBs (among other things; note that I personally prefer schema enforcement and strong typing);
- CouchDB and others started to integrate MapReduce-style support for distributed processing;
- Increased support for polygon/centroid indexing: PostgreSQL -> PostGIS, SQLite -> SpatiaLite
- Graph databases: RedisGraph, SparkSee, Neo4j, AllegroGraph and others - as well as Cypher as a de-facto query language standard (supported by Neo4J, RedisGraph and others)
Some senior DB people like Gerhard Weikum ( http://people.mpi-inf.mpg.de/~weikum/ ) switched from databases to Web mining, because they believe that is where the action is now: acquiring/learning structured data from unstructured sources.
I'm moving slow as I'm building it myself, and I don't even have joins yet in the language (although, I'm not sure I need them).
I believe current databases represent many mistakes. For example, how we think about privacy is a cluster fuck. The pattern of having a single root password to a database then exposes your data to anyone with network access. My opinion is that privacy, authentication, and access controls should be properties of the database layer.
A long term goal of mine is to build a nice layer for interacting with large documents which provide great direct product experiences, and then offload the larger scale aspects (i.e. cross document search) to dedicated indices or query engines. By focusing how product interaction happen and isolating to a user or document means fantastic control over privacy, audits, latency, and compliance.
It can't do everything yet that a full-functioned database like PostgreSQL, MS SQL Server, or MySql can do, but it is exceptionally fast and has a bunch of unique features. My benchmarks for hundreds of queries against some pretty big tables (e.g. 100M rows, 100+ columns) show it can be up to 10x faster.
It is a columnar store with a flexible schema so it can easily handle Json data where a single column is an array of multiple values. It creates a 3D table where each column/row intersection can have multiple levels. It is really easy to drill down to do analytics against subsets of a big table (e.g. only customers in California).
It also accomplishes its query speeds without needing separate indexes against any column(s). You get blazing speeds after creating a table and importing its data no matter what column values you query against.
The website (www.Didgets.com) has some short demo videos that display what it can do and how fast it runs.
Incremental View Maintenance
Materialised Views
Multiple indexed databases that allow for row, column, and search accesses very fast
Real-time replication
On the more technical side, I agree with brad0's list, especially incremental view maintenance.
https://wiki.postgresql.org/wiki/Incremental_View_Maintenanc...
Navigating the wave of AWS/Heroku etc replacements.
Fly.io, Netlify, etc