Other than the expensive part, is it really such a bad thing? I feel like relational databases are a pretty good fit for a wide set of use cases and have a huge amount of tooling.
Companies prefer well known products like Alteryx or Tableau because, despite the cost, it makes people easier to replace.
But i cant blame you for writing your own things. Im currently replacing a large SSIS-based etl proces with Python, because i'm sick of SSIS randomly breaking.
In short, is there any solution that "does everything you could possibly want" while ensuring you _never_ need to hire a data engineer? This is a holy grail that I don't think exists.
You have to normalize data taken from various sources of various age and complexity. So you really have to understand the data. You also have to really understand the questions.
I've worked with (and on) lots of these tools and projects; the complexity is never in the frontend, it's dominated by getting the data, getting the data right and into the right format.
If all you want in the end is a good looking dashboard on a website then you might as well build it yourself; because of the cost structure that can even cost less than buying one of the BI frontend tools (there's not a lot of difference in development time, the the BI frontenders are more expensive because they are rarer and the licencing is high).
The people and their spreadsheets was the easy part to control.
Welcome to my reality.
Would I love a data architect and a domain expert in my team? Yeah.
Will I run around booking meetings with everyone that even hints at working with data like a headless hen? Yeah.
Is this the normal procedure for Data Scientists in big and old companies? More so than I would like.
Oh! And I forgot that the security department will constantly deny your access to data you need (until you force their hand).
(Disclaimer: I work in data engineering at Amazon and use those tools in my day to day)
PostgreSQL on the other hand - so good, so free!
In theory, it could be used to provide that industrial strength abstraction layer between your Tableau/Looker/etc. and your bajillion weird and not-so-weird (RDBMS) data sources.
That would seem to make sense to me from the point of view of -- I would want my data visualization/analytics-type company to be able to concentrate on data visualization/analytics, not building some insane and never-ending data abstraction layer.
The part that surprised me was that Denodo could allegedly do a lot of smart data caching, thus speeding things up (esp hadoop-oriented data sources) and keeping costs down.
I'm guessing the other data virtualization providers can do similar.
They make it simple to get started and even without knowing what you are doing you can easily churn out something that works if it is simple, doesn't change often, doesn't need to scale and deals with small amounts of data.
But similar to http://blogs.tedneward.com/post/the-vietnam-of-computer-scie...
"It represents a quagmire which starts well, gets more complicated as time passes, and before long entraps its users in a commitment that has no clear demarcation point, no clear win conditions, and no clear exit strategy."
RDBMS are the root cause.
There are no major systems out there of even moderate complexity that aren't built on an rdbms.
I don't think this claim is accurate.
When one of these things change it changes the rest.
Counter-intuitively Datomic is in violent agreement with /u/rqmedes where he said "A better alternative is having the data, data model and business logic tightly bound in one place. Not separated in multiple "tiers"" – Datomic inverts/unbundles the standard database architecture such that the cached database index values are distributed out and co-located with your application code such that database queries are amortized to local cost. Immutability in the database is how this is possible without sacrificing strong consistency, basically if git were a database you end up at Datomic.