2,578 karma · joined October 15, 2012
E-mail: sixdimensional ATAT [dimensionsix DOT com] minus the ATAT and brackets.
I wonder if there is a logical overlap in the ideas between temporal table handling and this low level storage engine optimization.
I really understand what you are saying... I know the hype problem, I lived it. It makes me both frustrated and sad - because the hype is annoying, there is a lot of vaporware - but there is also something real that is happening too which is part of the story of the evolution of data architecture/infrastructure. My strong advice is, being open-minded is helpful - learn and take what was good/real and leave behind the stigma/hype. Something real and useful happened in terms of architecture, so take the benefits - but of course, don't compromise on delivering real, working solutions.
Regarding "data lakehouse", I struggle with the buzzword term also, but once again, I recommend looking at what is good/real and ignoring the stigma of the buzzwords. One way of looking at it is the literal translation - a data warehouse made from the components used to make data lakes. To be honest, it is a marketing term, but it is also an architectural pattern we had even before the term existed - for example - you could use a data warehouse product such as Vertica, and back it by HDFS - guess what, there's a data "lakehouse".. and most of the big database vendors can do this trick now - a full traditional data warehouse engine sitting on top of lake storage infrastructure.
There are several "real" use cases for data lakes. Precursor architectures could be seen to be "operational data stores" [1]. Data lakes are real, they are one approach to solving some problems.
These use cases could include: 1) raw, long term storage of large volumes of diversely structured data for staging/historical purposes; 2) data discovery/exploration of this data to identify patterns/models and relationships (this is both an AI and analytics use case for power users and BI/analytics/data scientists, etc.); and 3) an opportunity to change the paradigm of traditional ETL - instead of pulling from sources, one way you look at it is, allowing many diverse/distributed sources to push their data into the lake for powering analytics, exploration, AI model building, etc. It makes sense as part of lambda/kappa architecture as well - some of the "push" in can come from streaming sources as well.
Use case #1 is very much a "data infrastructure" kind of use case that we do anyway in data warehouses - especially those that do ELT (vs. ETL) - staging databases. If you want an architecture that actually helps make some sense of such a use case of data lakes more formally, one could look into Dan Lindstedt's data vault architecture [2]. While data vault modelling doesn't necessarily require "data lakes", the "raw" part of the data vault architecture use case overlaps nicely with data lakes.
[1] http://www.actinginbalance.com/intelligence-is-adaptability/
The overlap between PLM and MDM and PIM seems to be an area of complexity.
Do you think there is a clear direction here? Seems there is some overlap, but both are still needed as well as integration between these systems - since PLM usually doesn't master parties, financial data, etc. but rather engineering/manufacturing design item data.
"The concept of master files was transferred to some corporate computers as early as the 1960s and gained popularity during the 1970s and 80s when computers became more abundant. Master files contain essentially the same descriptive data – such as names, addresses, and basic summary information – as master data. Master files continue to be used as a part of some system databases and were a precursor to master data." [1]
[1] https://www.dataversity.net/a-brief-history-of-master-data/
In recent years there has been some backlash against ERP systems, e.g. Oracle. Also, master data management was seen negatively in an increasingly federated system world - one with the Bezos mandate, microservices, reduced centralization, etc.
I’ve seen a general unbundling and remarketing of enterprise solutions in ways to make them more palatable to the newer generations.
Therefore, to escape the legacy image, ERP -> master data management -> product data management -> product information management.
I think it has to do with the evolution from big centralized systems to federated/distributed/independent but interconnected ones, and the corresponding branding/marketing changes over time. Especially to avoid terms seen as “legacy”.
The marketing might be a bit short sighted, but that is an opinion. Maybe the branding sells more copies, and for most of these companies that is all that matters.
I worked for a company, where I really did have nearly 100% uptime at my local data center in our office. I wasn't a luddite, and the cloud wasn't something I was afraid of or didn't understand. I just, at the time, had an infrastructure in place before the cloud existed, and it worked well enough at a great price point. We were on the same electrical grid as a hospital, our infrastructure handled our scale, and just really didn't have problems for a long span of time.
Still, someone came in and said.. what, you are still doing stuff on-prem and not the cloud!? How legacy! How dated! And so the push to the cloud came next.
We migrated to the cloud (and some colo facilities for some equipment we chose to keep), and on the 3rd day of being freshly migrated, the cloud provider had a major outage and went down for longer than we'd ever had on prem. The next month, the colo went down and their diesel generators failed, and it was down for an entire afternoon.
Oh sure, there was some SLA money returned in that event from the colo...
I'm just saying, I've been on both sides. Tell them the truth - even the cloud has outages, but it is certainly more "convenient" to have hundreds of engineers work on fixing an outage at global scale, and all you have to do is wait for it to start working again - then it is to have to fix it all yourself.
Yes, since the inception of these kinds of tools, recovery from these corruptions became easier by restoring from an image. Still, in the early days, you could burn a lot of time doing anything of these things!
To be fair, that hasn't happened in a pretty long time for me - but in the earlier days it was commonplace. It used to be bleeding edge features that would do it, other times it was just because they took so long, that interrupting the process of upgrading or installing caused a mess. Or installing the newest version alongside the older version, etc.
Visual Studio is a massive and complex creature (much like other "big IDEs", such as Eclipse) - although it has become much more streamlined in more recent releases - considering how complex it is.
EDIT: To be clear, problems often involved corruption of registry values, library/DLL hell, small database or configuration files and caches, etc. that happened with Visual Studio in the scenarios above.
I have it too at my workplace, and other than a coat of paint on Sharepoint Lists, I concur, it is not amazing.
I guess I brought it up because it shows Microsoft being driven to do something, but much like Google, falling short of the target. Still, before AirTable and similar things, the giants were generally ignoring this area.
It took something like AirTable to wake the sleeping giants, that people wanted a low-code tabular data editor and storage engine, that wasn't really a spreadsheet. You know.. like FileMaker or MS Access.
I guess this goes to show something - I have thought for many, many, many years of building such an application in the cloud (having built "application generator" type applications in the past) - but, it may not have mattered if I had - because the big players might just come in and clone/copy it. It will be interesting to see how companies like AirTable can compete (and really hope they can).
I think we are crawling towards a future of low/no-code - or the dream of the world of 4GL [2] (or perhaps even beyond that) is coming closer and closer. Regarding 4GL, "those who do not study history are doomed to repeat it". Not saying it's a bad thing to be revisiting this, but it has been low-hanging fruit for so long now, you wonder why it took so long for the big companies to engage with it seriously.
[1] https://www.microsoft.com/en-us/microsoft-365/microsoft-list...
[2] https://en.wikipedia.org/wiki/Fourth-generation_programming_...
Still, seeing that a project is revenue-impacting or revenue-generating (has the relationship to the money-making potential of the company) is relative to the individuals who are viewing it. For example, if there is an effort you could do that generates 2x revenue, but those in charge of your company choose not to pursue it "because they are already making a ton of money and cannot see past the costs".
I guess this comes down to how good of a business leader one has.
I have so many times seen good projects/ideas, ideas that can clearly make money, even lots of money - but in the wrong company, with the wrong people, or attitudes, or beliefs - don't matter. These are the times when, if you are the person with such an project or idea, you ask whether or not you are the right "fit" for the organization - since they are not recognizing or prioritizing the value of your project/idea.
I suppose you could look at this as efficient capital allocation, if you assumed perfect business leadership. Perfect business leaders would only pursue the revenue generating activities that made the most sense.
I would argue that we are far from perfect, and that we often let ego, opinion, culture (or even "data") influence our decision-making about revenue-generating activities.. which brings us back to the point that culture is still relevant. Something that popped to mind related to this is the "Ultimatum game" from psychology/economics [1].
There are tangible and intangible costs and benefits to every pursuit (interesting example, externalities in economics). Focusing on only tangible benefits (money) does, to me, seem very common - rarely do I see intangibles being considered.
For example, how many times I see profitable companies making money hand over fist, but when you look at their operations internally, they are completely inefficient and chaotic, maybe even damaging. Like, if they did things even 10% more effectively (let's say at a bit of reasonable cost), they would make even more profit. But doing so takes hard work, they are already making so much money, so they have no desire to look further.
If the profit is so good, that ignoring the money "left on the table" doesn't cause enough pain for the subject involved - well, I think that also creates inefficiencies, complacency, etc.
I also think, when profit is "so good" that other efficiencies go by the wayside, the same behaviors you describe continue. I don't find that the other behaviors go away, really.
I once heard a quote that said, if you wanted any data project funded, convince the CFO that it makes money and they'll go for it. Much like the approaches of TCO, ROI, NPV, IRR, etc - prove the profitable case and you'd think people would buy in.
I have found something like this to be somewhat true - except in companies where they are already making so much money, they don't care about doing the right (or "better") thing. The "more money" they could make is imaginary to them, and they are happy with how much they are already raking in. Until it's super, super tangible/convincing.
The basic idea remains the same though - building dimensional data models, and then using OLAP/cube technology with MDX on top.
That said, if I think about it, my experiences with wide columnar databases were with HBase and Cassandra. The main goals when using those systems were usually three things: A) getting many attributes/variables being tracked in a single row as part of a measurement or observation of some event - and needing to write the data ridiculously fast (append only) and report on that data in real time; B) coming up with creative data processing patterns to aggregate data into higher level tabular structures, to make for blazing fast queries on these aggregates, which could be updating in real time, and C) being able to store huge volumes of data on a redundant cluster in an efficient format.
If you look at A, B and C above, these are, in a sense, the same reasons why people wish to implement OLAP cube technology, for the most part - aggregating huge results into aggregated views, lightning fast query responses and easy of navigating - "slicing and dicing" the data. However, OLAP cubes are most commonly associated with batch processing - there are real-time OLAP cube technologies, that basically work by doing change data detection on the sources.
I kind of believe that, you don't need specialized OLAP technology to accomplish the same thing - it's all about pre-aggregated and calculated data, efficient writing/reading and storage. I mean, OLAP technology is there to try to make that easier to do, but you can do the same thing with certain tricks in traditional dimensional models (accumulating snapshots, periodic snapshots, aggregate fact tables). Or in wide columnar databases (storing aggregates in wide tables). It's just maybe easier to use OLAP technology, otherwise, you kind of need to do all the scheduling, processing and aggregating of data yourself - but it's totally doable too and I have seen that work really well.
I personally have never been a fan of OLAP cube technology (MDX), even for how powerful it is, because it was so specialized - but that opinion was formed from a legacy of years of only being able to get these features from commercial vendors.
If you look at the legacy of SQL as a language, I actually believe that there was an attempt to unify OLAP queries with SQL by implementing "CUBE" and "ROLLUP" as part of ANSI SQL standard, but it was never as powerful as MDX and the OLAP engines that were built.
I think today, the world has changed, as we have open source OLAP and MDX options available - Pentaho Mondrian was popular for a while, not sure how much today. The current hot thing seems to be Apache Kylin. I also played with Apache Druid, in the past, although it doesn't seem as popular any more, as far as I can tell. Yandex Clickhouse is very popular also these days, for being extremely fast and open source. Again, Wikipedia has a good list of OLAP systems [3].
More to the point, if you use an event-based streaming approach (things like Kafka) - where the event comes directly from the source, combined with writing the data efficiently to something like an append-only, wide, column data store.. well, you have something pretty powerful. The only thing that a solution like this lacks, sometimes, is a powerful query engine - for example, Cassandra or HBase cannot do joins. There are ways to simulate joins with Apache Hive or other technologies, sitting on top of Cassandra or HBase.. but not sure I ever saw those work well.
I mean, the goal of aggregation is to avoid needing joins.. but, you'd be surprised how once in a while, you really wish you could just join this with that to come up with a new result - even in things like Cassandra or HBase. If you do your design right, you can avoid it.. but I find myself wanting it sometimes.
Really, in OLAP, everything is built to avoid JOINs or minimize them as much as possible. It's one reason, I believe though, that having JOINs in a dimensional model is still helpful. But maybe that's the old geezer who still likes SQL in me talking :)
Lastly, I thought I might mention - anyone who says that leaving all data without any kind of model structure or architecture (ex. flat or raw data only) is in a world of hurt, once you get to scale. This is what creates data swamps instead of data lakes. Data models are still as important as ever - snowflake, dimensional, multidimensional, etc. Pure raw data can be very useful for exploratory analysis, even machine learning models, but easy analysis for end consumers always requires some form of data modelling and architecture.
There is an opinion that we should throw data modelling to the wayside, and just feed raw data into AI and machine learning models - but I personally have never seen this as a reality yet.
I hope all the above might be useful!
[1] https://en.wikipedia.org/wiki/Online_analytical_processing
[3] https://en.wikipedia.org/wiki/Comparison_of_OLAP_servers
It's not about the columns being wide, per se that is important in Vertica - although that is a benefit. It is about columnar storage and optimized querying, over row storage and optimized writing.
It would be the other way around - SQL Server or other transactional databases for that part. Actually, these days, SQL Server supports column store indexes on top of transactional tables- which gives you HTAP - hybrid transactional/analytical processing. A few other systems do this (Oracle too).
If you do HTAP, in a way you could even avoid using Vertica for data warehouse/OLAP use cases. Or you can build a separate dimensional data model in SQL Server and keep everything in there.
But if you already have Vertica, which is quite fast and good at OLAP queries on dimensional models, use it for that (enterprise data warehouse), and feed data from transactional systems in.
Edit: Corrected Family -> Order - oops! Thank you pvaldes.
I had no idea something like this size and related existed in the sea! Incredible! I wonder now what the evolutionary tree looks like.
I work for a large corporation. They want to implement the Bezos mandate [1]. No direct database access between teams, API abstraction for everything.
OK, now let's think in onion layers (or hexagonal/clean architecture if you like). Think of layers of services with different purposes - data services (containing no application/business logic), application services (for business logic, orchestration, process), and UI/UX services (to power differing end user experiences).
Data services don't have to do much - be the data/repository layer, expose productive interfaces for CRUD. Need to read across data sources? Think of federated data services that can combine data on the fly, perhaps like GraphQL.
These kinds of tools are perfect for the first layer of services that abstract the database world from the application world. Just simple services, even ones that effectively let you mimic what SQL queries can do (filter, sort, page, etc.). Individual record oriented interfaces, and bulk oriented interfaces. The query side of CQRS (command query separation).
Many will say, "I don't need all this complexity and layers" - and sure, for smaller or simpler applications, probably not!
But, if you have to operate on any kind of larger scale, with multiple data sources, systems, etc., you end up needing the layers. And these types of tools automate some of the lower layers.
Perhaps when we talk about this, we shouldn't be focusing on "oh it's too complicated", and instead building frameworks or reference architectures that automate away the complexity - so it looks easy again, but now it is more flexible, perhaps easier to scale.
I believe that we are on the cusp still, of an almost fully defined, service based architecture (microservices and server less were just one part of the continuing development of that story). Federation is another part of that story oft ignored. Thinking of the onion as service layers is another part. Erasing the network boundary as a concern through much higher speed internetworking is another part.
Eventually we may come to see, that it is all a big "system", some parts just aren't connected to each other directly.
Sorry, got a bit rant-y at the end there :) Just passionate about sharing this world view with others - as I continue to see this architecture developing!
[EDIT] I wanted to add, it's not just that the use case for this is in a data service layer for automation - from a logical perspective I mean. In big companies such as the one I work for, we never get the resources we need, ironically. We are overwhelmed with demands, and must operate under the Bezos mandate rules. Tools such as Octo are not a panacea, but, they are a good compromise if you have to move fast, they are time-and-cost-savers. And they can get you surprisingly far.
[1] https://www.calnewport.com/blog/2018/09/18/the-human-api-man...
Your interpretation of the atomic columns requirement is a little different than my understanding. That requirement of normalization only applies to the "cells" of columnar data, it says nothing about encoding meaning into column names, which are themselves simply descriptive metadata.
I mean, for sure you wouldn't want to encode many values/meanings into a column name (some systems have length restrictions that would make that impossible, I'm not sure it makes sense anyway), but just pointing out that technically the spec does not make that illegal. Certainly, adding minor annotations within the name of a column separated by a supported delimiter does not, in my opinion, violate normalization rules at all. I mean things like "ID_" or similar.
Have you looked at INFORMATION_SCHEMA in SQL databases? [1] You mentioned SQL metadata and constraints, that is as close to a standard feature for querying that information there is, some databases do it using similar but non-standard ways (Oracle for example).
Also, not standard but, many relational databases support extended properties or Metadata for objects (tables, views, columns, etc.) - you can often come up with your own scheme although rarely do I see people utilize these features. [2] [3]
At some point it feels like we are more talking about type definitions and annotations, applied to data columns.
Maybe like, BNF [4] for purely data table columns (which are essentially types)?
[1] https://en.wikipedia.org/wiki/Information_schema
[2] http://www.postgresql.org/docs/current/static/sql-comment.ht...
[3] https://docs.microsoft.com/en-us/sql/relational-databases/sy...
I would say, yes this idea is known/very common, as data architecture is as much about the descriptive language we use as anything. I mean, "business glossaries", taxonomy, even just naming conventions [2] in coding, these are all related.
If you build enough databases/tables or even code yourself, you inevitably come across the "how to name things" problem [3]. If all you have to sort on for the known meaning of a thing (column, table, file, etc.) is a single string value, then encoding meaning into it is quite common. This way, a sort creates a kind of "grouping". Many database vendors follow standard naming conventions - such as Oracle, for example [4]. It is considered a best practice when designing/building the metadata for a large system, to establish a naming convention. Among other things, it makes finding things easier, as well as all the potential for automation.
You get all kinds of variations on this, such as, should the "ID_" come as a prefix or a suffix (i.e. "_ID"). One's initial thought is to use it as a prefix so all the related types group together, but then that becomes much more difficult if you want to sort items by their functional area (e.g. DRIVER_ID, DRIVER_IND, etc.).
One other place you see something similar is in "smart numbers" which is an eternal argument - should I use a "dumb identifier" (GUID, integer) or a "smart one" (one encoding additional meaning) [5].
I mean, basically, any time you can encode information in the meta-data of data, I think you can then operate on it by following "convention over configuration" (as mentioned elsewhere in the discussion comments).
The only problem I see is that such conventions can, at times be limiting - depending on the length of your metadata columns, and the variability you are trying to capture - which is why I believe, generally, metadata is often better separated and linked to the data it describes - this decoupling allows for much more descriptive metadata than one could encode in simple a single string value. Certainly, you can get a long way with an approach like this, but I suspect you would run into 80/20 rule limitations.
Using naming in this way is a form of tight coupling, which could be seen as an anti-pattern in terms of meta-data flexibility, in some cases.
[1] https://en.wikipedia.org/wiki/Metadata_management
[2] https://en.wikipedia.org/wiki/Naming_convention_(programming...
[3] https://martinfowler.com/bliki/TwoHardThings.html
[4] https://oracle-base.com/articles/misc/naming-conventions
The end result you speak of is a more flexible data platform, which can handle more use cases, including applying decoupled parts to different things.
A good example, in my mind, is Apache Spark - which is like taking a compute and SQL query engine, making it distributed and decoupling it from the underlying storage. Now, suddenly SQL or code can be used on anything you can fit into a data frame, even streams - wow!
Pulling out the transaction log - and wow, suddenly we have durable, potentially immutable queues we can use for even long term storage.
I was so excited by the line of thinking in that article. :)
Something the original post reminds me of a little is Crux [1].
TDD, like Agile and Scrum, is not a one size fits all solution. These are tools with process that have to be adapted to the people and the organization. That’s why the creators of these practices are declaring them “dead”, because people are blindly believing they are written in stone rules and that misses the intention. We made these rules to help ourselves and our organizations... not to overly constrain ourselves... add more discipline but leave room for flexibility that our jobs demand of us.
Have you ever had to write some calling code, to call some other code, to make sure it worked? Don’t call it a formal test, I mean, a chunk of code that calls your other code, and you verify that the expected output is what you thought it would be?
Coding is often a process of discovery. Architecture can be learned but applying it takes discovery and iteration. “Why do I need a certain architecture?”. Well, if you don’t know, code long enough, and do enough projects, and you might start discovering architecture all on your own. Then you might see how what you discover aligns with what others have discovered. Writing code can help you discover the right architecture, as long as you keep trying and keep iterating.
Many of us had to learn and discover everything on the job, because not everything was known - much of it was being invented and discovered while we coded. Is that as efficient as learning known patterns and applying them? Maybe not, but also I believe the only way you can truly internalize the knowledge of a pattern or architecture is by implementing it, even when you don’t fully understand it. Therefore, you need to “test” it out to explore and discover it, to internalize it, to really learn and understand it.
TDD in that respect can simply be seen as a helpful crutch to help you, as you explore, discover, iterate and learn. A “test”, whether it be a unit or integration test, is that chunk of code that calls some other code, so you can see if the other code works or not, especially in a mocked up context of how you think it should work later. TDD is just accepting at the beginning, “I only have a rough idea of what this should do, I’ll define a starting point to call into my code, even code I haven’t written yet, so I can explore and discover both that code’s external interface and also internal implementation, and see results sooner”.
Using TDD is like sketching the blueprint of how you think the architecture should work, or how you want it to work, and then trying to color in the code in the middle, to see if it really does work.
TDD isn’t an absolute. It’s just a helpful practice made by developers to try to help developers... and you will eventually do something like it, even if you don’t know what it is called... because that’s how we all code in the end.
Think, code, discover, test, integrate... and iterate.
TDD just suggests, start in a test, and then start thinking, and then continue as usual.. and let the testing framework help you assist you as you explore and learn.
I think you're talking about the difference between an "experience API" - that is, an API with the sole purpose of being support for user experiences/clients - and a "system" or "process" API, where the latter is for application or process integration between many systems. These are terms borrowed from Mulesoft, but I do like the terminology, I find it helpful for segregating concerns.
There are a lot of reasons people need separate experience APIs to power specialized UI/UX - especially with the needs of different client platforms (ex. chat bots vs phones vs desktop browser), separate from system/process APIs.
Shall we play a game?
So maybe not playing doesn’t really work.
I was thinking, in regards to some grandparent way up there, the same statement “don’t play” might have been true for Oculus in general.
What I mean to say is, don’t sell the company, ever. Then you can “control the outcome”.
Ah, but there lies another fallacy. You really can’t control the outcome even if you try to. Even if you don’t play, likely someone who wants to do the same thing as you, and exploit it, will find a way. Or maybe on their own, Oculus would have never found the right supporter who would honor privacy. Even if they had.. the below could happen.
For example, if Facebook hadn’t bought Oculus, maybe they would have bought the Vive product line from HTC (a bit far fetched) and compete against Oculus.. and then done the same privacy intruding measures.
So even if Oculus had held out and didn’t “play”, they might have been crushed anyway or the privacy problem could have just happened somewhere else.
I’m not saying we should give up trying to protect privacy and “play” the game... but that somehow in the competitive environment we are in, those playing the game are winning more over those who wish not to.