It is getting out of hand. They are interconnected and many people get confused. Any good summary out there for the data world?
Snowflake/BigQuery/RedShift. Think of them as databases that deal with a ton of semi structured data and are optimised to run analytics queries, typically with SQL. These are what you build a data warehouse on, generally, though you could also use a traditional RDBMS. Basically you collect data from all around the business and put it in one place so you can do joins or otherwise structure things in a more useful way. Imagine someone asks you to match up your Facebook ad creatives with how well those customers engage with your app using data from your own service databases. A well-formed data warehouse makes that kind of query trivial.
Tableau/Looker. These connect to your data warehouse and act as a layer that business or other people in the company can interact with. People who want to make charts and reports but who don’t want to or can’t write SQL, basically. Often this is called the BI layer.
Data lakes are like data warehouses but less structured. Data swamps are data lakes that are such a mess as a result that nobody wants to deal with them.
ETL you can just think of as a script that takes data from a source and shoves it in a destination, often from your production databases or third party services to your data warehouse. Typically the data you get out from the source doesn’t match what it should look like in the data warehouse, so transformation is done here too.
Segment I haven’t used but my understanding is it saves you interacting directly with the million different places you want to send data to and receive data from. So instead of installing (say) 10 SDKs in your mobile app and sending events individually to each, you just install Segment, send them to it, and it distributes to the others.
Caveat: this is all an oversimplification and flippantly described. Feel free to ask more specific questions though.
I don’t really think it’s particularly out of hand. Everything has its purpose, mostly. You’re just at that stage where you’re on the outside and it seems impenetrable. And each component has a few companies competing for the crown, which inflates the entities in the space, but not the number of different types of ‘things’ in there.
What is big data?
Is "big" there has anything to do with bigquery?
But in principle it just means utilizing data at a scale difficult to handle for traditional tools or approaches.
So yes, the ‘big’ in BigQuery just means it works at those scales.
As an example, say you’ve got a service you built to keep track of the people installing your app, which platform they’re using, device version, etc. You’ve also got a service that tracks their screen views and what they’re doing in the app.
Now, you want to be able to join those two data sources together so you can get better insights like ‘what do people on iOS do in my app first compared to those on Android?’. So you start thinking you should stick the data from these two services in one data warehouse, but how do you move the data where you need it to be?
One thing you could do is to make a simple script that runs every 24 hours, to export the data from your two service databases to files, and import it to your data warehouse that way.
This works fine, but what if you can’t wait up to 24 hours to get new info about installs and activity? Maybe you’re spending a ton of money on ads and want to be sure you’re not wasting it during a particular spike in traffic. Well then you might think about using Kafka to do it in real time. You can have one Kafka stream of install events—so every time an install happens, your install tracking service sends that event down the pipe labelled ‘installs’. Similarly, every time someone views a page in the app it gets sent down the pipe labelled ‘activity’ by the activity tracking service.
At the other end of the pipes, your data warehouse gets the events and sticks them in the relevant tables, so you can start querying them very quickly after they happen.
But! Your data warehouse doesn’t have to be the only thing sitting at the end of this pipe. You can have as many services as you want consuming the events. Want to trigger an email when someone installs your app? No problem, just make a service that consumes events from your existing ‘installs’ pipe and send an email whenever you get an event from it.
This helps you decouple services, because your installs tracking service doesn’t need to know anything about your email service, or any other service at the other end of the pipe. Nor does it have to know about the activity tracking service. It just has to send messages for the type of events that are within its own scope. What other services do with those messages is up to them.
Then the responsibles people look at the drawings and decide what to do, like giving you ice-cream if they see a lot of flowers on the drawings.
You have a lot of people who want to do the drawings and they can work together to make bigger and more beautiful drawings.