How to make MongoDB not suck for analytics
scaleapi.com
scaleapi.com
MongoDB was actually the third try. My first two attempts were BigQuery and Keen, neither of which worked out because they support only one index - time. Users want to slice and dice by various axes! And there's an obvious additional index you need - "merchant" - which column stores usually say propose setting up isolated partitions for. If you do that, you can't ask questions across the whole system!
We ended up with Postgres. It was actually faster than MongoDB for simple aggregations, and joins made it much better/faster for complicated queries. Of course it only works quickly if your dataset fits in RAM, but terabyte-size instances are pretty affordable and give you a lot of headroom.
That was a couple years ago. I don't know what they're using now, probably the same. It was a frantic few weeks figuring out what was going to work - each of those systems made it to production and quickly discovered to be inadequate in vivo. If you're in a startup, even if you're using exotic NoSQL systems like Google Cloud Datastore or DynamoDB - just use Postgres or MySQL for analytics. It will work long enough for you to figure out something else when you need it.
On one hand you complain about using technologies before you have done a prototype and evaluated the product. Then you blindly tell startups to just use MySQL/PostgreSQL without having any idea of their use case or whether it matches their query patterns.
If you are a startup the right way to go is to document your use case, understand what queries those use cases demand and then find the right database that satisfies it e.g. don't pick MongoDB if you are doing lots of joins and don't pick PostgreSQL if you are doing wide-table feature engineering type analytics.
Right tool for the right job.
This is not the case for most of the NoSQL databases where you'll pay for lack of certain features either by a) having to write a lot of code, or b) bad-to-crippling performance for use cases it wasn't meant to solve.
So, unless you're already very clear on what your exact use case is going why the spend time analysing before even getting your project off the ground?
Because if you don't know what you want you are almost guaranteed to pick the wrong technology.
That's one way to look at it...but a bit shortsighted.
Requirements can and do change, and a well designed model in an RDBMS will be far more extensible than a similar one in NoSQL document store. So RDBMS' aren't the "wrong" technology, they the safest bet; not to mention most modern relational DBs already out-perform mongo, so the point is sort of moot anyway.
Because that sounds like magic.
Also MongoDB destroys any RDBMS (minimum 10x faster) if you have embedded structures instead of joining against 10 tables in a normalised design. Hence the importance of understanding your query patterns and domain before selecting the database.
Real world example: Consider an Order table and a Visit table; conversion rates aggregate orders over visits. In Mongo you can denormalize some of the Visit data into Order, but what happens when you change the logic for computing conversion ratios? Or you want conversion ratios broken down by web browser, source tag, or any of the other data elements that live in Visit but you didn't denormalize ahead of time?
Can you give a common example of these? This article is referring to issues related to row vs column data stores, not sql vs nosql.
* Column stores like BQ and Keen don't let you efficiently slice and dice data by factors other than time. If you're slicing by customer or product, your queries become incredibly slow and expensive. You start writing hacky shit like figuring out when your customer's first sale was so you can narrow the time slightly, but that barely helps.
* MongoDB doesn't do joins. So you denormalize big chunks of your data, and now you have update problems because 1) you have to hunt all that down and 2) you don't have transactions that span collections. Also the aggregation language is tedious compared to SQL, requiring you to do most of the work of a query planner yourself.
* Some other person in this thread said MongoDB was faster than Postgres, but I found quite the opposite to be true. For the same real-world workload, basic aggregations on an index, we found Postgres to be much faster than Mongo. No idea what that other person is talking about.
I would argue that since both Mysql and PostgreSQL are JSON document stores with mostly the same capabilities when it comes to querying and aggregation I don't see the advantage of using MongoDB at all.
I wouldn't even use MongoDB for caching when redis does a better job at it. Logs? I don't see why logs cannot be shoved into a RDBMS. Prototyping? create a table with a JSON field and a primary key. Distributed file system? I don't know any business which uses gridFS as a CDN, full text search? PostgreSQL does it better. So what is the job your are talking about? PostgreSQL is so much powerful for analytics because of the power of SQL.
I agree with your overall argument of PostgreSQL & MySQL >> MongoDB(for querying and aggregation). But in all the experiences I’ve had doing analytical work with both; Postgres easily comes out ahead. If you’re starting from a blank slate, I’d definitely recommend it over MySQL. Just update/feature addition rate along with the better community quality are enough for me to prefer Postgres over MySQL.
With two engineers starting from scratch, we launched a product in three months that was making millions (per month, in profit) by six. When I hear "done a prototype and evaluated the product", I think you operate on a very different kind of timeframe. We needed a customer-facing analytics solution ASAP for the sales that our customers were already making.
This is why Postgres would have been the right choice from the start (mea culpa). It may not be the best solution, but it will be an adequate solution and get you through enough scale that you can worry about the million other holes to backfill. I thought I was being clever with BigQuery; it looked great on paper. Keen has better marketing but really the same problems. Mongo at least I was familiar with going into it, and that solution worked for a couple months, right up until the queries got complicated. Which in retrospect they were always going to.
In a runaway startup you're not going to be an expert in everything, or have time to plan out an optimal solution. Pick technologies that you can be confident will be "good enough for now" and give you time to find the boundaries of your particular problem domain. Postgres is a good axe to start with.
One of the top queries will be "show me my all-time sales". If your only index is time, you will touch your whole database every single time a customer asks for it...
It can join with the $lookup function these days. Although it is only to a "non-sharded collection". I don't know why it can't join to a sharded collection when the join is on the same shard though.
There is also the option of using $in with a list of things you have pulled down in another query.
Then there are client-side joins.
AKA what you are doing when writing your SPAs with their own state management. Server-side joins rarely make sense in that context.
For reporting/analytics.. yes. But these can be delegated to external system/databases optimized for that task. With elasticsearch for example you get very far very quickly without the need to write any SQL joins.
We didn't look at MongoDB due to too many people on the team having been burned by it in the past, but after looking at a lot of stuff we settled on Redshift with a Postgres database in front of it using dblink and foreign data wrapper. This allowed us to have extremely fast reads of common queries with only a 5 minute lag on data becoming available.
It's amazing what you can build with Postgres. I might write up a blog post about our exact strategy if that would be interesting to people.
Also it's SQL, what is preventing anyone from searching on any field they need? You don't need indexes for that. BigQuery only supports partitioning by a time-based column but that's more for cost control than speed, especially in your case where the dataset is small enough to be handled by postgres in the first place. Generally a mainstream RDBMS is the best choice for all things if the data fits, just because of the performance and usability available today.
table scans are fast in modern columnstores
I guess that depends on your expectations of 'fast'. Even with our smallish dataset, both BQ and Keen had multi-second responses -- frequently 10+s. It was totally unacceptable for user-facing analytics. And we had a lot of customers making a lot of queries - it started to get expensive fast.
I'm sure 10s responses would be very 'fast' for terabyte-sized data volumes. But that's not the problem we were trying to solve.
Keen isn't a columnstore, it's a custom database built on top of Cassandra where they take JSON records and split them into compressed batches with each unique property stored in the CQL data model, and it's processed by Storm workers. It's an outdated architecture compared to modern columnstores that can now handle unstructured/nested data really well.
BigQuery is designed for throughput instead of latency. There is a minimum 3-5 seconds to schedule your query across the server pool before it even starts processing. It's also a single shared cluster for all customers so performance is variable, but the trade-off is that 100TB also takes seconds to scan.
In my mind there has to be a decent "business intelligence stack". I'm not sure I'm coining that because I didn't get good search results from that phrase. Believe me I've been trying to find solutions. I believe there is big opportunity in building out this sort of stack that bridges data management and data analysis. Sure you can call IBM, Microsoft, Dell, HP but be prepared for big costs and huge software bloat. I would like simplified solutions and options that can fit with most industry standard tools.
I'm also willing to work with anyone on this as well.
So may one only use it with a subscription?
I got mongodb postgres foreign data wrapper[0] working in a previous life.
You can also use something like NiFi which supports MongoDB and will allow you to shift data out to Avro/Parquet on HDFS/S3 for your data scientists to use.
As for BI stack. Not sure what you mean. There are hundreds of tools which blend data management and data analysis. You can do this with Hortonworks (Atlas + Spark) or Alteryx for example.
But I will say that many moons ago when I did actually write stuff for Mongo. The oplog was a god send. You can "tail" the oplog, and get every transaction in near real time. We used this for updating Elasticsearch indexes etc in what is basically realtime, without having to poll or modify existing code at all.
Why not query the data directly in MongoDB?
Edit: changed "offline" to "non-realtime"
But even if you did that, you'll find that you'll need joins and aggregations that are painful to do in Mongo yet trivial to do in a system that is designed for them.
- hybrid transactional and analytical processing (HTAP), coined by Gartner, - hybrid operational and analytical workloads (HOAP), by 451 Research - Translytical, by Forrester
If that's the solution you want to explore, TiDB (https://github.com/pingcap/tidb), the open source distributed scalable HTAP database, might be able to help you. ETL is no longer necessary with TiDB’s hybrid OLTP/OLAP architecture.
Here is a use case about how it helps the largest B2C fresh produce online marketplace in China to acquire real-time intelligence:
https://www.datanami.com/2018/02/22/hybrid-database-capturin...
Here is a tutorial about how you can try TiDB/TiSpark on your own laptop using Docker Compose: https://www.pingcap.com/blog/how_to_spin_up_an_htap_database...
Disclaimer: I work for TiDB.
The past solution of the separate operational database and data warehouse poses great challenges for real-time analytics because it needs either data pipeline or the ETL process which could be the bottleneck of being "real-time", not to mention the waste of time, efforts and human resources maintaining multiple data warehouses. It was impossible for real-time analysis because, in the past, you would need a data pipeline, or message queue with the equivalent throughputs with your OLTP database, which I believe does not exist.
However, whether to adopt this hybrid solution depends on your specific usage scenario. For cases where users want to do real-time analysis in their data warehouse upon the same data table as in their OLTP database, TiDB is your choice.
Then they’ll build their BI models in some high level drag and draw system, and pay extra whenever they realize they didn’t get everything they needed in a cube.
The only place I’ve seen actual data scientists is at the university or at the 100% software companies that sell both the solution and the data cube. I’ve never met a real world analytic who could actually code. :p
You’d want to keep a separate dB for your analytics either way though, as they typically eat up quite a lot of load and you don’t want that to interfere with your production environment when you don’t have to.
Problem is that it doesn’t work for analysis and we are still using our old platform, which works just fine.
Seems to me that it’s on your devs to explain to the business why their poor technical choices now necessitate a substantial additional investment to get a usable solution. When they could have just used Postgres, and they knew it.
Quite often they will have also read somewhere that joins are slow and have managed to convince themselves that the solution is to avoid relational databases altogether. Or maybe that's just how they justify it.
If the devs want to use Mongo, it's their problem -- it shouldn't matter much to the analytics people, because they can just copy the data into a different database that fits their needs.
Certainly sometimes you are in a position where you just gotta take what other units in the org give you and deal with it.
But _someone_ in the org is hopefully in the position to be able to articulate why they are using mongo in the first place...
That's easier said than done when your database is over 10 TB big.
If you are really generating 10TB of data more than a few times a day, you can look into putting it in something like Kafka for real-time consumption by the analytics team instead of batch copying.
In general, you probably should have at least something in your stack which reads all changes from your DB, at the very least for backup reasons.
Automation here (in my case) is a net win on time.
It's sad that, for a backend DB, correctness can be trumped by marketing.
It's like if you ask "how do I drive my car downtown" and I answer, "Easy, just park at the station and take the train".
To answer your other question, their marketing goes a long way. I recently started at a new company, and the lead was proudly telling me how the project was developed using Mongo... So I start explaining how it's basically shit after using it professionally for a few years. His answer? But SQL doesn't scale well enough!
Pros:
- an 8 CPU installation with 64gb memory will probably be hundred times faster then postgres.
-it supports full sql
- It is super stable, even as docker container
Cons:
- it does not support nested data
- once you reach volumes of around 2Tb, you will probably have to switch to a paid version (I mean, you still can continue running on a 200gb ram box, but it will be suboptimal)
P.s. I am not affiliated with Exasol.
"Probably" not.
The way this usually goes down is that there may be a few synthetic benchmarks show a large performance benefit over existing established databases (x2, not x100), with any non-synthetic benchmark showing very poor performance (1/10th, 1/100th, sometimes even worse), and also often very unstable performance.
The product is then also usually beta quality, as it is hard to compete with the 36 years Postgres has been in development since its inception in 1982 (and that's not counting the 9 years of Ingres development, which Postgres—"Post-Ingres"—spawned from). Important features are usually also quite lacking.
If someone claims x10 or x100 performance improvement over established databases, they better have published a few papers about all the computer science research they must necessarily have done to get there.
Would you mind sharing some of the differences to, say, Postgres, and what to expect if moving from Postgres to Exasol? Porting my applications to Exasol to benchmark would be time consuming (synthetic benchmarks are very uninteresting), and without any information about what to expect, it simply wouldn't be sensible.
I tried to look at the website, but I am not interested in accepting a privacy policy just to get a white-paper, which frankly leaves me with no usable information at all. The rest of the website is basically empty, short of graphs without data and marketing "You want to do X? We can do that too! <no additional info>". The only real thing I could extract was "in-memory database".
To me, "in-memory database" would appear to be the catch that makes it an entirely different product than Postgres, catering to an entirely different payload with different pros and cons, rather than an faster all-round product. None of my tables fit in RAM anyway.
Random aside, how do you handle the consistency problems that can occur when you have multiple views when doing deletes?
I had data in PG and Mongo, but couldn't join it together. I asked about this on the forum, was told it's a known issue; and it seemed to end there.
I resorted to doing my analytics by hand in the end, MongoDB's aggregation framework is good enough. Create views from aggregation queries, and it becomes easier
The downside is that one needs a business license to use the BI connector.
Have you looked the postgres mongo fdw[0] before?
Baby's first ETL -- just scan the db with a cursor and analyze the data in a script -- tends to cover 90% of the use cases for BI db analytics with almost zero resource consumption anyway. Point being don't write a query to do analytics if your db can't answer your questions performantly, and don't build [latent, stale, slow] Enterprise ETL unless you really need it.
Disclaimer: I am not affiliated with Yandex in anyway, just a happy customer
Regardless of the choice of primary database, this is nothing new and just shows how a lot of startup technical talent seems to be discovering the same things all the time, usually with needlessly convoluted approaches, and writing blog posts about it.
If you have a Spark platform in place, it's a decent solution for this.
I suppose you could choose one of those options and sacrifice either your customer experience or your analytics, but it's probably better to use the best database for each use case.
Easy: you don't use Mongo