HSBC moves from 65 relational databases into one global MongoDB database

So in no way was the OP's article as catastrophic as it initially sounded.

> Historically, (...) HSBC did have an application core program environment, which had most of an application's core functionality. But it couldn't have a single programme environment running for all the countries, due to the differences in data models and databases.

> Another benefit is to use the same database for global data analytics and reporting. We don't need to translate into another data model or another database to run the analytics and reporting from that particular data.

So this "application core program" failed to provide an unified solution to their data warehousing needs, and now they believe they can solve this problem by unifying 65 databases into one MongoDB instance, "taking advantage" of its schema-less model.

It looks more like a global storage service than anything else to me.

"Local requirements for each country will be built into the application, but there's no need to maintain separate data models or separate databases anymore. We could easily design the global data model and database using the MongoDB JSON schema model. That brings data from all operating countries into one database and the application can run on just one database. Which is a lot of reduction in resource and maintenance cost. "

Are there any other schema-less databases out there that is better than MongoDB? If not I believe it's time for someone to build it.

[1] https://www.theguardian.com/business/2012/dec/11/hsbc-bank-u...

In short order, I cut the card up. Still get sad emails (which they provide no way to stop) begging me to use it.

Also, interviewed at MongoDB once. Couldn't shake the feeling of a cult, where everyone was careful not to speak of the product's serious limitations.

It used to be lots of fun and highly optimal to write all sorts of procedures and triggers because it was the only fast and reliable way to do things, especially when your in-database-code was essential the only gatekeeper against a multiple owner database schema.

The idea that you have one database schema that you are going to share with different applications seems bonkers to me at this point in time. Compute and storage resources are far cheaper than development, DBA and potential stagnation that a schema that doesn't have a single owner brings.

I'm a former Entreprise Architect from Banking Sector in Europe.

To be clear , this will never happen.

There has never been any directive that has had a "concrete" impact on IT Architecture of Financial Institutions.

Yet there has been dozens of regulations that urged banks to "simplify" their IT Systems.

90% of IT Staff are Baby Boomers with no background in Systems Design or Architecture , thus when new regulation comes in it follow this scenario 100% of the time :

- Find a vendor that sell a software that promise compliance with new regulation

- Find an integrator that promise integrations within the deadline

- Integrate the vendors software in Banks legacy stack of 3000+ monolithic apps

- Send report to regulator saying they have "redesigned" their architecture and made "investment" in order to take in account and that new regulation

Best examples of this is "PSD2" which has been the biggest fiasco of the industry , has of the last year only 18% of banks complied with the regulation.

France and UK said they would not "fine" anyone, because banks "aren't ready" and made "considerable" investment in it.

Regardless of what HN thinks , you won't solve Financial Institutions Multi Decade Legacy with a single Directive that would suddenly force them to use "ImMuTABLe Db ARChITectuRES"

It Directives have never worked and will never work.

The only way to enforce anything would be to remove their IT systems completely and have them use APIs provided by the regulators , and have the regulator become the sole provider of "Financial System".

They wont let that happen.

I will be tracking their technical reporting more closely over the next year.

https://youtu.be/GI5kwSap9Ug?t=63

But they already have relational models, right? Can't you port the DDL over to centrally managed instances?

Not to sound pedantic or old or curmudgeonly (because while I'm 45, I was already curmudgeonly and reading Fabian Pascal when I was 28). But c'mon.

All data has a schema, it just takes effort to discover it. A relational database is a set of facts about the world. Lack of effort in finding that schema and normalizing data, "conforming" it... it means effort from very frustrated people who have to clean up the mess later and make it conform.

FWIW SQL can seem like a clunky and dated way to talk about data. It is. But it's not the relational model's fault, it's SQLs's. Going "NoSQL" simply makes data modeling problems worse. It pushes the problem til later, and pushes reasoning about the data into places where it shouldn't have to be. Even more so when the underyling data storage tech is dubious, like Mongo appears to be.

Classic HN.

I'm not saying I'd choose Mongo specifically for this job (if for anything), but the lack of DB enforced schema might not be the biggest problem here.

Database alone isn't the deciding factor, how you used it in your application architecture matters the most. No database is perfect but some are more flexible than others.

Companies like IBM specialise in royally screwing companies through their licensing deals and with their legal enforcement teams. They are nothing like dealing with a normal startup vendor or with someone like AWS. They play hard ball.

So sure you can complain that they should've just stayed with DB2. But usually that means paying tens of millions which then increase each year since they know you won't migrate.

That's why companies take drastic decisions and move to the cloud even if it's technically not the best option.

if they've got a decent QA process and they can't get the new system to actually work reliably there's a fair chance this project would get stuck in QA for a year or two, blocked from going live and cancelled, so it wouldn't cause catastrophic data loss, just reallocate a bunch of money from shareholders to contractors / employees / vendors.

> microservice graph

banks have graphs of macroservices.

https://www.warrants.hsbc.com.hk/en/tools/search/ucode/00005...

I'm not sure about their choice of Mongo but the initiative seems like it should be helpful for customers like myself who have accounts in a few countries. We can access them all from one app but the experience is inconsistent at the moment. I'm cautiously optimistic.

Or it could be a real-time fraud system where you need every bit of customer data in one payload to feed into an ML model.

You can't do any of that if you are having to do multiple joins across federated databases.

Would really be interested to know how something like Mongo ends up getting seriously considered for something like this, I mean the whole decision process that goes in to selecting something like this.

[1]https://news.ycombinator.com/item?id=23285249

I don’t think you actually tried it. It returns user who’s id is “{$ne: null}” (string)

_ It's now a one service environment, one database and one execution path for all the countries. This is made possible because of MongoDB's document model and the ability to map all the different table requirements for each country into a single collection, using sub-documents. Everything is simplified into one collection using country specific identifiers._

How they will address race conditions, eventual consistency and so on is not addressed at all though. I guess they trust Mongo's new transactions will take care of that (at which point, it would be nice to know whether that actually worked and what the performance will be when compared to the old relational system).

What if... those models map to sql database schema’s? wouldn’t that be magical? Not since 2008.

Whats left is maintaining multiple databases. That sucks indeed.

Sql databases support bson as well these days.

What they probably want (or did) is make a base framework, suitable for all countries. And have each country develop its own stuff on it. No need to use mongo for that though.

One use case i can imagine is that forms and input just change and that old information never will be compatible again with the newer forms. Mongodb serves as a giant more or less queryable data bin. Then one can ask the poor dba to lookup something for a client with “client_id”: xyz. Then return the raw bson/json output. Might be sufficient.

At least HSBC are 'web scale' now!

Pretty common these days to just rely on Git and not DDLs.

Reference: https://www.youtube.com/watch?v=_nVk25ZvTkU

NoSQL has proven to be hard to get right than was first assumed and at the same time RDBMS’es have improved and adopted many ideas from the NoSQL world faster than expected, often with better results.

What’s no clear from the article is why MongoDB was picked, so maybe there’s something that made it the obvious choice, it’s just hard to see what the might be.

I’m skeptical about this - Google’s entire infrastructure is so large that surely they must be using a wide variety of db technologies?

I took a databases course that did mention that part of the Google search infrastructure runs on a tailor-made NoSQL technology, but it wasn’t really a doc-based approach, it was more akin to a relational DB but with flexible columns (can’t remember the details sorry).

At any rate I don’t feel that this is any vindication for NoSQL, as Google builds much of its underlying tech from scratch and its hardly comparable to the out-of-the-box solutions that the rest of us, even massive corporations like HSBC, have to work with.

Yes, keep an open mind, but please do not use a database that has a history of just losing data in mission critical software.

> there is no need to structure the data into tables and schema does not need to be enforced at the DB level.

There is no need to, just like your car doesn't need safety belts to function.

A database scheme is mostly a safety mechanism as it will prevent you from doing something stupid.

> Google has been running its entire infrastructure on a NoSQL database for years.

Do you have a source for this claim? I would be inclined to believe that the data for my Google account sits in a relational database somewhere.