Does each microservice really need its own database? I have recently proposed my team initially not do this, and I'm wondering if I am creating a huge problem.
If you share a datastore across multiple services, you have a service-oriented architecture, but it is not a microservice architecture.
Note that I'm saying this without any judgement as to the validity of either architectural choice, just making a definitional point. A non-microservice architecture might be valid for your usecase, but there is no such thing as 'microservices with a shared database'.
It's like, if you're making a cupcake recipe, saying 'but does each cake actually need its own tin? I was planning on just putting all the batter in one large caketin'.
It's fine, that's a perfectly valid way to make a cake, but... you're not making cupcakes any more.
The main premise is independent deployability. You need to be able to work on microservice independently of the rest, deploy it independently, it has to support partial rollouts (ie. half of replicas on version X and half on version Y), rollbacks including partial rollbacks etc.
You could stretch it in some kind of quasimodo to have separate schemas within single database for each microservice where each would be responsible for managing migrations of that schema and you'd employ some kind of policy of isolation. You pretty much wouldn't be able to use anything from other schemas as that would almost always violate those principles making the whole thing just unnecessary complexity at best. Overall it would be a stretch and a weird one.
Of course it implies that before simple few liners in sql with transaction isolation/atomicity now become phd-level-like, complex, distributed problems to solve with sagas, two phase commits, do+undo actions, complex error handling because comms can break at arbitrary places, performance cam be a problem, ordering of events, you don't have immediate consistency anymore, you have to switch to eventual consistency, very likely have to do some form of event sourcing, duplicate data in multiple places, think about forward and backward compatibility a lot ie. on event schema, taking care of apis and their compatibility contracts, choosing well orchestration vs choreography etc.
You want to employ those kind of techniques not for fun but because you simply have to, you have no other choice - ie. you have hundreds or thousands of developers, scale at hundreds or thousands of servers etc.
It's also worth mentioning that you can have independent deployability with services/platforms as well - if they're conceptually distinct and have relatively low api surface, they are potentially extractable, you can form dedicated team around them etc.
I think the dogmatic “you always need a separate database for each micro service” ignores a lot of subtleties - and cost…
This is really over sold. You could allocate another instance to a specific service to provide more CPU to it, or you can allocate another instance to your whole monolith to provide more CPU.
Maybe if the services use disproportionately different types of resources - such as GPU vs CPU vs memory vs disk. But if your resources are fungible across services, it generally doesn't matter if you can independently scale them.
Compute for most projects is the easiest thing to scale out. The database is the hard part.
If you did things the "right" way to begin with. You have to keep in mind that many people in the industry need to solve problems of their own making, and this then doesn't translate or make sense to other people.
Scaling is a great example.
It's common to see web applications written in horrendously inefficient front-end languages. Developers often forget to turn "debug" builds off, or spend 90% of the CPU cycles logging to text files. Single-threaded web servers were actually fairly common until recently.
Then of course the web tier will have performance issues, which developers can paper over by scaling out to rack after rack of hardware. Single threaded web server? No worries! Just spin up 500 tiny virtual machines behind a load balancer.
Meanwhile, the database engine itself is probably some CotS product. It was probably written in C++, is likely well-optimised, scalable, etc...
So in that scenario, the database is the "easy" part that developers can just ignore, and scaling the web servers is the "hard" part.
Meanwhile, if you write your front-end properly, then its CPU usage relative to the database engine will be roughly one-to-one. Then, scaling means scaling both the front-end and database tiers.
For example stored procedures that due to not splitting the db get "shared" between micro-services.
There are also different ways to share. Are we talking about different DBs on the same hardware? Different schemas, different users, different tables?
If you want to be so integrated that services are joining across everything and there is no concept of ownership between service and data, then you're going to have a very tough time untangling that.
If it's just reusing hardware at lower scale but the data is isolated then it won't be so bad.
Additionally, you can probably get by having low criticality reports fed through direct DB access as well. If you can afford to have them broken after an update for a time, it's probably easier than needing to run queries through the API.