Fortunately, with managed DBs like RDS it is really easy to run individual DB clusters per major app.
Being shared between applications is literally what databases were invented to do. That’s why you learn a special dsl to query and update them instead of just doing it in the same language as your application.
The problem is that data is a shared resource. The database is where multiple groups in an organization come together to get something they all need. So it needs to be managed. It could be a dictator DBA or a set of rules designed in meetings and administered by ops, or whatever.
But imagine it was money. Different divisions produce and consume money just like data. Would anyone imagine suggesting either every team has their own bank account or total unfettered access to the corporate treasury? Of course not. You would make a system. Everyone would at least mildly hate it. That’s how databases should generally be managed once the company is any real size.
Decades of experience have shown us the massive costs of doing so - the crippled velocity and soul crushing agony of dba change control teams, the overhead salary of database priests, the arcane performance nightmares, the nuclear blast radius, the fundamental organizational counter-incentives of a shared resource .
Why on earth would we choose to pay those terrible prices in this day and age, when infrastructure is code, managed databases are everywhere and every team can have their own thing. You didn’t have a choice previously, now you do.
You DO have to share data in other ways, usually datawarehouse or services, but that is not the same thing.
I’m not saying literally every source of data has to be shared and centrally managed. I’m also not saying “rdbms accessed via traditional client and queried via sql” when I say database. I’m just saying a shared database of some shape is inevitable.
Also, operationally it’s not “semantics” at all. You don’t get into (many) operational problems with analysts sharing a datawarehouse. You absolutely do with online apps sharing a rdbms, they aren’t the same thing.
A data warehouse is a type of database and is does need to be managed. Your assertion that it is easier to manage is orthogonal to my assertion that there will always be a central database to manage in an organization of decent size.
If you can't do something like determine if you can delete data, as the article mentions, you won't be able to produce an answer to how to deal with those problems.
This is rarely a problem when things are small, but as they grow, the bad schema decisions made by empowering DBA-less teams to run their own infra become glaringly obvious.
In the kitchen sink model all teams are tied together for performance and scalability, and some bad apple applications can ruin the party for everyone.
Seen this countless times doing due diligence on startups. The universal kitchen sink DB is almost always one of the major tech debt items.
Multi-tenant DBs can work fine as long as every app has its own users, everyone goes through a connection pooler / load balancer, and every user has rate limits. You want to write shitty queries that time out? Not my problem. Your GraphQL BFF bullshit is trying to make 10,000 QPS? Nope, sorry, try again later.
EDIT: I say “not my problem,” but as mentioned, it inevitably becomes my problem. Because “just unblock them so the site is functional” is far more attractive to the C-Suite than “slow down velocity to ensure the dev teams are doing things right.”
Full Stack is a lie, and the sooner companies accept that and allow people to specialize again, and to pay for the extra headcount, the better off everyone will be.
This is how you end up with the infamous "jira and confluence have two different markdown flavors" issue.
This will undoubtedly go over poorly, but honestly I think every data decision should be gated through the DB Team (again, if you have them). Your proposed schema isn’t normalized? Straight to jail. You don’t want to learn SQL? Also straight to jail. You want to use a UUIDv4 as a primary key? Believe it or not, jail.
The most performant and referentially sound app in the world, because of jail.
Uuids are really for external communication, not in-system organization.
They require periodic synchronization. What isn't a big deal at all and is required by many other database features.
PlanetScale uses int PKs [0], and they seem to have scaled just fine.
[0]: https://github.com/planetscale/discussion/discussions/366
[0]: https://www.percona.com/blog/uuids-are-popular-but-bad-for-p...
[1]: https://www.cybertec-postgresql.com/en/unexpected-downsides-...
[2]: https://www.2ndquadrant.com/en/blog/on-the-impact-of-full-pa...
DB team could act as an auditor and expert support, but they should never be fully responsible for DB layer.
That’s the point. Would you send a backend code review to a frontend team? Why do DBs not deserve domain expertise, especially when the entire company depends on them?
> they are not responsible for the whole product, just for the database
I assure you, that’s a lot to be responsible for at scale.
> DB team could act as an auditor and expert support, but they should never be fully responsible for DB layer.
Again, the issue here is when the DB gets borked enough that a SME is required to fix it, they effectively do become responsible, because no CTO is going to accept, “sorry, we’ll be down for a couple of days because our team doesn’t really know how this thing works.”
And if your answer is, “AWS Premium Support,” they’ll just tell you to upsize the instance. Every time. That is not a long-term strategy.
I wish and maybe there is a programming language with first class database support. I mean really first class not just let me run queries but almost like embedded into the language in a primal way where I can both deal with my database programming fancyness and my general development together.
Sincerely someone who inherited a project from a DBA.
Not quite embedded into the OS, but Django is a damn good ORM. I say that as a DBRE, and someone obsessed with performance (inherent issues with interpreted languages aside).
I have worked in many languages with many ORMs and this has been my personal favorite.
[0]: https://github.com/prisma/prisma/issues/5184#issuecomment-18...
One thing that has worked well for us is to alway include the top-most parent key in all child tables down yhe hierarchy. This way we can load all the data for say an order without joins/exists.
Oh and never use natural keys. Each time I thought finally I had a good use-case, it has bitten me in some way.
Apart from that we just try to think about the required data access and the queries needed. Main thing is that all queries should go against indexes in our case, so we make sure the schema supports that easily. Requires some educated guesses at times but mostly it's predictable IME.
Anyway would love to see a proper resource. We've made some mistakes but I'm sure there's more to learn.
With that said, this still sounds like a strange situation - most colleagues, acquaintances and people I consulted know they way around SQL and dropping down to 'dbset.FromSql($"SELECT {...' is very commonplace out of the need to use sprocs, views or have tighter control over the query.
But schema design is something else. I still take my time doing that.
Especially since our application is written with backwards compatibility in mind, so changing schema after it's deployed is something we try very hard to avoid.
But yeah, when hiring we require they are comfortable writing "normal" SQL queries (multiple joins, aggregation etc).
The neat thing is, you don't. Nobody ever avoids fucking up db design.
The best you can do is decide what is really important to get right, and not fuck that part up.
P.S. to the original person concerned about this though… for your own sake and your successors, please keep trying.
Just do the exercise of deciding what is really important first, so you can make sure you succeed for that stuff.