... and, you haven't ever seen a business requirement of what used to be a "software" to become a framework / plateform instead ? it happens all the time
But I've never seen it happen in any way close to resembling what the original architect thought would happen, and as a result, all those abstractions and generic implementations not only added time to the mainline development, but in the end actually got in the way of the abstraction that was needed.
If you can decide on a language, framework, critical libraries etc. then you should be able to decide on a database. It's probably more important than your application anyway.
This also improves efficiency in operations, not necessarily development. If you used a library/framework for database access anyway, it's not an extra expense. There's ultimately a portability concern even if "vendoring", it only imposes a cut-out to permit control of necessary change).
After a few unpleasant experiences I endlessly advocated we should always use an interface to access populated data objects and not interact with the database directly, not even running queries directly but always using at least lightweight IOC. I also advocated for testing where known result sets were fed through mock objects. After all, saved result sets could also be used to test/diagnose the database independently after schema/data changes. My experience predates a lot of ORM and framework responses.
Unfortunately later frameworks (intended to abstract these concerns) became ends in themselves, rather than a means to an end. These were used to satiate "enterprise-y" concerns (sacrifices to the enterprise gods). If you could afford to deploy operational Oracle, you would't necessarily flinch at the cost of the extra (often pointless) layers of abstraction.
At the same time, substituting Oracle with a lightweight DB in an environment where full-time DBA was developing a database with loads of stored procedures and DB-specific stuff wasn't something really feasible - no abstraction layer would solve that.
But as you said, some of the problems were when it got really DB specific. A simple layer that could be swapped for a simpler (and not performant, but didn't matter with little data) variant locally was nice.
DRY and YAGNI are two basic concepts I've always tried to work with. Do I need this piece of code more than once? Should be extracted/created in a separate place. Would I need this in other projects? Library.
YAGNI is most likely directly conflicting with most of what's mentioned, if the developer simply asks themselves: do I really need this? Will this project benefit from having separate layers of abstraction for things that are most likely never going to change?
I always think twice before writing a single line of code, with the main point of focus being if future me (or anyone who reads that piece of code) will be able to understand and change things, if needed.
Just because there is a block of code that’s being used in two different places doesn’t mean it should always be abstracted out. There’s a subtle yet mindful consideration whether these two consumers are under the same domain. Or if it should exist as a utility function across all domains. And if that’s the case, changing the code to make it generic, simple and without side effects is ideal.
I’ve seen too many of these mindless DRY patterns over time, and they eventually end up with Boolean flags to check which domain it’s being used in as the codebase becomes larger.
I also propose LAWYD - look at what you've done, a mandatory moment of honest reflection after drying out some code where you compare it to the original and decide if you've really improved the situation
And it sure is easier to just not argue about it and abstract everything. The real pain only comes later, after all.
If they absolutely have to be the same, wrap'em in a function. If they just happen to be the same, leave them as separate copies. If I don't know, make an educated guess.
Without DRY, this would be a perfectly acceptable practice. DRY gives me something I have in my head when I see this and refactor into something that is manageable. DRY gives me something I can point to and say "please for the love of god don't perpetuate this folly".
Long-lived OSS software is a relatively stable bet, but the point still stands.
Exactly, so make the changes when you need them, otherwise you are relying on hitting the abstraction lottery
I feel the industry has a long hangover from the 80s and 90s in terms of the holdup problem. Oracle, basically, created a massive externality of anxiety about vendor lockin that continues to impose drag to this day.
In the majority of cases, I think a simple “runs on Postgres” is even more valuable today (unless your product is a database abstraction layer).
This is a clear example of where YAGNI applies, I think.
Extra work plus extra boilerplate to maintain. No payoff.
That said, extra code + boilerplate is a great way to treat riskier software.
You can argue about that just the same as when client needs a feature X.
In my case, I was very often getting away with YAGNI or 'how about we implement nightly sync from pg to oracle' but not always.
One is that you're using two SQL databases and you're not using any advanced features. You started dev on MySQL, and then the company says "Thou shalt use Postgres" (or whatever). You don't need anything fancy in your own code to handle this. You're still making SQL queries, you're just swapping out the database engine. Technically, this is an example of dependency inversion (depend on the SQL abstraction), but you also didn't set out to do it - basically any programming language you're likely to use has the common database libraries use a common abstraction for sending queries. And you didn't specifically make sure you were writing generic SQL, you just happened not to need anything.
More commonly, you're switching databases for a specific feature. Maybe you realize PostGIS (or whatever) is going to solve a problem for you very well. But then you're changing how you model data, what your schemata are, and even how your code is architected and accesses things in the database. You're deciding to move certain logic from your code to the database engine - might even decide to move certain logic from the frontend into the backend, or change how request routing works, or something. This is a fantastic reason to move databases, but no amount of abstraction can prepare you for it, because you're fundamentally changing what the abstraction is. And you're deliberately abandoning SOLID because you're picking up a dependency on a concrete database.
But the real case I've seen is where you're switching databases (or data storage layers, more generically) to a different model - MongoDB to not-MongoDB, a C/P database to an A/P one, a relational one to a key-value store, etc. This is the above case but even larger. There is no abstraction you could possibly write that could encompass the old and new cases. It requires rearchitecting how your code works.
And then there are the most boring of cases - the ones where a database swap sounds doable in theory and the code is supposedly using an abstraction layer, but no one has ever verified that the code doesn't make assumptions about what database it's on and the code has gotten too big, so we just get an architectural exemption from "Thou shalt" and we run our own instance of the wrong database, because the overhead of running our own DB costs the business less than getting the swap wrong.
(Some public examples of these sorts of database migrations that come to mind: https://slack.engineering/scaling-datastores-at-slack-with-v... is about how Slack couldn't move away from MySQL and the architectural assumptions they made about it and had to rule out migrations to non-relational databases out of hand, and https://about.gitlab.com/blog/2018/09/12/the-road-to-gitaly-... talks about how GitLab moved from NFS storage to an RPC service, requiring a lot of refactoring of callers.)
FWIW, every single db abstraction I’ve ever witnessed was worth it - if only so that one could run tests in a sqlite and run prod in something else, or as a way to contain vendor lockin in the code (I’ve seen projects successfully migrate from a plsql-heavy system to mysql because the code was well segregated, and I worked at a startup that literally imploded bc the database was metastasized all over the place)
Anyway, as I put it, abstracting data storage is a no-brainer for me, and it saved my skin every single time. I don’t expect to convince anyone here to go do it. :-)
What were the reasons for the swap and what were the outcomes of it besides that it actually happened? Costs, performance, scalability etc?
I think hardly anyone would argue that those transitions are impossible, because they are indeed possible. The main question is usually if your abstraction can leverage the benefits of underlying implementation or it is just a common denominator. I can imagine a clever technical strategy in a startup, foreseeing future growth and starting with a simple DB before moving to a more complex in maintenance but scalable solution. This may work. Often it doesn't.
How do you avoid the issue where SQLite doesn't support all the things your real DB supports? (i.e., why is it worth running tests on SQLite as opposed to a local instance of the same DB software?)
I'll read some of these public posts....
WRT sqlite, the obvious advantage is test setup doesn’t require any local infra, and it forces your system to NOT use vendor-specific features. That school of design is essential for places that want to avoid lockin for whatever reason. This isn’t possible in every case - some companies have a great DBA team, for instance, or strong specialization around a specific db (cough Oracle shops cough), so obviously that flexibility wouldn’t be advantageous. As everything else in our industry, it depends :)
Abstractions are a way of trying to find a common ground conceptually. Agreed, abstractions should not be multiplied unnecessarily, but in the same way it's easier to learn Newton's Laws than read Kepler's data, a few well chosen abstractions help others organize and understand what's happening in a code base.
What's always amused me, in a tortured sort of way, is that SQL is already the abstraction. Then people go and layer an abstraction on top of it, such as some ORM flavor-of-the-day, which is obviously much less universal than SQL. In "SQL" I'm including vendor extensions too (Oracle, MySQL, etc.). It's easier to read up on vendor extensions than learn yet-another ORM.
Which do you think has had the longer shelf life: MySQL or some random Ruby ORM? If you've been doing MySQL since the '90s then it's largely the same as MySQL of 2021. That Ruby ORM? Probably hasn't seen an update since 2008. I can't even remember the names of all the ORMs I've had to use over the years.
class User {
static get(id) {
embed your SQL here
}
}Literally impossible to get any new clients to agree to run on oracle
That's what I mean when I say simple. Don't obscure your code with unfalsifiable assumptions reified, as related in the typical obscure vernacular...
I have had to deal with RDBMS-swapping for fairly large and transactionally intensive applications. I appreciated code that embodied SOLID to the extent that it reduced the amount of code to inspect and improved the quality of sizing up what needed to be done. However it was much easier to "lift" up systems to the new objective than "unravel" previous efforts to insure portability without a defined and testable objective of portability.
Ultimately the goal of SOLID is to be able to change any important aspect of the software independently from any other. If the DB is going to outlast the business logic you’re writing, there’s no problem having it depend on the DB concretely.
Separating the what and the how does have the effect of changing how you approach reasoning about a program though, so SOLID is not without penalty.
Edited for clarification.
I wonder if others here have applied some sort of domain driven design (domain models agnostic of database) without going down the whole repository route.
How can you be fed up with something that will never occur?