SQL Is No Excuse to Avoid DevOps
queue.acm.org
queue.acm.org
I've lived in the world of Rails and Django so long, I sort of take schema migrations for granted. It's interesting to see such a high-level person covering this ground; it reminds me of how much these frameworks have given us, and how so many organizations are still practically emailing around zip files of source code.
His Technique 2 feels harder to me: write the code to work on both old & new schemas, and deploy the schema change after the code change. I know this is what Google does though, and I think it's the only safe way to deploy rollback-able code changes. I'm curious who else is doing it, and especially how you are scheduling it. Especially with a CI/CD workflow, I'm used to getting things integrated and letting them get deployed "whenever". So how do you make sure the schema changes go out "later", and what gating is required first, and how do you get buy-in to spend time on that after the feature is already live? I'm not asking about the technical how-to, but the organizational/political how-to. For instance if you're using Pivotal Tracker, do you make a separate PT story? Leave the old one open until the schema changes are deployed? What does it concretely look like?
And there's lots more room for improvement over Rails and Django migrations.
Some of the problems:
- Annoying version number management
- Management of long "chain" of migration files
- Unnecessary coupling between ORM and migration tool
- Heavyweight process to generate and run each migration, which slows down local development
- Bad testability
Autosyncing local dev databases, and automatically generating migration scripts with a diff tool is a better approach, imho.
First, propose the new schema. You should check in a version of your schema that contains both the fields from your new and old code. You can't do certain changes in this phase - If you're deleting fields, you can't do that yet. If you're changing column types, you should be careful; Hopefully your DAO already abstracts this, but you may need to roll that code change out first.
Then, make your application use the new schema. Start writing to the new columns as well as the old. Prepare a migration that will copy any data from old columns that needs to be. Your application should use the old data as the source-of-truth during this time, but when writing, it needs to write both versions of those fields. At this point, you can still easily roll back, and may want to, if you decide to proceed a different way for development.
Next, flip the source of truth. Your new fields should be the primary ones, but the old ones still need to be valid. You should be prepared to roll back still - The previous release should still be usable.
Once you're certain you're not going to roll back, stop reading/writing the old data. This is your hard flip, but once it's started, you can proceed leisurely. It doesn't matter - You've entered cleanup phase. Once that software change is finished, you can do a second schema change to delete old fields. (This is where the significant value of Schemaless is - You don't need to do anything in this second change, just let old data naturally clear itself out. Schemaless actually requires more attention to schema than a Schema-driven database, for this reason, but the iteration time can be quite fast if you have a good schema-control system, like, say, storing protobuf definitions in your version control and following their update rules[1])
All in all, this requires at least two periods of schema change and one period that involves multiple software rollouts. It's not simple, but nor is it that complex - And if done right, it's easy to roll other software changes at the same time, never losing velocity (aside from that which is spent reaching this target)
All of this requires that your application is well formed. Your application needs to be properly modularized, use reasonable DAOs or otherwise have complete understanding of how the data is accessed. It may require coordination between two projects that move, while not in lockstep, with gates on the other.
[1]https://developers.google.com/protocol-buffers/docs/proto#up...
https://github.com/protocolbuffers/protobuf/blob/master/src/... // An UnknownFieldSet contains fields that were encountered while parsing a // message but were not defined by its type. Keeping track of these can be // useful, especially in that they may be written if the message is serialized // again without being cleared in between. This means that software which // simply receives messages and forwards them to other servers does not need // to be updated every time a new field is added to the message definition.
A bit more detail:
For those who don't work with protos, they're the thing that Google created to use instead of XML before JSON was a thing. To use them, you define a message schema consisting of fields, where every field has (1) a number that will uniquely identify that field in the message, (2) a type for the field, and (3) a name for the field. The protoc compiler then generates code that decodes and encodes that specific message type for you.
Now think about what happens when a new field is added to that message type and the code is re-generated. That change is probably going to be in at least three different binaries (and it's often more): the client, the frontend, and the backend. There's no way to guarantee that every single client, frontend, and backend picks up that change at the same time - it has to go out incrementally. So if client v2 talks to frontend v1 and gets routed to backend v2, frontend v1 damn well better not crash if it sees a new field.
It is utterly impractical when you're deploying to individual customer sites, for customers who agree to an upgrade on timescales ranging from three months to five years...
[edit] Part 1, the automated schema updates, I'm right alongside. I can't fathom how people can operate with manual schema updates all the time. But part 2 is IMO wishful thinking outside of a deployed environment your own team can upgrade pretty much at will.
I think we need a new name for developers who run their code in prod, as DevOps is diluted.
So, I’m this light name of the article is utter BS. As devs we deal with a lot of different databases, be is SQL, NO SQL or graph and dealing with them is part of our job. If all you do is run stuff and release - you don’t have Dec component, so you just Ops.
Been running on AWS for the better part of 8 years now and running multiple production environments for clients and as an employee.
Over the years, I have seen the term "DevOps" slapped on so many things I don't even know what it means anymore.
For some, it's a culture, for some, it's a team, for some, it's a role.
SRE is now getting the same treatment. No idea what it means most of the time.
That said, the easiest way to make yourself more 'in-demand' right now is to make it your title.
As someone who does ops and development, when I look around in the market and see companies hiring full time devops roles, I usually see big red flags. The only jobs that I tend to want lately are on development side at small companies or the operations side at larger ones.
A person who specializes in deployment automation, a company that has a culture in investing in deployment, a team that specializes in deployment automations.
As the article mentions, DevOps is a methodology. To quote the author, "The most concise definition of DevOps is that it is applying agile/lean methods from source code all the way to production."
How is your comment relevant to that?
I switched from dev to ops fulltime (as in not "DevOps") and I'm perfectly at home here. If anything, I'm the one bridging the gap between us and the development teams. My team thinks I'm an ops person who does development, but I don't really find that to be true.
There was no thought to high availability, security, scalability, reliable backups, etc.
I'm not saying all devs are like this, but that's just been my experience.
Behaviour depends on how incentives and constraints are structured.
Traditionally, operations had to keep things running and cared little about new functionality. Developers had to deliver new functionality and cared little about keeping things running.
Requiring developers attend 3am outage calls changes incentives, as does requiring operations to own (or get out of the way of) software delivery pipelines.
I disagree with this entirely. Devs _you have workd with_ might not have cared but it doesn't make the statement universally true. As a counter argument, some of the best software developers I've worked with were also very good at operating and debugging production software and the reverse has also been true.
I also don't buy that the two are a different mindset (at least in the domains I work in and care about). In my experience the very best people (whether they're working "dev" or "ops" roles or something in the middle care about the entire development and deployment lifecycle of the software they work on. Building a good experience in software includes thinking about reliability and availability and planning a reliable and performant deployment also requires that to be thought of in the application layer at some point.
Totally irrelevant to me, I've never seen a company like this in practice in my long career.
Typically they're shops that got very big very fast. They deal with problems of scale where schema changes are very expensive and data migrations can require weeks without heavy optimization.
They typically have tried to automate some stuff, but the crush of feature work to support the massively expanding business always came first. They usually have CI, but never enough tests (unit, integration, or functional) to deploy with any real confidence.
They used to deploy (by hand) every couple of days because it was easy. Then they just kept getting bigger and it became every week. Then maybe once a month.
Typically they'll talk about why they don't do continuous delivery or deployment because deploys are RISKY and the business depends on deploys being perfect.
I've seen this situation in video games (twice), a mobile app that had reached the 100 million user threshold, and an old school data marketing company.
It definitely is a thing that exists.
Sure some shockingly-behind-the-times companies exist, and may even be a non-negligible percentage. But I'm surprised this is is front-page HN because I assume that the engineers on this website know that you don't need downtime for a DB migration. It feels very common-knowledge among this community.
Time Management for System Administrators
The Practice of System and Network Administration
He is most likely paraphrasing many conversations he has had with people over the years
Never worked in government, I assume...
Niche industry, so still not very big corporations, probably explains it.
It's been so bad, I've completely distanced myself as a developer because of those that have far more political influence as to the use of the database and the effects it has on me. I can work on the API, Database and other services. I simply don't, I was hired as a UI/UX/JS Expert. That said, it's not always the most pleasant way of separating the people working on an application, compared to full-stack development.
I've been at this for over two decades, and tend to find it in lots of business development. Not so much in typical SV startups, but there's far more business development units out there than SV startups.
He either met a straw man or a troll.
WHAT ?
I'm working pretty much in an environment described in the article. Big bank, complicated database setup with multiple replicates and databases up to a size of ~25TB.
On a (pre defined) release weekend there are essentially two entities to be released: The application (obviously) and the database changes.
For any DB change a patch is created and tested through an environment of half a dozen instances. If bugs are found on already deployed db patches they are never fixed in the patch, but a subsequent patch (also tested through all environments) is created.
Needles to say that anything provided, which applies database changes is version controlled.
The idea to apply database patches manually is so damn outlandish to beggar believe.
Anybody doing this on a production database without a strictly controlled (and tested!) break-fix should be fired on the spot.
Instead of fixing the technical root of why there’s so much risk, most companies add the most possibly expensive resource out there to mitigate or understand it - a person. The more dysfunctional, the more people get piled on while nothing actually is fixed (this is a larger cultural enterprise problem that goes beyond database changes). Most decent data tier engineers I know don’t like to babysit DB changes yet almost everyone I’ve worked with had that as a large percentage of their time spent. Such a waste of time and talent.
But then there’s a few DBAs I’ve seen that basically don’t do anything productive by making everything that goes wrong a developer problem, refusing to help troubleshoot (even when the DB in question is not accessible to developers), and other not very devops-y behaviors (hostility-first operations that is). It’d be one thing if the DBs had some huge changes happen, but usually I’ve seen these folks hardly do anything visible to me as another infrastructure guy and the troubleshooting I wind up doing on behalf of powerless developers is something that would have taken someone more knowledgeable less than 5 minutes with a quick trace of DB queries in flight and some historical profiling.
But being change review Charlie is something that a lot of people do that, while necessary to some degree, becomes their entire job. That’s just frighteningly dull work to me.
In fact, the entire hospital had plans for multi-hour downtimes due to releases, not to mention department-specific contingency plans if the downtime took longer than expected.
Errors and mistakes happen, no question about that. And any ol' mistake certainly shouldn't be a firing offense.
But this behaviour is so grossly incompetent that I don't believe that it can't be helped with any training.
You're welcome to fire people if you think it's going to help. I'm just curious how you think a skilled, trained DBA is going to show up at your desk when that happens.
I agree with almost everything you said except this line. Usually, people with production access are (or should be) sensible enough not to touch the prod db with anything untested that might mutate things.
If they're not sensible enough for that, they shouldn't have had access in the first place, and instead of placing blame and firing people, I'd start thinking about /why/ that happened. Points to a more fundamental flaw in the Engineering. Firing won't fix anything.
Also note that the statement in question talks about an old methodology.
As for the article, good stuff. I enjoyed reading it.
MySQL perhaps, when even the addition of nullable columns without defaults or constraints requires a full table lock and copy?
That seems to be a common reason for stating online schema migrations are difficult (not impossible), but not a problem with many other RDBMS engines.
That won't save us from needing extra tooling like pt-online-schema-change or gh-ost for other types of schema changes in high availability environments, but it's a step in the right direction.
MySQL builds a shadow copy off the new table in the background, then swaps the new table in for the old table once the copy is complete.
DevOps is an analogue of Agile and Lean. Lean, which came from the Toyota Production System, doesn't just exist in one department and in one position. Saying that "DevOps is <x position> doing <x thing>" is like saying "Lean is a factory floor worker pulling an Andon cord when a product is defective". It massively understates all of the aspects of the method, and all the different roles involved, and how they get work done.
Nothing about these methods prescribes that you have to use X technology, or that there is just one position that does work in one way. They prescribe a method, and you choose how you implement that method, and lots of different kinds of people in an organization are needed to make it all work.
I started a Wiki recently to try to give some more in-depth understanding of DevOps (https://devops.yoga/overview.html). It's still pretty bare-bones, but I think you can quickly get the idea that DevOps isn't one particular thing, or one particular team, or one set of technologies.
Devops set about to deliver working software frequently.
Both optimise for frequent delivery. The theory and practice of how working is measured is where they differ.
Even ORMs make me nervous for similar reasons, since they essentially enable the application tier to pass into the database any/all queries and DML. But like migrations, I allow ORMs for the convenience they provide during development.
Ideally, the same build should be used in every environment and all the configurations such as db endpoint, username and password must be external to the built and fed from the environment. The migration command can be a part of application, but only invoked automatically in dev environment. Most of ORMs allow to configure whether schema initialization happens at the startup or not.
During deploy, the migration part can be executed separately in prod environment, before the actual deploy. Different tools provide different "hooks": e.g. heroku has release phase(https://devcenter.heroku.com/articles/release-phase), in spinnker you can add a separate stage for db migrations: https://blog.spinnaker.io/deploying-database-migrations-with..., which can use a different db user. In this case, migration functionality is disabled at startup in production environment and runtime db user need not have DDL privilege, because by the time it starts up, migration phase must have been finished.
In mysql (and I think also mariadb) ALTER TABLE cannot be rolled back, and also implicitly does a COMMIT on any in-progress transaction before altering the table.
In PostgreSQL you generally can do ALTER TABLE in transactions, but there are dome restrictions or exceptions. If the alteration is to add a value to an enum, it cannot be done in a transaction.
MS SQL Server seems to be OK with ALTER TABLE in the midst of a transaction, according to this article [1]. Oracle seems to be similar to mysql.
Even the ones that support this well, like MS SQL Server, have restrictions on other DDL, such as creating indexes.
I think I'd rather just assume no DDL in transactions and design my schema update procedure accordingly, rather than asking whoever is designing the new schema to try to limit themselves to changes from the old that avoid whatever statements whatever DB server we are using doesn't allow.
[1] https://www.mssqltips.com/sqlservertip/4591/ddl-commands-in-...
It also generally requires an abstraction between the DB variants.
That all said, I did not mean this to be that Docker doesn't provide some value. Just pointing out it is not a prerequisite. If that makes sense.
There are professional developers that use Select * EVER!
Maybe add write a simple Select, Insert and Update Statement to the FizBuzz test.
However, "SELECT * FROM SomeTable" can be used to determine if a table had data ( for "EXISTS" clause ) or to "COUNT" the rows of the table. It's industry standard.
You could also use something like "SELECT 1 FROM SomeTable" but it isn't natural.
If you always use column names, why does column order matter?
Another time when you don’t want to do select * is if you are using a database that does columnar storage like Redshift.
If you want to change it to SELECT 'EXISTS' then I wont be mad :)
There is nothing wrong with a select * when working with an unfamiliar data model for initial rough development to get your head around the data path. Just remember to refine your query to meet your performance requirements.
One would catch a lot of debate (never use an ORM), the other mostly doesn't (never use select *). Maybe most people that use an ORM don't connect these dots though.
function lookupFoo(baz) {
const sql = await db.init();
const result = await sql.query`
SELECT x, y, z
FROM foo
WHERE bar = ${baz}
`;
return result.records.map(mapResultsToFoo)[0];
}Hibernate doesn't for starters.
Old code from the 2000 PHP era is full of this. Never underestimate the horrors of "legacy" code bases, and never underestimate what people do in short lived "maintenance scripts" or "helpers" that are used well beyond the intended life span. Or "proofs of concept" aka "unbelievably dirty hacks to get a MVP" that end up being extended and extended to being the "real product" in the end.
For an MVP I will always use "select *" and friends simply for speed of development, but clearly communicate that this is in no way intended for production usage.
But it likely will be anyway.
I use a variety of ORMs and hand-rolled SQL, and often will do select . The times when I know there may be a performance impact, or I'm debugging an existing performance impact, hand-rolled SQL always is on the table, but often* the issue is either missing/poor indexes or simply table design that didn't foresee the current use case. There's been a couple of times where a stray select * was, really, pulling back unintended very large blobs and causing a performance issue, but it's been infrequent compared to index issues.
It's one thing to talk about good practices and another to build a large development team that can understand and follow them. By moving the responsibility to devs, you tax your senior devs, who have to monitor middle and junior devs more closely and sometimes micromanage them as a consequence.
A few schema changes a year are more easy to test, justify and —most importantly— track than many small ones going on at the same time. How do you even manage the schema version when there are multiple branches that require changes and you don't know the order they will be merged in? Instead of a version number, you start to use flags. How about deprecated features? How and when do you clean up? Something else that might be important, are performance optimizations.
I agree on the suggestions of the article and it's the way I try to write code that uses a SQL store, just saying that sometimes in real life you have to make things harder in order to stay safe.
Is "doing DevOps" impossible? "No! It's simply a difficult transition."
Oh, I see now. That makes it all better. Never mind that the author's "friend" never claimed it was impossible. Rather that it was fundamentally a bad decision.
It's like a parable, but a really bad one. I wonder when the author is going to bring up Agile and just assume we're all true believers and that anything that incorporates Agile must be good without question. Oh, there it is. Right there.
So the difficult transition of DevOps is presented here as necessary and appropriate because 1. Agile, 2. Faster, and 3. Because it avoids some incredibly horrible practices the friend's DBA team does.
1. I won't get into here. That's a different flame-war for a different time. But people should know better than to write articles with the unquestioning premise that Agile automatically justifies any and every difficult transition. 2. I'm highly allergic to justifications that involve moving faster. Most companies do not have a problem delivering value fast enough. They have a problem delivering any value at all, and delivering a reliable product at any pace. Faster time to value is the worst possible justification if you're trying to convince me to experience some pain.
I am a big proponent of DevOps, and I bake it in to all my work, both at my work-work and in my side projects. It's worth the difficulty to me because it allows me to reliably get my code from point a to point b. That it's perhaps faster (is it really faster if you add up the person-hours? I don't know. I don't really care) is a little extra niceness if it's there at all.
3. DevOps has basically nothing to do with the pain that's causing the situation described in this parable. DevOps won't fix bad DBAs or sloppy Devs. DevOps won't fix a manager that allows this kind of situation to happen in the first place. DevOps won't fix a company culture that allows Management to run this kind of a dev shop. It's like the "friend" here is dying of dehydration in the desert and this dude rolls up in his car and says, "Hey man, you really need to eat more vegetables and less red meat and exercise at least 30 minutes a day. It will totally change your life! Also, I have these essential oils that will cleans toxins from your spirit aura and help you focus on your life goals. Good luck!"
We use Liquibase, which is event-sourcing for database schema changes, and a database that supports online schema changes.
Surely it's the incremental cost you want to drive down. If there were only a fixed cost, it wouldn't matter how often you did it.
Either the author A. Met a crazy person, or B. Invented the anecdote for sake of what they beilieved to be a snappy title.
It is a bizarre premise, and its alive and well at an insane amount of shops today.
When I walked in the door at my last gig(and a few others), they were hand running all deployments, and often manually troubleshooting in production when it fucked up because they didn't have a QA environment.
I helped my current employer make this transition and it was similar to the description in the article. Controlling database change in source was foreign to them.
Also some of them would have some other problems to solve and would fire up the software on their laptop even a month later, meaning the venues would already have different DB schema. So to solve this I've done the following: -each venue would have a table for SQL statements to upgrade the software local database. Also this would specify what schema No is current one. -each venue would have a table for software upgrades, that's it inside a blob field the software was there for that specific version. The software on client laptops would also have a local database, used for aggregating reports from multiple venues mainly, but also to automatically perform a venue DB schema upgrade if required: -a table with SQL statements to upgrade the connected venue schema DB to latest one. -a table the software upgrades, the same blob as described above.
So, when the CTO (usually he was the one receiving the latest and the greatest) would get a new software version from DevOps, he would basically receive it by connecting his software to my DevOps DB. The software on his laptop would see that it has lower number than the connected DB schema No, access the upgrading fields, get the new software version in a temporary file and restart the software. The restart process would upgrade the software from that temporary file, run the necessary SQL commands to modify the local DB as well, if it was the case and launch the software with the latest upgrade. Now the CTO would connect to multiple venues, usually by just asking for the daily report on those and for each connected venue the software would see the connected venues schema No are older. Would run the necessary SQL commands to upgrade those venues, pushing the software latest blob as well and then get the records to compile the actual report. When the other members would connect to those upgraded venues they would receive the latest software as well same as the CTO received from my DevOps one. This ripple effect of upgrading software and schema was done without a single second of downtime from those venues.
Also the venues did ran a local software on their own, which was also restarted and upgraded as soon as it was detecting the changes, in the same fashion as the software on the client laptops was upgraded. In practice, since the venues were scattered around US and Mexico from Boston all the way down to California and Mexico City, took around 30 seconds each, as they were all connected through VPN. So the most downtime was experienced by CTO for like 2 hours (~100 venues), and usually the guy would do this during lunch leaving the laptop to do this while he was attending other matters. The rest were only seeing the message "Upgrade in progress, please wait" for only 30 seconds and was very acceptable. Also the local venue software was less then 5 seconds with this message up, since it was connecting to a local DB, not over VPN.
I did this in 2010, and while I departed in 2015 from that company this methodology is still in use without hiccups today by them.
> The command in array index n upgrades the schema from version n-1 to n.
> Thus, no matter which version is found, the software can bring the
> database to the required schema version. In fact, if an uninitialized
> database is found (for example, in a testing environment),
> it might loop through dozens of schema changes until it gets to the
> newest version.
1. Once you've accumulated years of migrations, it's unnecessarily slow to bring up a new dev/test db. The tool is potentially running many ALTERs against each new table, vs a declarative approach which just runs a single CREATE with the desired final state.
2. The migration tool does not have a way of handling databases that aren't in one of the well-defined versioned states. This can happen easily on dev databases, where a developer just wants to experiment before landing on the final desired migration.
3. Most of these tools require a developer to code (in a specific programming language, or a proprietary DSL) the migration logic, as well as its rollback. There's plenty of room for human error there, especially since most of these tools don't automatically validate that the rollback actually correctly reverts the migration.
In contrast, a declarative approach simply tracks the final desired state (a repo containing a bunch of CREATE statements), and the tool figures out how to properly transition from any arbitrary state to the new desired state. This permits the state to be expressed in pure SQL, with no need for a DSL or custom programming, nor any need to define rollback logic.
A declarative tool can also be used in an automated reconciliation loop, which can handle situations like a master failing mid-schema-change in a sharded environment. This is analogous to other declarative tools, such as k8s.
There are several recent open source projects using the declarative approach for schema management. I'm the author of one, Skeema [1], for MySQL/MariaDB. Others include migra [2] for Postgres, and sqldef [3] for MySQL or Postgres.
Although declarative schema management is only recently picking up adoption, it isn't a new approach. Facebook has been handling their schema changes using a declarative approach for over 8 years now.
[1] https://github.com/skeema/skeema
2. This is where having a ground-up script is helpful. If you can destroy/recreate a local dockerized db (as an example, if not specifically) that's far less of an issue.
3. Developers writing code against a SQL database, should understand the variant of SQL being used. This is also where things like code reviews can minimize the issues. I'd also suggest that issues would be caught in QA or UAT before production providing decent test coverage prior to production release.
As for your contrasting approach, what about the example in TFA where you go from a composite Name field into separate FirstName, LastName fields?
2. What if you want to capture the set of local dev changes, rather then destroying and starting over? Most migration tools that I've seen don't have diff logic built-in, so they have no way of saying "take whatever I did to the dev db, and capture that state in the filesystem, so that I can turn it into a pull request". Whereas in Skeema you can simply run `skeema pull development` and then `git diff` / commit / etc.
3. Agreed; but the more automated tooling, the better IMO. Otherwise it's easy for a code review to overlook something like the wrong table name or column name being used in a migration rollback, due to bad copypasta. And in an environment where schema changes occur quite frequently, I'd be surprised if QA always involves testing and confirming every single rollback.
re: your last question, that operation is a mix of DDL and DML... I'd agree that the declarative approach doesn't natively handle it, since DML isn't declarative. My personal advice as a database professional is to treat that example as a data migration, managed separately from pure schema changes. Migrations of this sort should be less common -- as well as riskier to execute due to deploy-order considerations.
FWIW, I'm approaching all this from a background of consumer-facing web/mobile tech companies, where schema changes need to occur quite often, and making them automated/self-service is game-changing. Processes/requirements will certainly be a bit different for traditional enterprise, finance, etc.
On a side note, it's really good to run into you here tracker1! It's been a long time. Are you still involved in the BBS scene?
On BBSing, on several BBS groups via FaceBook... I've been pretty hands off, would like to get a board up. I put bbs.io on freedns.afraid.org for dyndns support, and have BBS.land if I ever get around to doing anything with it. It's hard to find motivation after hours to work on stuff. Also been wanting to get my board back up.
Didn't realize you open-sourced tournament trivia. Was recently thinking of reaching out recently to try and get a bigger doormud instance running.
1. You can squash the migrations, or delete them all and start afresh. I have had to do this just once on a project after it was 4 years old, and it was easy to do (ie. it is relatively infrequent).
2. You can experiment by creating new migrations, and, when you’re done, rollback and create the one you want.
The third point I cannot address, as you were very much using Python/Django and I cannot imagine an easy way to use the database models cross-project. I would probably argue that you shouldn’t do this.
However, Django’s approach was largely declarative, but in Python not SQL. Further, the migrations were automatically generated as Python which made them easy to modify.
I've seen this at countless companies, and in my experience it gets ugly fast. You can either shoehorn all schema changes into Django (for example) even for non-Python projects, or you can have different processes for each language, but either way that's a losing proposition vs having a unified, language-agnostic tool.
The migration systems built into frameworks like Django and Rails are truly great for rapid development, but they can become a quite a burden later on.