Show HN: Rambler – A simple and language-independent SQL schema migration tool | Hacker News Reader

Show HN: Rambler – A simple and language-independent SQL schema migration tool

github.com

69 pointselwinar35 comments

I much prefer the methodology in Sqitch to the migration pattern.

While I wish the tool had been implemented in something other than perl (with 10 million CPAN dependencies), I like that DDL files are versioned and organized in ways more meaningful to the structure of the database rather than being bound to the timeline of the database.

If I am going to do migrations, a tool like Rambler would have to compare favorably to Flyway (https://flywaydb.org/), which is pretty solid.

i both like and dislike that the registry database is separate. i honestly can't with it how i feel about it.

The intrusiveness of the registry in terms of database configuration depends on which RDBMS you're using.

I use it with PostgreSQL and I don't use a separate database for the registry. I do have a separate schema for the registry in the same database as the data, but I would want that sort of organization no matter the case. If the database you use doesn't include some sense of "schema" below the database level of separation, I could see how that would be annoying from an administrative/maintenance point of view (or on hosted database solutions).

The testing framework of sqitch is pretty cool, I used a something to do database testing a long time ago, but I didn't though of integrating it in the migration tool.

Basically a copy of SQL migrate :

https://github.com/rubenv/sql-migrate

I'm not sure which tool came first in the Go community. While I'm not a big Go fan, this is precisely how Go can be handy sometimes, by allowing easy distribution of tools that don't depend on an interpreter or a JVM...

Wait, doesn't Go require you to install a Go runtime?

The point is you can release binaries for every plateform thanks to cross-compilation, no need to install Go on your machine. And no need to install a runtime no, the runtime is embedded into the binary.

Not exactly a copy, sql-migrate is based on goose, and does many things I don't want rambler to do.

I agree with the easy distribution thing, Go is awesome for this kind of things. I only wish the cross-compilation of binaries that embed C would be easier (XGo is the thing for tkhe moment).

I'm a biased fan of migration utilities like this (I've written a couple). I like using plain old sql for migrations, it makes more sense to me. Sure, you loose some DB mobility you get with the projects that have their own DSL, but then again I don't have to learn a new DSL.

I kind of wish I had kept developing http://jdc0589.github.io/mite/ the feature set is pretty solid even though much of its isn't documented in the site, but I got to the point I wasn't working with sql databases much anymore, so the interest kind of died off.

Rambler doesn't have any custom DSL, it use plain SQL. Just like Mite :p

yep, thats why I automatically liked it.

It looks interesting and I like the simplicity of the tool, looking forward for seeing more databases supported.

Besides needing the jvm installed, are there any differences between rambler and flyway command line [1] that make it a more suitable choice?

[1] https://flywaydb.org/getstarted/firststeps/commandline

Flyway author here.

Besides supporting many more databases, Flyway's parser is also much more robust with support for all kinds of database specific things like changing delimiters with MySQL, Oracle PL/SQL, PotsgreSQL COPY FROM STDIN, T-SQL, ...

And you can use truly plain SQL files like the ones generated from your DB's dump tool as a starting point, no special comments required.

(And of course you also have things like repeatable migrations, Java-based migrations (great for complex data transformations), API and build tool integration, ...)

All due respect to someone who has created a tool that does genuinely save me some work when deploying, but I can't internalise calling flyway a "migration" tool when failed deltas require such catastrophic manual cleanup. One typo or semantic error in a file and I get a corrupted stack that requires me to edit not only the file but the metadata table. I have started double-testing deployments to mitigate the awfulness, but are there any plans to ever make failed changes more graceful?

No need to ever manually edit the metadata table. That what Flyway's repair command is for.

However if failed migrations are a big concern for you, do yourself a favor and consider moving to a database that offers proper DDL transactions like PostgreSQL, SQL Server or DB2. Flyway runs every migration inside a transaction and this way changes become truly atomic without any sneaky implicit commits (I'm looking at you Oracle and MySQL).

The difference is in the number of features. Rambler doesn't do many thing to keep the overhead low.

how does it differentiate from

* https://github.com/mattes/migrate

* https://github.com/rubenv/sql-migrate

* https://github.com/pressly/goose

Goose does too much IMHO, and sql-migrate is mainly a wrapper around goose. I looked at it for inspiration, but disagreed with many choices.

Migrates is another thing, and I like it more except for the lack of configuration file.

I've been eyeing Pop[1], which is a data-mapper/ORMish library wrapping around sqlx[2]; and it happens to contain a new DSL called Fizz[3] that allows for database-independent migrations.

To be honest, I have always tried to avoid Go ORMs after a bad experience with gorm, but the whole package is looking very interesting for reducing the boilerplate involved in Go CRUD applications without having to buy into a complexity risk. It's worth a look.

[1] https://github.com/markbates/pop

[2] https://github.com/jmoiron/sqlx

[3] https://github.com/markbates/pop/tree/master/fizz

What was the bad experience you had with gorm, out of interest?

I don't remember the details, it was around a year and a half ago. But I do remember I had some trouble trying to optimize a few queries with gorm complaining about something and the other guy I was working with didn't understand it much and since we were early on development we scrapped it and went with sqlx.

I've come to drop ORMs altogether in Go, but I will have a look at this one, just out of curiosity.

Hmm -- most migrations don't need to be reversed, not sure what the 'down' section does in those cases.

The more important problem in DB migrations is transforming the content (usually involves application code) and resolving relational changes (i.e. 'in v2 all users are a member of a group').

Some large companies handle migrations by versioning the data and branching the code to handle every version. Another alternative is to 'migrate on read'.

Both of those options sound worse on paper than an all-at-once migration but one-shot upgrades can be expensive and dangerous on large databases. Dealing with data-versions explicitly also makes sense for data that's stored client-side and periodically synced.

> most migrations don't need to be reversed

What do you mean? That's like saying "most commits don't need to be reverted".

But git rollback is something you get for free. These 'down' commands have to be written and tested.

> Hmm -- most migrations don't need to be reversed,

This is true - but when something does break unexpectedly in a production migration, the last thing you want to do is figure out how to do your revert/'down' on the fly.

Okay, but are you testing every 'down' command on your prod DB? Large untested operations in prod may not be the best answer in an emergency.

Great getting a perspective from the trenches, thank you. Never heard of forking code for each migration.

> most migrations don't need to be reversed

I agree, in which case, you don't have to write a down section.

Anyway, rambler isn't targeted at "large companies with huge databases", which of course have more complex database migrations methods.

I wrote one a little while back, for MySQL and PostgreSQL, that's just a bash script. I've found it handy a few times.

https://github.com/dwb/dogfish

I had one of those at the very begining, but then I wanted to do things that are more pleasant to do in other languages, so I switched to go. But I can't agree more with your README, my focus stayed on getting out of the way: drop your requests into a file, dump your credentials in a configuration file, run, be done with it.

I've used (and committed a few small fixes to) Rambler in a few of my projects. My team and I chose Rambler over the alternatives for a few reasons, the primary one being simplicity all around, in the sense that there is no runtime needed for this tool, just a binary that you run with your deploy scripts.

But does this mean your migrations are limited to only one type of DB ? So if you switch from sqlite to oracle, you loose all your migrations?

In that case, I'll stick with my ORM solution.

The site clearly says: A simple and language-independent SQL schema migration tool and that it has first-party drivers for SQLite, MySQL and Postgres. I'm guessing it uses the lowest common denominator (the ANSI SQL standard), so any crazy FFI C extensions you wrote might be left in the lurch.

If you switch you vendor, you will probably have to fix manually many things, be it migrations, code, etc. An ORM-based solution is a thing, but then you lose the ability to use vendor-specific behavior, unless your ORM have specific extension for those, and then you're back at square one.

Reply on news.ycombinator.com