- Handles MySQL -> pg; MSSQL -> pg; And pg -> pg copy
- Very fast (lisp/"real threading")
- Syncs schemas effectively (this says data only)
- Has it's own DSL for batch jobs: specify tables to include/exclude, renaming them on the fly in dest, and cast dtypes between src and dest if needed. etc
I tried migrating a smallish mysql database (~10Gb) to postgres and it always crashed with a weird runtime memory error. Reducing the number of threads or doing it table by table didnt help.
I've used pgloader multiple times because I'm a huge Postgres evangelist for that exact use case multiple times without issue. Honestly - it's a favorite tool in my toolbox.
If it's "heap exhausted, game over", there's a solution for that - you need to tell pgloader to allocate more memory for itself.
It's worth getting it going, it's one of the greatest tools in migrating things into and out of pgsql I've ever used. If you have pgsql in your pipeline, get it working.
I used this and debezium to really make my ETL pipeline absolutely bulletproof.
This looks very cool, was not aware of it:
"Debezium is a set of distributed services that capture row-level changes in your databases so that your applications can see and respond to those changes."
https://github.com/ankane/pgsync/blob/master/lib/pgsync/tabl...