Hacker News Reader

Comment by hk1337 | Hacker News Reader

Agreed. Rails uses ActiveRecord and you'll most likely be trying to load millions of records through an ORM which will make you go insane trying to get it to work efficiently.

I have no problem with Rails or ActiveRecord but bulk upload of data through any ORM is a big no for me.

View on HN

> Rails uses ActiveRecord and you'll most likely be trying to load millions of records through an ORM which will make you go insane trying to get it to work efficiently.

You just need to use batches (that `.in_batches` in the link), it's an easy-peasy approach which is way more acceptable for your database than a single update of 500m records. I've been using it for years and it has never raised a single issue.

The downcasing in done in Ruby-land, thus the records had to be loaded in memory, on the worker. If the downcasing was done on the DB itself, then the records would not have needed to travel all the way to the worker, only to be discarded 200ms later. Better to run an UPDATE query direct against the DB instance, with IDs to control concurrency: UPDATE addresses SET city = lower(city), state = lower(state) WHERE id BETWEEN a AND b

Yeah, one of the first things I learned (Ruby 1.8) was that Ruby is slow. If you can avoid creating a bunch of Ruby objects and push the logic into the database you'll typically see performance gains. The big exception was Mongo, where more "advanced" features got shoved off to the JS interpreter which was an expensive operation – at least until the 'aggregation framework' became a thing.

Anyways. Before blindly doing string manipulation with postgres make sure that all the locale stuff is set correctly.

This is what the `in_batches` method already does. See [1], which is called by the `in_batches` method for update operations.

[1] https://api.rubyonrails.org/classes/ActiveRecord/Relation.ht...

You could use `in_batches` that way, but in the code from the article it looks like each call to `maximum`/`minimum` is causing a round trip, plus the `map` inside `MyService` is loading each batch into memory.

I agree with you if you focus on this very example, but I think to it just as an example. When I read this article, I replace <lowercase> with <some business logic implemented server side>.

Reply on news.ycombinator.com