I am planning to increase my catalogue to 1Mn. books as I get more users on the website.

Running a language model is challenging as it took ~22 hours for training on 30k books and I expect it to take at least a week on M1 GPU.

On Size, I expect 10Mn books to be taking ~16GB storage in my database.

Thanks for your comments and good luck to your data pipeline.