Can someone tell me how this is collected in SQLite
Presumably they have a script that does something similar to that process, and then writes the resulting data into a predefined table structure.
Yep, my process is similar. It goes...
- decompress (users|posts)
- split into batches of 10,000
- xsltproc the batch into sql statements
- pipe the batches of statements into sqlite in parallel using flocks for coordination