17 karma · joined April 5, 2024
Streaming: Can't hold it all in memory. Generate in chunks, write, release, repeat.
Format choice: Parquet with row groups is fast and compresses well. SQL needs batched inserts (~1000/statement). Direct DB writes via COPY skip serialization entirely is usually fastest.
FK relationships: The real bottleneck. Pre-generate parent PKs, hold in memory, reference for children. Gets tricky with complex graphs at scale.
Parallelization: Row generation is embarrassingly parallel, but writes are serial. Chunk-then-merge is on our radar but not shipped yet.
What does your stat product need, realistic distributions or pure volume/stress testing?
The difference from Faker: you don't write any code. Paste your CREATE TABLE, get data back. Faker is a library you have to integrate, configure field-by-field, and maintain as your schema changes. Different use case — more like "I need a seeded database in 30 seconds" vs "I'm building a test suite."
Fair point on pricing though, still figuring that out. Appreciate the feedback.