Show HN: Random Data Generator for arbitrary data types | Hacker News Reader

Show HN: Random Data Generator for arbitrary data types

github.com

5 pointsrixed9 comments

Datasino is a command line tool that can come handy when load testing a service or database.

Give it a data schema (many compound types supported, including recursive types), an encoding (JSON, CSV or ClickHouse) and a target (a file or a Kafka partition) and a data rate and it will generate a reasonably optimized (native code) random data source.

See the top of the README for some examples.

If you have any ideas on how to extend this tool so that it's useful to more people, please let me know.

Open articleView on HN

ClickHouse already has embedded tool for data obfuscation and generation: https://clickhouse.com/docs/en/operations/utilities/clickhou...

And here is an article about possible methods to solve this task and what method was selected: https://habr.com/en/company/yandex/blog/485096/

Interesting, I was unaware of this.

Not usable in our use case though: IIUC, clickhouse-obfuscator anonymise existing data, whereas we need to produce a giant volume of data (~1M large rows per second) in real time to test data ingestion speed of a data pipeline (that involves Kafka, ClickHouse and other tools in the way).

Yes, that's true. Although there is an option to run it in a loop over a smaller dataset as a seed.

PS. The task of random dataset generation is surprisingly deep and interesting, I welcome more tools in this area. Already digging into your repository...

Another important difference that I can see between clickhouse-obfuscator and datasino is that obfuscator takes a long way to make sure data, although anonimised, remains statistically realistic, whereas datasino is a dumb data filler/fuzzer: the output values will just be random bits. It's OK if you care about the data structure only but not so much when you need more "realistic" data.

Something I'd like to add to datasino is indeed a way to specify constraints on some fields, beside their type, so for instance a column that's supposed to store a time actually look like a time and not a random integer. I just haven't found a nice and simple way to specify those constraints yet.

Judging from the documentation datasino is easily the best data generator I ever came across! I will definitively give it a spin. Love your tech-stack: S-Expr, ClickHouse RowBinary, Kafka, nix and Ocaml. A man of culture!

Haha thank you, but how many data generators did you come across? It's not a very widespread genre, although it should be IMHO.

Seriously, any open-source references for command-line tools to generate data? In general people just build their own makeshift special purpose generators in my experience.

You are exactly right! There does not seem to be much out there. I usually do hash or otherwise transform real-world (e.g., https://github.com/awesomedata/awesome-public-datasets) data to retain basic properties like skewness utf8 well-formedness, cardinality, and such. Like many others, I often use the Enron eMail dataset, NYC Taxi, or Twitter data (for coherent use of Unicode). And then scale it either by combining attributes randomly or otherwise augmenting it. Earlier in my career, I wrote a regular expression inversion data generator; it quickly spiraled out of control complexity-wise ;-).

For relatively small data sets in CSV format, you can use the in-browser utility-- which is quite convenient (no installation, no security concerns) at https://zsvhub.com/playground/. A subset of the features there are available in a native command-line utility (all open-source-- see https://github.com/liquidaty/zsv), to which the "obfuscate" feature that is currently available in the playground will soon be available and is anticipated to be extremely fast (in keeping with the project's current status as the fastest CSV parser). Full disclosure: I am one of the contributors / authors of zsv/lib and zsvhub

Reply on news.ycombinator.com