Hacker News Reader

My use case is generating a very high rate (10k e/s up to 100k e/s) of JSON-NL events from samples of JSON-encoded log data (JSON-NL to be exact). Is this supported in OSS Gretel?

FYI, I'd built a hand-crafted generator using JSONNet templates and Golang, but I really wanted something that could model source data distributions accurately. The use case is large-scale load testing of customer workloads without requiring actual data.

View on HN

We're currently beta testing something that fits this use case directly. The models we have today are really great at capturing the original distribution, but they're not always the fastest. This new stuff will change that, feel free to reach out (maybe on our slack?) and we can see if we can get something working

Blog post is out now https://gretel.ai/blog/introducing-gretel-amplify

They get 43,300 records per second on this example, which seems to the right order of magnitude for you

Reply on news.ycombinator.com