Hacker News Reader

Comment by knodi | Hacker News Reader

Streams is something Go is very good at. I recently did project where I was download a large CSV zipped (50gigs) file and had to run analytics and collect metrics on the data. With io.Buffer i was able to download, unzip, parse and process the data all in realtime with ease. Love it, very powerful.

P.S: without any of the data hitting disk. :)

View on HN

What kind of memory footprint would something like this have?

I'm assuming you didn't use 50GB of RAM and the processing was line-by-line, which should be quite small.

I built an ETL that uses a Go stream for the transform. Once the data has been processed it's written to disk on our analytics servers. Since it can keep up with the stream, memory footprint is very small (which also means we never have to hit disk on ETL servers).

Go outperformed our previous solution by over 60x, making our new bottleneck disk I/O instead of CPU.

what language was your previous solution in?

PHP. It was what I used initially so I could have access to our internal libs since our main application is in PHP. After crashing the ETL server a couple times though, I decided to try moving the heavy lifting to Go.

When we benchmarked them, what would take PHP 30 minutes, Go could do in less than 30 seconds. Only the transform is written in Go, the rest of the ETL is still in PHP.

http://jaxbot.me/articles/benchmarks_nodejs_vs_go_vs_php_3_1...

This bubble sort benchmark between PHP and Go supports your claims.

Yes, the app only took up 150MBs max as the data was processed in chunks.

Reply on news.ycombinator.com