Hacker News Reader

Comment by tgtweak | Hacker News Reader

Strangely, or perhaps not, the most interesting part of this article to me was this:

"Using the Digital Backend instrument on the GBRT, Dr. Gajjar and the Listen team observed FRB 121102 for five hours. From this, they accumulating 400 terabytes of data"

That has to be one hell of an all flash array to take in that much data at a steady state of 22.75 GiB/s

View on HN

Fun fact: the Square Kilometre Array (SKA telescope) will gather roughly 3 TB/s when it's up an running.

Would it be a reverse DoS attack when the North Koreans try to steal the data and 3 TB/s crashes their Pentium II "mainframe"?

“There is no greater danger than underestimating your opponent.” ― Lao Tzu

Paper describing the instrument:

"The Breakthrough Listen Search for Intelligent Life: A Wideband Data Recorder System for the Robert C. Byrd Green Bank Telescope"

https://arxiv.org/abs/1707.06024

Relevant part:

"Each storage server contains 36 hard drives, each with 6 TB capacity. These are configured as three 11-disk RAID 6 arrays plus 3 global hot swap spares."

Impressive if you think in raid 6 you have double a single drive's write speed (not counting cache). They'd need 60+ of those arrays to ingest without back pressure.

A single spinning disk is already capable of >200MB/s sequential write:

shttp://www.storagereview.com/wd_black_6tb_hdd_review

100 of those in parallel will suffice to achieve >20GB/s, with roughly the same capacity of a few hundred TB.

Is use of 100 parallel disks at all standard? There has to be some overhead in processing, can you really just stripe across more disks and directly multiply the throughput? How much is that costing to run?

They probably filter and compress it, put it through a fiber glas system to storage systems of universities and so on.

Those store systems are huge and yeah are build to do exactly this.

I've seen petabyte-scale SANs for bioinformatics HPCs. Other sciences (esp. the high energy physics community) have similarly scaled storage and compute requirements.

7200 disks in 1 "array" for 30PB of storage. This cluster is described as their "playground"

https://cds.cern.ch/record/2015206/files/CephScaleTestMarch2... https://indico.cern.ch/event/594977/contributions/2407197/at...

what do they mean by accumulated data? Are they mapping the raw signal to some other format that necessitates this huge file size? Is it recording the signal at multiple wavelengths?

Reply on news.ycombinator.com