Hacker News Reader

Comment by bhickey | Hacker News Reader

An old friend of mine is one of the authors on this. Their instruments generate a lot of data. I don't know if this is the case for their array in Australia, but when he was working in South Africa they had to carry hard drives out of the country because there wasn't enough fiber capacity out of the continent.

View on HN

"Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway" - A. Tanenbaum of MINIX (amongst other things) fame

It's actually incredibly easy to beat the bandwidth of the internet with your car, with just a couple HDs.

I routinely drive HDs full of backups a couple hours when going between DC sites.

The latency will kill you.

These signals are 12 billion years old already.

Not necessarily: the car may be there much sooner than the last byte sent via the fiber arrives. Not everything is streaming video; some datasets make most sense only as a whole. Backups are an example.

I think the subtext is, it is easily 1000x more dangerous personally to drive the data along public roads, than to send it over a network. The value in avoiding such danger is perhaps incalculable.

Data can be encrypted to make it safe. Data will be duplicated onto disks before being driven away. It's not like there can only ever be one copy of the data.

I think they mean risk of injury to the driver.

Just send N+2 drivers for redundancy.

IIRC most crashes are down to unstable drivers.

Redundant Array of Independent Drivers. Acronym checks out. Proceed.

It's calculable: just hire a driver.

I recently found out that that is referred to as the sneakernet, and is actually still a preferred way to transfer a lot of data. https://en.wikipedia.org/wiki/Sneakernet

AWS has a little-known service for trucks transporting 100PB of data in shipping containers:

https://aws.amazon.com/snowmobile/

"Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway."

-- Andrew S. Tanenbaum (circa 1989)

What do their prices look like for 100 pb?

It's probably not bad compared to the monthly recurring you're about to get hit with for sitting in all of those bits.

Certainly less than they would charge you for the bandwidth if you sent them over the wire.

Damn. I once spoke with a friend who helped a museum in Rio de Janeiro to migrate its backups using AWS Snowball, but thats a whole new level.

https://aws.amazon.com/snowball/

Which one? IMS?

I always wondered who would actually use this. Apparently a company called DigitalGlobe[0], which has a large collection of satellite images.

[0]: https://www.networkworld.com/article/3154898/this-company-is...

For when your forklift migration needs a literal forklift.

In A Fire Upon the Deep, the Skroderiders' cargo was a one-time pad.

If you send someone several TB of random data on hard disk, you can now very securely send them several TB worth of encrypted data. It's many years of audio of reasonable quality, 1tb is about 72 days of raw 44.khz 16 bit stereo audio. With some speech codecs, that could be years of voice data. One time pad can be bigger than a notepad.

My lab still does this (or, will do it when we’re finally allowed back in).

We do a little bit of compression right on the instrument’s computer. After that, a briskly-walking postdoc can move a few terabytes a minute. The exact rate depends on whether they stop for coffee, but it always blows our internal 100 Mbps network out of the water.

I worked for a company CERN had bought equipment from a while back. I was on the technical support team. I had a direct path to engineering when CERN called. The amount of data in an ultra short time was seemingly endless.

So many "need more interfaces, more bandwidth...buffer tuning... buffer tuning".

When a card burnt up mysteriously I made a sci fi joke about black holes and CERN in my ticket to engineering. It was the one and only time I got them to make a joke back ;)

Now, it’s unfair to leave your audience hanging like that.

I wish I could remember the joke, but IIRC it was pretty terrible as I was just desperate to get something back from that engineering team that seemed like ... fun.

Fortunately they either didn't care for, missed or just never acknowledged most of my puns and really vague references.

Outside that one interaction the only thing I knew from them was that despite me being not the best technically compared to my peers, they asked I work specific tickets at times... but I think that was mostly because they knew I didn't make things up and would be honest and say "I don't know". Also I'd give them a heads up when I saw something hitting the fan to give them time to prepare ;)

I was doing my PhD when the LHC was being designed. It was interesting to listen to the discussions about signal and data management for the detections closest to the vertex.

The rate was so incredible that the triggers were incredibly optimized to get rid of the noise and keep the fist wave of useful data (it was then processed several times down the road).

The other interesting thing was to see how people were fighting for real estate in the detector, to be able to place their elements somewhere interesting.

I crawled once under one of the detectors (ATLAS) and imagining the several thousand tons above me was frightening.

The MWA has a direct fibre connection from site to a super computing facility in Perth. Although some processing (correlation and averaging) is usually done on site to reduce the data a little, there are some observation modes that send the raw voltages.

This is how Google Street view cars work too, because the sheer size of the high resolution photograpic data is so huge that it would make no sense to transmit it over a mobile network, vs just downloading it once the car finishes it's route.

They could be using peoples wifis on their way to transfer the data as they dash thru the neighbour street. ;)

I was just leaving radio astronomy when the MWA was in later stages of design/deployment, I remember the discussions around designing the data storage for it, and it was centered around the idea of having to fly the data set back and forth (At the time, around 2008-2011ish?) - so it was so large that it wasn't really feasible to use the Internet for each data collection run. On the order of many terabytes if I recall correctly.

That isn't as uncommon as you may think. Especially when talking about DR (disaster recovery) plans, sneakernet is certainly a valid way to get backup data from off-site because you never know what the network status will be when you need to enact your plan.

Event horizon telescope used HDD's being flown around to aggregate their data

Reply on news.ycombinator.com