Hacker News Reader

Comment by ck2 | Hacker News Reader

Should be interesting to see what they come up with.

Somehow I don't think they will find things better than multi-million dollar research like HAMR for write-once read-many, but we'll see I guess.

I wonder if there will be 100TB spinning read/write drives by 2020, an exponential leap instead of incremental

Is there any reason for them to stay with 5.25 or 3.5 inch design for a datacenter?

Why not go back to 8 inch for massive surface area? Or is that too much mass to spin.

View on HN

> I wonder if there will be 100TB spinning read/write drives by 2020, an exponential leap instead of incremental

The problem with spinning disks is that speed and reliability has not increased in line with capacity. There comes a point at which it doesn't make sense to make them any bigger (capacity-wise).

Just creating a filesystem on an 8TB disk takes hours. If you bring in disk (block level) encryption and the requirement to fill the disk with random data and then encrypt before creating the filesystem, you're looking at a multi-day task. Expanded to 100TB, you could be looking at a month just to bring a disk online.

On the reliability front - a spinning 8TB disk is probably about as reliable as a 1TB disk, so that means you have a 8x increase in probability of data loss, as well as ~ 8x more data to recover/re-distribute for every failure

> Just creating a filesystem on an 8TB disk takes hours.

Are you sure you mean TB?

    $ for i in $(seq 8); do truncate -s 1T 8tb.$i.img; losetup --find --show 8tb.$i.img; done
    /dev/loop2
    /dev/loop3
    /dev/loop4
    /dev/loop5
    /dev/loop6
    /dev/loop7
    /dev/loop8
    /dev/loop9
    $ sudo mdadm --create --level linear -n 8 /dev/md8 /dev/loop{2..9}
    mdadm: Defaulting to version 1.2 metadata
    mdadm: array /dev/md8 started.
    $ sudo time mke2fs /dev/md8
    [lots of output snipped]
    real    42m0.590s
    user     0m2.470s
    sys      1m0.440s
    $ du -sh . # just for fun
    65G     .

That's certainly not fast, but it's not "hours" either. It's also a magnetic disk that was actively being used by other processes. Given how much data it wrote, it averaged 25MB/s, which is not really that fast. An 8TB SSD would be much faster.

Is that defaulting to EXT4 or XFS ?

He used `mke2fs`, which defaults to Ext4.

> Just creating a filesystem on an 8TB disk takes hours.

Am I being dense here, but why would you create a filesystem on the hard drive? The hard drive should only store file contents.

There's all kinds of seeking and syncing and random access needed for the filesystem. Seeks from file contents can't be avoided, but ones due to the filesystem can be.

If there's no metadata-on-ssd + data-on-disk filesystem for Linux, there should be.

ZFS allows for that level of control: assign a cache device to the pool, then set your secondarycache flag to metadata. Yes the metadata will hit the primary cache first (memory), but it will spill onto the secondary as it is pressured out of ARC. I've got a SD card playing that role in my build machine, it works really well for keeping track of all the tiny files that make up the FreeBSD base and ports trees.

So what you're advocating is that instead of creating an ext4/zfs/xfs file system on each drive, have a sort of distributed filesystem that has raw access to each drive. And the i-node of the distributed filesystem could be always-available on RAM with SSD backing. My guess is, Google probably already does this.

Lots of distributed filesystems (I guess I'm talking about distributed object stores here rather than an actual clustered network filesystem) rely on a regular POSIX filesystem for the storage of blobs.

A huge amount of work has gone into making ext4/zfs/xfs etc fast and reliable, plus you get other benefits like journaling, filesystem caching, fsync(), metadata caching (a la ZFS), so there are credible arguments for not going down the path of NIH.

Interesting, I don't think much work has been done in this area. It's not a common way of dividing the workload, even though it sounds obvious.

> a spinning 8TB disk is probably about as reliable as a 1TB disk, so that means you have a 8x increase in probability of data loss

I don't understand that argument, not if you think at scale.

You need to store 8000TB. You can run 8000 x 1TB, or 1000 x 8TB. You loose 10% of the disks per year (to keep the math simple). So after a year, you have either had to redistribute 800 x 1TB or 100 x 1TB, which is the same to a software defined distributed storage.

There are operational upsides to fewer disks, power usage, manpower, rack space etc. But of course more disks are faster.

Google wants to have more options than just bigger vs faster available. Since they know exactly, and can influence, how their software behaves, they can gain much from using specialized disks that balance things differently.

> Just creating a filesystem on an 8TB disk takes hours.

Xfs creates such a filesystem in seconds. As another reply showed ext4 is quite a bit slower since it preallocates the inodes; more modern designs allocate them on the fly as needed.

> you're looking at a multi-day task. Expanded to 100TB, you could be looking at a month just to bring a disk online.

That's what adding value looks like.

In the long run anything that moves in DC is way less reliable than systems without moving parts. On the top of that, these systems also need more energy to maintain the movement. If you increase the size than you need better materials to keep up with the increased mechanical load. I bet by 2020 we are going to have better ways of storing large amount of data than a spinning disk. The new optical drive would be a good candidate for some use cases, mostly for archiving though:

http://www.sciencealert.com/this-new-5d-data-storage-disc-ca...

For continuous read/write workloads SSDs are the best for many reasons (energy efficiency, latency, IO bandwidth) and I think the world is moving that direction to replace spinning disks with these. I am hoping to see a bump in the SSD capacity as well. We are here now:

http://www.extremetech.com/computing/221303-the-worlds-bigge...

From the article:

"An obvious question is why are we talking about spinning disks at all, rather than SSDs, which have higher IOPS and are the “future” of storage."

"The root reason is that the cost per GB remains too high, and more importantly that the growth rates in capacity/$ between disks and SSDs are relatively close (at least for SSDs that have sufficient numbers of program-erase cycles to use in data centers), so that cost will not change enough in the coming decade."

Well at least for YouTube video storage, I imagine the P/E cycles are not a problem. There have been issues with longevity like on the Samsung 840 EVO drives, but if you can avoid that then you can build some incredibly dense, fast, reliable and power efficient storage on top of V-NAND.

Cost per GB for SSDs is definitely catching up to HDD [1] and I think is expected to match or beat HDD within the decade. Power efficiency should be better on SSD and MTBF will be much better than HDD, so I would have thought TCO for SSD would have HDDs basically beat already.

[1] https://cms-images.idgesg.net/images/article/2015/12/ssd-vs-...

Indeed, it will be interesting to see what Google can think up compared to the other big researchers.

Though I wonder if the goals are slightly different, as Google is focusing on "Cloud Storage", rather than single hard performance.

Which as they mentioned, the Cloud doesn't need to have great reliability (since data is assumed to be already redundant), this may allow them to decrease the cost of hard drives for exchange in cheaper disks that may fail more.

Doesn't sound like they are expecting to increase size/performance alone with this effort.

I'd imagine it's more likely to be the engineering tolerances required to keep the vertical movement to a minimum at double the radius? But I have no real insight, just a guess.

It does strike me though - why do disks do serial reads, couldn't the head read several tracks at once? Why not have several heads angularly spaced around the drive? Presumably such things have been tried and found lacking in some respect. Looks like I need to do some research ...

> couldn't the head read several tracks at once?

The paper in fact suggests this and also cites an attempt by Conner to introduce multiple actuator drives:

http://www.pcguide.com/ref/hdd/op/actMultiple-c.html

There were some disks with two heads once. They were mechanically too complicated.

On Tuesday Eric Brewer gave a talk on this white paper at FAST. In the talk (and probably discussed more in the paper) he proposed looking at alternative disk form factors. He seemed to be particularly interested in "tall" disks, say 2U, that had the same surface area as today, but with more platters, and potentially more heads.

Ah, going back to the beginning of micros, where drives were often full-height things that could easily break feet when dropped. The thing that worries me there is that having had a big full-height 9 gb seagate scsi drive back in the day is that you needed to give the drive about a hundred watts on its own to run it, with a spool up that sounded like a jet engine was about to take off. I know that motors have gotten faster since then, and platters have gotten lighter, but that's still quite a bit of mass to get up to speed.

The thing that interests me more is the idea of more read/write heads. Add a second, independent set of heads for read/write and you will decrease latency a good amount. Or, a second dependent set of heads which would not provide as much of a performance boost, but would still improve latency a good amount.

I for one am confident that SSD and its technology will be much more scalable and future-proof than spinning disks; stackable chips are a thing already. I'm sure that by 2020, there will be companies that can deliver a storage platform with petabytes of space in a 1U package. Just imagine one of those server things crammed full of ssd chips.

In the long run that may be true, but the same things they talk about (minus the mechanical head changes) will still hold in SSDs. The background activity is even worse for SSDs and the tail latency still very much exists.

Essentially the problem is that the HDDs and SSDs are built as black boxes and are not vertically integrated with a data center storage scheme. The major win will be when the HDDs and SSDs are slightly less self-sufficient and let the higher components in the stack manage them more in depth. Being able to tell the disk "do not bother with ecc for now" and try to get the data from your RAID and if that doesnt work come back to the disk and tell it "we really really want the data, spend as much as needed on this" is very strong in both getting the tail latency down and getting the reliability to stay at the same level.

For SSDs there is an added complication of garbage collection and the incredible amounts of over-provisioning needed to get consistent performance from them (at least 30% in an ME drive, the RI drives suck at consistent performance). If those can be reduced by more vertical integration that would reduce costs of the systems and make for better overall performance as well.

It amazes me that SSD drives still (mostly) conform to the same drive sizes as spinning disks.

My most recent computer build used an M.2 slot SSD, and plugging that tiny little card into the motherboard seemed like the future. No cables required, and it was even smaller than the RAM sticks.

> Is there any reason for them to stay with 5.25 or 3.5 inch design for a datacenter?

> Why not go back to 8 inch for massive surface area? Or is that too much mass to spin.

One problem AFAIK is that larger disks wobble too much, more the faster they spin. Which is why if you break open a 3.5" 15k drive you'll see that the platters are more like 2.5", and a lot of wasted space. For 7.2k you can use bigger platters, but how much bigger? I doubt 8" is feasible.

Of course you can reduce the rpm further, but that has downsides too.

Current disk densities are about 10^12 bits/square inch. To get 800 terabits on a single platter, it would need a diameter of about 32 inches. At 10^14 bits/square inch, only 3.2 inches.

Assuming a compound annual growth rate of 25%, densities will be 10-fold larger in 10 years, and 100-fold larger in 20 years. (This actually happened in hard disks between 1972 and 1992.) But that would take us to 2035, not 2020.

At those densities, a bit will have to be stored inside a square about 7 or 8 nanometers on a side. I don't know whether that's possible.

For 100TB disks to make sense the read/write speed would have to improve considerably, otherwise the disk will be useless other than for cold storage. I haven't really seen spinning disks speed improving much. It would take weeks to rebuild an array of 100TB disks at the current speed.

100TB should only be 5x (i.e. sqrt( 25 )) slower to access each byte sequentially than a 4TB drive (although shingling will screw up this calculation, since it provides more tracks, but not more density on each track).

Which is a long time, but not totally out of bounds for the way we're currently using disks.

The coming decade is going to be very interesting, SSDs are pushing forward on an insane trajectory, which could either fall apart as the benefit of node shrinks disappear, or accelerate through a virtuous cycle as unit volume increases.

It's even plausible that spinning rust is on death's door in that time frame. The price floor of SSDs is much lower than that of hard drives (i.e. how much does an 8gb hard drive cost to manufacture today in volume? As much as a 500 gb drive), meaning that both mildly performance conscious and price sensitive consumers both move entirely over.

Enterprise customers will still chase the price/gb and storage density of traditional hard drives, but without the consumer dollars to invest in R&D, that could start to stall, quickly leaving them with diminishing advantage and obsolete in the blink of an eye.

I am repeating a comment to another response but if you reduce the access time and at the same time increase the amount of data stored on the drive, you are locking yourself out of your data. Which is fine if it is just for archival. But if it is data supposed to be randomly accessed (youtube video) it doesn't help to have 100TB of data on a drive if you can only extract them at a single 50MB/s at a time. The more data on a single drive, the most likely you will a need to access several segments of this data at the same time.

Most of youtube (and photos archiving even more so) is 'write-once, read-never'. Only a comparatively small number of videos become popular.

or you use multiple drives at once to access the data?

I remember a washing machine sized IBM mainframe hard disk with tho sets of heads. That's one trivial way to double bandwidth and cut latency (it'll read like a RAID-1 and write as a RAID-0). Also, the more platters, the more data you can read on a single rotation.

You double bandwidth for a single access. But if you are storing 100TB on a single drive, it's unlikely to be 100TB related to a single user. You will want to make a lot of requests to this data simultaneously, a lot more requests than what you would to a disk containing only 1TB of data.

If they go with anything Arm and Platter I'd be shocked.

I'm imagining a 10 acre array of NAND

They are saying the paper they want arms and platters as it's more economical. An HDD is cheaper than an SSD of the same capacity at four to ten times and I don't think the electricity cost will offset this if you care more about capacity than performance.

It is right now, but silicon and precision mechanics scale with different curves. There is a point where spinning metal will no longer be viable as a competitor and I'll guess, based on a hunch, it's not that far into the future.

Flash is not getting cheaper faster than HDD's, once you control for write endurance. One of the big messages in the paper is that IOPS matter. If you only have a petabyte or two of data, you can use a brute force solution such as flash or 15k drives. But at Cloud scale (we used to say Google scale, but there other companies like Amazon and Microsoft and Facebook that run at these scales now) you just can't afford that much flash.

So the trick is to use every last IOPS that you can out of each and every disk, and that means provisioning and spreading out your data across an entire fleet of disks. The alternative, where you silo your storage in disk pools, and where some disks or disk pools have IOPS that are unused --- wasted, and where you make up for that by using flash for your hot and warm workloads is just a much more expensive way to do things.

It's for that reason that SMR drives aren't all that interesting. Sure, you get 20% more capacity, but at the cost of burning most of your IOPS for GC. Or if you move all of your SMR-friendly data to the SMR drives, that's cold data, and it means that you've wasted all of the IOPS that those SMR drives are capable of. Which means the rest of your disk fleet is now much hotter, and you might have to buy more CMR spindles to cover the cost --- at which point the cost advantage of SMR drives disappears.

(I helped to author the section about SMR drives, and why we are interested in hybrid SMR/CMR drives as a solution to this problem; read the paper for more details.)

If you get any response from disk vendors I'd be very interested to hear about it and be somehow involved. I've been asking them similar things for some time and got no response beyond a "we'll take a look" and never getting a reply back.

One such request that is also mentioned in your article is to get more granular responses in the IO responses, in a SAS disk you can get correctable errors that do not hamper the behavior but will get you information on the internal status and actions of the disks. SATA has a problem with this since any IO Error in SATA will cancel all outstanding IOs. Though I think that there are device logs and such now that can be implemented to provide some semblance of a similar result.

  > Flash is not getting cheaper faster than HDD's, once you control for write endurance.

Is the category of WORM-suitable data (like videos) not large enough to be interesting, or is there simply no WORM technology better than writable?

There's no WORM technology better than rewritable as far as I'm aware. Storage density is already well above the point where optical technologies are viable.

I think the question is why does P/E cycle count matter for storing YouTube videos? As videos are uploaded, you shard them with FEC across your SSD array, but once the blocks are written, when would you ever need to erase them? The only concern is longevity of a single block which requires some amount of P/E (re-charging the cell) but lets say that's just 1 full disk write per month to keep everything fresh...

OK. I was thinking more along the lines of PROM (i.e. random access) but I can imagine there's nothing now more useful than flash.

There is "antifuse", although it's a little slow to program. Commonly available on chips in small amounts for secure key storage, serial numbers, calibration etc.

I'm not aware of anyone selling standalone parts, but here is an IEEE paper: http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=124097... (512Mb in 2003!)

If Google really wanted to, they could make a couple of petabytes of slow-to-write memory by commissioning it themselves for low numbers of millions of dollars. But I suspect they value the rewritability.

Reply on news.ycombinator.com