Seagate just reinvented the disk interface using Ethernet
speakingofclouds.com
speakingofclouds.com
This has real promise so long as it stays as radically open as they are claiming it will be. When I can grab an old scrub machine, put a minimal debian on it and apt-get seagate-drive-emulator and turn whatever junk drives I've got laying around into instant network storage (without buying magic seagate hardware), I'm sold (and then might think about buying said hardware).
Apple has been using IPv6 for local network services for years now, like file sharing and Time Capsule backups, and it works great.
https://developers.seagate.com/display/KV/Kinetic+Open+Stora...
The important, actual TLDR: "Kinetic Open Storage is a drive architecture in which the drive is a key/value server with Ethernet connectivity."
The 99% use case of storage access requires no such frills, though.
Since the attachment for an AOE drive is Ethernet, the drive controller is just an Ethernet NIC...
Routing is also a non-issue since a single drive on the network is about as useful as a single drummer in a marching band, basically you're going to need at least three to make something with a bit of reliability, and more if you want efficient reliability. So between your actual storage 'processor' and the storage 'elements' you drop in a cheap bit of Broadcom silicon to make a 48 port GbE switch and voila, your much more reliable than SATA and much cheaper than FC.
I'm sure tho that the folks at Google are all over this. :-)
Imagine a CD player, turntable, receiver, preamp, etc., that all have only two connectors: power, and Ethernet. You wouldn't have problems anymore with running out of connections on the back of your receiver. That incredible rats nest of disparate wires and cables would be gone. No more RCA cables, coax cables, HDMI, optical cables, composite video, supervideo, component video, BNC, various adapters, etc.
No more fumbling around the back trying to figure out which socket to plug the RCA cables into, which is input, which is output, etc.
Start typing Uber A... into google and what do you get?
He's not alone ; this issue came up just yesterday: http://blogs.telegraph.co.uk/culture/davidbolt/100071146/how...
The Sonderkommando was a special group of prisoners at Auschwitz who had to prepare and cleanup the extermination of millions of prisoners. Their job was to carry the dead from the gas chambers into the ovens. They were recycled every 3 months or so by a fresh Sonderkommando, whose first job was to murder their predecessors. They also made history by starting the only counter-attack at the camp, blowing up one of the gas chambers.
You can write "Deutschland ueber alles" all day, that's a minor issue. But using Sonderkommando in the wrong context can cause quite a stir.
The verse of the Deutschlandlied that contains it is also no longer used, only the third verse is the national hymn.
It's quite unfortunate that the far-right in Germany has since appropriated the phrase, but it's been appropriated differently by English-speakers. Censoring people based on a usage that is foreign to them is a little harsh.
I'm an English speaker from the American midwest and I still think of it as an (I always assumed ironic) reference to Nazi supremacism when I see it. I'm not offended by it but it always struck me as a strange joke. "This thing is so great it's Nazi great!" Fantastic.
It's interesting to me that that's not the intent. It's a reference to the phrase "Deutschland Over Everything Else," which is either a Nazi thing or a merely German nationalist thing, depending on the context and intent. I can see why a German person might want to "take it back" and make it not mean something fascist, but why would anyone outside of that context even bother?
> Censoring people
Konstruktor was pointing out a flaw, rightly or wrongly, but not censoring. He doesn't have that power here.
Comments along the lines of "Backups? Snapshots? RAID? How they handling this then?"
The answer is: If you need those capabilities to offer up a traditional file system, you do as you do today: you layer it on top.
But many systems don't, because they already re-implement reliability measures on top of hard drives, as we want systems that are reliably available in the case of server failure too.
E.g. consider something like Sheepdog: https://github.com/sheepdog/sheepdog Sheepdog is a cluster block device solution with automatic rebalancing and snapshots. It implements this on top of normal filesystems by storing "objects" on any of a number of servers, and uses that abstraction to provide all the services. Currently sheepdog requires the sheep daemon to run on a set of servers that can mount a file system on the disks each server is meant to use. With this system, you could possibly dispense with the filesystem, and have the sheep daemons talk directly to a number of disks that are not directly attached.
For sheepdog RAID is not really recommended, as sheepdog implements redundancy itself (and you can specify the desired number of copies of each "block device" ), and it also provides snapshots, copy on write, extensive caching and support incremental snapshot based backups of the entire cluster in one go.
So in other words, there are applications that can make very good use of this type of arrangement without any support for raid etc. at the disk level. And for applications that can't, a key value store can trivially emulate a block device - after all sheepdog emulates a block device on top of object storage on top of block devices...
You could also potentially reduce the amount of rebalancing needed in the case of failures, by having sheep daemons take over the disks of servers that die if the disks are still online and reachable.
The biggest challenge is going to be networking costs - as I mentioned elsewhere, SSDs are already hampered by 6Gbps in SATA III, and 10GE switches are ludicrously expensive still.
The most famous ways of managing configuration is serializing a structure on EEPROM/Flash, or writing a string with lengths of the strings as delimiters.
Even if you assume, its for saving space etc. The way I see you will inevitably use space while you write the code to serialize and de serialize the configuration data.
So for example the output pins of a microcontroller are a memory location wired (through a buffer and some control circuitry) to the output pins. The PWM circuit is a counter and a comparator, where the comparator's inputs are the counter and a memory location.
You could write a key/value to memory location mapping layer, of course, but that's basically what vendor libraries, device drivers and operating systems already provide.
What is your ask? Get rid of hierarchy and use only a single flat directory on the SD card? Plan9 was close to the kind of vision you describe- configuration and state for applications lived live in the file system.
eMMC isn't meant to be removable, though.
Programming that way was fun, but I wouldn't want to use it on a system with megabytes of RAM. Embedded, it would be fun to implement what you describe on top of that, though.
SELECT * FROM sdc
WHERE Type='mp3';
Also, it would make sense in cases where you have... [whatever the equivalent for NUMA is for disks. NUDA? Things like hard drives with a limited flash cache.] Store the indexes on the flash or in RAM (periodically backed up to the disk, of course). Biggest issue would be wear on the flash, though.
http://thedailywtf.com/Articles/Announcing-APDB-The-Worlds-F...
I can imagine that once these are SSD drives, paired with reasonably powerful (likely ARM) chips, that we'll have massively parallel storage architectures (GPU-like architectures for storage). We'll have massive aggregate CPU <-> disk bandwidth, while SSD + ARM should be very low power. We could do a raw search over all data in the time it takes to scan the flash on the local CPU, and only have to ship the relevant data over (slower) Ethernet for post-processing.
I'd love to get my hands on a dev-kit :-)
so your idea might happen sooner than expected
The sprite breakdown identifies one core as managing the SAS commands, via DMA. Now how hard would it be to actually manage ethernet similarly, via a ARM core? If that's what they've actually done, that's really damn cool.
Hmm, which raises the question: how much RAM should a hard disk have? In a regular architecture, that database lookup could be meaningfully cached (and you could design and provision exactly to ensure your entire set is cached). Opaque K/V "disk" seems less appealing from this angle
Otherwise this will be hampered by the fact that the 6Gbps of SATA III is already too slow to take maximum advantage of many SSD devices (hence OCZ experiments with effectively extending PCIe over cables to the devices.
First of all: Hyperscale? I'm not a retarded non-technical manager or MBO, so I just stopped listening to your entire pitch. Second: You're still selling storage infrastructure, and I still have to support it. The expense just has a different name now.
"Companies can realize additional cost savings while maximizing storage density through reduced power and cooling costs, and receiving potentially dramatic savings in cloud data center build outs."
How does reducing my power and cooling costs maximize my storage density? Oh, by getting me to spend more money on your product instead of power and cooling. Nice try, buddy; give me the cost comparison or stfu.
Their whole pitch here is "throw away your key/value servers and use our key/value server instead". I wonder which will be more expensive: something I throw together with commodity PCs, or a SAN developed by Seagate.
On the other hand it opens up a massive opportunity for people to create their own storage fabric using these drives as building blocks. That means instead of having to hit up EMC for a big drive array, you will eventually be able to get an open-source implementation of same if you want, or one built on open standards so you're not locked into a particular vendor.
For companies like Facebook, Google, Apple or Yahoo that are storing petabytes of information, a drive subsystem like this is surely a dream for their engineers. Now instead of having to attach the drives to servers that do little more than wrap S-ATA or SAS into Ethernet for merging into a larger storage cluster, the drive does that all by itself.
Plus, imagine how Backblaze might be able to re-engineer their pod (http://blog.backblaze.com/2013/02/20/180tb-of-good-vibration...) to use this.
If all they provided was a disk shelf and loose cluster coupling, nobody would buy that.
The big win (for those companies which consider it a win) is in data management and namespacing. Storage virtualization, if you will. I can have a hundred disks spinning behind a storage controller which lumps them together into three storage volumes. I can serve file data off two of them (say, NFS on one and CIFS on the other) and stick some LUNs on the third and serve blocks (iSCSI). I can dedupe, snapshot, and migrate these data at will.
FWIW. Not belittling the Seagate announcement. Just clarifying why the article is correct in suggesting that EMC and NetApp aren't particularly worried about the announcement.
With this new technology it opens the door for innovation in this space. Using commodity components you can build out new EMC-type systems with a different focus. The amount of hardware engineering you have to do is minimized, and more of it can be done strictly in software. Lower cost, more rapid iteration, all that.
I'm not saying this will happen tomorrow, but in ten years the storage market could be turned inside out by this approach.
--
Case 1: Startup X makes a webapp cluster that looks up user information and returns results. It calls a library, which looks up a hash key to query a disk, and returns data.
Problem 1: Lack of load balancing. If there are three disks, and user FRANK is on disk two, and user FRANK's data is getting queried 50x more than the other users, that second disk is toast performance-wise.
Problem 2: No redundancy plus short lifespan of disk means when the disks die the user data goes too.
--
Case 2: Big Company Y creates a storage application layer to intelligently do things with the data. They have a small cluster of machines with apps that take queries and do things with the data, and manage the data using key/value pairs on disks attached to a private storage switch.
Problem 1: Dependent on ethernet (and its overhead, and latency) for each query doesn't perform as fast as other disk interconnects; have to use hacks to increase performance. Network management now critical component of your storage functionality.
Problem 2: Because the Virtual Memory Manager is no longer managing a filesystem cache, all key/value fields must be cached by the application, so you're re-implementing a VMM layer in your storage application. (Because nobody is stupid enough to not cache random disk queries)
Problem 3: Relational queries become almost completely useless. Performance drags due to all the individual queries, and you end up building a new cache layer just so your database can speed up searches, or at worst case end up with an index-only cluster of these disks.
--
As you can tell by reading Backblaze's site, there are lots of different uses for storage and different requirements for each. But one thing that's pretty widely acknowledged is it's more efficient to have a really long single piece of storage versus lots of very short pieces. I imagine Backblaze will look at this and go: Why don't we just make our own?
One important thing is that the disk is doing more work now -- you offload a bunch of what the filesystem has traditionally had to do onto the disk itself. That should mean less traffic, and lower latency. Maybe not higher throughput, though.
The interface is 2x1gigabit, so that's obviously slower than a 3-6 gigabit SAS or Sata interface. But maybe the offloaded work will be worth it? Especially if you are doing lots of "small" IO operations, the potential for lower latency might be a win.
It's a cost reduction at the end of the day, not a huge performance bonus. I am very interested to get my hands one and see how it plays out.
Remember that IP is layer 3 in the OSI model, and you could run it on top of other layer 2 implementations than Ethernet.
http://pl.atyp.us/2013-10-comedic-open-storage.html
I tried to post a comment on the NSOP (Not So...), but first I got "HTTP internal error" and then I got "duplicate comment" but it still hasn't shown up, so I'll post it here.
"The “private” bit is important; although various techniques have been created for shared (multi-master) access to the interconnect, all were relatively expensive, and none are supported by the consumer-grade drives which are often used for scale-out storage systems."
I was working on multi-master storage systems using parallel SCSI in 1994. Nowadays you can get an FC or SAS disk array for barely more than a JBOD enclosure. Shared storage is neither new nor expensive. It's not common at the single-disk layer, but it's not clear why that should matter.
The idea of network disks with an object interface isn't all that new either. NASD (http://www.pdl.cmu.edu/PDL-FTP/NASD/Talks/Seagate-Dec-14-99....) did it back in '99, and IMO did it better (see http://pl.atyp.us/2013-10-comedic-open-storage.html for the longer explanation.
"Don’t fall into the trap of thinking that this means we’ll see thousand upon thousands of individual smart disks on the data center LANs. That’s not the goal."
...and yet that's exactly what some of the "use cases" in the Kinetics wiki show. Is it your statement that's incorrect, or the marketing materials Seagate put up in lieu of technical information?
"they don’t have to use one kind of (severely constrained) technology for one kind of traffic (disk data) and a completely different kind of technology for their internal HA traffic."
How does Kinetic do anything to help with HA? Array vendors are not particularly constrained by the interconnects they're using now. In the "big honking" market, Ethernet is markedly inferior to the interconnects they're already using internally, and doesn't touch any of the other problems that constitute their value add - efficient RAID implementations, efficient bridging between internal and external interfaces (regardless of the protocol used), tiering, fault handling, etc. If they want to support a single-vendor object API instead of several open ones that already exist, then maybe they can do that more easily or efficiently with the same API on the inside. Otherwise it's just a big "meh" to them.
At the higher level, in distributed filesystems or object stores, having an object store at the disk level isn't going to make much difference either. Because the Kinetics semantics are so weak, they'll have to do for themselves most of what they do now, and performance isn't constrained by the back-end interface even when it's file based. Sure, they can connect multiple servers to a single Kinetics disk and fail over between them, but they can do the same with a cheap dual-controller SAS enclosure today. The reason they typically don't is not because of cost but because that's not how modern systems handle HA. The battle between shared-disk and shared-nothing is over. Shared-nothing won. Even with an object interface, going back to a shared-disk architecture is a mistake few would make.
Nerdy me likes idea of POE hub and bunch of drives doing their own thing.
Also pretty good time to start writing stuff to support this into Linux kernel and developing support apps.
my 2c
I wonder about that.
It's usually a lot cheaper to move computation to data, rather than data to computation. The model that Seagate is presenting here strikes me as wrong, because it assumes very fat pipes (or specialized topologies) for any non-trivial app. At the scale Google operates at, I just don't see this happening.
That, and I have a healthy distrust of networks. Instead of having a box with an OSS OS and dumb drives with small(er) closed firmware blobs, now you have the OS, all the network devices and their closed firmware blobs, and drives with large(r) closed firmware blobs, just to access your data. A lot more can go wrong. A lot more byzantine things can go wrong. Drives are dodgy lying sacks of fecal matter as is; this looks like it'll make things much worse.
The model Seagate presents could be useful for data that is rarely accessed, but I'm not really sold on that either.
Yes, it means your switch must not fail. But if you worry about your switch failing, you have that worry if it's not handling storage too, and you deal with it with redundancy. Moving the storage to hang off a switch does not change that - if you have a single switch and it fails, your servers are just as unavailable either way.
But hanging storage off your switches means it is possible to have servers take over drives of failing servers, which makes many other failure scenarios easier to handle.
In terms of pipes, yes, that is a concern for some uses. It won't be fast unless you go to 10GE, and 10GE switches are still hopelessly overpriced. But "most people" do not serve up gigabits of content, and could do just fine with slower drives hanging off cheap 1Gbps switches.
I already assume not only that my drives will fail, but that the network and servers will fail too. Which means I need to replicate data over many servers on different networks. In that case having the drives be directly addressable over TCP/IP is not an added complexity, and it opens up so many opportunities in improving flexibility of server enclosures etc.
Right, but having local disks reduces the sources of failure, reduces contention, reduces latency, reduces the complexity of failures, and is thus much nicer to work with. Computers and networks would be easier for us to debug if they had a binary works well/doesn't work at all, but we all know they don't. Especially networks.
The simplest and sanest architecture is keeping dumb disks local to where computation is running (and yes, that may also include duplicating data across several servers). Anything else is asking for more crazy classes of failure. Been there, bled there, not going back there.
JFS file system, and the only access we then allow to this totally self-contained storage building block is through HTTPS running custom Backblaze application layer logic in Apache Tomcat 5.5. After taking all this into account, the formatted (useable) space is 87 percent of the raw hard drive totals. One of the most important concepts here is that to store or retrieve data with a Backblaze Storage Pod, it is always through HTTPS. There is no iSCSI, no NFS, no SQL, no Fibre Channel.
2) Various posts pooh-pooh'ing this development (including the current top post) here are committing the classic mistake described in that book made by incumbents which leads to disruption by new entrants to the market.
3) Seagate is doing something right. It doesn't guarantee that they'll win the next phase of the storage battle but they are doing something radically different which has a plausible marketing story appealing to a large base.
If they do end up selling it as individual disks directly pluggable into standard racks (with some easy means of supplying power), this certainly would count as a potentially disruptive innovation. As I said before, whether market reacts positively or not is not certain, but it never is. But kudos to Seagate for trying it.
A technology is only disruptive if it provides some benefit over existing alternatives. Will Kinetic be cheaper than today's drives paired with tiny ARM servers (the storage-industry disruption that is already upon us)? We won't know until Seagate actually sets prices and tries to sell some, but it doesn't seem particularly likely. Will it be faster? Again, possible but not particularly likely. Will it allow higher density and/or lower power consumption? Will it be easier to build applications on top of this new and limited API than on top of the ones we already have? Without a credible positive answer to any of these questions, how is this disruptive?
If Seagate had done this right, especially wrt security and semantics beyond get/put, this might be a better building block for a whole system. That might be disruptive. But they've made such a poor start technically that if they make a dent at all it might well be someone else who achieves the actual market breakthrough. Once Seagate has done all of the marketing, that makes it far easier for someone with a better actual implementation to succeed. Look forward to the WD/HGST Object Drive, with an API that allows you to use one securely without a server to do read-modify-write and metadata management for you. Then I'll cheer.
On the oher hand I see an opportunity as shared storage for mobile and ligthweight devices. Using a single and simple protocol, compared to NAS, could open a new technology domain and market. Of course it requires also a good integrated authentication and access control system because on Ethernet this data might be open to the world.
It's been around a long time now, much longer than 1Gb was around until it started becoming consumer products. 10Gb still uses quite a lot of power, and consumer demand is virtually absent since the 10x speed we got from 100Mb to 1Gb has been "fast enough" for home users, and will be for many years.