Hacker News Reader

Comment by skissane | Hacker News Reader

IBM mainframes had key-indexed hard disks back in the 1960s – CKD (and later ECKD). Each disk sector could have a key field at the start, and rather than addressing disk sectors by physical location, you could tell the hard disk "give me the data of sector with key 1234", and it would go search for that sector and return it to you. (I think, you still had to tell it what disk track to search on, and it just did a sequential scan of the disk track...)

A lot more primitive than this, of course, but funny how old ideas eventually become new again.

View on HN

It's also interesting that nobody implements CKD on storage devices. This complexity runs on a massive POWER AIX box in IBM's storage systems, or in a similar commodity powerhouse running UNIX or an RTOS in similar arrays of the past couple decades.

You have a comparatively weak CPU (think ARM Cortex M series) on an SSD and drive companies have been notoriously bad at firmware development.

Frankly, I don't trust Samsung to implement a file system. I prefer to not even use their enterprise flash based on past trauma, but those are usually harder fails than the subtle fuckups they can pull off in an opaque FS.

You already are using a translation layer in the SSD, most SSDs have wear levelling, error correction, compression, etc. What your OS things it's writing to disk is very different to what ends up stored in the flash chips. I think your are fighting yesterday's war.

Fighting opaque and unsupportable complexity will always be the current war in technology.

That is exactly the reason I am using ZFS for important data -- one should not really trust the storage devices to be fully reliable.

Is is a real pity that so few filesystems systems have integrated data checksumming -- I'd love to use something other than ZFS for this.

SSDs are not the only storage device that lies about these things. Just about the only one that doesn't is tape, and I'm not even sure about those.

See also CAFS (Content addressable file store) [1], developed in the 1970s by ICL, which was at the time the UK's flagship mainframe computer company. It didn't make a huge impact on the market, probably because it was expensive to make and expensive to run. [1] https://en.wikipedia.org/wiki/Content_Addressable_File_Store

In "Moving targets: Elliott-Automation and the dawn of the computer age in Britain" Simon Lavington has documented[1] a computer built by Elliott Brothers (later Elliott Automation, then subsumed into ICL) named OEDIPUS that used content-addressable hardware storage. It went into production at GCHQ in 1954.

The computer also gets a mention in his 2011 book "Alan Turing and his contemporaries".

http://news.bbc.co.uk/2/hi/technology/8490464.stm

Thank you for sharing that. Reading material for tonight ;) I googled ECKD and it seems it is still in use and support by IBM (maybe through emulation).

It really only still exists for legacy reasons.

The original indexed file format under MVS, ISAM, heavily relied on the physical hardware keys on (E)CKD disks. In the 1970s, IBM replaced ISAM with VSAM, which didn't use that physical key feature and handled all the keys in software – and it was actually faster. (I don't know if the reason for it being faster was due to some fundamental flaw with the underlying idea, or just due to limitations of the particular implementation.) (Despite being deprecated, IBM continued to support ISAM until the mid-2000s, when they finally removed it from the operating system.)

IIRC, the legacy PDS (partitioned data set) format also uses it, but not the newer PDSE 1.0 or PDSE 2.0. (PDSE = partitioned data set extended). (PDS is like an archive file format, for object code it functions similar to .a files on Unix, for source code it was used to make up for the fact that originally MVS didn't really have the concept of directories, so keeping all your program's source modules in a single dataset was more convenient.)

I think the mainframe filesystem (VTOC) also uses it a bit.

There's probably a few other random things in z/OS that still need it, but the newer stuff (VSAM, PDSE, HFS/zFS, etc) doesn't really use it. I think if IBM really wanted to, they could add support to z/OS for running on industry-standard FBA disks instead of ECKD. (They did the same thing to VSE all the way back in the 1980s.) I think the main reason they don't do it, is not technical, but commercial – mainframes needing special SANs helps keep their storage business alive, if mainframes could work on industry standard SANs, they'd face more competition in that area.

This involves simply exposing how the spinning rust drive works internally to software.

On most magnetic drives there is no inherent relationship between the sector number and its physical location on the track and the drive simply waits for the sector with correct sector number in its header to come under the head.

> most magnetic drives there is no inherent relationship between the sector number and its physical location on the track

This is false. Sectors are approximately in block order. You can write a predictor which can predict the latency to read a block based on the last block read using that information.

You can also take the lid off a drive and watch it with a high speed camera to see where each bit of data gets stored.

On Prime minicomputers back in the 80's, there was a disk controller order to format a track. That's what broke it into sectors and put sector ID fields into each sector, in whatever way the disk controller wanted.

A read was in 3 parts parts: first do a seek over the right cylinder, then select a head, then read a sector. That least part read everything spinning by until it saw a sector with the correct sector ID.

Sectors were not in block order because if you read sector 1 then tried to read sector 2, it would have already spun past the head, meaning sequential sector reads would require a complete revolution of the disk.

So instead, sectors were interleaved with a skip. There were 9 sectors on a track, so the sector IDs might be written as: 0 3 6 1 4 7 2 5 8. You could read 3 sectors per disk revolution instead of 1.

There usually is some relationship, but does not have to be.

> but funny how old ideas eventually become new again.

I dislike this meme - things are generational, not cyclical.

Reply on news.ycombinator.com