Modern HDDs have gotten somewhat better | Hacker News Reader

Modern HDDs have gotten somewhat better

utcc.utoronto.ca

95 pointsgoranmoomin147 comments

I actually ran into this last year. We ordered about 2 thousand 8TB HDDs last year for a Ceph cluster and had an agreement with our server vendor that they are not supposed to substitute the HDD model. Well, seems like they ran out and a small fraction of the drives were of a different, older model. The performance difference was 2.5x for our workload:

Older model: 670 IO/s and 2.9 ms average latency

Newer model: 1680 IO/s and 1.1 ms average latency

We got the vendor to send us new drives and shipped the older model back to them.

Was the 'older model' in this case one of the models where it is known that manufacturers were shipping SMR drives without disclosing it?

No. Both models were PMR. I tried to find out more about what the differences are from the manufacturer, but didn't get far. Perhaps someone here can shed some light.

Older drive:

  model number: MB8000GFECR
  SATA Version: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
  ATA Version:  ACS-3 T13/2161-D revision 5

Newer drive:

  model number: MB008000GWWQU
  SATA Version: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
  ATA Version:  ACS-4

My guess at the time was that the newer drive had a deeper command queue, but I'm not actually sure, and I think both drives had the same amount of cache.

I'm not seeing any especially obvious differences between those models (aside from the ones you listed). Were the drive versions mixed in alongside each other behind the same storage controller? If so, I would wonder if perhaps the controller's handling of different SATA versions played a role.

It's possible. Yes, the drives were sprinkled around in the inventory, and we had some servers where there was a mix of old and new drives in the same box, behind the same controller.

We use the controller in HBA mode, so the controller shouldn't be doing anything fancy, but you might be right.

I've seen controllers in HBA mode pull some oddball performance shenanigans when presented with differences as small as running different drive firmware versions across connected disks, so it really wouldn't surprise me. SATA drives have a pretty bad reputation for firmware releases introducing unexpected and undocumented behaviors. I used to be on the customer success team for a hyperconverged distributed storage product, and such things were the bane of our existence. Over the years the inventory of product-supported SATA devices shrunk considerably as a.) more and more quirks were discovered in products and b.) flash prices continued to drop to the point that spinning disks were in less and less demand, putting downward pricing pressure on SAS devices.

Those model numbers seem to be the HP part numbers. Don't HP rebadge drives from other vendors?

If so, then it would probably be useful to see if the vendor drive model numbers are available.

For the dramatic increase in iops, I wonder if there are more head+actuator assemblies on the newer drives or similar? Along the lines of these:

https://www.seagate.com/innovation/multi-actuator-hard-drive...

These are both Seagate Exos drives, one is a generation newer than the other. I'm pretty sure both have the same number of platters, thus same number of heads.

And yes, HP changes the firmware and model numbers... super annoying.

> had an agreement

>We got the vendor to send us new drives and shipped the older model back to them

not much of an agreement then, shows where the power lies

>MB8000GFECR

Seagate ST8000NM0055 at 2-4x the price

>MB008000GWWQU

another seagate

> because of course disk vendors quote everything in the smaller SI units).

SI prefixes should be (and are) the rule rather than the exception. It only made sense to measure ram in base-2 units because they are manufactured according to base-2 addressing.

HDD’s have never used base 2 addressing.

Another example is network media. Serialized data transfer speed is measured in {K,M,G,T}b/s (bits) or B/s (Bytes) where a Gigabit is exactly 1000000000 bits and Gigabyte is exactly 1000000000 bytes.

In the early days of computers ram was extremely scarce and expensive. It was such a limitation that it was usually the very first question you asked about a computer’s capabilities.

I remember in 5th grade asking a fellow student who’s parents had just bought an Atari 800 “How much ‘K’ does it have?”

This intense focus on ram may have led people to assume that it’s odd nomenclature would apply to anything computer related.

Well but they did use 512 B sectors

Base 2 is used in the world's most common desktop operating system(s) so it's no surprise that people get confused. They buy a hard drive that's supposed to be 1TB and when they install it their computer says it's 0.9TB, that's surprising to those not in the know.

Despite hard drive manufacturers using correct units, people still blame hard drive manufacturers for their disappointment.

Apple switched macOS to base 10 a while ago so the UI matches the hardware.

Android and desktop Linux also use base-10. It's a Windowsism at this point. Though some Android vendors specifically use the wrong unit in their software.

I have to disagree. Computers measure storage in base 2, this is why it is people's expectation.

Marketers measure in base 10 because it lets them inflate the stated capacity in a way that isn't legally fraudulent.

>"Computers measure storage in base 2"

What do you mean? 'Computers' don't measure storage, they execute programs. Humans (often programmers) are the ones who want to know about storage.

Those executed programs measure storage.

'Measuring' is a human paradigm.

So is everything else related to computers. Humans made computers. Computers are a human paradigm.

No, many things related to 'computers' are determined by physics and chemistry. In any case, 'computer' is being used loosely (and poorly) here, which is why I have been putting it in single quotes. 'Computer' can mean many things.

https://en.wikipedia.org/wiki/Processor_register#Size

If you look at register size, you'll believe that different processors use different 'bases', depending on whether they're 8-bit, 16-bit, 32-bit, or 64-bit...

And it just happens to be a coincidence those are all powers of 2?

I understand why we use binary, and the implications on processors, but I still think it is wrong to say: "Computers measure storage in base 2"

Almost all numbers inside computers are base 2 (for floats, the exponent and fraction are one such number each).

Base 10 is something that is primarily used for presentation to humans.

We all know this, this is HN....

Fine, the humans that use computers measure storage in base 2. What a pedantic hill to die on.

There were also 12, 24, 36 and 48 bit computers. 32 Bit just won because of System/360.

Computers access things in powers of 2 because we settled on a byte being 8 bits. We then proceeded to doing things in even multiple of bytes. A cache line, row in DRAM, registers, and a page of memory in your MMU are all some even multiple of bytes. And back in the day when we couldn't put 300 million transistors on a chip, because the systems work in binary, it was also quite useful to be able to do things like calculate the address of a pixel in the frame buffer by simply shifting the row index and adding the column index, meaning that something like 1024 pixels wide was much simpler to implement than 999 pixels wide.

There's lots of reasons why computers so often use powers of two and thus K being 2^10 instead of 10^3. But the word measure is probably not the best.

Isn't that "base-10"?

HDDs, RAM, network speed were base-2 (e.g. 1TB = 1024GB) for most of my life until SSDs became common and even Google now says "1 Gigabyte = 1000 Megabytes" to my exasperation.

Technically it’s gibibyte(s).

AFAIK network and HDDs were always measured in si units, only RAM was measured in powers of 2.

For hard drives it was complicated, since their size is a function of the number of cylinders, heads, and sectors. Since the sector size is normally 512 bytes, they were a multiple of that. See e.g. this table of old hard drives:

https://www.computerhope.com/hdquantu.htm

E.g., if you take the Quantum Bigfoot 1.2 GB model:

2492 cylinders * 16 heads * 63 sectors * 512 bytes per sector = 1286111232 bytes

The size it was sold as, is definitely closer GiB (as in power of two):

1.2 * 2*30 = 1288490189

With rounding, they could sell it as 1.3 GiB with the current marketing trick of using SI units.

So the old branding strategy seems to have been: compute the size in bytes, then find a reasonably MiB or GiB value (powers of two).

The gap between SI prefixes and their base-2 approximations grows as sizes get larger.

Before 1GB HDD’s the file system overhead was a much more dramatic effect so it didn’t really matter to consumers which prefix convention they used or how they rounded.

As I see it the GB numbers are powers of 1000 rounded down:

2492 * 16 * 63 * 512 / 1000 ^ 3 = 1.286111232

4092 * 16 * 63 * 512 / 1000 ^ 3 = 2.111864832

4994 * 16 * 63 * 512 / 1000 ^ 3 = 2.577383424

Using GiB you get different numbers if you look past the first element in the table:

2492 * 16 * 63 * 512 / 1024 ^ 3 = 1.197784423828125

4092 * 16 * 63 * 512 / 1024 ^ 3 = 1.966827392578125

4994 * 16* 63 * 512 / 1024 ^ 3 = 2.4003753662109375

So I stand by my point, they have always used SI units. It's only ram (and software) that used incorrect definitions of k, M, G and T.

The whole power-of-10 thing for storage goes back forever-- to IBM mainframes and to the first hard drives used in PCs.

E.g. Seagate ST-506, first 5.25" hard drive.

"The total formatted capacity of the four heads and surfaces is 5/10 megabytes (32 sectors per track, 256 bytes per sector, 612/1224 tracks)."

612 * 32 * 256 = 5013504, or 5.01 megabytes or 4.78 mebibytes.

> HDDs, RAM, network speed were base-2 (e.g. 1TB = 1024GB) for most of my life

No, network speeds are the exception in that they have always used base-10 for a long time. For example, the basic ISDN data rate of 64kbit/s is exactly 64000 bits/s (8000 samples of 8-bit PCM per second); a 9.6k modem is 9600 bits/s; a 10 megabit Ethernet link is 10000000 bits/s; and so on.

It's absurd to claim they're doing this for any other reason beyond deceptive marketing. It's like an advertising version of a dark pattern: a decision which has a sheen of reasonableness, but is clearly designed to deceive and exploit your customers.

Nowhere else in the computer is the base-10 version standard, and it shouldn't be. The sentence "a 32-bit CPU register can address up to 4 gigabyte of memory" is clearly reasonable, the sentence "a 32-bit CPU register can address up to 4.29497... gigabytes of memory" is not. It's not just ram though: all basic data types (bytes, ints, floats, pointers, SIMD registers, whatever) are sized to powers of 2, having a storage unit that is not is insanity. Or, to bring it back to hard drives: what's the max file size of a file stored on a FAT32 filesystem again?

It's just bad and deceptive, and hard drive manufactures should not be allowed to play these kinds of games. You should stop carrying the water for their shitty practices.

The sentence "a 32-bit CPU register can address up to 4 gigabyte of memory" is sloppy, the sentence "a 32-bit CPU register can address up to 4 GiB of memory" is not.

> The sentence "a 32-bit CPU register can address up to 4 gigabyte of memory" is sloppy, the sentence "a 32-bit CPU register can address up to 4 GiB of memory" is not.

"Up to" is marketing. From my experience "up to" starts from 0.

32 bits can address 4GiB. That's how 2^32 works. It can also address 2GiB, or 1GiB. I think the wording is fine.

Like saying a car can go up to 200mph, no one thinks the car is unable to go 50mph.

How many meters are there in a kilometer? 1024?

There are 1000 meters in a kilometer. However, there are 1024 meters in a kibimeter.

Kibi, the prefix that comes from kibitzing.

I wonder why RAM manufacturers are so generous that they don't engage in deceptive marketing, while everyone else does.

It goes back to the '60s and '70s.

Back then, a "byte" was not necessarily "8 bits", but whatever the smallest unit of addressable memory was. This was the same as or an even fraction of the word length of the computer. Common word lengths back then included 12, 18, 24, and even 30 bits. 8-bit bytes would only fit evenly into one of those.

Storage devices like hard drives are not addressed like memory, and the logical segmentation of bits into bytes/words/clusters was a function of the software, not the hardware. One storage device could be built for use in an 8-bit or 12-bit computer, resulting in very different numbers if you tried counting its size in bytes on these different machines.

Nowhere else in the computer is the base-10 version standard

As the parent said, networking. (Which also suffers from an unfortunate bits/bytes mismatch.)

The real problem with storage IMO is that software used base-2 measurements and vendors used base-10 and eventually the discrepancy got big enough to be noticeable.

For me, the egregious part isn't even the difference in base 2 and base 10 but rather that windows uses base 2 but displays it as base 10 units. (Internally uses Gibibytes but displays the unit as Gigabytes, for example)

> Nowhere else in the computer is the base-10 version standard

Well, you're simply wrong. CPU frequency is measured in GHz, that is 1,000,000,000 Hz, not 1,073,741,824 Hz. Consequently, my CPU has a nominal frequency of 3.0 GHz, not 2.793967724 GHz. Since frequencies are denominated in powers of 10, bandwidths are as well: The L1 cache in my CPU can read 64 bytes/cycle (I think), so it has a read-bandwidth of 192 GB/s, that is 192,000,000,000 B/s. My Ethernet device can transfer 1 Gbit/s, that is 1,000,000,000 bit/s, because it's clocked at 125 MHz (that's 125,000,000 Hz) and transfers 8 bit per clock cycle.

> all basic data types (bytes, ints, floats, pointers, SIMD registers, whatever) are sized to powers of 2

Yeah, and it pretty much ends with basic data types. Most of my structs aren't powers of two in size, even though their individual elements are 1, 2, 4, or 8 bytes in size. Almost none of my files have sizes related to powers of two. My SSD has a size of 931.51339 GiB – so tell me again how powers of two are more correct or more useful here?

> Almost none of my files have sizes related to powers of two.

They are all rounded up to 512 or 4096 bytes by your filesystem.

> My SSD

SSDs operate using erase blocks sized in mebibytes, and expose sector sizes of 512 B or 4096 B to the OS. [1] HDDs a similarly low-level-formatted with 512 or 4096 B sector sizes.

[1] https://spdk.io/doc/ssd_internals.html

> They are all rounded up to 512 or 4096 bytes by your filesystem.

That's an implementation detail, and neither I nor my programs care about it. We do care about the actual length of the file, which isn't related to powers of two, except in niche cases.

> SSDs operate using erase blocks sized in mebibytes, and expose sector sizes of 512 B or 4096 B to the OS.

While true, this is also an implementation detail, and I don't think about the size of my SSD in multiples of sectors or pages or kibibytes – "976762584 kiB" just isn't a useful number for me, but "1000.2 GB" is manageable. Specifying it as "931.51 GiB" isn't more correct, so it makes sense to default to powers of 10 instead of powers of 2.

The situation would be different if HDDs and SSDs always or typically had exact sizes like 512*2^30 (= 512 GiB) or 3*2^40 (= 3 TiB) – you know, like RAM – in which case using powers of 2 would make sense. But that's not the world we're living in.

> The situation would be different if HDDs and SSDs always or typically had exact sizes

I just checked two of three SSDs available to me. Two were exact multiples of power-of-two gigabytes -- 200 GiB, 120 GiB. One was 500.3 GB.

SI units might be more useful to a consumer (and I am not arguing they aren't). But the argument that storage systems have "nothing to do" with powers of two is not valid.

HDD sizes are set by manufacturers according to the IDEMA "LBA Count for Disk Drives Standard"[0] formula:

  LBA_count = 97696368 + 1953504 * (Reported_Capacity_in_GBytes - 50)

So, for 120 GB HDD:

  (97696368 + 1953504 * 70) * 512 / 1024^3 = 111.79 GiB
  (97696368 + 1953504 * 70) * 512 / 1000^3 = 120.03 GB

For 200 GB HDD:

  (97696368 + 1953504 * 150) * 512 / 1024^3 = 186.31 GiB
  (97696368 + 1953504 * 150) * 512 / 1000^3 = 200.05 GB

For 500 GB HDD:

  (97696368 + 1953504 * 450) * 512 / 1024^3 = 465.76 GiB
  (97696368 + 1953504 * 450) * 512 / 1000^3 = 500.11 GB

Doesn't look like powers of 2 at all. Are you sure about your sizes?

[0] http://www.idema.org/wp-content/downloads/2169.pdf

Logical Block Addresses were simple back when all blocks were 512 bytes. Now we have 520 (T10-PI), 4096, etc, so your math falls apart.

For disks with sectors of 4096 bytes, the formula is the same except the constants are accordingly 8 times smaller.

Regarding sectors of 520 bytes, usable bytes are still 512. Quote from the above referenced document:

> HDDs formatted with T10 PI (Protection Information) have 8 bytes added to the end of each sector. However, the user usable sector size will remain at 512 bytes or 4096 bytes. The 8 bytes of PI will increase the overhead of the disk sector density similar to overhead of ECC bytes in a given sector.

> They are all rounded up to 512 or 4096 bytes by your filesystem.

Not always. Some filesystems allow small files to be compacted into other structures: the MTF can hold small in NTFS, ext* and btrfs support similar inlining of data in inodes for small files, other filesystems (ZFS, Reiser, btrfs again) support more general sub-block allocation (with techniques such as tail packing or block sub-division). Sometimes this is only for small files not “oddly” sized larger files, but things like tail packing can apply to the latter too.

I may not be the biggest fan of it ever but it is a fact that this is how hard drives have been measured since the days when a 5.25" full height MFM 10MB or 20MB hard drive in an IBM XT or clone was state of the art technology.

your comment seems to insinuate that this is some new sneaky marketing practice of HDD manufacturers to trick the consumer.

Right. It’s an old sneaky marketing practice of HDD manufacturers to trick consumers, which is still a sneaky marketing trick, just not a new one.

at this point I think it's something we will have to deal with, like, I'd prefer to buy carpet by the square meter, but fact is in canada it's still sold by the square foot just like in the USA, with weird non metric units.

Give it a little more time and it'll be like 2x4s. At one point they were actually 2" by 4", but now they're 3.5" by 1.5" and that's the standard size and we're stuck with this dumb little detail where the name doesn't match reality and everybody just accepts it - but it confuses virtually every single person on the planet at LEAST once or twice in their life.

if you really want to see something that's irretrievably stuck in american measurements, take a look at the national engineering standards and vendor sources for telecom tower components... for anything from a light duty 40' aluminum self support you might put in your yard for ham radio all the way up to gargantuan 2000' guyed broadcast towers.

canada has no hope of doing anything in metric here, everything is in US units, and probably will be until the year 2200

2x4s are 2" by 4" before they're finished. If you search S4S (sanded 4 sides) 2x4s, they will actually be 2" by 4". If you get the wood with rounded corners, they take a quarter inch off each side to "finish" it. It's been that way for 100 years.

Sneaky, eh?

I don't think that's right.

> The limited availability of lumber and the rapid pace of housing construction made other methods like concrete-block housing viable. This pressured further compromise because thinner 2x4s were a way to compete in price with wood alternatives. Size standards, maximum moisture content, and nomenclature were agreed upon only as recently as 1964. The nominal 2x4 thus became the actual 1½ x 3½, imperceptibly, a fraction of an inch at a time.

http://www.harvarddesignmagazine.org/issues/45/nominal-versu...

So it seems the reduction was was driven by cost saving, not a product of the finishing process, and it's actually only been standardized that way for 58 years.

What is 2x4?

a north american measurement of wood

It refers to pieces of wood used for building in North America, which comes in 2 inch by 4 inches by x feet, where x is generally 8 or 10.

So, basically a 50x100mm piece of lumber?

(I rounded those values because if the US were to go metric, the dimension would slightly change, rather than being exactly 50.8 by 101.6 mm)

Except, per my original point - it’s called a 2x4 - but it’s not actually 2 inches by 4 inches, which would be as you say 50ish x 100ish mm. In reality it’s actually standardized to be 38 x 89 mm

Guess 40 by 90 then.

What I tried to convey, in a stupid way, is that there wouldnt be an exact translation of the measurements into metric, but that the dimensions qould slightly change in switching over.

and now I realize that that point was irrelevant anyways. Sorry for derailing the thread with useless replies.

https://en.wikipedia.org/wiki/Lumber#Dimensional_lumber

"Dimensional lumber is available in green, unfinished state, and for that kind of lumber, the nominal dimensions are the actual dimensions."

> For example, a "2×4" board historically started out as a green, rough board actually 2 by 4 inches (51 mm × 102 mm). After drying and planing, it would be smaller by a nonstandard amount. Today, a "2×4" board starts out as something smaller than 2 inches by 4 inches and not specified by standards, and after drying and planing is minimally 1+1⁄2 by 3+1⁄2 inches (38 mm × 89 mm).[8]

Your quote just means when you're buying green wood and it says it's 2"x4" it will be exactly 2"x4". It doesn't mean a 2x4 started out as a 2"x4" green board.

> Historically, the nominal dimensions were the size of the green (not dried), rough (unfinished) boards that eventually became smaller finished lumber through drying and planing (to smooth the wood). Today, the standards specify the final finished dimensions and the mill cuts the logs to whatever size it needs to achieve those final dimensions.

I was originally pointing out that the naming is simply a standard and convention, 'dimensional lumber'. All that matters is that if I buy a '2 by' almost anywhere in the United States, it will be close to the same size and characteristics.

It wasn't sneaky way back then. A "byte" has not always been defined as "exactly 8 bits"; it used to be hardware-dependent. 6-bit bytes were common in systems with word lengths that were multiples of 6 instead of the now-common 8.

You have to be very familiar with tech to have any idea about base-2 vs base-10. Universal standards to make computing units understandable would very clearly go with base-10, which is the status quo in units elsewhere (metric). Base-2 is just an implementation detail, and one that the vast majority of the world simply does not care about.

Maybe not today. But they did in previous decades.

> very familiar

Only slightly familiar. Binary number systems are explained in introductory first books on computers. I knew binary nearly 10 years before I got my hands on a computer.

Yes, but let's not allow the Americans to "reinvent" the SI-prefixes for computers..

If you want base-2 then use kibiBytes (kiB) and move on..

You need to have some domain knowledge to know -- did someone think the power of 2 was important here?

For communications they mostly haven't (we tend to deal with decimal clock rates, bandwidths, etc-- and even when we have 128-QAM or whatever that's going to not result in powers of 2). For memories and caches we generally like them to be powers of 2 to use all the bits of our addressing and decoders well. For secondary storage, ... we tend to have a non-round number of lumps that are a small power of 2 sized, and we might as well use decimal.

This is subtle and has been woefully inconsistent. It would be better to just use powers of 10.

(Me of 20 years ago would be very disappointed in myself for writing this comment).

> Or, to bring it back to hard drives: what's the max file size of a file stored on a FAT32 filesystem again?

That is not about hard drives, that is about the abstraction layer of FAT32, which uses a 32-bit quantity for size.

FAT32 could be used for a storage device that stores things based on powers-of-10 sizes. It's just an implementation detail if anything happens to do it with powers of 2 underneath.

> FAT32 could be used for a storage device that stores things based on powers-of-10 sizes

Not without wasting lots of space. FAT32 only allows sector sizes of 512, 1024, 2048, or 4096 bytes.

[1] https://www.cs.fsu.edu/~cop4610t/assignments/project3/spec/f...

> It's absurd to claim they're doing this for any other reason beyond deceptive marketing.

Your comment is a good example of "I so want this narrative to be true!" without any backing.

Back in the 90's, my computer studies textbook had 1 KB as 1024 bytes, and 1 MB as 1,000,000 bytes. It explicitly stated that it is not 1024 KB. What's their angle?

> It's absurd to claim they're doing this for any other reason beyond deceptive marketing. […] It's just bad and deceptive, and hard drive manufactures should not be allowed to play these kinds of games. You should stop carrying the water for their shitty practices.

The confusion of binary and decimal prefixes has been around for decades:

* https://en.wikipedia.org/wiki/Timeline_of_binary_prefixes

This is not some recent conspiracy to boost profits or whatever.

If this troubles you, wait until you learn how many bytes are on a 1.44MB floppy disk.

It holds 1,474,560 bytes.

That is 1.44 * 1000 * 1024.

zero people in 2022 use floppy disks (and yes I still have mine)

Anecdata: I have a lovely old Canon Cat in working condition that I want to sell to someone who is interested in this piece of computing history. But I have somehow misplaced the box of floppy disks that came with it!

I know they are here somewhere. Looks like I have some housekeeping to do.

When I find them, that number will become nonzero...

Redefining meanings is like defining π = 3.

SI metric prefix standards really really matter to an engineer. The creation of a new indistinguishable unit which varies depending on what was measured (so that 1k != 1k) is mind-bogglingly idiotic in this old engineer’s opinion.

As a child I too felt the pain of feeling “shortchanged” by hardware manufacturers’ marketing practices with hard drive sizes, and I deeply understand the glamour of using 1024 based decimals for declaring total memory. However the cost of differing values for metric prefixes (k, M, G) is confusion, mistakes, and arguments. I also think that any card carrying geek knows their measurements, and cannot be easily deceived. Another argument for why software engineers are not. Disclaimer: last gig was JavaScript.

No, it is absurd to claim that something which predates using base 2 based counts being common is designed to con people who work in that sort of unit.

Communications theory used 1,000s and such to state transfer rates as with any other scientific unit long before modern drives. That stuck when things were recorded to mediums such as tape, and every disk ever has been measured on 1,000s or 1,000,000s or …

There have been some oddities along the way: punched cards were all over the place and were measured in characters not bits/bytes, many comms methods too were measured in characters/symbols or modulations rather than bits & bytes, many comms and storage (tape, disk, and RAM/ROM) formats are not nice round powers of 2 in raw form due to check bits, parity bits, and stop bits.

The one and only occurrence I can think of where 2^10 was used officially for a physical storage measure is the 1.44Mb floppy format which was a strange bastard of multiple divisions: it was 1440KiB, so 1.4410001024, so really was either ~1.38MiB or ~1.47MB.

> Nowhere else in the computer is the base-10 version standard, and it shouldn't be.

Nowhere else in science nor enginerring is 2^10 the standard instead of 10^3, and it shouldn't be. In computing we've adopted 2^10 that because some things are easier for us to think about that way, and 2^10 and 10^3 are close enough for many estimating or quick calc purposes, but why should science that predates that practise being common be forced to change simply for our convenience?

Yes I support moving from imperial to metric where that hasn't already been done, but not because I encountered metric first (I didn't) so the other is unfamiliar but because it makes sense. You might argue that to programmers or less technical tech users 2^10 makes more sense, but they are less important in terms of science and engineering then, well, the rest of science and engineering.

> what's the max file size of a file stored on a FAT32 filesystem again?

2GiB, or approximately 2.147GB. Many people incorrectly calling it 2GB does not change that G as a prefix means 10^9 and not 2^31 - otherwise we have to accept other things like “u” being a correct spelling of “you”!

It isn't a grand conspiracy to defraud us. We chose to give those prefixes a double meaning, it isn't the rest of science & engineering's fault that we sometimes get confused by that choice that we made and they didn't.

> HDD’s have never used base 2 addressing.

The two most common sector sizes on HDDs are 512 bytes (a byte being 8 bits) and 4096 bytes. Base 2 units make a lot more sense with these power-of-two sector sizes than base 10 units.

Ethernet packages are 1518 bytes or 12144 bits. Maybe we should start measuring network speed in power-of-12144.

No honestly: Kilo should always mean 1'000, Mega should always mean 1'000'000 and not 1'048'576.

> Kilo should always mean 1'000, Mega should always mean 1'000'000 and not 1'048'576.

And that's why we have "kibi", "mebi", etc.

So Kilo means times 1,000. Kibi means times 1,024 and so on.

>HDD’s have never used base 2 addressing.

Still the "minimum" unit, the sector or "block" is 512 or 4096 bytes, so the multiple/power of 2 is "ingrained" in the media.

Any HDD spec sheet will give a base-2 definition for "MB". They use this when reporting cache size.

Funny enough RAM is still used to segment products in price classes even though its cheap. Want to buy a entry level mac? 8GB here you go, now you can only have a couple of tabs up and god forbid you open up photoshop while browsing. It wont run out of memory but it will page to disk making switching programs slower. Next you have an OS update which conveniently consumes more RAM.

Hard drives (both rotary and SSD; floppies [1] and CDs [2] also) are also manufactured according to base-2 addressing. They are low-level formatted at the factory in sectors which are power-of-2 sizes. [3] This has been true for at least 40 years. [4]

[1] https://en.wikipedia.org/wiki/Disk_formatting#Low-level_form...

[2] https://en.wikipedia.org/wiki/CD-ROM#Sector_structure

[3] https://en.wikipedia.org/wiki/Disk_formatting#Low-level_form...

[4] https://en.wikipedia.org/wiki/Disk_sector

> Another example is network media. Serialized data transfer speed is measured in {K,M,G,T}b/s (bits) or B/s (Bytes) where a Gigabit is exactly 1000000000 bits and Gigabyte is exactly 1000000000 bytes.

interesting network trivia: a LANPHY and a WANPHY 10 gigabit ethernet connection are not quite the same thing. The latter is designed to fit within legacy DWDM transport systems designed to carry OC-192 so its data rate will fit within 9953.28 Mbit/s or slightly under that.

As files have gotten bigger though, I have found myself in situations where I've had to explain why, if it's taking longer than you expect for a file to transfer, about 7% of that is unit conversion errors.

So while there is a bit of waste and we could do some work to speed up the transfer, it's still going to be 10% less than you think is attainable.

But even so, the gigabyte sins of the hard drive makers are still less than half of those of the network card makers who used to advertise baud instead of bytes, so the fact that the data is going across as 5:4 encoding means that you divide the Kbps by 10 - not 8 - to get the actual throughput.

Fuck those guys.

> But even so, the gigabyte sins of the hard drive makers are still less than half of those of the network card makers who used to advertise baud instead of bytes,

Actually, they generally specified bits per second, even if they called it "baud".

> so the fact that the data is going across as 5:4 encoding means that you divide the Kbps by 10 - not 8 - to get the actual throughput.

Stop bits and start bits. E.g. a 9600 bit per second v.32 MODEM operated at 2400 baud (2400 4 bit symbols per second), and had an end-user throughput of 960 characters per second.

"preach". RAM is sort of relevant to this though.

Clock speed is base10. Line speed is base10. Storage is base10. But ram is base2, because it's almost always constructed as a grid.

Tapes are essentially unitless, how long is a piece of string. The earliest magnetic harddrive I can find is the IBM 350 storing 5 million 6-bit characters. Where 5 million is 5 and a bunch of zeros. it's not 4.75 .. mibilion?

Where things get weird, is when we start dividing tracks into sectors. For convenience we went for sectors that were convenient to page into memory. Every harddrive since that has had a base10 number of base2 sectors. From that point onwards, you can be team base10 or team base2, they both end up wrong.

And this is how we end up with the 1.44MB floppy, where 1.44MB is 2880 * 512kb. Or 1.44 * 1000 * 1024.

Why does the storage size unit of measure have to do with addressing? That drive addressing uses the LBA-48 (bit) addressing has nothing to do with the fact the minimum addressible unit size, the physical sector is a base-2 value, i.e. 512 bytes or 4096 bytes.

Still seems the storage unit size should be IEC based, not SI.

Fun fact: The dimensions of a magnet (a bit of data) on an HDD are actually rectangular. It's about 3-5x wider than it is long. This favors read/write speed, since you cover more bits per rotational length. Aside from the obvious speed boost you get from it, it's also done to make it easier to stay on track.

One of the huge limitations with bit width is the ability to keep the actuator arm stable within the 15 or so nanometers because of the air turbulence. That's why they fill the drives with helium now, less aerodynamic drag/turbulence means you can either cram more platters in (which increases the turbulence back to where it originally was) or make narrower tracks since you have more precision.

Is it too hard to maintain vacuum? The comparison I can think of is a thermos, which I'm told uses a vacuum layer between the eg coffee, and the outside.

Vacuum won't work with current technology. The heads use air currents to fly a few nanometers over the platter. If you put a running drive in a vacuum chamber the heads will touch down and everything will be destroyed.

https://en.wikipedia.org/wiki/Flying_height

230MB, 100MB, or 50MB per second doesn't make that big a difference for anything I'd use a HDD for. What I want is lower pricing. The 12TB drive I bought two years ago hasn't changed in price. The 8TB drive I bought four years ago is $30 more expensive now than it was then.

I think we might be hitting a point where technology improvements can no longer outrun inflation.

All kinds of "legacy" products go up in price-- RAM a couple generations back, etc.

SSDs have won. Spinning hard drives are going to ship fewer and fewer units. The economies of scale are not going to improve ever again and are already getting worse. Further, the market just doesn't care about paying for capacity for the most part.

We peaked in 2010 with 650 million drives/year. Current unit counts are about one third of that. Compare to SSDs with about 400 million units/year.

All I want is a cheap tape drive for backups. Seek can be slow, as long as throughput is high.

I too would like a tape drive. It's frustrating that tape is so cheap, yet the drives cost so much for home use.

If you go used on LTO you can get a drive a few generations back for not insane prices, but it's still not entirely competitive with additional spinning rust.

Yeah, it seems once you go back far enough where the drives are reasonable, the capacity is just too low to be cost-effective vs buying a HDD or two.

i miss the days when optical drives were practical for backups

Same. There's something fun about optical disks.

The drives I looked at were cheaper when I checked a few months ago? Have prices gone up again?

This is my pain point. For those of us who are aware the cloud exists and would rather do tape anyway, why on earth is a recent vintage tape drive so expensive?

I was talking spinning disks - tape is too niche now

It's been a year since prices recovered from the earlier madness & got back to where they were. It's not totally stable, but prices as I see them have been going slowly down since.

I personally have some Exos 16TB's. There's a decent price history available[1] reaching back to mid-2019. Make sure to tick on "3rd party new". These aren't going to be the best prices available in many cases, but it's reasonably indicative.

[1] https://camelcamelcamel.com/product/B07SPFPKF4

I always buy enterprise drives - much higher reliability and lifetime and usually I have found they benchmark higher. They are also way heavier and almost always use better bearings on all surfaces. Some people complain they are super loud but the three types I have randomly purchased over the years have been just as quiet as any other spinning disk (as in I do not notice the sound at all once its inside my desktop)

For 1 drive, the differences are perhaps not much. When you have a cluster with 2k HDD drives, the difference in handling both squential and random workloads (see my other comment) can add up.

A 2.5x difference in IO/s adds up quickly.

As for HDD prices, neither 12TB nor 8TB were top of the line at the timeframes you mention. If you buy largest available drives, or a model lower, the prices have been going down. Not as much as we all would like, but overall the trend has been moving in the downward direction.

The helium filled drives also appear to be much, much more reliable. Only thing that worries me about them is that if you need to do disaster recovery you are pretty much out of luck unless you can get the original manufacturer to do it or find a highly specialized place.

- adjusted for inflation?

- did you buy those drives in the "sweet spot" of capacity per dollar? 8 and 12 TB drives aren't the sweet spot anymore

- there's also this covid disruption thing

- when was the factory fire in ?vietnam?

- and how much competition is there? Are we down to two companies basically?

Oh wow, you're right. I bought two 12TB WD drives in March 2020 for $175 each. The same drive on Amazon is now $218.

The pair of 8TB I had previously are within $5 of what I paid in 2017.

Might be temporary? How about if you look at different drives like seagate enterprise or HGST or Toshiba?

I purchased several 8TB WD Red's about 18 months ago, when they were ~$300. Can now find them for ~$180. Same story for IronWolf.

This is the result of "more bits-per-inch", which is the natural result of going from 4TB-per-drive to 20TB-per-drive.

Yes, I'm skipping a few steps, but that's the fundamental issue at play here. There's more platters, more bits per inch, more read / write speed at the same 7200 RPM.

Its not a lot, but its a steady increase as hard drives keep getting denser and denser.

At 7200rpm, would increased data density on the disk account for this? These new ones are HAMR…

Yes, exactly. The drive has to be faster to keep up with the platters.

TL;DR: Modern 7200RPM 20TB HDDs can do 230MB/sec sustained—which was news to me!

Yes, but that appears to be if you're doing sequential reads. Works for a handful of use cases, but not most normal desktop or server workloads.

Sounds ideal for large arrays for video post use. We used to string 16 drives together to get speeds of what a modern m.2 SSD can do as a single drive. Only, the arrays gave you the capacity to do useful things rather than the toy-like capacity of 2-4TB SSDs. Doubling the speed of spining rust might be interesting when building RAID-6 volumes for edit bays

But even if in "normal server workloads" it's comparable to older drive you still get the benefit during things like RAID recovery where it very much will be sequential operations dominating. So it's an improvement which can really matter for unlikely but very critical server operations.

Seems like around the 6TB mark is where they broke 200MB/s.

Possibly up to double (480MB/sec) that soon for large sequential read workloads. Seagate has their multi-actuator Exos 2X14 Mach.2 drives which should start hitting some level of channel availability within the coming weeks.

For video editing, ETL and other read-heavy workloads, they have a point.

They are actually presented as two drives, so in order to benefit you have to overlay a RAID 0 or some other form of striping.

I've also seen a lot more reasonably priced gear include 2.5gbit LAN, which is a good pairing if you're home rolling a NAS for example.

Only takes you 8 days to read it, funny right? ;)

230MB/s, not 230Mbps; one day, not eight.

There's one that can do over 500MB/s : https://www.storagereview.com/news/seagate-exos-2x14-hdd-del...

You've been out of the loop for a while. Certain HDDs have been able to achieve such speeds for about a decade, if not more. Most of their (slow) gains over the last 10 years have been in capacity.

Is the status quo on drive firmware lying about what's actually been flushed to media still as terrible as it was, oh, twenty years ago?

A friend used to work for Apple and said that one of the reasons they had apple-specific versions of various mass market SCSI and IDE drives was because Apple's firmware actually flushed data to disk when told to do so, and didn't lie about whether data was flushed or not.

If there is un-written cached data there is a capacitor bank that allows this data to be quickly written to non-volatile flash and rw heads to be retracted in the event of power loss. They test this feature pretty heavily, esp. in enterprise drives. Source: used to work for an HDD manufacturer. So, they might be lying to you but it is pretty useless information on for modern drives.

Basically, "flushed" means safe. Whether it's written to internal DRAM/SRAM (with power-loss prevention), NV flash, or the actual platter is irrelevant. When in history did the lie start to not matter?

The other interesting thing about modern data centre HDDs is they have a media cache which significantly accelerates synchronous random write IOPS - OOTB these drives get 75 IOPS with fio fsync=1 bs=4k randwrite, but with a WCE=0 they can do 400 IOPS.

So if your app is fsync heavy (such as Ceph) then you can switch on this media cache by setting the drive to write through mode (WCE=0).

SATA SSDs have a similar quirk.

I'm never buying another HDD in my life. There are a few tasks that take weeks on an HDD but only hours on an SSD. And the price difference is often negligible.

Using a single Prometheus node with redundant drives rather than redundant nodes with single drives is an odd choice to me. Why replicate at block level when layer 7 has support?

A multi-node Prometheus setup is significantly more complex to design and operate than a Linux software RAID mirror, especially once you throw in Grafana, Alertmanager, and so on. Distributed anything makes it harder. The layer 7 may 'support' this, but it's not a plug and play setup.

(It also requires more hardware.)

(I'm the author of the linked-to entry.)

Articles that could have been a tweet

How are the random reads/writes? AFAIK this contributes a lot to a device feeling responsive.

I stopped using HDDs all together like 8 years ago and I haven’t had any issues at all

Reply on news.ycombinator.com