Hacker News Reader

Comment by usernew | Hacker News Reader

Umm sorry, no. I'm a storage engineer with one of the leading enterprise vendors. SSDs are already "short-stroked." They have much more space available internally than you can see through the controller externally. The more "enterprisey" the drive, the more of that hidden space there is.

A wise sysadmin buys a drive with more of that hidden internal space, instead of getting a larger drive with less hidden space. The logic inside the drive is much better at zeroing/out and background trim on that internal hidden space, than host-addressable blocks that are currently unused. That's because the drive has no idea if you're about to use them, while the hidden space is guaranteed to be always free.

In fact - a fun fact. Flash has been used as a Write buffer on storage arrays for a couple of decades.

View on HN

Most of this hides the problem. It still exists though.

I have a piece of logic in my indexing code that essentially transposes an ~100 Gb multi-value dictionary on disk. If you do this the naive way with random writes, the write amplification makes it a complete non-starter. All the caching layers and buffers fill up with completely disjointed 8 byte writes and it takes ages to write.

What I've ended up doing is to in an intermediate stage write the data to be written into a series of files, containing pairs of offsets and data (up to like 100Mb each); and then going over the files one by one and essentially evaluating them as assembly instructions.

Both passes have relatively good data locality, and despite essentially writing 2.5X as much shit to disk, it takes hours rather than weeks to do this operation.

Short stroke in a SSD era is more about trying to keep your data within a page of a SSD. Depending on the flash drive you are using the controller may or may not do some of it for you. One of the easiest perf gains I can sometimes get from a program that is write intensive is to put a small amount of write buffer into the mix. Depending on your OS that can help a lot too. Basically keep my code out of the kernel and off the bus and keep the write block to something that closely resembles what the drive considers a block. If you are doing a bunch of small 8 byte writes randomly in your file you probably are having a bad time as you may quickly exceed the amount of buffer the drive has for that sort of thing. It will start backing you off which bleeds up into the kernel space then very quickly into your program. Keeping them together can help, as you found out.

I see this sort of issue in a lot of programs. As it is a dead easy problem to make in your program. You need to write something out you just splat it out somewhere. With a dozens 1/4/8/16 byte writes instead of one big write. Basically not thinking about how that data is getting into your files. Most of the time that is just fine and not that big of a deal. But as your data set grows or you want better perf you have to worry about it. I usually use something like filemon and can see what is going on. You can see the pattern where there will be a large stack of I/O with hundreds of very small read and writes. While SSDs are an order of magnitude faster than the older drives. They still have their command structure and kernel context switching you have to deal with. You in some cases want to minimize that as it can become a large portion of writing and reading data. As with most optimizations (in this case the drive is faster) we just moved where the bottleneck is (to the kernel and bus typically).

Short stroking data doesn't writes within a page. It gives the FTL more time to shuffle data around in pages when garbage collecting and erasing pages. Erasing a page tends to take milliseconds of time vs microseconds of time for reads, tens to hundreds of microseconds for writes. NAND flash is typically written in smaller pages (512 bytes in older products vs 4KB-16KB ) but erased in larger pages (32KB+). It's only NOR flash that supports smaller byte granularity write sizes. NOR flash is not as dense as NAND, so it doesn't get used much beyond embedded applications.

exactly. Also My point was more about keeping yourself out of calling into the kernel and paying the price for the context switches. Which can add up quickly.

Context switches between processes are bad, but going into the kernel for an irq using things like aio and io_uring which can populate events into a result ring buffer doesn't add a ridiculous amount of overhead. Optane DIMMs avoided that, but the consequences were such that the hardware had a lot more complexity. Doing something like an FTL at DDR4 speeds and latency is very difficult. Plus you don't really want to expose hardware that requires wear leveling directly to applications, as a buggy app can wear out (damage) the hardware prematurely.

A wise sysadmin buys a drive with more of that hidden internal space, instead of getting a larger drive with less hidden space.

The pricing of enterprise SSDs is... not exactly fair for the performance you get.

that's like complaining a bentley is too expensive and not good at moving your piano.

when you get fined $1mil/min by the SEC for downtime, and data loss or corruption can cost you in the hundreds of millions, or someone dying at a hospital, enterprise gear is cheap. reliability is the key and for what you are paying. now yes, there may be a specific consumer drive more reliable than a specific enterprise drive. so which do you buy? well the enterprise array vendor tested the crap out of everything in every combination and workload and environment, and picked one for you, and it comes with full support and SLAs that you can use to meet gov regulations.

performance for an enterprise drive is not something you usually consider, at all. in fact, did you know that when I quote a storage array, I can't even specify the drive type of vendor, and who knows what will get shipped? Ionly specify drive size.

your performance comes from all your workloads clumped together, spread over a thousand of these drives connected with infiniband, with dedupe and compression on the backend, and hundreds of terabytes of RAM. the perf of an individual drive is not relevant.

but yes, sticking it into an AMD server you bought on newegg when they had a sale is not its purpose, and is a very bad deal.

now when we talk about wise sysadmins, we're talking about guys who know their stuff, and do "important big stuff." not a guy at a small business ordering from newegg. and that guy - he shouldn't be coming up with any storage policies because he lacks the needed large-scale experience.

I sold a 1PB usable-effective (after 3x dedupe/compression) space storage array last year. It was $2mil, after a 65% discount. If one time in its 5year lifecycle the "enterprisey" stuff on that array prevents about 10 seconds of downtime, it paid for itself.

Sure, and judging by System Z sales there's still a demand for earthquake-resistant machines with hot-swappable CPUs.

But for most of us, infrequent downtime is acceptable, and most single-machine downtimes are either automatically mitigated or have very limited impact. In that scenario, getting a prosumer SSD and over-provisioning it can be a sensible choice.

* Hetzner joined the chat

There are many other ways to get reliability at the software level, usually via redundancy. Even enterprise level SSDs doesn't guarantee you free of data losses and downtimes, you have to back them up regularly and preferably set up high availability for the database.

cool. so I have a few thousand VMs of various OSs and hypervisors, a few hundred home-built applications with an average of 10 components each, some mainframe, some ibm-i, a bunch of linux, solaris, aix, and HP-UX server, and I'm using mysql, sql server, postgres, oracle, and a couple of DB2. there may be an informix DB or two somewhere - I don't know, I manage storage. about ten different volume managers, about 30 different filesystems - and some raw disks of course. 10PB total space, it needs to be metro cluster replicated ten miles away for active/active, and async in a different state can be about a minute behind max.

So, write me some software that's going to make all that work and make the storage highly available. make sure it works for everything.

you're exposed to solutions that make that work, daily. when you swipe your credit card buying condoms - you have no idea how much happens so your transaction doesn't get lost, corrupted, or errored out.

> you have no idea how much happens so your transaction doesn't get lost, corrupted, or errored out.

Maybe he doesn't, maybe he does - you don't know nor do I.

I'm pretty sure this is how IBM salesmen used to respond when confronted with those newfangled Unix systems which were starting to appear here and there, nibbling first, then taking larger bytes out of their market share. Instead of the litany of diverse systems they'd have thrown LPARs, SYSPlexs and ESMs around but in the end it still came down to the same thing: this stuff is too complicated to be left to amateurs. They were right, in a way... until those amateurs grew their wisdom teeth and took a large part of their market away from them.

Yes, "enterprise" stuff is complicated - often overly so [1] - and it has its place. This does not make it the only viable solution to these problems, something will eventually come up to eat your lunch just like IBM saw its herd of dinosaurs being overtaken by those upstart critters from the undergrowth. Maybe some smart software system which "guarantees" data reliability and availability without the need for "enterprise" storage devices? It wouldn't be the first time after all.

[1] https://github.com/EnterpriseQualityCoding/FizzBuzzEnterpris...

It depends on what kind of performance you're looking for and how you value that performance.

Under sustained long duration (ie: hours to days) of continuous full load performing small block writes, even the highest rated consumer SSD drives will begin to show significantly reduced throughput in comparison to pretty much any enterprise SSD drives.

Additionally, it's totally normal to see >=5 drive writes per day and 5 year warranties on enterprise drives. Consumer drives usually are rated significantly below 1 drive write per day and rarely for more than 3 years of warranty. If you're performing a lot of writes, you're going to need to replace worn out SSDs and that's not free in a business (remote hands, downtime, etc). So buying less durable storage has costs which don't show up on the original purchase invoice but do need to be factored in within a business setting. The warranty period is the best indication of how confident the drive vendor is about drive durability.

More total space for the same price changes that equation.

Spending 2+X per GB means rather than 5x the same X it’s 5 times 1/2 or less total space. And that’s before you consider write amplification issues with having dramatically less total SSD space.

Enterprise SSD’s have a few benefits, but they shouldn’t be your default choice for all servers.

I always complained to our IT group about storage - until they showed me an invoice for the drives they are buying.

wait till you see your Cisco bill

> Consumer drives ... and rarely for more than 3 years of warranty.

Looking at my list of consumer SSDs and their warranties:

Samsung EVO 850: 2y

Samsung EVO 960: 3y

Samsung EVO 860: 5y

Samsung EVO 970: 5y

Samsung PRO 970: 5y

For comparison with spinning rust HDDs:

Seagate Ironwolf: 3y

WD Red (both EFAX & EFRX): 3y

But take a wider sample of consumer SSDs, a large majority are <=3 years. And look at enterprise SSDs.

Seagate Ironwolf and WD Red are not enterprise spinny hard disk drives, instead look at Seagate X18 or WD HC560.

Yes, I know, I was comparing warranty on consumer SSDs and consumer HDDs.

Consumer drives usually exclude wear out in their warranties making the number of years pretty much irrelevant. What is does show is that vendors are more confidence in their FTLs and hardware nowadays compared to the first high volume SSDs on the market more than a decade ago.

I am well aware that flash is overprovisioned. One consumer SSDs the overprovisioning is quite small (maybe 5-10%), so short stroking the SSD will have a much greater effect than on enterprise SSDs. It also has the nice side effect of giving the drive more free space to work with which allows the garbage collection room to be more gradual and efficient, which can improve performance for more write intensive workloads.

Actually, many storage arrays use battery backed RAM, not flash as a write buffer. Flash does not have the endurance needed to serve as the write buffer for a large storage array. Some products that use battery backed RAM for this purpose will dump the contents of that RAM onto flash. I worked on a messaging appliance that used that approach, albeit with supercaps to provide the hardware time to dump 4GB of DRAM onto a compact flash card. Supercaps were easier to monitor and maintain. There are also plenty of hardware RAID cards that use batteries for their write buffer as well.

Edit: there are also persistent memories like MRAM that don't need power to retain their contents which are used in this space as well.

yes, big arrays (symmetrix, shark) have battery backed everything - enough to take all the cache and dump it to disk when power is lost. terabytes of RAM. they also often use use a flash cache behind the RAM - that was the primary use for Optane. The reason for this, is sustained writes instead of peaks. If you let more and more sit in cache, eventually you have write folding. That results in less write IO on the backend, and you're able to have a higher sustained peak.

smaller cheaper arrays (unity, pure, isilon) and HCI (nutanix, vxrail) that don't have terabytes of RAM - they have gigabytes, and pretty much always use flash as a write cache. In fact, I cannot name one that doesn't.

No one in enterprise storage cares about the indurance of flash. All that means is that twice a year, a vendor engineer comes out to replace a few flash drives under your support contract.

And flash has been used as a write cache behind a smaller RAM cache, for two decades, by all major storage vendors.

Reply on news.ycombinator.com