NVMe is not a hard disk
blog.koehntopp.info
blog.koehntopp.info
I do believe that's been the author's experience. However, I think he may be unaware that's not everyone's...or even most people's, experience.
The parents point is really spot on though: most websites aren't at scale where every stateful api has redundant implementations. But the author's point does have merit: inevitably, something goes wrong with all systems - and when it does, your system goes down if you trust your HA config to work. If you actually did go for redundant implementations, your users likely aren't even gonna notice anything went wrong.
It's however kinda unsustainable to maintain unless you have a massive development budget
He's also doing wasted work his redundancy has bugs that a consumer has to worry about working around.
I think the author was specifically talking about RAID-1 redundancy and is advocating that you can leave your systems with RAID-0 (so no redundant drives in each server), as you're gonna need multiple nodes in your cluster anyway... so if any of your systems disks break, you can just let it go down and replace the disk while the node is offline.
but despite being offtopic: redundant implementations are -from my experience - not used in a failover way. they're active at all times and load is spread if you can do that, so you'd likely find the inconsistencies in the integration test layer.
Aurora works like that. Read replicas are on standby and become writable when the writable node dies or is replaced. They can be used, of course, but so can other standby streaming replicas.
The larger your infrastructure, the smaller the relative efficiency win that's worth pursuing (duh, I know, engineering time costs the same, but the absolute savings numbers from relative wins go up). That's why an approach along the lines of "redundancy at all levels" (raid + x-machine replication + x-geo replication etc) starts becoming increasingly worth streamlining.
Another, separate consideration is types of failures you have to consider: an availability incident (temporary unavailability) vs. durability (permanent data loss). And then it's worth considering that in the limit to long durations, an availability incident will become the same as a durability incident. This is contextual: To pick an obvious/illustrative example, if your Snapchat messages are offline for 24h, you might as well have lost the data instead.
Now, machines fail, of course. Doing physical maintenance (swapping disks) is going to take significant, human time scales. It's not generally tolerable for your data to be offline for that long. So local RAID barely helps at all. Instead, you're going to want to make sure your data stays available despite a certain rate of machine failure.
You can now make similar considerations for different, larger domains. Network, power, building, city/location, etc. They have vastly different failure probabilities and also different failure modes (network devices failing is likely an availability concern, a mudslide into your DC is a bit less likely to recover). Depending on your needs, you might accept some of these but not others.
The most trivial way to deal with this is to simply make sure you have a replica of each chunk of data in multiple of each of these kinds of failure zones. A replica each on multiple machines (pick the amount of redundancy you need based on a statistical model from component failure rates), a replica each on machines under different network devices, on different power, in different geographies, etc.
That's expensive. The next most efficient thing would be to use the same concept as RAID (erasure codes) and apply that across a wider scope. So you basically get RAID, but you use your clever model of failure zones for placement.
This gets a bit complicated in practice. Most folks stick to replicas. (Eg. last I looked, for example HDFS/Hadoop only supported replication, but it did use knowledge of network topology for placing data.)
The reason why you don't want to do this in your application is because it's really kinda complicated. You're far more likely to have many applications than many storage technologies (or databases).
Now, at some point of infrastructure size or complexity or team size it may make sense to separate your storage (the stuff I'm talking about) from your databases as well. But as Kris argues, many common databases can be made to handle some of these failure zones.
In any case, that's the extremely long version of an answer to your question why you'd handle redundancy in this particular layer. The short answer is: Below this layer is too small a scope or with too little meta information. But doing it higher in the stack fails to exploit a prime opportunity to abstract away some really significant complexity. I think we all know how useful good encapsulation can be! You avoid doing this in multiple places simply because it's expensive.
(Everything above is common knowledge among storage folks, nothing is Google specific or otherwise it has been covered in published articles. Alas, the way we would defend against bugs in storage is not public. Sorry.)
Image you have two servers, each with two 1TB disks (4TB physical storage). And you have two distinct services with 1TB datasets, and want some storage redundancy.
One option is to put each pair of discs in a RAID-1 configuration, and so each RAID instance holds one dataset. This protects against a disk failure, but not against server or network failures.
Your other option is to put one copy of the dataset on each server. Now you are protected from the failure of any single disk, server, or the network to that server.
In both cases you have 2TB of logical storage available.
In either case what you're describing isn't really the 'cloud' alternative.
it is not. if you have 10 DB hosts with the same brand of nvme, and they fail under a workload, what good is it that you have 10 hot-hot failover hosts? you just bought yourself a few days, or hours if you are specially unlucky and using consumer grade.
For that you have time delayed replicas and of course backups. But time delayed replicas are usually much faster to restore than a backup.
That provides capacity, but also redundancy. Better redundancy than at the disk level - fewer resources are shared.
https://blog.koehntopp.info/2021/03/24/a-lot-of-mysql.html Here is how we run our datatbases.
That's one of the perceptions I'm saying isn't always true. Especially in big, non-tech companies that have a mish-mash of crazy legacy stuff. Traditional H/A disks and backups still dominate some spaces.
Only really huge services would care about downtime of this degree, or horizontal scaling.
At 400 MB/s, you restore a terabyte in 45 minutes, and then you need to replay the changes that happened since the backup (in MySQL: replay the binlog), which will take aproximately another 15 minutes.
"One hour of MTTR or provisioning time per Terabyte of data at 400 MB/s sustained linear I/O speed" is a useful rule of thumb.
Having a replica that you can promote reduces that to seconds. Having a time delayed replica where you just roll forward the binlog reduces that to minutes.
You can work with modern databases without using replication, but that in most cases just shows you suck at operations.
When I ran large-scale Exchange and database systems, we would always get into big fights with the storage people, who believed that SAN disk with performance tiering, replication, etc was the way to solve all problems.
The problem at the time was that the fancy storage cost $40+ Gb/mo and performed like hot garbage because the data didn’t tier well. For email, the “correct” answer in 2010 was local disk or dumb sas arrays with exchange dag.
For databases, the answer was “it depends”. Often it made sense to use the SAN and replication at that layer.
This is repeated multiple times in the article, and I refuse to believe it is true. If NVME/SSDs never overwrote anything, they would quickly run out of available blocks, especially on OSs that don't support TRIM.
Rather than change the paint color in a hallway you have to tear down the house and build a new house in the vacant lot next door that's a duplicate of the original, but with the new hallway paint.
To optimize, you keep a bucket of houses to destroy, and a bucket of vacant lots, and whenever a neighborhood has lots of "to be flattened houses" the remaining active houses are copied to a vacant lot and the whole neighborhood is flattened.
So, things get deleted, but not in the way people are used to if they imagine a piece of paper and a pencil and eraser.
- lines on page
- pages of paper
- whole notebooks
"When a datacenter catches fire, we just rope it off and rebuild one town over."
This is a very important property of SSDs and is a large reason why log structured storage is so popular in recent times. The SSD is very fast at appends, but changing data is much slower.
(Also you need the translation layer even for write-once since flash generally doesn't come 100% defect free. Not sure if manufacturers could try to get it there, but that'd probably drive the cost up massively. And the translation layer is there for rewritable flash anyway... the cost/benefit tradeoff is in favor of just living with a few bugged cells.)
FWIW, there are WORM microsd cards available but it looks like they still use flash under the hood.
Because of the Internet age there probably is not much place for write once media anyway, even it would be somewhat cheaper. But maybe for specialized applications or if it would be much much cheaper per GB.
Because the older Flash aren't as stable when miniaturized as you'd expect. Current flash is a direct descendant of these, they only are more stable because the cells are much chunkier and thus with lower leakage.
That being said, it's not clear how innovation has occurred in this direction in the storage space.
It could theoretically make them cheaper. But I guess that there wouldn't be enough demand, so you'd be better off having some kind of OS enforced limitation on it.
There is a question on stackoverflow about this topic with many answers but no definitive conclusion. There seem to be some papers touching the subject but at a glance I couldn't find anything useful in them.
[1] https://superuser.com/questions/4307/what-lasts-longer-data-...
My layman intuition (which could be totally wrong) is that trapped electrons have a natural tendency to escape due to pure thermal jitter. Whereas magnetic materials tend to stick together, so there's at least that. Don't how much of this matches the actual electron physics/technology though...
I don't have intuition either, but I don't think this explanation is sufficient
From the physics point of view, aren't both cases the same thing?.
Isn't magnetic regions a state of the electric field? so if I move electrons in and out, the electric field should be changing as well
So flash memory is about the electrons, while magnetic memory is about the ions.
"The level of charge in each cell must be kept within certain thresholds to maintain data integrity. Unfortunately, charge leaks from flash cells over time, and if too much charge is lost then the data stored will also be lost.
During normal operation, the flash drive firmware routinely refreshes the cells to restore lost charge. However, when the flash is not powered the state of charge will naturally degrade with time. The rate of charge loss, and sensitivity of the flash to that loss, is impacted by the flash structure, amount of flash wear (number of P/E cycles performed on the cell), and the storage temperature. Flash Cell Endurance specifications usually assume a minimum data retention duration of 12 months at the end of drive life."
You have to be careful how you interpret this bit. "Normal operation" here assumes not just that the SSD is powered, but that it is actively used to perform IO. Writes to the SSD will eventually cause data to be refreshed as a consequence of wear leveling; if you write 1TB per month to a 1TB drive then every (in-use) cell will be refreshed approximately monthly, and data degradation won't be a problem.
If you have an extremely low-write workload, the natural turnover due to wear leveling won't keep the data particularly fresh and you'll be dependent on the SSD re-writing data when it notices (correctable) read errors, which means data that is never accessed could degrade without being caught. But in this scenario, you're writing so little to the drive that the flash stays more or less new, and should have quite long data retention even without refreshing stored data.
I mean... This is general purpose HW here. Write once SSD is a workflow more than an economically tenable use-case in terms of making massive size write once then burn the write circuit devices.
No, it's worse than that. The fact that it's an overly subtle distinction is the problem.
SSDs are fast while write traffic is light. From an operational standpoint, the drive is lying to you about its performance. Unless you are routinely stress testing your system to failure, you may have a very inaccurate picture of how your system performs under load, meaning you have done your capacity planning incorrectly, and you will be caught out with a production issue.
Ultimately it's the same sentiment as people who don't like the worst-case VACUUM behavior of Postgres - best-effort algorithms in your system of record make some people very cranky. They'd rather have higher latency with a smaller error range, because at least they can see the problem.
Maybe there is a better analogy or paradigm to view this through.
So -- you keep track of vacant lots and "dead" houses (abandoned but not flattened); whenever you've got spare time you will copy blocks with some ratio of "live" to abandoned houses to new lots so the new block only has live houses.
These pending / anticipatory compaction/garbage collection operations are what I refer to as "buckets" -- having to compact 300 (neighborhoods) blocks to achieve 300 writes is going to result in glacial performance because of this huge write amplification (behind the scenes the drive is duplicating 100s of mb / gb of data to write a small amount of user modifications)
As you might imagine, there are lots of strategies to how to approach this problem, some of which give you an SSD with extremely unpredictable (when full) performance, others will give a much more consistent but "slower" performance.
The RAS command eradicated the entire row, like 1024 bytes or so. This is because the DDR4 cells only have enough charge for one reliable read, after that the capacitors don't have enough electrons to know if a 0 or 1 was stored.
A row close command returns the data from the sense amps back to the capacitors. Refresh commands renew the 0 or 1 as the capacitor can only hold the data for a few milliseconds.
------
The CAS latency statistic assumes that the row was already open. It's a measure of the sense amplifiers and not of the actual data.
The "fact" that you can do it in your program without disturbing bytes around it is a convenient fiction that the hardware fabricates for you.
Pretty much only cache is RAM proper these days (aka: all locations have equal access time... that is, you can access it randomly with little performance loss).
The distinction seems to be how big the chunk is not uniformity of access time (is a symmetrical read disk not a block device?)
So yes. DDR4 has surprising similarities to a 512byte sector hard drive (modern hard drives have 4k blocks)
>> What’s the difference between a cache line and a row in RAM?
Well DDR4 doesn't have a cache line. It has a burst length of 8, so the smallest data transfer is 64 bytes. This happens to coincide with L1 cache lines.
The row is 1024 bytes long. Its pretty much the L1 cache on the other side, so to speak. When your CPU talks to DDR4, it needs to load a row (RAS all 1024 bytes) before it can CAS read a 64 byte burst length 8 chunk.
-----------
DDR4, hard drives, and Flash are all block devices.
The main issue for Flash technologies, is that the erase size is even larger than the read/write block size. That's why we TRIM for NVMe devices.
In terms of performance analogy though, hard drives do not get random access to blocks, but RAM does. The practical block size of hard drives is sequential reads of 100kiB+ due to seeks.
I think this makes the comparison unhelpful since the characteristics are still very different.
If flash was the same way, and it could rewrite an entire erase block with no consequences, then you could ignore erase blocks. But it's nowhere near that level, so the performance impact is very large.
There are only 10,000 erase cycles per Flash cell. So a lot of algorithms are about minimizing those erases.
So, the drives do indeed never "overwrite" data - they mark the block as unused (either when the OS uses TRIM, or when it writes new data [for which it picks an empty block elsewhere]), and put it in a queue to be erased whenever there's time (and energy and heat budget) to do so.
Understanding this is also quite important because it can have performance implications, particularly on consumer/low-end devices. Those don't have a whole lot of spare space to work with, so if the entire device is "in use", write performance can take a serious hit when it becomes limited by erase speed.
[Add.: reference for block sizes: https://www.micron.com/support/~/media/74C3F8B1250D4935898DB... - note the PDF creation date on that is 2002(!) and it compares 16kB against 128kB size.]
Security implications too. The storage device cannot be trusted to securely delete data.
To zap a drive you ask it to securely drop the self-encryption key. The data will still be there, but without the key it is indistinguishable from random noise.
On NVMe/PC type applications with a controller driving the flash chips… I have absolutely no idea. I'm curious too, if anyone knows :)
When it comes to what non bare flash drives do, you can start here: http://www.vldb.org/pvldb/vol13/p519-kakaraparthy.pdf
This paper is imperfect and the following citations are worth skimming. There's a cohort of similar papers chasing the same basic question in recent years that aren't densely cited amongst each other.
Go here next: https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.46... but note that's just a jumping off point to the more recent papers.
It's hard to gain a full understanding of this layer because it's the basis of intense competition, hence held closely by controller manufacturers.
I'm far from world expert on this, but have read a lot about it and can answer with what I know to the best of my ability.
I didn't downvote your sibling response, but I did ignore it since it provided neither any sources nor any context for why I should trust your knowledge. Apparently others were less kind on your short statement.
With the additional information in this post, I'm much more willing to accept it into my head — thanks for answering this!
It might also help to keep in mind that both regular disk drives and solid state drives remap bad sectors. Both types of disks maintain an unaddressable storage area which is used to transparently cover for faulty sectors.
In a hard drive, faulty sectors are mapped during production and stored in the p-list, and are remapped to sectors in this extra hidden area. Sectors that fail at runtime are recorded in the g-list and are likewise remapped.
Writes may usually go to the same place in a hard drive, but it's not guaranteed there either.
This is because for SMR HDDs, each block can either be SMR (higher density, EXTREMELY SLOW WRITES like <10mb/s possible, erases will remove multiple blocks just like flash memory), or normal (standard density, normal write speeds).
The controller abstracts this away and does writes as normal, but while the drive is idle, the controller in the background, converts these standard blocks into SMR blocks.
This is also why SMR HDDs support TRIM.
Seems it's increasingly a bad idea to presume the implementation of internals.
Flash can write blocks at a granularity similar to a memory page (cells, around 4-16 KB). It can erase only sets of blocks, at a much larger granularity (around 512-ish cell sized blocks).
The FTL will try to find free pages to write your data to. In the background, it will also try to move data around to generate unused erase blocks and then erase them.
In flash, seeks are essentially free. That means that it does no longer matter if blocks are adjacent. Also, because of the FTL, adjacent FTL are not necessarily adjacent on the physical layer. And even if you do not rewrite a block, it may be that the garbage collection moves data around at the PHY layer in order to generate completely empty erase blocks.
The net effect is that positioning as seen from the OS no longer matters at all from the OS layer, and that the OS layer has zero control over adjacency and erase at the PHY layer. Rewriting, defragging, or other OS level operations cannot control what happens physically at the flash layer.
TRIM is a "blatant layering violation" in the Linus sense: It tells the disk "hardware" what the OS thinks it no longer needs. TRIM'ed blocks can be given up and will not be kept when the garbage collector tries to free up an erase page.
TRIM is useful, it gives the GC important information.
TRIM is not that important as long as the device is not full (less than 80%, generally speaking, but it is very easy to produce pathological cases that are way off in either direction). Once the device fills up above that it is crucial.
> The net effect is that positioning as seen from the OS no longer matters at all from the OS layer, and that the OS layer has zero control over adjacency and erase at the PHY layer. Rewriting, defragging, or other OS level operations cannot control what happens physically at the flash layer.
I don't agree with this. The "OS visible position" is relevant, because it influences what can realistically be written together (multiple larger IOs targeting consecutive LBAs in close time proximity). And writing data in larger chunks is very important for good performance, particularly in sustained write workloads. And sequential IO (in contrast to small random IOs) does influence how the FTL will lay out the data to some degree.
If you feed your NVMe a stream of 1GB writes spread out at completely randomised OS visible places (LBAs), the FTL may very well write it sequentially and you get the solid sustained write performance.
Conversely, you may try to write 1GB of sequential LBAs, and your FTL may very well spread it out all across the physical blocks simply because that's what’s available.
What I'm saying is that sequential reads and writ workloads are good, but whether the OS considers them sequential or not in terms of LBAs is irrelevant. The controller ignores LBAs and abstracts everything away.
My understanding could be wrong, so please correct me if I am.
Whereas with larger/sequential writes, there's commonly no need for read-modify-write cycles. The entire previous erase block sized chunks can just be marked as reusable with new content - the old data isn't relevant anymore.
This is pretty easy to see by just running benchmarks with sustained sequential and random write IO. But on some devices it'll take a bit - initially the writes are all in a faster area (e.g. using SLC flash instead of denser/cheaper mlc/tlc/qlc).
Of course, if all the random writes are >= erase block size, with a consistent alignment to multiples of the write size, then you're not going to see this - it's essentially sequential enough.
I sync my photos to S3 (a mix of raw and jpeg, sidecar rawtherapee files) across a few devices so Glacier is prohibitively expensive in this regard, but I still pay <$100 a year for more stuff than I could ever store locally.
What's confusing is that the calculator has "Restore Requests" as the # of requests, "Data Retrievals" as TB/month, but there's also a "Data Transfer" section for the S3 calculator. If I add 1 restore request for 10TB of data (eg: restoring my full backup to the NAS), that adds about $26 for that month. Totally reasonable.
However, if "Data Transfer" is relevant, and I can't tell if it is or isn't, uploading my backup data is free but retrieving 10TB would cost $922! Is that right?
This is what has always deterred me from using AWS. It's so unclear what services and fees will apply to any given use case, and it seems like there's no way to know until Amazon decides that you've incurred them. At $10/month for storage and $26 if I need to restore, I can just set this up and I don't need to plan for disaster recovery expenses. But if it's going to cost me $922 to get my data back, I've got to figure out how to make sure my insurance is going to cover that. This isn't a no-brainer anymore. Also, what assurance do I have that the cost isn't going to be higher when I need the data, or that there won't be other fees tacked on that I've missed?
With Glacier as it is usually used you dont read data directly from the Glacier storage, it has to be restored to S3 where you then access it. That is the restore charges and the delays, so you can pay a low rate for the bulk option that takes up to 24 hours to restore your data to S3. But the real cost is the bandwidth from S3 back to your NAS/datacenter/etc, which brings it up to about $90USD/TB.
Other fees would include request pricing, some low amount per 1000 requests. So costs can go up a bit if you store 1 million small files to Glacier vs 1000 large files. There is also a tipping point (IIRC about 170KB) where it is cheaper to store small files on S3 than Glacier.
Depending on your data and patterns it can be better to use Glacier as a second backup which is what I do. All my data is backed up to a Google Workspace as that is "unlimited" for now. The most important subset (a few TB) also goes to Glacier. Glacier is pay as you go, there isnt some "unlimited" or "5TB for life" type deal that can change. If Google Workspace ever becomes not "unlimited" or something happens to it, I have the most important data in Glacier and its data that I have no qualms paying >$1k to get back.
But for me restoring from Glacier means that my NAS is dead (ZFS RAIDZ2 on good hardware) and Google Workspace has failed me at the same time.
I'll check out Google Workspace; that sounds like the right level of kludge for me, since this is the first time I've ever bothered to try to setup off-site backups. I only started using RAID a couple of years ago.
Workspace make more sense for large amounts of data where you can take advantage of the "unlimited".
Backblaze B3 might be what you are looking for as an in between option.
Note that the official name is "S3 Glacier", so from AWS's public perspective, it is S3.
There's another (unofficial!) calculator at http://liangzan.net/aws-glacier-calculator/ you can toy with.
That's right. AWS charges offensive prices for bandwidth.
There are alternate methods to get data out for about half the price, or you can try your luck on using lightsail and if they don't decide it's a ToS violation you could get the transfer costs to around $50.
I use a Windows free tool called FastGlacier. I set up an IAM user on my AWS account for my backups, and use those creds to login. Then it's drag and drop! You can even use FastGlacier to encrypt/decrypt on the fly as you upload and download.
Glacier is cheap because the retrieval times are very slow - something like 1-12 hours depending on the tier.
I have about 100GB of critical data. Personal documents, photos and some music I don't want to have to search for if the house burns down. It's something like a dollar a month. Less than a cup of coffee.
Of course, tapes require a tape drive and its cost would require a lot of data to compensate the cost, but at a such high cost of retrieval it would not take much data to equal the cost of a tape drive.
Glacier is OK for a couple of TB, but for tens or hundreds it would not be suitable.
But the less you expect to use it, the less this matters.
So I'd put the break-even point a bit higher. Tape is good for 100TB or more but for tens it's hard to justify a tape drive.
Also it's important to remember to get those tapes offsite every week!
Sources:
https://geizhals.eu/?cat=zip&xf=1204_LTO-6 (tapes)
https://geizhals.eu/?cat=hde7s&sort=r (disks)
(For completeness, the tape library ran me 500€ on eBay used, but they normally run about twice that. It's a 16-slot library, which coincidentally matches the breakeven - filled for 40TB it's 820€, the same in disks would've been 800€, though I would never buy the cheapest crappy disks.)
FWIW, a major reason for going for tapes for me was that at some point my backup HDDs always ended up used for "real". The tape library frees me from trying to discipline myself to properly separate backup HDDs ;)
For example, at Amazon in Europe an LTO-7 cartridge is EUR 48 for 6 real TB (EUR 8 per real TB), while an LTO-8 cartridge is EUR 84 for 12 real TB (EUR 7 per real TB).
At Amazon USA I see now $48 for LTO-7 and $96 for LTO-8, i.e. $8 per TB if you buy 1 tape. For larger quantities, there are discounts.
Older formats like LTO-6 have the advantage that you may find much cheaper tape drives, but you must handle more cartridges for a given data size.
Currently the cheapest HDDs are the external USB drives with 5400 rpm, which are much slower, about 3 times slower than the tapes, but even those are many times more expensive than the tapes (e.g. $27 ... $30 per TB).
That was a better than usual deal but $15/TB is the norm for low end drives with a bit of patience in the US, at least until Chia's influence hit extremely recently.
Depending on the business system you might want your server to be backed up multiple times a day to just weekly, and if you are doing multiple times a day you do want it to be decently fast or to have some extra capacity so as not to slow down normal operations while the backup runs.
The price per TB is currently about $7 or less (per real TB of LTO-8, not per marketing compressed TB), so there is no chance to approach the cost and reliability of tapes using HDDs.
The problem with tapes is that currently the tape drives are very expensive, because their market is small, so you need to store a lot of data before the difference in price between HDDs and tapes exceeds the initial expense for the tape drive.
Nevertheless, if you want to store data for many years, investing in a tape drive may be worthwhile just for the peace of mind, because when storing HDDs for years you can never be sure how will they work when operated again.
You'd actually need two tape drives to be reasonably secure, because getting a SATA adapter is much easier if your current setup is broken and you need to access the data either many years later (when technology moved on) or right now.
Having your data on a few hard drives combined with some check-summing and somewhat regular checking is probably easier. Or a combination of both - two different storage systems with two different failure modes are a lot safer. But when you're going that way the break even moves even farther away.
So we're still very interested in things like hard disks, NVMe, etc.
This post's point that even with that setup, it's still nice to have volume-based storage for quicker image replacements is interesting, I'm not experienced enough with cloud setups to know if that makes sense (eg how long does it take to upgrade an EC2 instance that has data on disk? upgrade your OS? upgrade your pg version? does ephemeral storage vs volume storage affect these? I imagine not…)
and rarely if ever true at personal/family level (most people don't think their phone/tablet/laptop could lose their data; most people don't think of their data past photos they took this week - "Oh it's OK to lose... wait, my tax documents are gone? Photos of my baby from two years ago are gone? My poems are gone? My [.. etc] is gone???"?).
The biggest thing I have realized (as have others), is that traditional IO wait strategies don't make sense for NVMe. Even "newer" strategies like async/await do not give you want you truly want for one of these devices (still too slow). The best performance I have been able to extract from these is when I am doing really stupid busy-wait strategies.
Also, single-writer principle and serializing+batching writes before you send them to disk is critical. With any storage medium where a block write costs you device lifetime, you want to put as much effective data per block as possible, and you also want to avoid editing existing blocks. Append-only log structures are what NVMe/flash devices live for.
With all of this in mind, the motivations for building gigantic database clusters should start going away. One NVMe device per application instance is starting to sound a lot more compelling to me. Build clusters with app logic (not database logic).
In testing of these ideas I have been able to push 2 million small writes (~64k) per second to a single Samsung 960 pro for a single business entity. I don't know of any SQL clusters that can achieve these same figures.
Traditional OLTP workloads do not tend to have the concurrency to actually saturate the NVME. You would need to be 40-way parallel, but most OLTP workloads give you 4-way.
Multiple instances per device are almost a must.
Completely agree. There is another angle you can play if you are willing to get your hands dirty at the lowest levels.
If you build a custom database engine that fundamentally stores everything as key-value, and then builds relational abstractions on top, you can leverage a lot more benefit on a per-transaction basis. For instance, if you are storing a KVP per column in a table and the table has 10 columns, you may wind up generating 10-20 KVP items per logical row insert/update/delete. And if you are careful, you can make sure this extra data structure expressiveness does not cause write amplification (single writer serializes and batches all transactions).
> If you build a custom database
> engine that fundamentally stores
> everything as key-value, and then
> builds relational abstractions on
> top
On top-line gear this isn't an issue, they don't signal a write cache (by virtue of either having a non-volatile cache or enough of a power reserve to flush the cache). Which then prevents the OS from actually doing more expensive for fdatasync()/O_DSYNC. One also can manually ignore the need for caching by changing /sys/block/nvme*/queue/write_cache to say write through, but that obviously looses guarantees - but can be useful to test on lower end devices.
> Multiple instances per device are almost a must.
Isn't actually unproblematic in OLTP, because it increases the number of journal writes that need to be flushed. With a single instance group commit can amortize the write cache flush costs much more efficiently than with many concurrent instances all separately doing much smaller group commits.
> Append-only log structures are what NVMe/flash devices live for.
I would think this is also good for filesystems like ZFS, APFS, and BTRFS, yes? I had an inkling but never really looked into it. Aren’t these filesystems somewhat similar to append-only logs of changes, which serialize operations as a single writer?
I guess I'm different from most people, good news! When building my new "home server" half a year ago I made a raid-1 (based on ZFS) with 4 NVMEs. I rarely appear at that city, so I brought the fifth one and put it into an empty slot. Well, one of the 4 nvmes lasted for 3 months and stopped responding. One "zpool replace" and I'm back to normal, without any downtime, disassembly, even reboots. I think that's quite useful. When I'm there the next time I'll replace the dead one, of course.
We then install SDS software, Cloudian for S3, Quobyte for File, and we used to use Datera for iSCSI. Lightbits maybe in the future, I don't know.
These boxen get purchased with 4 NVME devices, but can grow to 24 NVME devices. Currently 11 TB Microns, going for 16 or more in the future.
For local storage, multiple NVME hardly ever make sense.
But a disk can only be a hot spare in one pool, so to have a "global" hot spare it has to be done manually. That may be what that poster was doing.
It's these reasons that I've decided to just keep a couple of cold spares ready that I can swap in to my system as needed, although I do have access to the NAS at any time. If I was remote like GP, I might decide to use a hot spare.
This is a good insight, but you need to be sure the disks are independent.
that's why it's not recommended when running ceph.
[0] https://www.digitalocean.com/community/questions/poor-fsync-...
You can build in redundancy at the component level, at the physical computer level, at the rack level, at the datacenter level, at the region level. Having all of them is almost certainly redundant and unnecessary at best.
I'd rather have ZFS or something equivalent in software. Software can be inspected, is (hopefully) battle-tested for years by many different companies with different workloads & requirements, and worst-case scenario, because it's software, you can freeze the situation in time by taking byte-level snapshots of the underlying drives as well as a copy of the software for later examination/reverse-engineering, something you can't do with a hardware black box where you're bound to the physical hardware and often have a single shot at a recovery attempt (as it may change the state of the black box).
Have you heard of the SSD failures about a decade ago where the SSD controller's firmware had a bug that bricked the drive past a certain lifetime? The data is technically still there, and would be recoverable if you could bypass the controller or fix its firmware, but unless you had a very good relationship with the manufacturer of the SSD to gain access to the internal tools and/or source code to allow you to tinker with the controller you were SOL.
But yes, hardware RAID controllers and ZFS don't go together.
> most database-like applications do their redundancy themselves, at the application level …
If that’s not the case for your storage (doesn’t sound like it), then the author’s point doesn’t apply to your case anyway. In which case, yes, RAID may be useful.
> I guess I'm different from most people, good news!
The earlier part of the sentence helps explain the difference: "That is, because most database-like applications do their redundancy themselves, at the application level..."
Running one box I'd want RAID on it for sure. Work already runs a DB cluster because the app needs to stay up when an entire box goes away. Once you have 3+ hot copies of the data and a failover setup, RAID within each box on top of that can be extravagant. (If you do want greater reliability, it might be through more replicas, etc. instead of RAID.)
There is a bit of universalization in how the blog post phrases it. As applied to databases, though, I get where they're coming from.
1: LSI has a NVMe RAID controller for U.2 chassis, preparing for a non-RAID future, just in case.
The total cost might be similar but you have increased reliability over SBC/controller/uplink failure.
Of course there are tradeoffs on performance and ease of management...
This is doubly true for physical or self-hosted systems.
It looks like this: https://blog.koehntopp.info/2021/03/24/a-lot-of-mysql.html
It seems like these solutions don't have a strong authentication/encryption-in-transit story. Are vendors building this into proprietary products or is this only being used on trusted networks? I think it'd be solid technology to leverage for container storage.
One thing I’m noticing is that most of these storage protocols do, in fact, assume converged Ethernet; that is, zero packet loss and proper flow control.
Is this also the case with NVMe over TCP?
It seems to me applications taking advantage of NVMe are focused on building out deep queues of operations which may smooth out issues introduced by the network. But only way to know is to benchmark.
Why is this better than just connecting the drive via PCIe bus directly to the CPU?
At some scale, it's nice to separate the two. You don't care much about where the disks live vs. where your compute is running. You can evict work from a node, have it reschedule, and not have to worry about replicating the data to another machine with excess capacity. Though I'm no authority on this topic.
That being said, dual head stacks (as opposed to just dual actuators on a single head stack) would almost certainly not fit within the footprint that ordinary drives do, which would mean not being able to use standard disk enclosures for them... which would be a non-starter. Alternatively you shrink the platters, which means less capacity - and given that hard disk drives have been stuck at a certain areal density limit for a while I imagine nobody's particularly keen on losing lots of capacity for a small IOPS benefit.
IDK - maybe if HAMR goes mainstream in hard drives then we can make that tradeoff without the drive seeming like an inferior product.
Yes. You'd probably need to use 2.5" platters for a 3.5" enclosure. Also not sure if the mechanics would be precise enough to allow one head to read what the other wrote. When IBM did this, densities were MUCH lower.
As for capacity vs IOPS, 10K and 15K RPM drives rarely have the same densities as the slower large capacity units. I don't think I ever saw a 10+TB drive faster than 7200 RPM.
Edit: followed the link, it is 2.5:1 oversubscribed
major router and switch manufacturers aren't even making new line cards with multiple 40GbE ports on them anymore, they're tech from 6-7 years ago. You can buy either something with dense 10GbE SFP+ ports, or something with QSFP/QSFP28 or similar 100GbE ports.
If you want something for a test/development lab, or are on a very tight budget, sure.
Today you'd rather use Arista 7060, 7050cx orJuniper 5200 as ToR. You'd not build 1:1, but plan for it in terms of ports and cable space, then add capacity as needed.
Almost nobody in an Enterprise environment actually needs 1:1, unlike hosters or hyperscalers renting nodes to others. Even then you'd probably being able to get away with a certain amount of oversubscription.
For large processing of data in general requests (maybe like a traditional SQL query) AI might typically require the retrieval data to map to vector formats. But it's not applicable to add those to RDBs (you're not doing any sql upfront, retrieval will be too slow), nor compute those ad-hoc (business app use cases, not general CV or NLP models, nor analytics).
AI/ML inputs/vectors need to exist somewhere where it is fast access vs abstracted software over that access; so using embedded dbs (leveldb, rocks, badger, lmdb, etc). Anecdotally, I get top performance over 8-24 physical ssd attachments (TBs of data), nvme raid-0 single mount on individual nodes, where we need to gather anywhere from 1 - potentially 10s of thousands of vectors every request. It's never more performant to distribute that and nothing is really gained with persistence. Today's platforms are somewhat geared for fast recovery (though writing TB's of data isnt simple, you can model automation & scale horizontally). Still, app platform infrastructure isn't really geared for AI application architectures like this. E.g., Kubernetes is too abstracted, but I imagine Google/GCP will, intentionally for AI workloads, enable CSI drivers and more robustness around Local SSDs on GKE.
That being said, nvme seems nice with no major footguns, so I guess NVME over X is in principle at least a good idea.
Ethernet is absorbing all the features of other networking technology, and adding them to it's own perfection.
I have no idea what this is even supposed to mean. It's like somebody combined some buzzwords thought up by a fresh business school marketing graduate working in the 'cloud' industry with an attempt at actual x86-64 hardware systems engineering.
The whole premise of the first half of the article seems to be 'you don't need to design a lot of redundancy and fault tolerance', the second part then goes into a weird explanation of NVME targets on CentOS. I hope this person isn't actually responsible for building storage systems at the bare metal level supporting some production business application.
The conclusion seems to be that it's not important for hardware level data redundancy because existing database software already handles duplication in application code. I don't understand how that conclusion was reached. Hardware level redundancy like raid1 seems useful because it simplifies handling a common failure case when a single HDD or NVME fails on a database server. Hardware redundancy is just the first stage in a series of steps to handle drive failure. I do agree that a typical stateless server doesn't need raid1, but afaik it's not standard practice for a stateless web application to bother with raid1 anyway.
Correct.
> Hardware level redundancy like raid1 seems useful because it simplifies handling a common failure case when a single HDD or NVME fails on a database server
Nobody needs that if your database does replicate. Cassandra replicates data. MySQL in a replication setup replicates data. And so on. Individual nodes in such a setup are as expendable as individual disks in a RAID. More so, because you get not only protection against a disk failure, but depending on deployment strategy also against loss of a node, a Rack or a Rack Row. Or even loss of DC or AZ.
Edit: typo
For metal, chip credit cards I can't shred, I put them on a brick in the back yard, and torch them with a weed burner. There's a psychological bonus here, if for example Amex made me change cards through no choice of my own because they were hacked.
This wouldn't be "compliant" for a flash drive, but it would be effective.