antongribok

While I totally agree you can easily handle this with Ceph, I would never do it with 2x replication on spinning rust. You can maybe do 2x replication with SSDs if you really know what you're doing and you have data replicated on multiple Ceph clusters.

Also, your math does not quite add up if you want to store 425TiB. You'd be filling up the Ceph cluster to 95% which would be impossible (you'll start losing OSDs way before that).

If this was me, I would definitely do this on Ceph, but I would opt for a more traditional erasure coding setup of 8+3. For that, you'd need 12x SX134s as a minimum starting point.

I suppose you could start with an EC profile of 2+2 or 4+2...

I do have a serious question on Hetzner:

Do they offer any private networking options? That page only lists a single 1 Gbit connection. I don't think that would work well with Ceph. It's doable, sure, but I wouldn't want to support such a cluster, not with 160TB raw per box.

And incidentally Ceph supports both S3 and Swift (which I think is the best S3 protocol alternative):

https://docs.ceph.com/en/latest/radosgw/swift/

This fits the saying very well:

Suicide is a permanent solution to a temporary problem.

How does this compare with dm-flakey [0] ?

[0]: https://www.kernel.org/doc/html/latest/admin-guide/device-ma...

Not the OP, but that's how I use Signal. It's my default SMS app on my Samsung phone.

I have my SMS history there going back multiple years from multiple phones.

I'm curious where others in similar situation are moving to.

I switched to Samsung from SanDisk about 7-8 years ago (when I first discovered this cheating on one of my SanDisk USB sticks) and haven't had any issues.

BTW, there is no need to buy anything fancy for RPi, I just buy the cheap stuff, but I buy much higher capacity than what I need. Overprovisioning this way is cheaper and more reliable in my opinion vs. buying something more exotic.

For example for the most basic RPi install, I'll buy a 64GB microSD card. The card may only ever see less than 10 GB worth of data, but I know it'll last for years with plenty of chatty logs written to it daily.

Source: I've been running RPis from the very first version, and have never had a single card fail on me, ever.

I can confirm... Slack works. Sadly, still hogs way too much RAM.

This is truly very impressive.

If we're talking simple math on magnification:

It's very easy to find macro lenses that have 5:1 magnification, meaning that an object is projected onto the sensor 5x it's size. Taking 1.7 mm and projecting that onto a Canon R5 45 MP sensor with 5x magnification gives us something that would be around 1934 pixels in length (just under 1/4th the width of the sensor that is 8192 effective pixels wide).

I think you can convert 5x into 10x or 15x pretty easily without much loss in image quality.

That's not the problem, the problem is getting enough light (without boiling your specimen, which they briefly mention) and nailing the focus. At such magnification, both problems are challenging.

The fact that it actually comes with a Raspberry Pi is incredibly cool to me.

Oh, and of course there's a review of it from Jeff Geerling: https://www.youtube.com/watch?v=-epPf7D8oMk

Actually, this thing has some pretty decent specs... my initial reaction was that it was overpriced, but now I'm thinking this is a decent deal for what you get.

North-East USA does not have high speed rail. Amtrak Acela is not high speed rail.

I miss the old El Reg! I sincerely hope you find a balance.

Yelling! Yahoo! Headlines! were always easy to parse and will be missed.

Ceph is incredibly reliable. I run a ~ 250 PB Ceph install with the equivalent of 1.5 medium-paid full time engineers[0], and I sleep very well at night.

Failure after hardware failure, it just keeps running and running. We currently fail at a rack level, but no reason we can't do that at a building level with enough hardware.

0: We have more people on our team, but we also deal with other things that take a lot more of our time.

I love my LG Gram 17 from 3.5 years ago (I think it's a 2018 model).

Costco sells them, and when they go on sale, the value can't be beat IMHO.

Great screen, amazing battery (even after 3 years), zero issues on Linux (Fedora), love the dual NVMe option.

I'm thinking of upgrading to a 16" model, but I'll probably wait until another year.

Edit to add: Costco now carries 14 different models.

At the very end of the clip he says: "psychologically this was hard to do, because so much of their identity gets wrapped up in these educational badges...".

This resonated with me, but because of an opposite experience, I didn't go to a good school, and for a long time had imposter syndrome any time I started a new job. It took many years to internally get over that.

I've not had a single PG inconsistency on my RPi cluster in the one year it's been running. Weekly errors are at work.

Most of these are for actual disk media failures.

For 100,000 disks what is a normal failure rate in your opinion?

Each node is completely separate, housed in a good quality aluminum enclosure with a fan, and sitting on top of an external USB Seagate 2.5" portable drive (either 4TB or 5TB), connected via USB 3 cable. I'm pretty sure these drives are SMR, but they've been good to me, and they're fast enough for my needs.

Power is provided either using official RPi power supplies, or a couple of multi-port Anker USB power supplies that I had previously. A limit of 2.5 amps does not seem to cause any issues.

Currently everything is connected to a single switch, but I move things around my office sometimes, and sometimes have the RPis connected to two different switches.

Right now, everything including the 1 switch is connected to a single APC UPS, and that thing is super old, so that's another SPOF.

My clients currently are a few wired desktops and laptops over wifi, all connecting via CephFS. I haven't tested with librbd or krbd, I imagine it wouldn't be fast.

The RPis are mostly 8GB, but I do have a couple 4GB, and one RPi 400, which is kind of hilarious.

Everything is running Ubuntu 20.04, Ceph Pacific, and deployed from the first node with cephadm.

I use only Samsung microSD cards, either 32GB or 64GB. I don't think it matters what kind, but getting bigger cards makes me feel like they'll last longer. Most of the nodes have the var partition on the external drives (on a small partition at the beginning of the drive), but I do have a few where I didn't set that up early on, and haven't gotten around to redoing it.

I partition the drives and put LVM on manually, and tell cephadm to use the specific LV instead of the bare drive.

If you want any kind of performance, definitely set your expectations very low, but for me this works. I can stream at least a pair of 4K movies off this simultaneously, and I also run an instance of Paperless-NG off this over a CephFS mount and haven't had any issues.

I just setup a test cluster at work to test this for you:

4 nodes, each node with 2x SAS SSDs, dual 25Gb NICs (one for front-end, one for back-end replication). The test pool is 3x replicated with Snappy compression enabled.

On a separate client (also with 25Gb) I mounded an RBD image with krbd and ran FIO:

  fio --filename=/dev/rbd1 --direct=1 --sync=1 --rw=write --bs=4096K --numjobs=1 --iodepth=16 --ramp_time=5 --runtime=60 --ioengine=libaio --time_based --group_reporting --name=krbd-test --eta-newline=5s

I get a consistent 1.4 GiB/s:

  write: IOPS=357, BW=1431MiB/s (1501MB/s)(83.9GiB/60036msec)

When Ceph migrated from Filestore to Bluestore, that enabled data scrubbing and checksumming for data (older versions before Bluestore were only verifying metadata).

Ceph (by default) does metadata scrubs every 24 hours, and data scrubs (deep-scrub) weekly (configurable, and you can manually scrub individual PGs at any time if that's your thing). I believe the default checksum used is "crc32c", and it's configurable, but I've not played with changing it. At work we get scrub errors on average maybe weekly now, at home I've not had a scrub error yet on this cluster in the past year (I did have a drive that failed and still needs to be replaced).

My RPi setup certainly does not have ECC RAM as far as I'm aware, but neither does my current ZFS setup (also a 6 drive RAIDZ2).

Nothing stopping you from running Ceph on boxes with ECC RAM, we certainly do that at my job.

For production, we have two basic builds, one for block storage, which is all-flash, and one for object storage which is spinning disks plus small NVMe for metadata/Bluestore DB/WAL.

The best way to run Ceph is to build as small a server as you can get away with economically and scale that horizontally to 10s or 100s of servers, instead of trying to build a few very large vertical boxes. I have run Ceph on some 4U 72-drive SuperMicro boxes, but it was not fun trying to manage hundreds of thousands of threads on a single Linux server (not to mention NUMA issues with multiple sockets). An ideal server would be one node to one disk, but that's usually not very economical.

If you don't have access to custom ODM-type gear or open-19 and other such exotics, what's been working for me have been regular single socket 1U servers, both for block and for object.

For block, this is a very normal 1U box with 10x SFF SAS or NVMe drives, single CPU, a dual 25Gb NIC.

For spinning disk, again a 1U box, but with a deeper chassis you can fit 12x LFF and still have room for a PCI-based NVMe card, plus a dual 25Gb NIC. You can get these from SuperMicro, Quanta, HP.

Your 3x60 disk setup sounds like it might fit in 12U (assuming 3x 4U servers). With our 1U servers I believe that can be done with 15x 1U servers (1.5 PiB usable would need roughly 180x 16TB disks with EC 8+3, you'll need more with 3x replication).

Of course, if you're trying to find absolute minimum requirements that you can get away with, we'd have to know a lot more details about your workload and existing environment.

EDITING to add:

Our current production disk sizes are either 7.68 or 15.36 TB for SAS/NVMe SSDs at 1 DWPD or less, and 8 TB for spinning disk. I want to move to 16 TB drives, but haven't done so for various tech and non-tech reasons.

I would say the minimum is whatever your biggest replication or erasure coding config is, plus 1. So, with just replicated setups, that's 4 nodes, and with EC 4+2, that's 7 nodes. With EC 8+3, which is pretty common for object storage workloads, that's 12 nodes.

Note, a "node" or a failure domain, can be configured as a disk, an actual node (default), a TOR switch, a rack, a row, or even a datacenter. Ceph will spread the replicas across those failure domains for you.

At work, our bigger clusters can withstand a rack going down. Also, the more nodes you have, the less of an impact it is on the cluster when a node goes down, and the faster the recovery.

I started with 3 RPis then quickly expanded to 6, and the only reason I have 9 nodes now is because that's all I could find.

Aside from running Ceph as my day job, I have a 9-node Ceph cluster on Rasberry Pi 4s at home that I've been running for a year now, and I'm slowly starting to move things away from ZFS to this cluster as my main storage.

My setup is individual nodes, with 2.5" external HDDs (mostly SMR), so I actually get sligtly better performance than this cluster, and I'm using 4+2 erasure coding for the main data pool for CephFS.

CephFS has so far been incredibly stable and all my Linux laptops reconnect to it after sleep with no issues (in this regard it's better than NFS).

I like this setup a lot better now than ZFS, and I'm slowly starting to migrate away from ZFS, and now I'm even thinking of setting up a second Ceph cluster. The best thing with Ceph is that I can do a maintenance on a node at any time and storage availability is never affected, with ZFS I've always dreaded any kind of upgrade, and any reboot requires an outage. Plus with Ceph I can add just one disk at a time to the cluster and disks don't have to be the same size. Also, I can move the physical nodes individually to a different part of my home, change switches and network cabling without an outage now. It's a nice feeling.

Ceph is incredibly stable and resilient.

I've run Ceph at two Fortune 50 companies since 2013 to now, and I've not lost a single production object. We've had outages, yes, but not because of Ceph, it was always something else causing cascading issues.

Today I have a few dozen clusters with over 250 PB total storage, some on hardware with spinning rust that's over 5 years old, and I sleep very well at night. I've been doing storage for a long time, and no other system, open source or enterprise, has given me such a feeling of security in knowing my data is safe.

Any time I read about a big Ceph outage, it's always a bunch of things that should have never been allowed in production, compounded by non-existent monitoring, and poor understanding of how Ceph works.

I'm on Fedora 36 laptop that started from Fedora 25, and at some point in the middle I moved from Dell XPS to LG Gram.

To migrate machines I didn't use rsync, used dd instead.

Zero upgrade issues, however I generally wait 2-3 months after release before upgrading.

Sage Weil co-founded Dreamhost. While there he developed Ceph, before Inktank was founded.

Wow, I haven't heard of these "catastrophic failures" previously... that's a bummer.

The previous generation certainly worked.

Is there an article that explains this in more technical detail?

Does Elbrus count?

https://www.techradar.com/news/this-russian-cpu-annihilates-...

Supporting 16TB of RAM is pretty impressive actually.

These are both Seagate Exos drives, one is a generation newer than the other. I'm pretty sure both have the same number of platters, thus same number of heads.

And yes, HP changes the firmware and model numbers... super annoying.

It's possible. Yes, the drives were sprinkled around in the inventory, and we had some servers where there was a mix of old and new drives in the same box, behind the same controller.

We use the controller in HBA mode, so the controller shouldn't be doing anything fancy, but you might be right.

For 1 drive, the differences are perhaps not much. When you have a cluster with 2k HDD drives, the difference in handling both squential and random workloads (see my other comment) can add up.

A 2.5x difference in IO/s adds up quickly.

As for HDD prices, neither 12TB nor 8TB were top of the line at the timeframes you mention. If you buy largest available drives, or a model lower, the prices have been going down. Not as much as we all would like, but overall the trend has been moving in the downward direction.

No. Both models were PMR. I tried to find out more about what the differences are from the manufacturer, but didn't get far. Perhaps someone here can shed some light.

Older drive:

  model number: MB8000GFECR
  SATA Version: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
  ATA Version:  ACS-3 T13/2161-D revision 5

Newer drive:

  model number: MB008000GWWQU
  SATA Version: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
  ATA Version:  ACS-4

My guess at the time was that the newer drive had a deeper command queue, but I'm not actually sure, and I think both drives had the same amount of cache.