antongribok

I actually ran into this last year. We ordered about 2 thousand 8TB HDDs last year for a Ceph cluster and had an agreement with our server vendor that they are not supposed to substitute the HDD model. Well, seems like they ran out and a small fraction of the drives were of a different, older model. The performance difference was 2.5x for our workload:

Older model: 670 IO/s and 2.9 ms average latency

Newer model: 1680 IO/s and 1.1 ms average latency

We got the vendor to send us new drives and shipped the older model back to them.

I'm pretty sure since that time a judge ruled in her favor and the previous probation history is no longer a threat to her maintaining custody to her kids.

Can't find where I saw that right now.

Elgato Stream Deck XL comes with 32 buttons.

Are you saying you want even more than that?

Given the single GigE network and SMR drives, it's not very fast, but fast enough for my needs.

More importantly, it is more reliable than running ZFS on a single box, which is what this Ceph cluster is replacing. I've had 100% uptime for 11 months, and I've been able to move the PIs physically around my house in this time.

Also, the power utilization is lower which is actually very impressive. I'm using 2.5" SMR drives, so that helps.

Most of my data is on an erasure-coded pool with a 4+2 setup, and the PIs are fast enough. CephFS has been very reliable on the laptops as clients, and reconnects within 1 second after sleep.

I'm also using compression, and it's not a problem on the PIs.

Laptops:

   LG Gram 17" 16GB RAM and dual NVMe disks - Fedora, main machine for work
   Dell XPS 13 - Fedora, wife's laptop
   Mac Book Pro 16" 2019 - Official work laptop, used as a paperweight
   Toshiba Tecra 740CDT - Recently revived, Knoppix (kind of retro playground)

Desktops:

   Ryzen 2700X w 64GB RAM 6-disk ZFS - Fedora, backup storage, video transcoding, photo editing
   Xeon E3-1230 w 16GB RAM - OPNsense, house firewall

NAS:

   9x Raspberry Pi 4s, (mix of 8 and 4 GB RAM) - Ubuntu 20.04 with Ceph Pacific used as main network storage with CephFS

I actually bought an PRi 4 8GB 2 weeks ago at my local MicroCenter, so they're definitely making (and selling) them.

I decided to check on my phone on a whim. They showed 8 in stock. By the time I got there, 30 minutes later, they had 3 left.

That PiLocator site does show the last time they've been in stock, and from my non-scientific assessment, it looks like availability is slightly getting better.

I used the very first original Pi B when it first came out as a small "shadow IT"-type desktop at work around 2014. We could only use Macs or Windows, and I needed something sane running Linux.

That version is much slower than today's Pi4, and I ended up using it mostly for command line sysadmin type stuff using a terminal and SSH. I used it for 1 year, before switching jobs.

While not running a desktop, today I have 6 Pi4s running a Ceph cluster and serving CephFS in my house. It's actually quite easy to set up, and very usable. Not super fast, because I'm using SMR drives, but definitely very usable as a very reliable NAS. Definitely more reliable than any ZFS setup.

I know of several large retailers that allow their employees to be paid via a debit card.

If you don't setup direct deposit before your first paycheck, instead of mailing you the paycheck you get a debit card. You can then withdraw the full balance out of it every two weeks without any fees.

This doesn't help with all of your banking problems, but it's a good start.

One more very satisfied user of an LG Gram 17. A beautiful LG 2560x1600 screen, amazing (also LG) battery life, less than 3 lbs.

I really like the keyboard and touchpad, more so than on a Mac (but that's very personal). The RAM can be upgraded to 40 GB, and there is a second M.2 slot if you want disk mirroring or striping.

I run Fedora and have zero problems. (I generally upgrade 3 months after a release.)

The model that I have has a 72 Wh battery and newer 17 and 16 inch models now come with an 80 Wh battery. Mine is 3 years old now and it's aged much better than I expected. I can watch a 4K movie over a network CephFS mount and after 2 hours, my battery is at 80%.

I bought it on a whim at Costco, and I seriously think I'll get another one next time (maybe a 16"). Costco drops the prices to $350-$400 off every few months, and they're a steal for what you get.

Counter anecdata: I've got a Ceph cluster running on multiple RPi 4s using external Seagate 4 and 5 TB portable hard drives. So far zero issues for about 1 year since I bootstrapped this cluster..

I use a combination of official RPi 4 power supplies and an Anker 6-port USB power block that I'm pretty sure tops out at less than 3 amps.

More importantly, everything is plugged into APC UPSes.

We're using LVM cache with Ceph on spinning disks for object storage workloads in a large, on-prem private cloud. In other words, this cache is behind the S3 backend, but in front of spinning disks.

When we built this, we had a choice of using all-SSD, spinning disk only, or a combination of both using LVM cache.

LVM cache gave us a lot better performance per $ than using all-SSD.

As for what type of actual application workloads? What do you use AWS S3 for? You name it, we have it:

Backups, container registry, data warehouse, pictures, PDF docs, config data...

See this blog post for some data at the end:

https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-tes...

The post is from 2014, but the performance data was updated for many years after that.

As you can see, there is not really 1 brand to single out.

That Micron P400e was a pain to swap out in production, and if you use consumer drives there is even worse out there.

Note: this post is on something very specific about Ceph, but the idea translates to any other workload that does syncs (including LVM cache scenarios).

I have about 50-100 PB of object storage workloads sitting on top of LVM cache for the past 5 years or so in production. It works extremely well.

Before going into prod, we tested LVM cache, and bcache, and LVM cache worked better for our workloads. Also, the tooling is significantly better than bcache.

One thing I love about LVM cache is if you are already running with LVM, you can add the caching layer on the fly without any downtime.

Regarding the note at the end where the author sees poor performance. I recommend taking a very close look at the SSD model. I've seen enterprise SSDs that perform worse with sync IO than spinning disks. When we buy hardware, we tell our vendors the exact model of SSD they can use. If we haven't qualified the SSD, we don't put it in production.

If you're already using Python, there is a very good module for this:

https://pypi.org/project/sslyze/

Can use it from the command line too:

  python -m sslyze news.ycombinator.com

For a while the easiest and fastest way to identify a 1U server in a rack of 40 was to SSH in and type:

  eject

Video of this guy from Wired:

https://m.youtube.com/watch?v=SEBBj2BIBm8

I've been working for about the same amount of time. My first job was right after high-school working in an office, where I was hired because I was fluent in a second language and they needed help translating documents. Slowly, during times when I was not busy, I fixed the office computers, redid the LAN setup, and built a Linux firewall to share an internet connection. I also built a CRM database on top of Microsoft Access that became central to everything the company did. About 10 years after I left I ran into an old coworker that was still there and they told me they still use that crazy Access DB.

When I found out, that moment was as close to being proud as I've ever been about something I've done in this IT field. There were other brief moments when my team has won some award, or I've given a talk at a conference and received positive feedback.

But all of this pales in comparison to me being proud of the spouse, parent, and friend that I am to people in my life today.

To learn a bit about Docker?

I've been using Xonsh for the past 3 years. I really can't go back Bash.

I write lots of small scripts and utilities for myself, perhaps on average one per week. Random stuff, like: slicing json data, renecoding home videos with ffmpeg, querying APIs with requests module, loops, dictionary comprehensions, etc...

If some script needs to be deployed to production, it's fairly easy to rewrite it in plain Python.

I've resisted installing it on our production systems, but I'm thinking more and more of using XXH for this instead.

Is this not the correct way to do it?

  $ sudo nvme get-feature /dev/nvme0 -f 7
  get-feature:0x7 (Number of Queues), Current value:0x7f007f

In my case, this was in New York.

This was a Christian monastery, but not Catholic.

I agree with everything you said, except your last centence:

"You don't get to pick up friends you have fun with in the evening."

At some point in my youth I visited a monastery on weekends, because I was friends with some seminarians studying there. People knew how to have fun. Perhaps not every day, only on some days of the week, but you certainly could hang out with friends. There was at least one or two people in the group with a car, and there were favorite spots people liked to go to.

I'm sure there are a lot stricter places, but for what it's worth, I know more than one person from this group (class?) that went on to become priests, working at churches today.

OT, but I just wanted to say that your RedHat interview was interesting:

https://news.ycombinator.com/item?id=28484534

After many years thinking about it and procrastinating, I finally built a Ceph cluster with 4x Raspberry Pis (8GB RAM, 4 or 5 TB external portable drive for each Pi).

The setup per node is super simple, Ubuntu 20.04, install Docker, and then one command to add the new node to the cluster, and one more command to add a disk.

(Ceph version 16 switched to cephadm for deploying and managing a cluster and it's a lot easier now than with previous versions.)

With just 4 nodes I'm already saturating a 1Gb connection on my client (Ryzen desktop).

Perhaps the most surprising thing for me was how stable and forgiving a native CephFS client is on my Linux laptop. After waking up from sleep, it takes about a second to reconnect.

This weekend I'm planning on expanding to 7 nodes. With 7 nodes total power usage should be ~31 watts (not counting network switch), which is less than my current 6 disk ZFS RAIDZ2 setup in my desktop.

I'll add an LG Gram 17 to that list.

19-20 hours of battery, and a nice 16:10 screen. Also, super light.

Everything on default Fedora just works out of the box.

I can't decide what I'm more impressed with:

The 110% code coverage, the downloads per month, or the license.

Yurchenko is the name of both a specific vault and a vault family in artistic gymnastics.

The Yurchenko was named after Soviet gymnast Natalia Yurchenko in 1982 during a competition in Moscow.

This reminded me of DSL [0]. I used to use it as my router 15 years ago.

The box was loud while booting (because of the floppy), but once booted was completely silent as everything ran from RAM. The Pentium CPU had just a heatsink with no fan, and no fan on the power supply.

[0] http://www.damnsmalllinux.org/

I (with 2-3 other people) run over 100 PB on Ceph clusters in production, supporting some critical functions of a Fortune 50 enterprise.

We use Ceph for block and object workloads (no CephFS). Most of our clusters are still on Luminous (v12) and Filestore (XFS), and only our newer clusters that are being built now are on Nautilus (v14) Bluestore. We plan on migrating to Bluestore this year (and likely next year).

I'm on-call basically all the time, but we'll have maybe 1 issue per year where we have to act immediately. Most failures that would happen on a Saturday can wait until Monday to be acted on.

The smallest cluster that we'll build is 6 nodes. Our largest cluster now is 120 nodes with 1920 OSDs. We might build a larger cluster this year.

Back in Ceph Hammer days, I had a 6-node cluster lose 3 nodes, one node at a time, over a course of a couple of days with zero downtime.

We deploy with Ansible, and have our own, very paranoid and opinionated playbook for doing a rolling cluster reboot or restart, that checks multiple things before moving on to the next node.

Reading this takes me back to when I started playing with storage professionally.

For me it was in 2004, also using 3Ware controllers. I was running on RedHat (before RHEL) and XFS before it was common on Linux, and similarly had memory issues when trying to repair filesystems.