bddicken

That database architects blog is a great read.

Thank you for the info! Do you have any good references on this for those who want to learn more?

It would be incredibly cool if this were used in high school curricula.

For this particular one I used d3.js, but honestly this isn't really the type of thing it's designed for. I've also used GSAP for this type of thing on this article I wrote about database sharding.

https://planetscale.com/blog/database-sharding

You are welcome!

> Your "1 in a million" comment on durability is certainly too pessimistic once you consider the briefness of the downtime before a new server comes in and re-replicates everything, right?

Absolutely. Our actual durability is far, far, far higher than this. We believe that nobody should ever worry about losing their data, and thats the peace of mind we provide.

The nerdbaiting will now provide generational benefit!

I heavily, heavily abused d3.js to build these.

I respect it. In my very biased opinion, it's worth enabling for this article.

Thank you! I started this back in October, but of course have worked on plenty of other things in the meantime. But this was easily 200+ hours of work spread out over that time.

If this helps as context, the git diff for merging this into our website was: +5,820 −1

The visuals add a lot to this article. A big theme throughout is latency, and the visual help the reader see why tape is slower than an hdd, which is slower than an ssd, etc. Also, its just plain fun!

I'm curious, what do you do on the internet without js these days?

Author of the blog here. I had a great time writing this. By far the most complex article I've ever put together, with literally thousands of lines of js to build out these interactive visuals. I hope everyone enjoys.

OP here - thank you! I'd actually like to more in the future with your data sets. It's so cool that you publish this.

I'd like to do another vis that includes failed drives. Please, keep helping me brainstorm.

Great suggestion.

Yeah, when drives fall off the bottom that represents decommissioning.

Thats good feedback. It definitely gets crowded, and sluggish due to the number of dom nodes. I didn't spend much time optimizing performance.

[OP] Interesting! I just tested it on Android/FF and it works. Could be a version-specific thing. Could also be just taking awhile to load (it has to download a 32MB json file).

I actually built this primarily with chatGPT o1.

One of the things LLMS are really good at is writing scripts for processing and pairing down data. I wanna do a blog post talking about how did some of this, maybe coming up!

[OP]

Backblaze writes about these data sets, and includes some more conventional graphs. For example:

https://www.backblaze.com/blog/backblaze-drive-stats-for-202...

I wanted to do something more fun!

At PlanetScale, we have a really nice solution for this in MySQL using SPANN + SPFresh. The way we've implemented it allows for pre and post filtering, full compatibility with where clause filtering, and has the same kind of acid compliance you'd expect from a relational database. You can read about it here:

https://planetscale.com/blog/announcing-planetscale-vectors-...

Tweet OP here: Very good analysis, thanks for sharing.

Author here: Yeah, the mod was added to reduce risk of loop(s) getting optimized away.

Author here: Yeah, I was surprised that there doesn't seem to be many options for extra optimizations in the Go compiler. Would be curious if Go experts have more insight or other compiler recommendations.

Yeah :/

That's what I had on my local machine, and I did not upgrade for this test. I have heard that things have gotten better in python 3.13. Regardless, I did include PyPy that shows it can at least be reasonably fast.

Just pointing out that what you're paying for is actually 3x these resources. By default you get a primary server and two replicas with whatever specification you choose. This is primarily for data durability, but you can also send queries to your replicas.

Cool, thank you for the input.

Author here - You can see the raw data as a table here: https://www.cloudping.co. Sometimes visualizations like this are a careful act of balancing practicality with cool-factor.

Author here - thank you!

Author here - If you have a resource that provides good data on Azure or GCP latencies, please send them my way.