Hacker News Reader

Comment by rulionti | Hacker News Reader

Interesting insights and refreshing to see large scale providers still relying on „old-school“ tech!

What I did not fully grasp - isn’t Vitess it’s own database system built on top of a K/V store similar to e.g. CockroachDB? Here it sounds like they only use pieces of Vitess in front of regular MySQL primaries with sharding?

It’s also curious that they sharded based on domains. I assume at some point the QPS on specific domains (e.g. Gist) will be so high that partitioning no longer is effective?

edit:// Looks like Vitess is indeed a set of tools on top of MySQL that allow for these type of scaling operations. It is not re-implementation of MySQL using different underlying technologies.

View on HN

    refreshing to see large scale providers still relying on „old-school“ tech!

Somewhat OT but Github is really "old school" - they don't have an SPA frontend and render most things server side the classic Rails way, with lots of UX enhancements coming from judicious use of JS and websockets for interactions. That's one of the things I find very endearing about their platform.

That approach unfortunately also makes keeping state updates consistent very difficult. Some pet peeves - when reviewing code, if you set up some file filters then scroll down to trigger more files to load, those newly loaded files are not affected by the previously set filters. Or occasionally the page would helpfully offer to reload to show new commits, but clicking that link loses the current white space and file filter settings.

SPA frameworks aren’t a silver bullet here, and you can definitely fix these without using one, but these bugs are definitely much harder to make since the DOM is rendered from a single source of truth instead of being patched piecemeal.

That’s a good point. I’ve faced a couple of issues like that doing pure server side rails and mostly dealt with them by not having “infinite” scroll, though I guess that’s not an option in all cases. I think Stimulus can go some way in making experiences like that possible without a full SPA.

> In addition to vertical partitioning to move database tables, we also use horizontal partitioning (aka sharding). This allows us to split database tables across multiple clusters, enabling more sustainable growth. We’ll detail the tooling, linters, and Rails improvements related to this in a future blog post.

They make a clear distinction between "vertical partitioning" and "horizontal partitioning", I think there are probably bigger wins to be had, and probably bigger challenges in the horizontal partitioning than the vertical.

All of this work only got them a 2 factor reduction in load, whereas with an ideal horizontal solution, they could scale it infinitely.

I remember reading a post from Jeremy Zawodny back in the early 2000s that was basically: "Start virtually sharding your DB data so that when you have to physically shard it, it's a lot easier."

The whole of Alibaba websites (Aliexpress, TaoBao, Tmall) rely on a single MySQL database for stock count keeping.

Also they developed their own storage engine for MySQL (X-Engine) which is based on LSM data structure and using FPGA to accelerate levels compaction [1], whitepaper [2].

"In addition, FPGA hardware is used to accelerate the compaction process, further maximizing the performance of the system. This marks the first time that hardware acceleration was applied to the storage engine of an OLTP database."

[1] https://www.alibabacloud.com/blog/new-insights-into-x-engine... [2] https://dl.acm.org/doi/pdf/10.1145/3299869.3314041

404

https://www.alibabacloud.com/blog/new-insights-into-x-engine...

Vitess is to MySQL as Citus is to Postgres

And Microsoft owns Citus and Github.

Vitess can be thought of as a very smart MySQL proxy with additional MySQL management features built in.

Centralized vtgates proxy to vttablet services which run on the same host as MySQL, the tablet then queries the local MySQL server. On top of this, a lot of magic can be built

What you’re describing sounds more like TiDB. Like a MySQL flavoured cockroach DB with its own K/V store TiKV.

Reply on news.ycombinator.com