The goal of Fullstaq Ruby is to democratize the fight against Ruby memory bloat. Democratization means that as many people should be able to reap the benefits as possible. It's 2021 now, and expecting users to compile Ruby or Jemalloc from source is no longer realistic. Compiling anything is no longer a non-scary, low-friction thing to do.
But I'm not convinced it's a product and not a service, or a task for someone on staff for larger organisations.
As but one thing: to be a product it needs to have some universal value for a significant chunk of scenarios (possibly with some configuration options). But at that point, it's not clear why it shouldn't be a contribution to the upstream project. Is there, possibly, an "Enterprise Edition" or "Cloud Deployment Pro Plan" in the works?
I'm also (and that's neither here or there but I can't help myself) not entirely sure how democracy got involved here. While the rule of law and a participatory citizenry do tend to lead to obvious benefits, "reaping" evokes a one-way process that is sorta antithetical to the cooperative nature of democracies? Dunno, maybe it's just me. It's also on some buzzword bingo cards, although I can't quite say in what specific context...
From my experience with users and the industry at large, the amount of people who are comfortable with compilation and sysadmin-y tasks are rapidly declining, proportion-wise. The industry is heading towards ever-more specialization. Many many backend developers nowadays don't want to think about infrastructure at all, they just want to focus on business logic. There are a huge amount of backend developers who have never seen './configure && make install'.
In an organization with sufficiently advanced human capital, yes there is someone who can take care of that. But the existance of such a person can be taken less and less for granted nowadays even in very large organizations.
There is also the factor of: should we do this work ourselves? Lots and lots of developer tooling nowadays are extremely slick. We've been spoiled. I have been spoiled. I can still write C++ but I don't want to bother with './configure && patch && make install' anymore. The standard nowadays is higher. Why should I spend a day installing a custom-patched Ruby when someone else can do that for me and all I have to do is to add an APT repo? Especially when I always have better things to do?
> it's not clear why it shouldn't be a contribution to the upstream project.
This is explained in the FAQ: https://github.com/fullstaq-labs/fullstaq-ruby-server-editio...
There is also a discussion here: https://news.ycombinator.com/item?id=27870740
> Is there, possibly, an "Enterprise Edition" or "Cloud Deployment Pro Plan" in the works?
There is not. There are absolutely no paid features, nor plans to monetize. There are even enough reasons to not monetize.
See the FAQ: https://github.com/fullstaq-labs/fullstaq-ruby-server-editio...
And the project vision, which is community-based: https://www.joyfulbikeshedding.com/blog/2020-05-15-why-fulls...
> I'm also (and that's neither here or there but I can't help myself) not entirely sure how democracy got involved here.
Democratization, not democracy. Democratization is about making something available to as many people as possible. Which is a different concept from democracy.
Given that so many people are uncomfortable with compiling, or with LD_PRELOAD, or with anything outside of "bundle install", asking people to "just compile Jemalloc 3, make sure to apply this patch, then modify your systemd init script to include LD_PRELOAD" is too much to ask. It shuts down an entire range of people from benefiting from Jemalloc. With democratization, I seek to combat this.
The real argument is that installing jemalloc separately and running Ruby with an environment variable is too hard.
The Jemalloc version matters a lot. For reasons that are not yet clear, significant memory savings are only achieved with Jemalloc 3, not with Jemalloc 5. Your distribution only ships one Jemalloc version. So likely you need to compile Jemalloc 3 yourself. Here you are already entering compilation land.
But Jemalloc 3 no longer compiles by default on some modern distributions, such as Debian 10. Fullstaq Ruby fixes this by patching Jemalloc for you.
Furthermore, which Ruby binaries are you using? The ones provided by the Linux distribution are perpetually outdated. Another of Fullstaq Ruby's value proposition is that we supply binaries for the latest Ruby version, quickly. We packaged Ruby 3.0 on the same day it came out.
"Fullstaq Ruby vs LD_PRELOADing Jemalloc yourself": https://github.com/fullstaq-labs/fullstaq-ruby-server-editio...
> For reasons that are not yet clear, significant memory savings are only achieved with Jemalloc 3, not with Jemalloc 5.
So the real advantage of this package is that it bundles a 6+ year-old, unsupported version of Jemalloc, because the more recent versions found in current OS distributions don't yield memory savings in practice- for unknown reasons. This doesn't instill very much confidence.
I would be much more excited by efforts to investigate the jemalloc > 3.x changes so Ruby can work optimally with current releases packaged in modern Linux distributions, rather than double-down on a workaround that requires bundling an increasingly-ancient version of the software.
I should also add - as mentioned by the jemalloc author [1], the addition of the time-based purging feature is likely responsible for memory-usage differences between jemalloc 3.x and 5.x, so you can reduce `dirty_decay_ms` and `muzzy_decay_ms` to get 3.x-like memory usage. I have been using this configuration in production since 2018 for significant memory savings in Ruby using jemalloc 5.x.
First, "for unknown reasons" deserves more nuance. The vague, high-level reason is clear: Jemalloc 3 behaves differently from Jemalloc 5, having different algorithms and data structures. What I mean by unknown is not so much an indication of incomprehensible arcane magic, and that things can collapse at any time.
What I mean is that it's not known in what way the algorithms and data structures are different. Consider that before I did my 2019 research on why Ruby memory bloating occurs[1], Ruby apps suffered from memory bloat "for unknown reasons". That didn't mean that before 2019, all Ruby apps were houses of cards waiting to fall over.
It's like saying "I don't understand why this Linux kernel upgrade made things faster" -- the kernel developers know but they have better things to do than to answer your questions. And the fact that knowledge about a new optimization in the Linux kernel is not widespread, does not mean that that kernel version is unstable.
Nobody truly understands every single detail about all parts of the stack. Yet I can build reliable, high-available web apps just fine without understanding how for example how 5G works and why users on 5G can access my app faster than on 4G.
The differences between Jemalloc 3 and 5 are not explicitly documented anywhere, and to find out requires research. I intend on doing that some time in the future, but not now. Jemalloc 3 is proven to work, it's proven to be stable. The combination of Ruby + Jemalloc is proven to work well, not only because we've had several years of user feedback now, but also because Github has tested this combination for years now even before Fullstaq Ruby.
The pragmatic thing to do is not to prioritize figuring out exactly how Jemalloc 5 works. It's to continue the packaging work to make Ruby + Jemalloc 3 available to the public. Jemalloc 5 can wait.
[1] https://www.joyfulbikeshedding.com/blog/2019-03-14-what-caus...
First, my lack of confidence in depending on Jemalloc 3 in production is not only the 'unknown reasons' underlying such a frozen dependency, but also due to the fact that this particular dependency is over six years old and unmaintained. Not only does this lack more recent security/bug fixes and features, but also makes it more complex to integrate with up-to-date Linux distributions (e.g., your need to maintain custom compilation patches instead of simply depending on the OS's jemalloc package).
> What I mean is that it's not known in what way the algorithms and data structures are different. [...] The differences between Jemalloc 3 and 5 are not explicitly documented anywhere, and to find out requires research.
As I mentioned, the jemalloc developer already highlighted the exact differences back in 2018, and he even provided a MALLOC_CONF environment variable to use that makes memory usage in jemalloc 5 behave like jemalloc 3:
> You could verify this by setting dirty decay and muzzy decay to 0 in the MALLOC_CONF environment variable (i.e. MALLOC_CONF="dirty_decay_ms:0,muzzy_decay_ms:0", unless I've typoed something).
There is also a very readable page on performance tuning in the jemalloc 5 documentation [1].
In your research, have you ever tried running jemalloc 5 with this configuration? I did this back in 2018, tested/verified against my production workload, and have been running Ruby on jemalloc 5 without any issues since.
All that's involved is installing your OS's 'jemalloc' package and setting two environment variables (LD_PRELOAD and MALLOC_CONF). Simple enough and more confidence-inspiring than maintaining a patch against a six-year-old frozen dependency if you ask me.
> The pragmatic thing to do is not to prioritize figuring out exactly how Jemalloc 5 works. It's to continue the packaging work to make Ruby + Jemalloc 3 available to the public. Jemalloc 5 can wait.
I disagree about the relative priorities- I spent a day tuning Jemalloc 5 for my team's Ruby application back in 2018 [2] and it's been a done issue for us since then.
[1] https://github.com/jemalloc/jemalloc/blob/dev/TUNING.md
[2] https://github.com/code-dot-org/code-dot-org/pull/24676#issu...
By the way, Fullstaq Ruby does not only provide Jemalloc-patched versions. We also provide an unpatched version (and also a version with only malloc_trim) — in which case Fullstaq Ruby's main value add becomes DEB/RPM packaging only.