Fullstaq Ruby: Ruby, optimized for production
fullstaqruby.org
fullstaqruby.org
“There will be no paid version. Fullstaq Ruby is fully open source. It is also intended to be a community project where anyone can contribute. There are no monetization plans.”
Thought I would mention that in case the parent’s comment implied something different. It’s more that the core Ruby team are reluctant to.
The goal of Fullstaq Ruby is to democratize the fight against Ruby memory bloat. Democratization means that as many people should be able to reap the benefits as possible. It's 2021 now, and expecting users to compile Ruby or Jemalloc from source is no longer realistic. Compiling anything is no longer a non-scary, low-friction thing to do.
The real argument is that installing jemalloc separately and running Ruby with an environment variable is too hard.
The Jemalloc version matters a lot. For reasons that are not yet clear, significant memory savings are only achieved with Jemalloc 3, not with Jemalloc 5. Your distribution only ships one Jemalloc version. So likely you need to compile Jemalloc 3 yourself. Here you are already entering compilation land.
But Jemalloc 3 no longer compiles by default on some modern distributions, such as Debian 10. Fullstaq Ruby fixes this by patching Jemalloc for you.
Furthermore, which Ruby binaries are you using? The ones provided by the Linux distribution are perpetually outdated. Another of Fullstaq Ruby's value proposition is that we supply binaries for the latest Ruby version, quickly. We packaged Ruby 3.0 on the same day it came out.
"Fullstaq Ruby vs LD_PRELOADing Jemalloc yourself": https://github.com/fullstaq-labs/fullstaq-ruby-server-editio...
> For reasons that are not yet clear, significant memory savings are only achieved with Jemalloc 3, not with Jemalloc 5.
So the real advantage of this package is that it bundles a 6+ year-old, unsupported version of Jemalloc, because the more recent versions found in current OS distributions don't yield memory savings in practice- for unknown reasons. This doesn't instill very much confidence.
I would be much more excited by efforts to investigate the jemalloc > 3.x changes so Ruby can work optimally with current releases packaged in modern Linux distributions, rather than double-down on a workaround that requires bundling an increasingly-ancient version of the software.
I should also add - as mentioned by the jemalloc author [1], the addition of the time-based purging feature is likely responsible for memory-usage differences between jemalloc 3.x and 5.x, so you can reduce `dirty_decay_ms` and `muzzy_decay_ms` to get 3.x-like memory usage. I have been using this configuration in production since 2018 for significant memory savings in Ruby using jemalloc 5.x.
First, "for unknown reasons" deserves more nuance. The vague, high-level reason is clear: Jemalloc 3 behaves differently from Jemalloc 5, having different algorithms and data structures. What I mean by unknown is not so much an indication of incomprehensible arcane magic, and that things can collapse at any time.
What I mean is that it's not known in what way the algorithms and data structures are different. Consider that before I did my 2019 research on why Ruby memory bloating occurs[1], Ruby apps suffered from memory bloat "for unknown reasons". That didn't mean that before 2019, all Ruby apps were houses of cards waiting to fall over.
It's like saying "I don't understand why this Linux kernel upgrade made things faster" -- the kernel developers know but they have better things to do than to answer your questions. And the fact that knowledge about a new optimization in the Linux kernel is not widespread, does not mean that that kernel version is unstable.
Nobody truly understands every single detail about all parts of the stack. Yet I can build reliable, high-available web apps just fine without understanding how for example how 5G works and why users on 5G can access my app faster than on 4G.
The differences between Jemalloc 3 and 5 are not explicitly documented anywhere, and to find out requires research. I intend on doing that some time in the future, but not now. Jemalloc 3 is proven to work, it's proven to be stable. The combination of Ruby + Jemalloc is proven to work well, not only because we've had several years of user feedback now, but also because Github has tested this combination for years now even before Fullstaq Ruby.
The pragmatic thing to do is not to prioritize figuring out exactly how Jemalloc 5 works. It's to continue the packaging work to make Ruby + Jemalloc 3 available to the public. Jemalloc 5 can wait.
[1] https://www.joyfulbikeshedding.com/blog/2019-03-14-what-caus...
First, my lack of confidence in depending on Jemalloc 3 in production is not only the 'unknown reasons' underlying such a frozen dependency, but also due to the fact that this particular dependency is over six years old and unmaintained. Not only does this lack more recent security/bug fixes and features, but also makes it more complex to integrate with up-to-date Linux distributions (e.g., your need to maintain custom compilation patches instead of simply depending on the OS's jemalloc package).
> What I mean is that it's not known in what way the algorithms and data structures are different. [...] The differences between Jemalloc 3 and 5 are not explicitly documented anywhere, and to find out requires research.
As I mentioned, the jemalloc developer already highlighted the exact differences back in 2018, and he even provided a MALLOC_CONF environment variable to use that makes memory usage in jemalloc 5 behave like jemalloc 3:
> You could verify this by setting dirty decay and muzzy decay to 0 in the MALLOC_CONF environment variable (i.e. MALLOC_CONF="dirty_decay_ms:0,muzzy_decay_ms:0", unless I've typoed something).
There is also a very readable page on performance tuning in the jemalloc 5 documentation [1].
In your research, have you ever tried running jemalloc 5 with this configuration? I did this back in 2018, tested/verified against my production workload, and have been running Ruby on jemalloc 5 without any issues since.
All that's involved is installing your OS's 'jemalloc' package and setting two environment variables (LD_PRELOAD and MALLOC_CONF). Simple enough and more confidence-inspiring than maintaining a patch against a six-year-old frozen dependency if you ask me.
> The pragmatic thing to do is not to prioritize figuring out exactly how Jemalloc 5 works. It's to continue the packaging work to make Ruby + Jemalloc 3 available to the public. Jemalloc 5 can wait.
I disagree about the relative priorities- I spent a day tuning Jemalloc 5 for my team's Ruby application back in 2018 [2] and it's been a done issue for us since then.
[1] https://github.com/jemalloc/jemalloc/blob/dev/TUNING.md
[2] https://github.com/code-dot-org/code-dot-org/pull/24676#issu...
By the way, Fullstaq Ruby does not only provide Jemalloc-patched versions. We also provide an unpatched version (and also a version with only malloc_trim) — in which case Fullstaq Ruby's main value add becomes DEB/RPM packaging only.
But I'm not convinced it's a product and not a service, or a task for someone on staff for larger organisations.
As but one thing: to be a product it needs to have some universal value for a significant chunk of scenarios (possibly with some configuration options). But at that point, it's not clear why it shouldn't be a contribution to the upstream project. Is there, possibly, an "Enterprise Edition" or "Cloud Deployment Pro Plan" in the works?
I'm also (and that's neither here or there but I can't help myself) not entirely sure how democracy got involved here. While the rule of law and a participatory citizenry do tend to lead to obvious benefits, "reaping" evokes a one-way process that is sorta antithetical to the cooperative nature of democracies? Dunno, maybe it's just me. It's also on some buzzword bingo cards, although I can't quite say in what specific context...
From my experience with users and the industry at large, the amount of people who are comfortable with compilation and sysadmin-y tasks are rapidly declining, proportion-wise. The industry is heading towards ever-more specialization. Many many backend developers nowadays don't want to think about infrastructure at all, they just want to focus on business logic. There are a huge amount of backend developers who have never seen './configure && make install'.
In an organization with sufficiently advanced human capital, yes there is someone who can take care of that. But the existance of such a person can be taken less and less for granted nowadays even in very large organizations.
There is also the factor of: should we do this work ourselves? Lots and lots of developer tooling nowadays are extremely slick. We've been spoiled. I have been spoiled. I can still write C++ but I don't want to bother with './configure && patch && make install' anymore. The standard nowadays is higher. Why should I spend a day installing a custom-patched Ruby when someone else can do that for me and all I have to do is to add an APT repo? Especially when I always have better things to do?
> it's not clear why it shouldn't be a contribution to the upstream project.
This is explained in the FAQ: https://github.com/fullstaq-labs/fullstaq-ruby-server-editio...
There is also a discussion here: https://news.ycombinator.com/item?id=27870740
> Is there, possibly, an "Enterprise Edition" or "Cloud Deployment Pro Plan" in the works?
There is not. There are absolutely no paid features, nor plans to monetize. There are even enough reasons to not monetize.
See the FAQ: https://github.com/fullstaq-labs/fullstaq-ruby-server-editio...
And the project vision, which is community-based: https://www.joyfulbikeshedding.com/blog/2020-05-15-why-fulls...
> I'm also (and that's neither here or there but I can't help myself) not entirely sure how democracy got involved here.
Democratization, not democracy. Democratization is about making something available to as many people as possible. Which is a different concept from democracy.
Given that so many people are uncomfortable with compiling, or with LD_PRELOAD, or with anything outside of "bundle install", asking people to "just compile Jemalloc 3, make sure to apply this patch, then modify your systemd init script to include LD_PRELOAD" is too much to ask. It shuts down an entire range of people from benefiting from Jemalloc. With democratization, I seek to combat this.
These changes are specific to this context and are unlikely to be accepted into the more generalised Ruby core.
Same team as Phusion Passenger, so excellent background.
I also spoke to Matz during EuRuKo 2019. And this is the impression I got.
Also found a discussion on jemalloc https://bugs.ruby-lang.org/issues/14718
- Shyouhei Urabe did a marvelous job of resisting pressure from several well-known developers representing multiple high profile ruby companies/projects.
- Everyone eventually came to their senses in https://bugs.ruby-lang.org/issues/14718#note-89, and concluded that the best way forward is to use glibc with MALLOC_ARENA_MAX=2.
My personal feeling is that while the "Jemalloc FTW" section in http://oldblog.antirez.com/post/everything-about-redis-24.ht... might have been correct back in the jemalloc 3.x days, a lot has changed on both jemalloc and glibc since then. Furthermore, there is also musl in the equation these days.
I actually have a mention of this in my book on deploying Rails (https://deploymentfromscratch.com/). Here's a one liner with ruby-install:
$ sudo dnf install -y jemalloc-devel
$ ruby-install 3.0.0 -- --with-jemalloc
Then check you have it:
$ ruby -r rbconfig -e "puts RbConfig::CONFIG['LIBS']"
... -ljemalloc
Furthermore, it's not just "dnf install jemalloc-devel". The Jemalloc version matters a lot. For reasons that are not yet clear, significant memory savings are only achieved with Jemalloc 3, not with Jemalloc 5. And your distribution only ships 1 Jemalloc version. So likely you need to compile Jemalloc 3 yourself.
But Jemalloc 3 no longer compiles by default on some modern distributions, such as Debian 10. Fullstaq Ruby fixes this by patching Jemalloc: https://github.com/fullstaq-labs/fullstaq-ruby-server-editio...
Another caveat is that LD_PRELOADing Jemalloc is more effective than compiling with --with-jemalloc.
Furthermore, installing Ruby is one thing. Keeping it security-patched without having to constantly monitor things, is a whole other can of worms. Here's where Fullstaq Ruby's second value proposition comes in: we supply DEB/RPMs and do the tiny version patching for you, you don't have to go through the whole compilation process and gem migration process every time there's a new Ruby tiny version.
See also this FAQ entry, "Fullstaq Ruby vs LD_PRELOADing Jemalloc yourself": https://github.com/fullstaq-labs/fullstaq-ruby-server-editio...
Sam Saffron: "After spending a bit too much time thinking about this, I would like to recommend against any jemalloc related changes " https://bugs.ruby-lang.org/issues/14718#note-89
Besides being the author of Fullstaq Ruby, I am also the author of the first copy-on-write-GC-friendliness patch for Ruby. Consider the fact that it took ~5 years for the copy-on-write-friendly-GC feature to find its way back to upstream Ruby.
If you lived in 2008, would you be willing to wait 5 years in order to save 33% memory?
Your original statement basically equates quality with authority: you believe a project is qualitatively good if and only if it's officially blessed by an authority you trust. And that belief is fine. Most people either aren't able to, or don't care enough to, verify quality themselves, so we defer to trusted authorities. Nothing wrong with that approach in general.
The thing I'm trying to tell you though, is that in case of MRI, there have been multiple past cases when the lack of official authority blessing did not in fact mean a lack of quality. This isn't hypothetical: there's a proven track record. My own track record is proven as well.
What is in it for you? Well, reduced memory usage, improved performance, faster and lower-hassle installation and security patching!
There is always a risk, yes. But all I'm trying to say is that the equation "lack of authority blessing == lack of quality" is proven not to be true for a Ruby project such as this. Whether you want to take the risk of trying it out, that's entirely up to you.
There has been 8 - 10 glibc release since then I am not sure if things have improved or not.
> Wow, a lot of comments. What Fullstaq Ruby is doing is difficult to add into mainline, so my overall impression is I'm thankful someone's going to the trouble of doing it themselves.
> It's not quite right to say that on Linux with jemalloc it uses less memory. In reality, it would be more correct to say that using a super-old version of jemalloc that for some reason used less memory leads to reduced memory usage (for unknown reasons). So it can't go into mainline.
> That said, I can understand that some people are just happy to see reduced memory usage, so if those people want to keep using the super-old version of jemalloc with reduced memory usage, I have no objection. Go for it, is my feeling.
Why MRI ruby likes to keep unused memory pages around is only, that they don't like to add that compile-time option and probe for jemalloc. It's too trivial to worry
Edit: yep, exactly that.
Memory bloat is a real problem, even for mid-size Rails apps.
But it might not be in their interest, if customers are suddenly able to downgrade to cheaper Dynos with less memory, so we‘ll see.
> Memory bloat is a real problem, even for mid-size Rails apps.
Yes, but the solution isn't releasing the memory back to the OS, that's just papering over the cracks. If you're running (say) 10 Rails processes on a machine with 10 GB RAM, and particular request paths/background jobs whatever pushes the process size to 2 GB RAM size, you can release the memory back to the OS after you're done, but you still have the underlying problem which is the process gets to that size in the first place, and if all the processes hit 2GB at once, now you're 10 GB in debt, either in swap, or OOM errors.
The solution you should be looking for is installing something like Scout (or NewRelic, but IMO Scout is better for memory stuff), looking to see what is bloating the processes, and in 99% of the time fixing it using some the Rails' batch find methods (seriously, it's ActiveRecord objects 99% of the time, bonus points for in-memory CSV reading or write buffers).
Rails applications don't have to balloon in memory.
Fullstaq Ruby author here. I have done research on the memory bloat problem[1], and this statement is not true. With the way memory allocation in glibc's ptmalloc2 works, multithreaded processes can reach bloaty proportions even if they don't do much work. I started investigating this whole issue because I had a trivial Ruby HTTP proxy server: it had no reason to use 1.3 GB of memory, it doesn't do anything besides forwarding small requests. That's when I found that ~1 GB of that 1.3 GB of memory wasn't even due to anything that my app did by itself, but it's just how the memory allocator works.
[1] https://www.joyfulbikeshedding.com/blog/2019-03-14-what-caus...
Can you share the method you are using to do it? Not challenging, actually answering. Googling I'm not sure which thing I'm finding is the trivial one you use/recommend.
https://github.com/postmodern/chruby#rubies-1
(You don't need to use ruby-install with chruby, you can use whatever method of installing rubies you want. The chruby README even includes instructions for `ruby-build`. But normally whatever method you use, you install to `/opt/rubies` or `~/.rubies` which is where chruby will look. But fullstaq ruby insists on installing to it's own locations. But possibly chruby will be fine with that if you just tell it where that is... but you might need to do it for each ruby you install, which woudl be annoying. You could probably write a little bash shim which just checked the fullstaq install location for which rubies it had, and then just added them each individually?)
See the FAQ: https://github.com/fullstaq-labs/fullstaq-ruby-server-editio...
And the project vision, which is community-based: https://www.joyfulbikeshedding.com/blog/2020-05-15-why-fulls...
But there are various projects for Python.
For example, there's Pyston [0] which was started by Dropbox. Or Cinder [1] which was started by Facebook. Or PyPy [2].
[0] https://github.com/pyston/pyston
It's not a project officially blessed by Ruby core. Upstream Ruby is actually "good enough" for most. Most people don't feel the pain enough to try something which appears as unofficial. Perhaps their thinking are similar to yours: I'm hesitant because not everbody else is already using this.
I want to make this an official project. But before I do, I want to make sure that the project is sufficiently ready: healthy, democratized, not reliant on myself: https://www.joyfulbikeshedding.com/blog/2020-05-15-why-fulls...
I am also not marketing this as hard as I should. I lack free time.
Another reason is because of the beta label. The only reason why the label is there is because I have a specific, ambitious vision on how "final" should look like (i.e. more features), not because it's unstable. Feedback from users regarding stability has been very good so far. I might be shooting myself in the foot marketing-wise by not removing the beta label earlier, but I'd rather do a good job than to hurry.
Support for doing this in python was added in PEP-445 (https://www.python.org/dev/peps/pep-0445/) and implementation details can be found here from 2013 (https://bugs.python.org/issue3329). For more info see: https://docs.python.org/3/c-api/memory.html
You can see some commentary on using malloc-trim here too -> https://www.softwareatscale.dev/p/run-python-servers-more-ef...
...long story short: basically, you can already, but it doesn't really make much difference for most workloads.
Or, if your question is, why doesn't python get optimisation love and become fast like the javascript runtimes?
dunno. I guess we just don't have any big FANG companies pouring money on that, because they don't have a use for a faster python. A faster python does not make numpy, tensorflow or pytorch run faster because those are not written in python.
So I guess, there's a lack of value-for-effort in trying to 'fix' the problem of 'pure' python being slow.
The team also includes core developers Eric Snow and Mark Shannon (of the faster-cpython [1] proposal).
[0] https://pyfound.blogspot.com/2021/05/the-2021-python-languag...