Rewriting the Ruby parser
railsatscale.com
railsatscale.com
> Chad Fowler and I over basically 2 weeks, took the Java Debug Wire Protocol specification [...] turned it into a DSL in Ruby, and used that DSL to generate the packets for sending and receiving data. So we used the DSL in Ruby as a generator to generate Ruby code, as the whole protocol and then I used that at Darpa. They were trying to say “we could freeze the agent society from within the agent society will send messages, and it took about 7-8 minutes for all the messages to propagate and everything to freeze and then go quiet. I had a Ruby process that was running all 300 VM’s were underneath it, and I could freeze it in about a half of second. All 300 of them! And you actually could watch the CPU use because we had a monitor and the CPU use has dropped to zero. And it freaked them out. And what was great was you could turn it back on, and all the agents came back on. But time had been lost. And it was like alien abduction lost time, ten minutes went away, “what happened to us?” It was a bizarre thing because they were agents and they were planning on things and all of sudden 10 minutes just went away. But it was interesting to show how Ruby could actually be used as this kind of harnesses to wrap around things like systems.
So he took a data format designed for human readability and converted it to a data format that's designed purely to be read by Ruby and people are surprised that it's faster?
But I mean sure, WOOHOO, what a great achievement!
It continues to amaze me the pace at which kddnewton and co. are able to work. His tweet[1] from a few months ago is really relevant here. Being full-time employees and not just open-source volunteers makes a huge difference!
[1]: https://twitter.com/kddnewton/status/1639258413120073730
On a macro side, i am always surprised with the length and effort company will go to keep working with solution design and selected at their young age, and try to scale them well beyond the braking point. And i feel like most of this effort to keep ruby alive could have been used to slowly transition part of the infra to something JVM/.Net/Rust based for much more bang for engineering buck.
For what its worth though, we do also have Go, Rust, Scala, etc. in different places in our stack where its needed.
> The end of the day I think the way to get tech too scale, is for companies / academics / etc to really just put the work in.
This is true in the abstract, but my point is mainly about the comparative effectiveness of different strategies when it come to building scalable infrastructure (or scaling up infra): Keep the bulk of the existing code and invest in making the underlying compiler/interpreter/tool chain better or progressively migrate the code to a tool chain with better scaling capabilities from the get go.
From experience, the nature and semantic of a language severely what is "reasonably" possible in the runtime in term of safety (like code loader in java), performance (both CPU and memory) and tooling. Now improvement are possible, but they tend become exponentially expensive as time goes on.
Now, obviously there is always a tension between developer productivity and infra concerns, but i do believe that we have better compromise point on that line with newer language and framework.
I totally agree that in the 2000's the experience of most "enterprise framework" sucked very hard, and the emergence of language such ruby/python etc... was a god send : a response to the overly rigid and ceremonial way of the past. But with time, we have been able to understand better what makes a good programming experience, distill that into better designed languages and framework which offer "better" compromises. For example :
- Instead of dynamic vs static type , we have progressively typed and type inference - Instead of GC vs non-GC we have rust borrow-checker - Instead of runtime meta programing with have DSL and macro's
Even more important, i believe that a lot of the experience come from the tooling around the language, and there again the "cargo/dotnet/go" cli approach with a single coherent entry point for both package management and framework scaffolding ease a lot of the pain of the old way.
With all of that we now have languages which offer a better compromise on the dev. prod vs infra/performance...
> JVM is blazingly fast
I would say blazingly faster... But compared to C++ (or even rust) java is still quite slow. Especially for anything compute intensive.
That probably explain your surprise. This question comes up extremely often on HN, the answer is that even if you were to stop all feature development, re-platforming a large organization like Shopify would take years, not even considering all the re-training, lost knowledge etc.
And you'd spend these years having to support multiple platforms so that would only pay off much later, and in the meantime your competitors continue iterating.
In general it's best to stick with the devil you know, unless the new platform interoperate very well with the old one.
> That probably explain your surprise
While i don't write "business logic", i have been involved in a lot of project to babysit, maintain, refactor and/or improve business code bases. Sometime the runtime can only do so much and you have to adjust the user level code. And from experience it's never as bad as some make it to be.
> And you'd spend these years having to support multiple platforms so that would only pay off much later, and in the meantime your competitors continue iterating.
Again i see this very often as well. Software engineering seem to over emphasizes the cost of platform transition while downplaying the operating burden of not modernizing.
Given that I work on the Ruby Infrastructure team at Shopify, I think I have a well informed view on both the cost of transitioning and the operating burden.
It is also a lie to say java is "quite slow" compare to c/c++
Why ?
Every thing depends on the use case, and one case always find special case as counter example and most engineering solution/choices.
Generalization are still useful and sometime "true". C++ (and native languages) are faster than java simply because it was designed this way. C++ chose speed at the price of complexity, safety and build time. Java on the other hand focused on simplicity and safety.
I understand the idea of wanted to bring nuance, but the"it depends" can also become an excuse for bad tech choices.
> There are plenty of cases where Java will be faster than c++ purely due to the available libraries.
Hard to believe, but in that case i would say that you are comparing libraries, not the languages.
If that were true, C and Rust (which are typically faster) would be just as complex as C++ (but no language is).
It chose to jump into the OOP fad while staying low level ("what if we had C with classes?"). That's the cause of most of the complexity. The rest is age, backwards compatibility handcuffs, and kitchen sink stuff from other languages like move semantics and functional stuff.
> Java on the other hand focused on simplicity and safety.
Java chose to jump into the OOP fad while being high level. Ruby and Python did the same thing, and they're also simple as a result. That or low level and not OOP are the correct combos if you don't want to wind up with something as byzantine as C++.
I don't think that this assertion follows from my statement.
It's totally possible that there exist simpler languages which give exactly the same level of performance and expressiveness as C++ while being simpler. Nobody said that the C++ design was optimal with regard to the complexity/zero cost abstraction ratio. Bjarne him self think so (https://www.stroustrup.com/quotes.html), and both carbon and cppfront are effort in that direction.
The point was that when comparing (as of today) java vs C++ we shouldn't be surprised that a language which has "zero cost abstraction" as a core principle, and which is willing to be arbitrary complex ended up being faster.
With regard to the language you mentioned, C is simpler because it does less... And rust is much more recent and completely rethinks the native language landscape. I am not sure that we had the understanding (or even the tech) necessary to create rust 20 years ago.
But more importantly, i don't really buy the premise, of "are typically faster". Some concrete example would be nice, otherwise from experience this statement is wildly incorrect.
> It chose to jump into the OOP fad while staying low level ("what if we had C with classes?"). That's the cause of most of the complexity
Disagree.
> The rest is age, backwards compatibility handcuffs
Very true, if we remove backwards compact. C++ would be simpler.
> like move semantics and functional
Adding features paradoxically can simplify a language by providing a coherent/unified version of previously distinct usage pattern. uniform initialization is the canonical example. I would say that move semantics also simplify the language by folding resources reuses patterns inside RAII.
Same for C++ lamda, just simpler syntax ...
So you seems to say that we can choosing to forgo low-level control (ruby,java,python) or general abstraction structure(which is what OOP is really) to would produce a simpler language.
But C++ is exploring a different design question, trying to have in the same language low-level control, and abstraction that are general enough to expression complex design, and which can be efficiently deconstructed as to not impact performance. And the C++ community seems to be willing to pay some level of complexity for that.
No, you're not, because comparable libraries in C++ to do many activities, such as at-scale stream processing, simply do not exist in C++.
I do not doubt that there are domain where the best libraries are in a particular language, and this language might not be C++.
I am just not convince that java is better than C++ with this regard. (as they are more domain where the best C++ is faster than best java one).
But i think that's beside the point, we are comparing language here, so the existing/quality of characteristics of libraries shouldn't be the main focus (beside the stdlib of course).
Now there is more to choosing a tech. stack than just the programming language, and that include the libraries, current knowledge of the team etc. And i would even venture to say that those might be more important than the programming language per say. But i do believe we can still factor those concern out and compare programming languages, keeping in mind it's only part of the story.
The JVM takes bytecode and generates assembly for execution. It also profiles that code and improves it over time.
Sometimes long running java bytecode will be faster than statically compiled C++. In many/most cases, long running Java code on good JVM will reasonably close to C++ for some measures of "reasonable".
Garbage collection pauses, memory usage, JVM startup/warmpup is all more detrimental to Java in the speed comparison.
But "quite slow" basically, to me, implies it is 10-100x slower than C++. Sure you can come up with various benchmarks (lies, damn lies, and benchmarks) to make some corner case.
In general, the JVM is probably about 2-3x slower than C++ compiled code. It certainly is not "quite slow".
This project could be one approach to ameliorating that challenge, I'm very pleased to see shopify willing to invest in this, and hope it ends up effecting the ecosystem as hoped.
There are many languages that are hard to parse that are very successful. C and C++ are the obvious examples: there are many dark corners around the preprocessor, type annotations, etc. C++ is an absolute nightmare to parse.
C# is pretty complex too. Python has its weirdness around indentation.
Honestly, most popular languages end up acquiring a decent amount of grammatical complexity over the years and it doesn't seem to significantly hinder adoption. Humans are quite good at reading complex text and syntax.
I think the real tax on Ruby is its pervasive use of runtime metaprogramming. It's Ruby's most exciting strength and enables much of the joy and excitement that Ruby is known for. But it makes static analysis so hard and becomes less and less valuable as team and codebase size increases.
JavaScript was a huge success not because it was a great language, but because everyone wanted to write for the browser.
> There was little choice for other features they offered (like Unix being written in C and bundling a C compiler) which.were the compelling reason to use them.
Pascal was around at the same time, is grammatically infinitely more elegant... and is dead.
I think a PEG framework with a few features (like an easy way to implement operator precedence just by stating it with either numeric values or X > Y statements) could be revolutionary. Python is almost there but not quite.
So in a nutshell Raku‘s „Grammar“ construct is like RegExes on steroids and renders defining DSLs or other special purpose languages so much easier than in any other modern language.
Obviously, this isn't a blanket statement - metaprogramming has some amazing success stories (homebrew and vagrant both have excellent DSLs), but more often than not I appreciate code that's just "one layer" of abstraction.
I even had ideas like what you mentioned about encoding precedence and associativity explicitly: https://github.com/haberman/gazelle/blob/a12a123129dfb7e1f3e...
But I'm no longer as optimistic. The main problem is that nearly all languages have syntax that cannot be easily formalized using declarative abstractions like CFG or PEG, and must fall back to imperative code. And once you have a mix of imperative and declarative code, most of the benefits of a purely-declarative abstraction go away -- or at least the benefits that mattered to me.
Or even one line :)
Factor...
: unless ( ? quot -- res ) swap [ drop ] [ call ] if ; inline
unless: func [expr block] [either expr [] block]Except you also don't have the brackets to keep things organized.
def save = File.write(name, self.to_yaml) unless invalid?
def save = File.write(name, self.to_yaml) if valid?
The reason is that new grammar creates a new language, and the divergence of the new language from the base language creates a cultural barrier that inhibits communication. It's harder for new engineers to be productive, and it's harder to collaborate.
Customizing your grammar works best if you're a lone wolf, a one man band. You can lever up your productivity by custom-designing something perfectly suited to both you and your preferred problem domain. But nobody else will understand it.
... open parenthesis character is ambiguous in this context. To get around it they made their grammar more ambiguous and then enforced that the actual grammar was enforced in their tree builder.
I'd change the second "more ambiguous" to "more lenient"
i.e. lenient meaning "grammar accepts more strings", ambiguous meaning "grammar is invalid"
I have seen this issue in Python's grammar as well, and it's mitigated by the new PEG parser
But Python's pgen rejects such grammars as invalid ... The other strategy is just to pick an arbitrary interpretation
Fast parsers that use a schema are also doing model checking.
[1] https://github.blog/2023-04-06-building-github-with-ruby-and...
There might be one or two additional big companies that are similarly funding employees to contribute to core infrastructure in significant ways.
But not too many more than a handful, I agree. And they are carrying a lot of weight in ruby ecosystem for sure.
I feel like this era of open source in general is one of very shifting patterns of contribution for sustainability. One or a few big companies paying people to keep the thing alive is definitely one that seems to be increasing.
Perhaps since the company(ies) in question didn't originate the product(s) in this case, it doesn't feel like they "own" them exactly (not like "open source" products originated and developed by only one company where the product is their business itself -- not sure which category O'Caml fits in), but the risk in depending on only one or two companies (where the product is _not_ their actual business itself) is that if the company decides it no longer wants to make the investment, it can definitely be disastrous for the product. Shoppify just did a bunch of layoffs -- I don't think they hit the people contributing to ruby too hard, but they easily could have, except perhaps Shopify too realizes that if they stopped keeping Ruby alive it would be disastrous for their own business.
- without rails nobody would use ruby
- without X corp, ruby's dead
Fact is, we're seeing less and less usage, and more and more distillation of the current userbase along the golden paths laid out by DHH.
Is it wrong for rails and thus ruby adoption to slow down? Not at all, people should use what they like, however I think Rails and Ruby are in this negative spiral where:
- rails is mostly needed for prototyping and crud apps
- this work is typically done by juniors
- rails devs are at this point largely seniors, not juniors
- rails devs pay the bills with other tech or by maintaining legacy rails apps.
There are startlingly few deviations from this, and either everyone majors in rails with a minor in javascript and C++ or they just get happy with their current gig and settle. I wish it weren't the case, but Ruby just hasn't done enough to differentiate itself from Rails, and when compared to neo-PHP or JS there's just not a lot of attractive parts of the golden path Rails provides. It's off-tune for this generation of choice and ubiquity.
We don't need that level of scaffolding anymore and in many cases there are other tools that handle that with more versatility.
If you look at PostgreSQL then a lot of dev work comes from EnterpriseDB and a handful of companies too.
The thing with Rails is that it doesn't scale terrible well to "Twitter scale" (if I'm not mistaken Twitter has dropped all usage of Rails) so there aren't that many well-known companies running it, but the overwhelming majority of companies are not "Twitter scale" and it's not really an issue for them. There's a long list of smaller outfits that are not in the "top 50" using Rails quite happily.
People focus too much on "What is {Twitter,Facebook,Google,Amazon,Netflix,...} doing?" Who cares? You're never going to have the same problems they have. And whatever they are doing is not necessarily representative for the entire industry.
This furthers the point. By stating a lot of companies are looking for Ruby (something that doesn't match my experience when looking) is not a testament that it is hot and in-demand, it is a testament that those roles are not being filled. Senior devs don't make senior dev money doing junior dev work. My assertion is that the majority of Rails is CRUD development that only gets difficult when you step off the golden path- ergo, those positions go unfilled and outnumber their statistical representation in what would be called 'production Rails applications'
I'm not really sure what your point about senior/junior devs or "roles are not being filled" is.
(aside: please don't delete your post and post exactly the same identical post again to clear the downvotes on it).
It's true that Twitter switch to JVM langs, but it's not true that Ruby doesn't scale (or couldn't have to Twitter's level if they'd kept it). Twitter was early days for Ruby and things have improved a lot, but the only scaling challenge with Ruby is the cost of app instances. I use Elixir/Phoenix now and run 1/4 of the app instances I used to and with much less memory required per instance. (in one app it's 1/10 the ruby instances!) It's traditionally opex cost that hurt Ruby scalability, not technical, and very few companies will ever see the level of success where the cost of servers gets prohibitively high (compared to dev dev cost).
If that's the case, then they are misguided.
> You can scale anything with enough hardware
No, some architectures or implementations can give you diminishing returns or a hard cap. Not everything can scale horizontally ad infinitum.
> rails devs pay the bills with other tech or by maintaining legacy rails apps
That "maintaining legacy rails apps" job just doesn't exist with our PHP apps. Once properly tested and deployed PHP will generally work perfectly and smoothly for years with zero maintenance unless someone finds a bug that was missed in testing. You can pretty much setup a cron job to apply security patches and that's it. Maintenance done.
The only downtime I can recall was when our datacenter installed buggy firmware on their storage array and brought down every virtual machine we had with them... which was a pretty rare and unusual event (by the time they fixed it, we had already moved everything to another datacenter... and it seems we weren't alone because they went bankrupt).
With ruby that wasn't our experience at all. We found our production code would regularly just stop working and resources had to be pulled off other tasks with zero notice to figure out why/fix the issue. It was a productivity nightmare. Maybe that's improved now, but from reading the Shopify and GitHub blog posts they both seem to be taking on mammoth amounts of work to ensure their systems are reliable.
At the companies I work for (more than one) we expect anyone qualified to do work like that to be dedicating all of their time to other things. All of the apps we built in ruby were rewritten from scratch in PHP and we haven't had any regrets.
I've never really liked PHP and I actively hate JavaScript, but I've come to accept those two languages are just more practical than anything else. I'm definitely keeping an eye out for that to change though - Swift is look promising for example.
My current app runs on AWS ECS. Upgrades are largely just updating my Dockerfile or merging a pull request from Dependabot. We have pagerduty and it only goes off when a 3rd party API is down, usually resolving itself.
> - this work is typically done by juniors
> - rails devs are at this point largely seniors, not juniors
> - rails devs pay the bills with other tech or by maintaining legacy rails apps.
- and those prototyping apps have very often gone through multiple teams, some or all of them probably outsourced. Potentially true of any codebase, but it's true of an exceptionally high proportion of Rails codebases.
I'm at the point where I'd want a stupid premium to come in on an existing Rails codebase, and I'd want a day or two with it before saying "yes" even at that. They're great if they've been maintained by professional, expert teams their whole life, but god-awful messes remarkably resistant to analysis, otherwise.
I like Ruby a lot but most of the jobs are in Rails, and after initial infatuation followed by repeated exposure over 15ish years, I've come around to pretty much hating Rails. Too much implicit magic, too much memorization, too opaque to tools that might help overcome those first two problems.
If you are a senior level Rails developer none of this is true except maybe memorization and that seems to be a pre-requisite for any senior engineer in any language. It's trivial to debug rails applications with debugger and reading the underlying source code. Everything you need to solve problems is a binding.pry or `bundle open` away.
> that seems to be a pre-requisite for any senior engineer in any language
In many languages and language-ecosystems, there's little point to memorizing e.g. method names and signatures that you're not using so often that memorization happens naturally, because your tools can remind you when you, fairly seamlessly, when you need to know. A lot less memorization goes a lot farther in those worlds, than it does in Rails, and the pain of encountering something one is not familiar with is near-zero. Coming back to them after a year or two—or five—away's not a big deal. The brain-space required for Rails is unusually large, and the rate of rot in Rails skill is high. Ramp-up time in an unfamiliar Rails codebase is rough, and requires assistance from those already "read in" to avoid a bunch of wasted time tracking down which gem provides such-and-such dynamically-named object or method or what-have-you. "Which library is this even from?" is not a question that ever reaches the level of conscious thought, in many other languages & frameworks.
Getting up-to-speed on an unfamiliar Rails codebase is full of little side-quests that simply aren't needed elsewhere, and you have to hold a lot more in your head to remain productive in it, than other systems require. This is obviously not impossible, but... oof, why?
All that written out... there's a chance I'd still pick it for a new, solo project, depending on the task. It's fine as long as you are very-familiar with the entire codebase, and some of its gems are major time-savers. I get why companies, and especially move-fast prototyping startups, end up with it, I'm just very done onboarding to existing Rails codebases, personally, without some serious pain & suffering compensation.
I see this people complain about this but I don't understand why. First of all I've seen highly competent engineers complain about methods in Rails that exist from basic inheritance in Ruby. A concept that they probably learned when they were 10 years old. This is how object-oriented code is written. Pretty much every game is written the same way. Yes the dynamic methods that are generated can be annoying, but not what I see people complain about.
Second, they're trying to code a language like Ruby in a text editor and complain. If you tried to write Java or Scala in a text editor you would also have a bad time. So, yea I don't get it to be honest.
Nowadays it has to compete with nodejs, phoenix framework, django and laravel. all of which are within 80% of developer productivity while being vastly more performant. I use phoenix framework myself and while I wish there were more packages available, developer productivity is good enough and we can get away with far less machines to do the job.
Anecdotal I know, but from the moment that it appeared on my resume in 2012 it's been non-stop. Probably 80% of everything I hear about.
Ruby and it's ecosystem brings the closest thing to natural Aspect Oriented Programming that I've seen in the wild, which is why it's so much more productive than everything else I've tried.
- I wanted to make an index page where a user could make edits to the items being displayed and make regular show/edit pages for the item so if a user was on that page they could edit it. This is actually really useful: a user can bookmark the page for a specific item, or open it in a new tab, or in general do things browsers let you do but apps struggle with. Making two different edit components would be stupid, and I thought this would be one of those things that I could do better in React than in Rails. After looking around a bit I quickly found that if I used standard REST-y routes and wrapped the key parts of the view with turbo frames I could get exactly what I wanted, and it worked out seamlessly. In general, I've found that Rails' heavy emphasis on REST was a good architectural choice, and every time I disagreed with it and went another way I ended up regretting that choice and reverting to REST.
- In the Java world, you typically have thin models + a service layer which has the business logic. This is apart from other layers such as Repo/DAO etc, which I was already replacing with ActiveRecord, but I was initially resistant to putting all of my business logic in the models, especially if it involved logic across them. But it also hurt discoverability. I was working on a project I was expecting other people to work on with me, and I wanted to make it easy for them to figure out what they could do with a model object. The solution to this came from DHH himself. I've lost a link for this, but he said that if you make service objects, you should add a method on the model that acts as a way to reach the service. This keeps the model "fat" while also separating out logic into simple, unit-testable classes.
One of the smaller things I appreciate is keeping all of the routing details in rails routes. I know other frameworks like Django also do this, but I really didn't like this about most Java frameworks and microframeworks in other languages.
In general, if you're interested in seeing how to architect Rails apps, I say study how 37Signals does it. Playing around with Basecamp convinced me of the practicality of the Rails way and taught me a lot of interesting and useful patterns.
This is news to me. Is this really true?
Longtime rails dev here. The reason smaller companies are using it is because you can move much faster working on a full stack rails app that gives you SPA-like functionality without needing to incur the performance or operational cost of a dedicated front end.
I've worked for both rails shops and JS shops, and the productivity achieved with Rails is staggering compared to React in a small team environment. Guillermo Rauch tweeted a few months back that SPAs were a zero interest-rate phenomenon and I completely agree. Just because a bunch of companies jumped on the JS hype train doesn't mean that they were all making the right decision.
Guillermo Rauch is selling Vercel, whose strategy includes first-class tooling for SSR, convincing people to move their SPA to SSR, and then locking them in on their platform.
[1] https://rauchg.com/2014/7-principles-of-rich-web-application...
Referring to SPAs as a "zero-interest-rate phenomenon" implies that SSG/SSR models are more efficient in terms of financial cost of deployment. I don't agree this is necessarily the case, and I think SPAs can be developed and deployed sanely and cheaply also.
Vercel is doing some amazing things, but it's also innovating in ways that occasionally lead to "lock-in", in the sense that moving away from Vercel would involve a lot of friction. So I think it's fair to point out that you have a financial interest in convincing people to adopt delivery models that your business streamlines.
The continuing work on the language and its performance is impressive, but Ruby (and Rails) themselves have the honour of being stable, tried-and-tested solutions for rapid application development. Is it as exciting as the latest and greatest serverless lambda framework in Typescript? Not really. Is it a dependable workhorse? Absolutely.
At some point you might be successful enough to justify a rewrite into something else, but a simple Rails app will take you a long way with little effort.
I hardly hear about new projects being started with Rails as well...
It's used in many companies, new and legacy.
I'd be very surprised if the decrease since then is out of line with the rest of the industry, regardless of tech stack.
I don’t think this makes it dead or dying though. It’s stable and entrenched while JS has taken the place of the golden child.
One complaint I’ll grant myself is that library development is a little less prolific this days. Again, there are well-established solutions to a lot of problems in Ruby so you’ll have a go-to collection of gems, but it’s more often the case these days that something doesn’t have much library support and you’ve got to roll your own.
It may be that after years of working on other technologies, you just aren't passing the filters anymore. When we were aggressively hiring Rails devs in 2021, we specifically searched for the seasoned Rails devs that could hop in and get going. I've found less appetite outside of the big shops (Shopify/Github/etc) to pick up junior Rails devs which isn't great.
I think a technology must continue to change and adapt to remain relevant. I see Shopify as the main driver of current improvements to Ruby (the core features of the most recent release, and the upcoming release, were mostly due to Shopify as I understand it). I don't see smaller companies doing these improvements and I think without them Ruby will (slowly, because it's used in some big places) fade away.
They're in a different class to heavy hitters like Shopify and Github, who will gain a lot more from investing in gradual improvements to Ruby's runtime at the scale they operate at.
I'll contrast it with JavaScript, which has tried to assimilate every language pattern under the sun over the past decade and is intensely difficult to maintain a stable stack with, even if it's better now than it used to be.
So are fax machines.
The demand for Ruby developers is higher than the supply.
The new GC, and JIT, along with a few Ruby Cores are all Shopify employees.
Flash forward several years, and the amount of changes in Ruby are huge! Ruby is as fit as ever for modern development, and I'm really happy about that.
I’d love to see where Ractor goes but I worry it will remain niche, like with Refinements.
The pipe operator implementation proposal that they team came up with was wrong though and I'm glad they didn't do it. We don't need alternate syntax for `.`, we need ability to chain arguments together in a syntactically pleasant way in a functional style so we don't have to write `first(second(third(arg)))` we can write `arg |> third() |> second() |> first` which is much cleaner and reads left to right like it should
Would've been a bit more concise with method references, almost introduced in 2.7, but alas.
(But, well, people tend to want "something that looks like operator, preferably something that looks exactly like |>" and reject every other possibility)
That still wouldn't have solved passing additional args, so maybe { object.method_name(_1, args)} is the next best thing. Though it perceives non-atomic due to wrapping block.
```
first = ->(x){some code...}
second = ->(x){some code...}
third = ->(x){some code...}
(third >> second >> first).call(arg)
```
I agree it's not as clean as what you propose, but much better imo than traditional nested calls (and Haskell's `.`).
Ruby being a type 2 lisp is a fun one - creating a class and and a factory function with the same name, with argument forwarding:
class Animal; …; end
def Animal(…); Animal.new(…); end module HashExts
refine Hash do
def symbolize_values = transform_values { _1.to_sym }
end
end
module Test
using HashExts
def self.new_h = Hash.new
end
puts Test.new_h.symbolize_values
# => undefined method `symbolize_values' for {}:Hash (NoMethodError)As far as I know the teething issues around refinements are ironed out but they remain an obscurity.
You can also use whatever React/Vue on the frontend if you really need to.
I would like to use some Typescript framework but none are "there yet" with regard to productivity.
Scaling rails in some scenarios is quite challenging, but in most cases you can leverage caching to solve performance challenges.
> Over the years, processors and C compilers have gotten much better using a couple of techniques. These include pipelining, inlining functions, and branch prediction. Unfortunately, the parsers generated by most parser generators make it difficult for any of these techniques to apply. Most generated parsers operate with a combination of jump tables and gotos, rendering some of the more advanced optimization techniques impotent. Because of this, generated parsers have a maximum performance cliff that is extremely difficult to overcome without significant effort.
Although generating parsers, and finite automata in general, using a table-based approach is common, it has long been recognized that using tables/data for this purpose (as opposed to generating executable source code directly) is not a good idea, precisely because it inhibits compiler optimizations. I think the current situation is simply a consequence of the fact that parsing abruptly stopped being sexy multiple decades ago.
Much better parser generators are possible, and LR, specifically, has much untapped potential left.
Rubocop yes if it ends up using YARP as a new backend.
Either way, I would suggest you check out ruby-lsp, which is definitely going to benefit from this speed, and soon.
Just added several ruby grammars here https://github.com/mingodad/plgh/tree/main/ruby they are converted (mainly using https://www.bottlecaps.de/convert/) to an EBNF understood by https://bottlecaps.de/rr/ui to generate navigable railroad diagrams.
Copy and paste the EBNF on https://www.bottlecaps.de/rr/ui on the tab "Edit Grammar" the click on the tab "View Diagram" to see/download a navigable railroad diagram.
Also,
> CRuby actually ships with 90 encodings (as of 3.3)
This is asinine.
The encodings is a bit historical, to my knowledge Ruby is the only major language that was developed by someone whose language did not work with ASCII. So the first encodings written in Ruby were the old Windows pages.
>This is asinine.
Overall there are even more: 103. But then again 1.9.2 only has 85 and 95 overall. Also one of those new 'overall' ones is the EBCDIC code page for US/Canada.
That makes a lot of sense. ASTs tend to be a very dumb but fairly verbose data structure. And, in C in particular where everything is more verbose, there ends up being a ton of boilerplate in the AST nodes.
In a language like Ruby with a very rich syntax, you end up needed a ton of different AST nodes.
Generating those from a simple declarative format makes maintaining that much easier.
This makes it sound like it's some template language specific to this project, but ERB is the dominant templating language for Ruby because an ERB implementation is in the standard library.
I'm glad to see big steps towards this.
I think they might an interesting intersection here we ML, where can could learn the comment mistake pattern made by real user and either error correct better, or at least provide pin point accurate error messages.
What’s the progress on Ruby/GraalVM that Shopify has heavily invested in?
As you might imagine, there's a lot more to production deployments than simple Ruby compatibility. At Shopify, we've recently modified our CI system to support TruffleRuby and are actively running projects against TruffleRuby in CI now [1]. The TruffleRuby 23.0.0 release coming out in the next day or so includes quite a few compatibility issues we've worked out getting a large Rails app booting. That project is on the order of months, not decades.
YARP will make adoption of TruffleRuby easier. Absent a language specification, implementations like JRuby and TruffleRuby have to match what CRuby is shipping and that by necessity means they lag after CRuby releases. Parser changes are amongst the hardest things to port over. YARP eliminates most of the challenges there.
[1] -- https://railsatscale.com/2023-06-12-truffleruby-in-shopify-c...
Disclaimer: I'm on the TruffleRuby team at Shopify.
What does ‘booting’ exactly mean in the context of a web app (Rails)?
In my experience, the Rails boot stage accounts for most of the compatibility work. If you can't boot the app, you won't be able to serve requests and likely can't run tests. E.g., it was at this step that we learned we needed to support more of the native extension API to get memcached running (†). Once booting we did run into some other compatibility issues, but the lion's share came up during the Rails boot process.
† - As an aside, the memcached gem hasn't seen a release since 2014. Its C code attempts to detect the Ruby version and alter how it sets things up. It predates the availability of TruffleRuby so there isn't any TruffleRuby-specific detection logic in it. Our extension compatibility made it look like we supported CRuby's internal object model and that was causing the extension to try to allocate objects in a CRuby-specific way. A small change to a macro fixed the problem and the rest of the gem ran fine. It's one of those things that we can't fix until we see, but once we do we can fix it permanently.
Also, any thoughts on the recent license change of GraalVN?
The GraalVM commercial license change seems like a positive change to me. But, I'm not a lawyer and I'm not speaking on behalf of Shopify. To me, it looks to be in keeping with how the Oracle JVM is licensed and I think that simplifies a lot. The commercial version has some really nice improvements for the native binary and that's been mostly closed off to Rubyists.
I wonder if Shopify will ever give up on trying to fix Ruby and instead, do what FB did with PHP and just create their own derivative (Ruby) language that addresses their needs better.
Facebook has far more employees and wasn’t going to get PHP improved to their liking.
https://onlinephp.io/benchmarks
https://serpapi.com/blog/benchmarking-ruby-3-1-yjit-ruby-2-7...
Look at PHP 7.2 onwards, which is a similar timeframe, and the improvements are fairly modest.
This is a new implementation of an existing standard.
> We recently got approval to merge this work into CRuby, and are very excited to share our work with the community.
How many languages would even entertain such a contribution? How many languages are so tightly bound to their parser that writing a new one wouldn't even be workable? How many languages have user tooling that works extensively with parsed code snippets? How many languages have multiple implementations? Of those, how many are working together on an effort to share a common parser?
There's plenty of valid reasons to use something other than Ruby. An open source project that grew organically over 25 years deciding that it's time to pay down some technical debt and improve the ecosystem at the same time is probably not one of them.
If your company has enough engineering overhead and capacity to stop delivering features and try to fix your core language for a year, then you're in a different ballpark from most other tech companies. If you're NOT in that boat, then "Rails scaled for Github and Shopify" is not a message you should take home.
And they didn't stop delivering features for a year.