I'm curious to know more what sorts of challenges they have around managing dependencies across engines — I think what we were doing was fairly vanilla Rails, and we didn't have the opportunity to run into those sorts of issues.
Long story short, Rails and dependency inversion equals lots of friction. The whole framework is built on the assumption that it's OK to access everything from everywhere, and over the years we've built lots of tooling on top of those assumptions.
E.g. we heavily use https://github.com/Shopify/identity_cache with active record associations that cross component boundaries.
We also have a GraphQL implementation that is pretty closely coupled to the active record layer and _really_ wants to reach into all the components directly.
All of those problems can be overcome, but this is definitely an area where we have to working against "established" Rails culture, and our own assumptions from the past.
From a deployment perspective are your engines deployed and scaled independently?
- same database - same runtime - same deployment - same repository
That said, I don't think this is an either/or. It's a spectrum. you can have components within the same runtime and repository that have separate databases, or components that are using the same database but live in separate repos, etc.
From one monolithic app towards fully separated microservices is a spectrum, and I think developers should be enabled to move freely around that spectrum.
I call them Micro-SDKs.
Do you envision any extension points to the way engines are implemented that could better enforce boundaries? In our engines, there was nothing that referenced another engine's resources, leaving the main application to handle route mapping and ActiveRecord associations between app models and engine's models.
I feel like the use-case for engines has long been around supporting framework like functionality (Devise, Spree, etc), but I wonder if there are changes to be made that better support modularization for large apps.
Can you expand on that? I'm not sure I follow.
I strongly suspect, but cannot prove, that we would have been better off simply transitioning to "macroservices" -- breaking the monolith up into several (as opposed to dozens) of reasonably sized pieces.
• We were encouraged to componitize everything. When I left, we were up to a few dozen components, and the number was climbing rapidly. I'm not sure if the approach itself was the problem, or if the flux during the transition period was the real pain point.
• We had no real enforcement of interfaces between components. It was so easy to break things in other peoples' components.
• Theoretically that breakage would be caught by tests. But to catch that breakage, you needed to run the complete test suite (30-60 minutes) rather than simply testing your own component
• Essentially, it felt like we were suffering all the disadvantages of microservices, with the exception of coordinating deployments; from a devops perspective it was still just a single monolithic deployment
• We still had many of the problems associated with monoliths, such as slow deployments, long test suite times, and extremely high per-instance RAM usage
• Various small tooling and debugging issues related to using Rails but going too far "off the Rails"
I'm looking forward to digging into the linked article and learning how Shopify solved those issues. They seem to have quite a bit of engineering firepower at their disposal. Our management did not allow us to dedicate a lot of resources to internal engineering concerns like this.
(We essentially had one guy figuring it all out himself, and due to internal politics he was forbidden from considering a microservices or "macro services / multiple monoliths" approach. He was talented and did the best he could, considering)
I should have included this in the blog post: The number of components _needs_ to be kept small. Shopify's main monolith is 2.8 million lines of code in 37 components, and I'd actually like to get that number _down_.
I like to compare this to the main navigation that we present to our merchants. It's useful if it has 8 entries. It's not useful if it has 400.
In a way, components are the main navigation to our code base. A developer should be able to look at what's in our "components" folder and get a general impression of what the system's capabilities are.
I like to compare this to the main navigation that we
present to our merchants. It's useful if it has 8
entries. It's not useful if it has 400.
As you said in the article, one of the benefits of components for you was that it truly forced you to think about a proper separation of concerns. In hindsight, that's an area where we really missed the mark for a variety of reasons, some methodology-related.
We practiced a rather strict version of Scrum. Management paid a lot of attention to our velocity from week to week: we needed to rack up those story points.
But, outside of the tiny team dedicated to the component effort, there were no story points to be had for supporting that effort. Therefore we were in fact incentivized not to support it. I remember one sprint where I did some refactoring work in order to achieve a better separation of concerns. It negatively affected our velocity for the week and that was noticed.
So, we were receiving a schizophrenic message from management. We were all to support the component effort.... but on our own time, apparently?