From Ruby to Node: Overhauling Shopify’s CLI for a better developer experience | Hacker News Reader

From Ruby to Node: Overhauling Shopify’s CLI for a better developer experience

shopify.engineering

107 pointsmooreds101 comments

Cool. But god, the Shopify CLI / developer experience has degraded so much in the past few years. With the newest iteration, they want a full app re-structure just to work with the CLI properly. It is absolutely brutal.

Sure, before they had basically no tooling, but then they had some basic ones that everyone was basically happy with, and then they just decided to iterate too much.

(saying this as a ~ 8 year long Partner / Expert / App Developer)

I've spoken to some Shopify employees and they all agreed that the developer teams are a bit of a mess. It seems like nobody managing the developer products are actually using the end results, otherwise they'd realize the hoops needed to work through (unless you are starting a brand new project right now --- and hoping that they don't decide to change it all, like they have 5 times in the past 3 years)

A lot of Shopify Partners I've talked to feel the same way. The tooling and API has become harder to work with over the past couple of years.

The previous designs (which worked very well for a long time) seemed to have been all thrown out/deprecated to such a large degree it has caused a lot of long time devs to be scratching their heads.

They sure do like to start over, huh? What's funny is that Slate was canned while still in beta: https://github.com/Shopify/slate#-slate---end-of-support-jan...

One of the things I find fascinating is that in the days where it was Liquid templates, and Vercel etc. were barely a twinkle in people's eyes, Shopify had a huge value add - that you or an app development consultancy could iterate on customizing interfaces here and there, and deploy changes from the CLI or directly on the Shopify platform, without needing to worry about deployment, payment processing, etc.

But so much has changed now. Deploying a customized frontend has a negligible cost, and the API surface that Shopify powers could be powered by a myriad of tools, with Stripe etc. doing large amounts of heavy lifting on the administrative side.

Were contractual obligations, and not wanting to bite the hand that feeds, the only thing preventing longstanding Shopify app developers from joining together and creating an industry-standard alternative API surface that could be deployed in different ways?

My partner runs a medium sized ecommerce store on Shopify and her priority is having a system that is simple and stress-free.

And whilst she appreciates all of the Shopify Apps they are secondary to the fundamental process of taking, processing and shipping orders.

The key to the whole ecosystem are non-technical users like her not the app developers.

And people who think they will be the next big store :)

How does something like this happen? Shopify should be absolutely on top of its game here, seeing as user experience seems to be one of their main priorities?

Nowadays it's not that hard to get UX/DX right if you are actually committed to it.

My read on this is they just wanted to cut out Ruby and do everything in Node. That's fine as a decision, but rest of the article feels like just trying to justify it after the fact. Like someone at the top decided, now lets pretend it's a good decision.

As you mentioned, if they're not OK with CLI, they could refactor in Ruby. The whole "embracing functional programming" and MVC architecture (i think Rails when i hear MVC) as reasons to go to Node, and not Ruby, is nuts.

If they were worried about dependencies, they'd use Rust or Go as they mentioned.

> Like someone at the top decided, now lets pretend it's a good decision.

It might not even be a top thing, if they migrated internal eng from Ruby to node internally I would not be surprised if this was dev-driven either

Interesting how Shopify went from the most pro-ruby company to what looks like unreasonably rapid divestment from it. Wonder what's going on.

Ruby is still the primary backend language at Shopify. Node is popular amongst partners.

Besides this article, what gives you that impression?

I seem to recall reading recently that Go was showing up in their org. May be mistaken or misremembering.

We’ve been using Go since 2013. We recently started using Rust, but it’s mainly eating away at Go’s usage rather than Ruby’s. Ruby continues to be our first-choice language for most problems.

Thanks for the clarification, I appreciate it!

That’s how I read the article too. We wanted to move from ruby to node. They even admit that the most well know language is ruby in their org. Seems like a pretty serious waste of resources to me.

This might be the first article I've read where Node's module resolution algo is touted as a benefit.

Just how complex is your command line app if you need to have multiple transitive versions of the same library?

Coming from the same company that is so heavily invested in Ruby that they contributed to a new JIT for it (YJIT), built a static type checking layer (sorbet) for it, and built a module system that prevents you sharing code between those modules (packwerk), etc.

Unrelated but it is reminding me of my attempt to integrate with Zapier. Zapier doesn't document an API anywhere, you have to use its 'CLI app'. The CLI app of course just calls undocumented HTTP APIs behind the scenes.

Minor note, Sorbet was written at Stripe.

Agreed. Every year or two Shopify reinvents the wheel. Ruby => Go => Rust => Node => Wasm?

It will happen again. I believe they have terrible tech leads / managers who push/approve wrong things. They also grew exponentially during the pandemic, so that grew their manager's incompetence exponentially as well.

This is where I'm at right now. We have 40000 items on the store and I'm writing an updater for it, and the entire official ecosystem is a dumpster fire.

I've found some ok golang api clients which seem to do the trick so far.

i'm a shopify partner and i agree with this sentiment. changing the cli is one thing, but making it require a full app restructure is so risky that i'm holding off on doing it until the very last minute. besides, it looks like they are pushing more towards embedded apps. that is fine, but you need to also cater to app developers with standalone applications.

finally, i do think it will change again because the structure for theme app extensions have changed and now i'm not even sure if my project will work if i update that section.

the migration documents are very bad and they remove old documents. for example, it used to be that you need to add a folder called `theme-app-extensions` and it's very different, and there is no migration docs for that. they just want you to use cli 3 out of the blue.

It increasingly feels like the decisions taken and the work done at Shopify API / Libraries / CLIs / Docs etc are done for promos, rather than thinking about DevEx of people who try to build reliable product on their platform.

I notice this pattern of shoving nodejs / js, especially react through people's throat for the apps developed on the platform. Polaris, Session tokens and all the boilerplate apps the CLI generates are great examples of that.

I would rather having Polaris as CSS framework + some JS bits in vanilla-JS, Auth + Session tokens as just as a small JS library. But instead I am forced to use React. I know Polaris is also available as CSS only but that's almost impossible to use as it's not designed for human use really. Or Bridge exists as a library but for many cases it's just way too bloated and documentation for non-react version of those things are not so good.

> ”Go and Rust allow distributing a static binary, but at the cost of fewer people at Shopify being inclined to contribute due to the lack of familiarity with the language.”

If learning a language as simple as Golang is a real barrier for Shopify engineers, then I’d say they need to reevaluate their hiring processes. Especially given that they’re talking about CLI apps as opposed to something like distributed systems.

I’ve hired multiple engineers who had zero experience with Elixir at all and it’s generally just taken a couple of days to get up to speed. I’ve had the same experience on client projects, too. I don’t think Elixir is as difficult to learn as JS or Rust, but Golang might even be simpler given how small the language is.

I work somewhere with plenty smart engineers, and plenty of in-house golang expertise. I've seen several small internal tools rewritten from Python to Go, and contributions by the wider team totally dropped off. It used to be absolutely trivial for anyone to make changes, and the "deployment" of such scripts was basically "get the code reviewed and merge it." Now there's a whole Bazel song and dance, compilation as part of the dev cycle, producing binaries for distribution, and it's introduced just enough friction that people no longer make the little tweaks that used to be dead simple. Say what you will about loosey-goosey dynamic languages, but they work superbly for things like CLI tools.

I haven't used Bazel for CLI apps and may have significantly underestimated how much other complexity they have around their deployment processes. That said, it's also very possible and common to make Node apps a pain to work on and distribute, too.

IMO the big advantage of languages like Go in this space is that you can ship a binary to your users and that will Just Work™. Shipping a Python tool tends to be significantly more complex either for your users who will have to set up Python + dependencies, or for your build (e.g. the Python AWS CLI just ships an entire Python environment, which is a "solution" of sorts I guess, but it's not very easy or simple to do).

I don't think Go has to increase more friction in the dev workflow, but it probably does require a little bit (one-time) more effort to get it right.

Is golang really that easy to get started with?

Easy to learn, hard to master.

I've been looking into Go, but two old colleagues of mine that use it basically turned me off on it. The amount of complaints I hear about quirks and things that take more time than other languages make me not want to pick it up.

Ruby is a powerful language and its frameworks try to magically solve a lot of boring problems for you. I see “fewer … inclined to contribute” and I think they did or could learn the language but aren’t eager to spend so much more time on handwritten grunt work. (At least that’s my own attitude.)

I don't understand going from Ruby to JS. Go or Rust sure, but Node for CLI? What is gained by doing that lateral move?

Developers who'll work right out of school, is my guess. Rails is great for the people who used it during that generation, and still is, I'm told. JS offers cheaper and more plentiful talent, and doesn't require as big of a mind shift as rust. If they become more JS centric, given the overlap in JS and Rust ideology, I could absolutely see them involving Rust soon after.

If that's true that a company as big and established as Shopify can't afford to attract, onboard, and train ruby devs, it really feels like the beginning of the end for ruby.

Ruby will be fine, this is more of an impact on Rails as the centerpiece for getting-to-MVP fastest. It once was the case, and might still be in some capacity, but everyone with that expertise is on years 10-15 in their career and don't come so cheap because they've put in the work to earn more and are well within their rights to say they aren't going to work all nighters.

I wouldn’t say that at all. I know several successful indiehacker types who started in the past five years and have had success with Rails. Several of them started with JS first, but Ruby is still growing on an absolute basis, despite having a smaller portion of the market as a whole than it did a decade ago.

I suspect a part of this is that startups that use Rails (or Phoenix or Laravel) don’t need as many devs as startups doing a mix and match of many libraries instead of a full stack web framework.

I don't think this has anything to do with what they can afford at all. More like someone in leadership thinks Node is cool and collected a bunch of unvalidated reasons to support it.

Tooling is much better, for starters. And they plan to use TS, not JS, did you even read the article?

Let me get this straight: you believe that Node's tooling is better than Golang's tooling?

The entire Node ecosystem is commonly described as a dumpster fire.

It's not that bad - it's more of a meme than anything else as long as your sensible. Personally I dislike how Golang handles local packages. I'd rather use Typescript than Haskell in production!

it's in the article: "Hydrogen developers expect the npm install command to resolve all the dependencies that they need to work on a project. With Ruby as a dependency, that mental model breaks and they can easily run into issues where the CLI refuses to run because of additional steps needed."

If the bulk of their cli users are already node users/devs then that actually makes perfect sense.

Interesting. TIL about the Open CLI framework that they all seem to be moving to: https://oclif.io/

CAC also quite good: https://github.com/cacjs/cac. Used by Vite and many others.

its from heroku! we even did a conference once! https://www.heroku.com/events/2019/oclifconf (and i gave one of my most random talks ever https://www.youtube.com/watch?v=1_w1YWCHXFg)

It's from Salesforce/Heroku. The documentation is definitely lacking, but it's battle-tested and under fairly active development.

I moved the Zapier CLI to oclif a few years ago over a bespoke version, since it handles a lot of the nitty-gritty of displaying text in terminals. It's a pretty nice setup and allowed us to focus on the important things.

It's great too -- spiked out a CLI using it in my last company. It was really easy to set up and program in.

Same here. I wonder why it hasn't gotten more adoption.

Regarding one of the problems they cited--If anyone is interested in deploying a stand-alone ruby CLI executable using a modern Ruby (3.1), the fork here can do that, but you need to compile the rubyc for each platform first (not used the linked ones in the readme).

https://github.com/ericbeland/ruby-packer

Also, there was an issue with the most recent X-Code, so I had to actually downgrade my local X-Code to compile rubyc.

I wonder if Shopify had known they could build the ruby CLI into stand-alone binaries (no Ruby install or Node needed) if they would have still gone the Node route? Not that I blame them for not knowing--I had to fork the motor-admin fork, and update a few things to get it working. The original ruby-packer only works with ancient Rubies.

Good luck! I've found any cli with any kind of runtime dependency other than maybe sh eventually becomes difficult to run. I think they all tend to end up getting packaged with an isolated copy (node/ruby/python) of whatever runtime they need.

Questionable decision making. Why not use Go when minimizing runtime dependencies is first priority? I’d probably just have cleaned up the Ruby code and used something like Ruby-Packer to create binaries. When typing is so important why not use rbs? Makes no sense to me.

Agreed. Go is the obvious clear winner here for a CLI executable.

Rbs doesn’t hold a candle to TS.

Or they could have ported their Ruby to Crystal. That would have given the same benefits of Go when it comes to runtime, but with the benefit of only needing to do minor refactoring of the code instead of a full rewrite.

Nice. But there is so much more work to be done. From a customer perspective, it's great. However, really feel as the entire developer experience is in need of an overhaul (And it seems pretty clear that they know that too). In saying that, even if things don't change, it's still the best option for a lot of merchants.

The CLI is nice, but developing Shopify apps+themes is still a wild west.

It's possible to write CLI tools with support generic plugins in static languages: drop in {{name}}-{{extension}} executables in a private vendored plugins/bin directory. docker for 1, cargo for another.

At MAANGs, Rust is taking over and Go is waning. Go allows for quick productivity from 0 but it plateaus due to an unwillingness to add features. Rust has a Haskell/Nix/(fancy tool) problem but it has a good productivity curve that keeps going. Sometimes worse is better as UNIX people might recall.

Looks like it is mostly deployment issues they are fighting. I suggest to use a true deployment tool like GNU Guix.

I always got the vibe Shopify developers didn’t want to rock-the-boat to much with Tobias (founder/ceo) on tech stack, given his past role as a core Rails developer.

Developer experience seldom correlates with user experience.

Heroku did the same

more background info

https://blog.herokuapp.com/evolution-of-heroku-cli-2008-2017 Their version 6, started with Ruby, then go/node, then pure node.

https://github.com/heroku/cli "This is the next generation Node-based Heroku CLI. The goals of this project were to make plugins more flexible, remove Ruby as a runtime dependency, and make the CLI faster. [...] It has identical functionality to the old Ruby CLI."

The original article has no relation to Heroku, that's why the downvote.

Oclif is from Heroku so in a way Shopify's CLI is downstream of them

If only Heroku had any customers remaining to use it

The free tier shutdown was definitely not the best marketing. Their prices are steep in a docker/kubernetes world. They could have had at least left the free tier for Ruby projects as it’s their origin.

I'm begging you, just use a language that compiles to a single binary. It can self update in place, be easily installed on developer machines and CI pipelines without requiring a whole additional toolchain in every container. Google, Heroku, AWS, Shopify all do and it's an awful developer experience.

> I'm begging you, just use a language that compiles to a single binary. It can self update in place

Is there a library that does self updating in place for binaries? Go is the langage that comes to mind when talking about single binary, so maybe a go library?

Maybe check out https://github.com/keygen-sh/keygen-go? Let’s you add auto upgrades to any free or commercial Go app (with support for licensing too).

(I’m the founder of the licensing API.)

What's the mechanism for self-updating in place? Something more intelligent than just downloading and copying the file over itself? A running program itself is a copy of the file on disk, I think. So that sounds fine. But this still depends on getting permissions right though I think. Also, does a binary always know where it lives (across Win/Mac/Linux)?

Download-and-copy is the way. A single binary CLI can just copy over itself and exec into the new version. You can preserve the permissions of the original file.

With scripting-language CLIs, you either have to deal with the module install process or use an external packaging solution.

This works pretty good, but we've ran into issues where the downloaded file isn't code signed and then this becomes a big issue. AFAIK, if this is a public CLI then homebrew will codesign for you, if you are distributing a private CLI then you need to solve the code signing issue yourself. Not many blog posts on how to handle this...

Exactly, I get that they initially used Ruby given the whole company is build on it. But overhauling the CLI and NOT going for a single binary is beyond me.

Seems odd on the face of it. They say it is because the CLI already depended on Node and they couldn't escape it. So, they moved from depending on Ruby and Node to just Node I guess.

Huh? To run a node application, you need node installed on the container, as well as the JS files. Is that a big deal?

You don't even need that: https://github.com/vercel/pkg.

I think there are equivalents in the Python ecosystem.

I have used PyInstaller for this before, which has the major caveat of it not being a cross-platform solution. You can only build binaries for the platform you're currently on. However, you can work around this by configuring GitHub Actions to build binaries for you for Windows, macOS (x86) and Linux.

Yes, because now I need to install node, probably run npm install, etc…

there's no npm install and there's no etc.

All the files needed to run a node application are in node_modules. If node is on the container, you can just rsync the scripts.

I think you're comparing distributing stand alone binaries, vs distributing the source code for a node app, and then installing its dependencies, and maybe doing some transpiling. That's not a fair comparison.

Rust is amazing for CLIs (Go potentially as well, too little experience with that personally!), but if you're already quite comfortable with JS & have existing investment in it for CLI or related code, then ´npx <tool> ...` while asking the user to install Node (or Bun) is an acceptable tradeoff in most cases. Updating via npm is also easy enough for users, though it's not self-updating of course, but basically no CLI will do that fully automatically for you. Not even Cloud SDK CLIs. And for good reason.

I just find working with Node to be gross.

Look at this:

https://github.com/Shopify/cli/blob/main/package.json

but wait...this too:

https://github.com/Shopify/cli/blob/main/packages/app/packag...

https://github.com/Shopify/cli/blob/main/packages/cli-hydrog...

https://github.com/Shopify/cli/blob/main/packages/cli/packag...

https://github.com/Shopify/cli/blob/main/packages/create-app...

etc, etc, etc

https://github.com/Shopify/shopify-cli/blob/main/Gemfile

What am i missing? Why do you have to rely on all that junk to run a CLI? It feels like people choose Node because creating a Rube Goldberg machine creates job security.

The project looks extremely messy, so many packages inside packages. It feels like java, but less performant

One reason is because ruby has a large standard library. Also people just pick what they know - there are more js devs than ruby devs. Also neither ruby nor node are a good choice for a cli, so it is kind of a moo point.

> there are more js devs than ruby devs.

Not at Shopify there ain't.

It's like a cow's opinion.

They think of it as a benefit: "Node’s module system allows having multiple versions of the same transitive package without conflicting with each other"

You are missing a few things about the Ruby based CLI:

1. dependencies for Gems are specified in the gemspec file and not the Gemfile. See https://github.com/Shopify/shopify-cli/blob/main/shopify-cli... for example. There's a few non-development dependencies.

2. since it's difficult to package up a Ruby gem for distribution, maybe dependencies were vendored directly in the codebase: https://github.com/Shopify/shopify-cli/tree/main/vendor

This isn't meant to be a comparison of the number of dependencies or anything. Just pointing out a few nuances to how the Ruby dependencies were handled.

The dozens of ESLint dependencies are really what frustrate me... I wonder when someone will make a "batteries-included zero-config" JS linter with the most useful rules included, since that's what people seem to want in JS crazy land. See: Vite, Parcel, and the endless complaints about Webpack configs. Of course, people will inevitably want plugins and extensibility and you end right back up with Webpack again.

xo and eslint-config-xo

https://github.com/xojs/xo

All they need now is a sexy website!

That's not Node's issue but NX's monorepo. Having a mega monster root package.json just makes your ci/cd incredibly slow, one way to solve it, is to have per app package.json

PSA: The singleton context configuration is probably the most disgusting thing I've seen in Ruby culture. Please do not bring this practice to node -- I've already witnessed this crime in shopify SDKs and stripe.

Honest question: Why is it a bad pattern? And what are some better alternatives from other ecosystems?

The singleton pattern generally tends to be hard to write tests for. Singletons are basically a global object with static methods.

The dependency injection pattern fits better. DI is basically passing around objects which conform to an interface. This way you can mock out those objects more easily.

One issue that a lot of patterns run into in Ruby (and other highly flexible languages) is that a lot of patterns exist to help you deal with the restrictions the language puts on what you can do with it.

Ruby has enough sharp knives that many patterns are pointless, or trivial enough that they're idioms instead, or that the downsides of the pattern are drastically limited.

(example: command pattern vs lambdas; delegate vs blocks)

Having singletons can make sense, if the abstraction has to be leaky...

(examples of abstractions that must be leaky: AWS vs GCP) (examples of abstractions that can be tight: Segment vs Rudderstack)

...because then you're never really in a position where you need multiple objects, or to swap out the implementing object, during production runtime. So you don't benefit much from DI on prod.

And in Ruby, when you're in the tests, there's a handful of nice and handy shenanigans you can pull to remove the downside / AKA make it trivial to mock the object.

AKA Ruby is flexible enough that you need DI like a contortionist needs a backscratcher.

I agree in general, but Ruby is flexible enough to handle mock/stab everything so it's not tend to big issue.

So who's left, basically Github?

Left with what?

Using Ruby primarily is my best guess.

Stripe, for one

Shopify's monolith is Rails.

They seem to be more involved with contributing to Ruby than any other company, contributing an incredible amount, including yjit and object shapes and more.

Ruby on Rails is better than NodeJS for eCommerce still.

I don't know why they switched, but I haven't used ruby on rails that much.

Reply on news.ycombinator.com