Wondering what's the state of the art for managing the processes in local dev in monorepos in 2024? Or other tools for managing a monorepo I might be missing in general!
It is admittedly geared towards JS/TS stacks, but supports other languages by running commands via `dmno run -- yourcommand` which will inject the resolved config into your command as env vars. Generating types and adding helpers for other languages should not be too tough. We'll follow user demand as to which to implement first.
We are currently working on a tool that will help you run and manage your local dev services as well, but it is not ready yet. In the meantime, you can check out https://tilt.dev which is oriented around docker and k8s but does have some ability to just run arbitrary scripts. As for running scripts in general, I've had some success hacking together pnpm and turborepo, even to run scripts in non js/ts services. Lastly, I'll mention https://moonrepo.dev/ as it has more native support for polyglot repos, and its notion of task inheritance keep things fairly clean, especially in a larger repo with many services.
packages.foo = pkgs.callPackages { };
devShells.foo = pkgs.mkShell { inputsFrom = [ packages.foo ];
This isnt perfect I know with all the housekeeping, but it balances itself out in terms of flexibility. For example, I can trivially create a relatively simply "assets" package/derivation that is both just a folder a files assured to be somewhere during a build that needs it, as well as be a simple lisp package that interfaces with those files. And prod builds can become almost entirely nix derivations that have inputs from across the repo and are defined by simple inline scripts: "nix run .#prod-server".
The first monorepo I worked in was a python/typescript monorepo. It largely grew organically, and was heavily iterated on. We had no tooling to manage the complexity, and constant bumped into sharp edges that made life difficult.
- Dependencies change and silently break applications or developer environments.
- Figuring out how to start up applications you didn't work on could be difficult because you needed to figure out the tool chains and how to invoke them. This painpoint came up everytime toolchains changed.
- Figuring out where dependencies were used was really hard. This lead to lots of zombie code, we weren't sure if it was alive or dead.
With that being said, bazel does seem to solve most of the sharp edges we had with our ad-hoc/no tooling, but it comes at a cost. Bazel is not trivial, and won't allow you to do some things that the underlying toolchain might have been okay with. So that may leave you making code changes to fit inside bazels constraints.
Bazel tooling also isn't equally mature across all languages. The go tooling is pretty mature, and there are great generators that 'make things work', the situation for frontend is less mature, but getting better. It will add additional friction that webdevs might not love.
At the end of the day though, I would choose to use something like bazel (or one of the alternates), and the friction it brings over not having it.
Example bazel frontend. https://github.com/bazelbuild/examples/tree/main/frontend/ne...
> The retroactively named Buck1 was a capable build system and is still in use today at Meta (although many users have migrated). Buck2 is a rewrite that aims to keep the best bits of Buck1 (with a high degree of target compatibility) but also borrows ideas from academic research and build systems, including Bazel, Pants, Shake, Tup, and more.
https://buck2.build/docs/about/why/
I'd say the two main tradeoffs to consider is do you want something very general-purpose, or is something more specific to certain language(s) appropriate? and do you want mature ecosystem, or bleeding edge solution?
Bazel/buck2 are very general and language agnostic, while something like nx is more focused on JS with some support for other languages. Bazel is pretty mature at this point, while with buck2 you might need to do more trailblazing yourself.
This. At work, we use Bazel for the Java monorepo, and it's great (although pants2 seems like it might be better? Dunno, never used it...). Folks can convert individual classes to Kotlin, as they like to do, and it all just works.
The Go monorepo still uses standard Go tooling, with a couple makefiles and some scripts: there are certainly things that bazel/pants2 would do better, but the Go tooling is just so nice (and so fast) that there's a much higher activation energy required to give it up.
[Edit: adding...]
One note: probably the biggest thing about a monorepo is that you really need to invest in tooling to make your CI still go fast. Typically, that means (a) tracing out dependencies of a particular change, and testing only those things, and (b) partitioning the tests up intelligently and running them on lots of different machines.
IIUC, you could potentially get around having to do any partitioning or dependency tracing yourself by just using Bazel, with a bunch of Bazel workers running in the cloud, and then just do `bazed test //...` and let Bazel figure it out.
Otherwise, you're going to want to find or write something that divides your tests up.
Note that it's written by nx.dev (TS monorepo tool), so well worth taking the specific comparisons with a large grain of salt. Useful to understand what is out there though!
Although I'm not sure process management is touched on specifically. I've used docker compose for a similar purpose in the past, with source code in a volume to support live reloading.
We have a root Taskfile that references our JS and Go taskfiles and wraps / abstracts all tasks. It's good for dependency management and task orchestration (`task lint` will run lint in both JS and Go for example, and `task test` does the same).
TypeScript sharing between packages can be a lil' funky, but so far it's super solid. Happy to share more if it would be helpful.
The nice thing is that this just relies on npm, which means setup is simple and straightforward, and we benefit from all of the stability and continual improvements npm brings to the table.
I just used git in my home folder and it "just worked." Code reviews were pretty simple; the folder/file in the code review matched what was in your home folder.
It was pretty nice with a good .gitignore setup, so you could have your own dotfiles on the vm. There were also some really good scripts (you shared them by branch `git checkout jims-scripts -- ./scripts` or by commit—you didn't need review to commit new script files, but it was encouraged).
I'm running a hand-rolled monorepo with ansible, terraform, and several language projects (JS/TS, Swift, Ruby) that each have a makefile to standardize entrypoints for CI/CD. Docopt is nice because it's self documenting so you can better understand how all your apps are deployed.
For me personally I like that it updates Typescript references automatically and makes docker image setup far easier.
Thinking back on my work experience I’ve worked on:
1. Pure JS/TS 2. JVM/Kotlin/Java 3. Polyglot Kotlin+Python 4. Polyglot JVM+Cpp+TS+Python
I’ll address each one and what I used at the time.
But first I’ll address my dev flow for a greenfield project. Initially after some cursory documentation I build a scaffold. Over the years my scaffold has pretty much boiled down to two main directories, packages and services. Packages expose functionality, services consume functionality from packages and distribute it over RPC. It’s served me well.As for service standup, it goes: local host to incubate it, docker compose to integrate it and develop , kubernetes to deploy it.
For the pure TS code base, at the time I used yarn workspaces and lerna. This worked well at the time and I had no complaints from what I can remember. I think I remember having to volume mount my whole repo into each individual service to get hot reloading working, which was no fun.
For Pure JVM stacks, I found gradle was the way to go. It seems like a pain in the ass at first but once you grok it, and stay on a stable version it rocks. You can orchestrate whole workflows in it.
Polyglot Kotlin and Python. So I went about this a couple ways until I settled on something I liked. First I said, ahh yeah gradle can do everything, so I wrote a poetry wrapper and had my python services pretend they were like JVM services. This ended up not working so well in the long term. Especially because python library sharing is a lot different than JVM. The I went for a hybrid approach where there were multiple layers. Gradle for JVM code, poetry for python code. This was good but switching context kinda sucked and there were too many commands to remember, and ctrl+r history and tab only gets you so far.
For the last one, Polyglot JVM, Cpp, Python, JS/TS… I think I finally have something I’m happy with (for now at least), like I feel productive and I don’t dread having to do like anything in the codebase. The final touch was, take the gradle for JVM, poetry for Python, use turbo repo with pnpm in a web directory for all your JS/TS code. I then tie everything together in a combination of gradle scripts but more importantly, a python tooling package that uses click cli and poetry. The python tooling package has logic for anything I want to script, and I expose a CLI for it. Then I have one of my modular make files expose the functionality by calling into the python package.
Seems like quite the contraption but it works.
Judging from other comments I might want to check out bazel.
As a python dev, I've been frustrated with existing solutions, which is why we built Tach to solve some of the problems we've faced, particularly around organization and isolation - https://github.com/gauge-sh/tach
Pants is another great build system out of Twitter that has seen widespread adoption - https://www.pantsbuild.org/
Even if you only have like, 2 languages, often it is possible to stitch the tools together, if you don't go too far off the trodden path, and you understand the two well.
But that leads to the two most important questions: "Do you need to support multiple languages" (N = 3 or more I'd say) and "Do you need to support incremental caching" (including test results.) Once that happens, it becomes very difficult to properly express dependencies and cache artifacts, and it becomes increasingly difficult to expect every developer to consider all the possible interactions between the tools. That means builds take longer (no big cache), and it's easier to break them (missing dependencies) or make them slow (overly conservative dependencies).
If the answer to both of these questions is "Yes", then you need to take serious looks into solutions like Nix, Bazel, or Buck2 to manage your dependencies accurately and cache the results of your builds incrementally. All of these are, quite frankly, going to be difficult to adopt if you are not familiar with them. All of them can save you an incredible amount of pain and effort, too. It's sort of inherent to the problem space; building software is often more complex than we think -- part of the reason why simple single-language tools have so much appeal! -- and has a wide array of nuanced requirements.
Note that the common wisdom is that Nix/Buck/Bazel are often only needed at very large scale. That's true in the sense they are nearly mandatory, but there's a flip side, which is that it's often easiest to adopt those tools early on at a small scale, when there is the least amount of technical debt and smallest amount of impedance mismatch, and you can sort out those problems proactively. It's much easier than solving them all after the fact.
All of these tools will require some amount of dedicated engineering to keep them well maintained and oiled with all the modern conveniences (LSP support, packaging end-result artifacts, etc.)
it works.
...but I've no idea how. i don't dare step off the beaten path. when it stops working, i call for help, loudly.
I also take more of my experiences here from the open-source world, where "The build broke for me, what happened?" is often met with a response somewhere between "Works for me" to "You're on your own" or "Just rebuild everything." So, if you at least have someone dedicated who can help you when things go wrong, you're already ahead of the game!