Devpod: Remote development environment at Uber
uber.com
uber.com
It was a network terminal. Your files and entire session were on the server. Your “local” terminal consisted only of a network interface and enough compute power to display your session. The way they had it set up was that you could insert your Sun employee ID – the same card used to get into the building – into a slot in the terminal. That authenticated you to the server and displayed your session instantly. Want to show a colleague something you’re working on? Just put your ID into their Sun Ray and show them exactly what you were doing. That was cool! It was a frictionless way to demo and collaborate.
Turns out lots of software depends on at least minimal hardware acceleration.
Nvidia had a successful product as well before stadia. Hell I remember reading a use AWS GPU instances as a gaming machine blog posts before stadia.
But a few yrs in the big players in the cloud gaming industry figured out some critical latency issues around the controllers and optimizing delivery of the video feed. To the point where lag was almost non-existent on a good fiber connection. The datacenter stuff was obviously the core innovation though.
I still prefer to download my Xbox games to my series s/x but I spent a year playing Xbox cloud games exclusively and I could 100% see that being the default for a big part of the casual audience.
RDP doesn't perform even 0x better than the competition, like parsec.
For example, I'd say that I need:
A dropdown for a single option, with options: A, B, C
Actually, I think I'm just describing an analogue to HTML with the equivalent of CSS provided by the platform, but for native desktop toolkits (hopefully without the complexity of a browser engine).
I mean, that is a thing xpra can do. Or, x11vnc can do it with some hoop-jumping.
Hopefully never.
I had my own server in the DC that I could then connect to from my desk using a multi-monitor SunRay terminal. At home I had a SunRay connecting in to the office with VPN. I could move between terminals by just inserting my smartcard in whichever terminal I was at. There was even a company creating a SunRay laptop called Gobi that I tried using , but a regular laptop with the software client was a much better experience.
These tiny machines were just way too slow to handle even the tiny amount of work they had to do. Also, everyone knows that X over network is just not made for modern applications ("modern" in the year 2000!). I worked with Matlab, and had a lot of fun trying to rotate 3D plots with a few thousand points. It was just unbearable.
Then of course the "single point of failure" thing. Network problems? No one can work. Main server has a drive failure? No one can work. Main server needs an upgrade? No one can work.
The Sun Rays had super-poor USB support. My ergonomic keyboard had no auto-repeat when connected to these things, absolutely impossible to fix or even debug. Then of course there was Sun software: although they invented Java, their JVM was leaking like a sieve and everything Java had to be restarted regularly. The Sun coreutils were just very limited compared to the GNU counterparts. We complained endlessly, and in the end, IT budged and we all got our dedicated Linux machines.
I remember that famous Larry Ellison speech in the mid 90s about how thin client/networked applications will be the future. It’s apparently what helped make him famous because no one cared about enterprise DBs in the tech media:
https://tedium.co/2018/04/12/larry-ellison-network-computer-...
No. I don't think everyone knows that at all. At my university we used X terminals connected to a central Sun SPARCcenter 2000 over ethernet and it was fine. We used MATLAB, Maple V, SAS, etc etc.
* low utilization, so you need to auto-shutdown or auto-scale down systems but sometimes jobs did need to run overnight or on the weekend so you needed an interface/user training to avoid upsetting users/killing their jobs.
* local hardware is cheap and powerful - employees are already issued really powerful laptops and some teams just went out and bought their own really, really powerful workstations. It is hard for the 'cloud' to compete with this.
* bin-packing workloads was hard, Kubernetes probably solves this better.
* fast-customization was hard, we had docker but it was hard to train users to update/fork the dockerfiles to keep the environment reproducible. A lot of users were more scientists than engineers and so weren't great at using version control.
* persistence / shared-storage IOPs are expensive - shared storage is a nice to have and a number of teams made a lot of use of it, it also made it easy to migrate users around but it's expensive. Local disks were also very painful/slow/buggy to deattach/reattach (maybe this is better now).
* latency - we needed instances close to our team and at the time the metro clusters near us were like second class with low capacity and limited features.
* specialty hardware like GPUs
* multiple paradigms, we also needed long running dev/staging environments, spark clusters, or other software, some managed or licensed to run in specific way, and it was hard to get these all managed the same way, clustered on the same nodes without introducing other issues - but w/out this costs would spiral. Again now that kubernetes is like the defacto cluster manager this might be easier now.
I guess the thing that makes this make sense for Uber is the ginormous repo? This probably is a good "big ball of mud" solution to a bazillion tiny projects, each with varying degrees of documentation making it impossible to run.
One day we as an industry will figure out how to make it so that our dev setups work and run well in a multitude of environment (hopefully without that solution involving "we are pinning to a specific Ubuntu docker image"...)
A lot of this will be solved if the people making our OSes ever decide to build a real distributed OS (either SSI or something more like P9). The OS is supposed to make programs easier. Why we keep trying to reinvent the wheel inside every application, and tie them together with duct tape, I don't know.
I'm not so sure. Here's a quick comparison:
Desktop:
Dell XPS 12ᵗʰ Gen Intel® Core™ i5-12400 Windows 11 Home Intel® UHD Graphics 730 8 GB, DDR5 256 GB SSD
$669.99
https://www.dell.com/en-us/shop/desktop-computers/xps-deskto...
Cloud:
Linode 8 GB 8 GB 4 CPUs 160 GB
$0.06 / Hr
Suppose I work every working day of the year for 10 hours (an overestimate). I would pay 250 * 10 * $0.06 = $150 for the Linode. So the Dell desktop takes 4.5 years to pay for itself.
Granted this is a fairly rough comparison but I don't think it's obvious that physical machines are more cost efficient that cloud machines.
You can't physically connect a local screen and a keyboard to a remote Linode instance, so the $0.06/h is on top of the $669.99 (which in your example doesn't include screen, input, etc).
So in that exact setup you end up with a sub-par local computer that connects to a sub-par remote computer (and creates all kinds of headaches associated with remotely accessing compute resources); where the remote computer can't do a single thing that a ($669.99+$150) $820 local computer wouldn't do much better.
All to save you how much cash? How many hours' worth of $ in engineer salary? $2000-4000 will get you a really powerful M1/M2 Mac, and if you need more than 128GB of ram then you can indeed spin up an instance - and instances with >128GB of memory are nowhere near close to $0.06/h.
Optimize for developer productivity, that is your bottleneck. Uber's problem is less of "our local machines are bad and/or expensive", more of "our setup is so convoluted, and we have so many devs, that we need to apply economies of scale to tackle the problem".
If we take this as the basis for comparison, then the same dollars will buy 1-2 (working hours only) years of Linode 96 GB with 20 cores, so it seems like cloud services indeed lose competitiveness at the high end.
I'm not sure what you mean. The "Dedicated 32 GB" plan is 6x my quoted price. Why wouldn't the comparison be to the midpoint of the "Dedicated 8 GB" and "Dedicated 16 GB" plans (which have 4 and 8 CPUs respectively)? That would be around $0.14 per hour, about 2.5x my quoted price.
Anyway, it seems highly unlikely that local development, which uses CPU very burstily, needs a dedicated plan.
Someone needs to think about the boundaries between different parts of a system. If those boundaries are defined by functions, classes, packages or services, it doesn't matter that much. Yes, it is always a pain.
Shipping the entire forest of modules in a single repo seems like a good compromise. It stops being a good compromise when you start creating a complicated network of soft dependencies between those modules. And when you have everything in a single repository, it's hard NOT to do that.
By the way, there's nothing wrong with monoliths. They're just better if you design them as such. There's nothing wrong with microservices as well. It shouldn't be a surprise though that neither of them are actually silver bullets.
I understand why these solutions were created. I see it as an ephemeral thing. Once someone figures out the tooling and practices to automate all of that painful management locally, developers will always prefer that. Then when it's fast and easy, we'll break it once again (like we did with classes, packages, containers, etc).
Monorepos are a tooling nightmare. Monorepos are a bandwidth black hole.
Monorepos make some things possible that are just not possible or very very hard when you have multiple independent repos that are built and tested independently.
Namely,
a) they allow you to test the effects of a change on all the components that depend on you, before you merge your change. This reduces the noise caused by regressions (API or behaviour) introduced in one component that is depended on by many consumers.
b) they are a practical way to ensure that all components have up to date internal dependencies: by placing the burden of API and behaviour breakage to the author of the change, you don't end up having hundreds of teams each struggling to keep up with dependencies that keep breaking their builds when you update them and consequently hating the teams that release those changes.
None of these things is a big deal unless you're a huge company with hundreds of teams.
I think in theory there could be some tooling and workflow that could provide all or moat of the benefits of monorepos, without the downsides.
Until then, monorepos are likely going to be a bad choice for small companies.
In my experience, a monorepo significantly makes things easier and saves time everyday for all developers involved when compared with multi repo.
Bazel is extremely cool! But you end up with, like, a handful of people on the team who can write Bazel and eveyone else cargo culting their way through it.
There are a lot of JS "monorepo management" tools but they all seem concerned about the release phase for a lot of libraries that really should just be one library instead of 300 npm packages or whatever.
> repo=project
This has gotten better lately. At least, good enough for small-medium scale projects.
It's not that difficult and docs are outstanding. It can and it will be worse with your own Bash-isms that likely won't have any docs at all.
There is another aspect where if you are using a language like Python or JS then you have to kind of swim upstream to get an existing project onto Bazel. Far from impossible but if you just look at the default Bazel stuff without pulling in third-party libs it's pretty tedious to get a project with a good amount of dependencies working.
bandwidth:
I meant "bandwidth" as literal bandwidth. When your codebase becomes huge, your VCS repo size becomes enormous and it becomes harder and harder to keep a full checkout on all the development machines, especially if they are over the WAN (e.g. at home on your laptop).
This has fuelled solutions like sparse checkouts (like MS vfs for git, now scalar) remote development (like TFA; but also Google's cider and srcfs etc).
tooling:
naïve monorepo tooling (which I've seen in various companies I worked for) simply perform a full build of the whole monorepo for each CI execution. At first this is just fine since you can parallelize builds and call it a day; but after a while, the builds just don't scale anymore ; flaky tests become an increasing frustration etc.
The tooling that can help scale large monorepos does exist, but requires buy in and comes with its own learning curve and tradeoffs. One well known such tool is bazel (https://bazel.build) and bazel remote builds and remote caches. These tools are hard to set up, although folks at https://www.buildbuddy.io/ can help smaller startups by offering a managed service.
(Again, I'm talking about really large monorepos. A monorepo which includes a dozen or so modules, for which you can easily perform a full re-build on a single CI worker and on your laptop is not the kind of repo that creates tooling nightmares.)
For small-medium scale, monorepo has been a blessing after dealing with multi-repo systems for years.
That said, I cannot pretend I don't see the problems with suboptimal tooling working in a medium size codebase. "big" and "medium" and "small" are quite subjective things.
At $work we have a monorepo whose git repo grew to 1GB in size and where the CI turn-around is so high, and the glitches so often, that it often takes hours or even days to land some code to prod. Developers instinctively react by making bigger and bigger changes because the very thought of going through PR/review/CI/merge cycle once again terrifies them. It's all compounded by a security policy that forces a code review approval every time the source code changed, including when you have to apply fixes to build failures induced by a component you don't own.
All of these things can and should be be fixed. But this is work, and is not urgent work so it's not often done at the same pace other stuff is done. This induces fatigue in the team and, as usually things go, people tend to blame the most easiest thing to blame: the monorepo.
That's why I try to phrase the problem to be a tooling problem and not a monorepo problem. Clearly if you don't have a problem with your monorepo, you either already have good tooling, or you don't need good tooling.
It's also getting a lot better, gradle works amazingly well for a monorepo even with dozens of developers committing to it every day with shared caching, nx/turborepo/others are making the story for front-end/TS much better too.
The issue that monorepo is solving is regarding write operations: FOO depends on BAR, which depends on BAZ. If you need to change BAZ to develop what you want on FOO and they're all on multiple repos, you'd have to pull request your way from the bottom to the top of the dependency graph. This is what causes the friction that monorepos avoid, and this is the hard part of such workflow to automate with multiple repos.
You might spend one week building next version of your component and 3 months updating all of the dependents. Then do it again. And again. You are working at a fraction of your productivity. I wonder if that's why Google needs thousands of engineers.
Whereas I just updated stripe from 2.x.x to 5.x.x in one of the projects I'm working on, because new version has features that I needed. I never wasted time updating to other versions until I had a need to.
It also limits your ability to break backwards compatibility in new versions, because we all of course are coming up with great designs right from the beginning.
I get the security and performance benefits of keeping all dependencies up to date, but man, the time sink and limitations seem so not worth it.
That is what the visibility [1] in Bazel solves. You can't import other people's code unless they say you can by making their code visible to yours.
Maybe you need to dockerise Mongo, MySql and 5 other dependencies - I can get this, but I don't get it why the rest the code should still be running in the cloud. Python, Rails, Node? Why? Developers should be able to run 1 shell commands to install node.
Dev experience excuses are just excuses for a bad setup. So, fix your setup, please.
Not being able to run the project natively is big red flag for me and when I move jobs will be my 1st question.
Spinning up a VM with an image containing all the development tools is a much smoother experience most of the time. The only reason why I don’t use it where I work is because I use vim and network adds too much latency for me.
That's why they use devpods. You moved entire configuration to the cloud. There is absolutely no reason why it should be run in your local environment.
Instructions to new employees would say things like "run this thing, scroll up past dozens of pages of stdout noise and manually deal w/ the errors buried therein by looking up relevant FAQs in some doc somewhere"
The scripts would touch every technology imaginable, from brew to npm to arc (phabricator's cli) to proprietary tools and no single person understands how setup scripts work in their entirety.
One exercise we'd get new employees to run through was get them to brainstorm about how some system ought to work. The lesson was that just about any idea they could come up with would have already been tried (and failed).
I'm told that devpods aren't even the first time we tried cloud dev envs. Presumably lots of lessons were learned from previous attempts at improving dev envs.
The devpod flow is a lot smoother. I had my laptop replaced recently and was up it running again in an amount of time that felt like cheating.
Okay, but if you can define/script your environment enough to run in a pod, couldn't you just run that locally? You already have to solve the manual steps either way...
It’s a construct of the order of operations.
One core issue with local is the variety of OS’ and local build tools that would fundamentally mess with the centralized scripts. Getting company-wide setup scripts to work on top of existing laptop config was a continuous challenge. Hence, having a consistent baseline (OS flavor, system-level packages) on top of which the company-wide “setup script” is added followed by “developer-customizations” seems to work great.
Central teams can manage the first couple of steps and individual user-specific configuration can be managed much better in a decentralized manner.
It's easier to apply best practices to a greenfield project written in a modern language (hence devpods) than trying to comb through a decade+ of tech debt written in bash.
It IS ever-changing. One library among the zillions of local dependencies that you need to build something changes, and you have to go through the dependency hell.
If entire software world valued backwards compatibility and vigilantly guarded it, that wouldnt be a problem. But in the package hell that we are living in today, every other day a package update brings some incompatibility or breaking change for this or that other thing.
Now, it's great if you can avoid complex setups in general cuz complex is harder no matter what! But if you're starting from a complex setup, having easy ways to roll out changes is an important step in actually doing the simplification work to get to where you want to be!
Three reasons from my perspective:
1) There's no setup steps. You just open your editor of choice and everything is set up for you. All the build tools, linters, specific versions of software $XYZ, etc.
2) Large VM (16 core, 96GB of RAM in my case) speeds builds and tests up dramatically.
3) Zero productivity lost if your laptop breaks. Just grab a new one from IT and you're up and running exactly where you left off with zero effort.
> Not being able to run the project natively
What do you mean by this? It's just running on a remote server rather than your laptop.
It's great too because if your own environment breaks, you can compare to a working one to fix it.
Just because a company might abuse it doesn't mean we should avoid it.
IE. it's no slam dunk. Like any other tool of this sort is will be used for good and ill.
Why would your personally customized development setup even need to be on the remote host? You pop up your IDE, connect to the remote host via SSH and you are set.
Currently I just have a git repo with my setup mostly in it (sans executables) with a way to get it going on a new machine. It works but is rather hackish and requires a bit of work to keep in sync.
For people who knew how to run vms or even chroots, this was not a big issue
It really feels like it’s local, I enjoy it
But we're at the point where we can still have a lot of analysis on the machine, and offload the slow analysis and building onto a more powerful computer, so that remote development is faster even with latency. And also getting to the point where internet connectivity is really fast so latency is low.
Like with most systems, the important part of remote development is that it's done well. And it seems like most employees are comfortable with Uber's setup.
Although, I do have to say that most companies really shouldn't use remote development, only if they have some excuse like they're Uber sized so that the benefits outweigh the costs. I've done remote work at uni, and they're servers and integrations aren't nearly as good so it's a chore; it's faster to use Mutagen and just develop locally then build/deploy remotely.
Because, making local environments exactly identical to the actual production environment is nigh on impossible. There still will be minor differences. And, to maintain the local development environment, crap ton of work will go to however that environment is maintained.
If engineers are maintaining it themselves, each of them will literally waste time on maintaining the local dependencies needed for the local environment - frequently encountering blockers due to package management hell that we are living in these days breaking one thing or the other. If you have 100 engineers as an example, your organization will lose 100 man-hours each month to such local development environment issues.
If you go the route of having infra or dev experience teams etc maintain them, then that team will be spending that effort to keep the scripts and whatever being used to keep the remote local environments in the engineers' computers up to date and working.
Instead, that infra team can just prop up dev versions of their infra/cluster/whatever, give the engineers access to that environment through a vpn etc, and voila - you instantly removed a lot of that lost man-hours.
Moreover, you will not never encounter any totally unexpected bug or performance problems that could end up coming to being from there being unforeseen incompatibilities in between local environments of the engineers and the actual prod environment.
> So, fix your setup
Life is not long enough for hundreds of engineers being spending their time on fixing totally unnecessary package management conflicts that are created by the utterly insufferable package and dependency hell that we are living in today. If you like suffering through that dependency hell, good for you. Most of us prefer to ship code and make things happen.
All good until you have an outage. Then your development team’s productivity drops to exactly zero while you fix it and your entire production environment is now potentially vulnerable to defects you can’t fix until the development environment is fixed (better hope it stays running). This is when companies realize that the development environment is actually a service and it needs higher SLA targets than production, but it will never get the attention that it needs to achieve those targets (because it’s just dev, right?).
But compare dev productivity lost due to service downtime with that of each new SWE in your org burning time to A) setup their own unique snowflake of an environment and B) futzing and debugging it when it breaks or there's a software update.
Anyone good is doing that even if you tell him not to. One size does not fit all.
Dev servers are arguably much simpler to provide without issues.
None of that is possible in a centrally-managed dev VM setup. When the VMs go down, you send your dev team home until it’s fixed. You’re still paying their salaries while you pay another team to make them productive again.
Reason 743 why monorepos create more problems than they solve
There are mitigations and a couple of hours of dev downtime is imo not the end of world. Sure, prod incident could overlap with a devpod incident but sounds like they still have the choice to do local dev.
Lost revenue is usually more impactful than the occasional lost developer time.
Better to have hundreds, or * gasp * thousands of engineers having to fix local package dependency problems on their computers every other day. Much better than those engineers having to keep that gigantic context loaded in their mind so that they fix those local problems.
That effort spent for maintaining local environments and navigating the package management hell is effort not being spent on creating actual code.
Does it mean also having a single unified production environment build?
I have been managing a stack composed of 500+ repositories, communicating through webservices, files and ABIs for many years now, and never quite hit much of the issues cited as reasons to switch to monorepos.
Having multiple small independent environments for each deployed service is a feature to me. It reduces the surface of bugs and regressions introduced by new dependencies.
Not having to update dependencies globally has been a feature as well. It allows to prioritize which environments to migrate first. I found that big bang dependency updates burden is the #1 reason of _not_ updating a dependency, while allowing a per-service dependency migration ensures we can be fast to update the most important and supported services.
Switching between repositories has never really been an issue to me. I found that if the structuring of projects in repositories make sense, rarely do you have to work across more than a few of them at the same time.
Each repository is its own package, with its own dependencies. Features that cross repository boundaries are much less frequent than isolated ones, and updating dependencies to other projects is part of each project's PR anyway.
I never tried monorepo because I never quite felt the need to. To me it seemed to be a step backward to end up with a megafat repository, where individual service history would be lost in the overall monorepo history. The deployment seems like a nightmare too, having to update the whole stack at once because you then have no idea which individual service changed between releases.
Not to mention I really don't want people to spend time migrating project X - that does its work perfectly without issue since 5 years - to the latest version of LibFooBar just because project Y wants it.
What am I missing?
At some level a monorepo is just a way to stick all your code in one giant directory and manage it all under one VCS repository.
You could still do separate build tools per project, separate vendoring if you really wanted and so on.
However you may find that being able to simply import other first party code by path instead of doing some cross repo dependency process is a massive win.
Edit: apologies, this was meant to be a reply to the top level comment.
I don't quite get the benefits of "depending by path".
The engineering seems huge, you now have to create a magic meta build and testing system so that only individual components that changed are rebuilt/tested. That seems like a scaling nightmare at best.
Also, dependency management is hardly an issue on most modern stacks anyway. Javascript, rust, python, etc, all have private package hosting tools that are trivial to deploy.
It makes working in all the languages mostly consistent, and provides a nice platform for optimizations like only testing code which changed.
I think the mistake is smaller companies adopting it without understanding the large amount of investment it requires. If you have a mono-repo mostly in one language, and you hire someone who is going to work in another language then you could be in for a world of pain. Unless you resource the language support.
Monorepos allows us to centralize important dependency upgrades. E.g. fixing log4j vulns is a lot easier when you can patch everything simultaneously. Same for tzdata (2022g gave very little heads up) Auditing for npm supply chain attacks was a lot simpler in monorepo than microrepos. Etc.
Monolithic version control doesn't have to mean monolithic everything.
Our web projects can be deployed independently of each other, and we leverage tool like yarn workspace focus and bazel for granular installs and builds/tests/etc.
It doesn't have to mean monoversions either. We support multiple version of libraries, though we prefer coalescing them as much as possible to facilitate effort centralization. Finding out that your library change will break downstreams before you land the change is a feature.
We had microrepos before and the main problem is that to this day I still get some random team coming to me for help w/ some rediscovered 7 year old repo that doesn't even build anymore cus lockfiles weren't a thing back then.
At a large enough org, you'll inevitably see the full spectrum of team quality, from the really good teams to the one intern/contractor getting thrown into the deep end of some unloved ancient thing. You want a common denominator that lets you do things like patch vulns in unstaffed projects.
I've done fairly large migrations both to and from monorepos. Each has pros and cons. For us and companies like Google, monorepos work well with our organization model. For others it may not.
monorepos solve none of the problems you listed. you are just basking in the short lived light after a big refactoring.
only thing monorepo does is make it easier to update some shared code and have all code which uses it run tests beforr pushing the new lib version. with many repos (micro repos is false speach to justify monorepos) you first publish the shared lib and then find out downstream failures.
anyone using monorepo in a way you can't build a small pieceocally is doing it very wrong.
How does CI work in practice? how do you avoid rebuilding and retesting the whole repo at every change? That would be an insane waste of resource and time.
I was resistant at first, but have found it to be a worthwhile trade-off
It can also cache execution of unchanged transitive steps, so you can skip builds/tests that were already run previously (e.g. you could skip most of a large 2nd CI run if all you did in a code review was edit one file.
You can also parallelize execution across cloud nodes.
> E.g. fixing log4j vulns is a lot easier when you can patch everything simultaneously.
This is true, though the refactoring action done can also be distributed and carried out via automated Pull Request creation to multiple repos. You still will deploy the changes over a period of time with a degree of parallelism.
> We had microrepos before and the main problem is that to this day I still get some random team coming to me for help w/ some rediscovered 7 year old repo that doesn't even build anymore cus lockfiles weren't a thing back then.
You can solve this by forcing a CI build for every repo to be run periodically.
> For us and companies like Google, monorepos work well with our organization model.
What are the concrete aspects of the organization model that make both orgs favor monorepos?
The problem monorepos attempt to solve isn't a technology problem, it's more of a people problem. For example, say you cron CI job fails. Then what? Someone needs to look at it.
It's easier for someone to fix things they currently have context for (e.g. if I upgrade Python or update some security-related config and something breaks, I can reason it was my change that broke it), vs an unsuspecting contractor getting around to some backlog task 6 months after the fact with no context.
Organizationally, we can shard tasks to match areas of expertise. We only need one Node.js expert, one tzdata expert, one JRE expert, etc, to upgrade each of these, instead of everyone needing to obtain above average familiarity w/ obscure FFI bullshit or whatever in each technology.
No, the way you store the source code for projects is independent to how you ship them.
>Not having to update dependencies globally has been a feature as well.
For a large upgrade where a bot can't fix what will break it is typical to introduce it and have both versions in the monorepo at the same time. You still migrate each project one at a time and eventually remove the old version. Of course you could never finish migrating and have to support both versions at the same time.
>I found that if the structuring of projects in repositories make sense
I don't see how that makes any less sense than putting projects into folders.
>The deployment seems like a nightmare too, having to update the whole stack at once because you then have no idea which individual service changed between releases.
If your landing page is edited there is no reason that you should be deploying a new version of your mobile app. Deploying everything overran every commit is a tooling issue.
>Not to mention I really don't want people to spend time migrating project X - that does its work perfectly without issue since 5 years - to the latest version of LibFooBar just because project Y wants it.
Then pay the cost of supporting two versions of LibFooBar in your monorepo.
You can completely emulate what you do with multirepo with a monorepo. A monorepo gives you extra things like a single revision that can let you see the state of everything from when a build was made or making it easy to depend on the latest version of libraries without having to constantly bump it either manually or via bots.
Some advantages off the top of my head:
- Let's say I am getting an error in production, and it was built from version X of the repo. In 3 seconds I am navigating the source tree of all of my tens of thousands of dependencies at that exact version.
- While developing, doing experiments, or reproing a bug, I can trivially make temporary changes to any dependency. It's a zero effort thing, so I often jump into any dependency without hesitating. For example for adding some extra logging.
- Step debugging into code from any library dependency is trivial.
- Making changes to a library let me use the build and test system to find if it breaks any user. Because the build and test system is completely consistent across the monorepo, I can easily dig into any user's code, fix it, and run their tests with my changed library.
The "Software Engineering At Google" book page discusses more pros and cons: https://abseil.io/resources/swe-book/html/ch16.html#version_...
Every benefit (minus atomic commits*) can be had immediately on micro-repos with a for-loop to do the monorepo-thing in each one. If bazel is what you want, great, use bazel! In microrepos! Want consistent dependencies? Enforce them! HEAD must build? Wonderful, nothing's stopping you from doing that! It's all solvable, and quite easily.
What you actually want is reliability and consistency. A monorepo gives you one political entity, with a clear adoption path, to argue with when enforcing those kinds of requirements. So they're much more likely to actually be achieved. In a big company, that may be worthwhile... but oh boy are the downsides large, and the only way to deal with them is massive eng effort and money.
* Atomic commits are a completely false promise. Your code on multiple machines doesn't change execution atomically, and being able to split a breaking change is a good thing, and massively harder to support in monorepos - it lets you adopt changes gradually, rather than forcing it on everything at once. You know, the same thing that every safe-change-practice handbook says you should do. Except in monorepos apparently.
Is that all? These 5 less minutes of bookkeeping are the killer feature of monorepos?
Because, as far as I see it, there's an insane amount of engineering to make a monorepo work even at small scale. Are these 5 minutes per multi-project PR worth it?
I wouldn't be surprised these 5 minutes are largely offsets solely by the additional hours of CI testing time introduced by having to run tests on the whole monorepo at each commit instead of just the project that changed.
This is rarely the case. Most monorepos shard tests such that wall time is minimised and have dependency analysis that only runs tests affected by the changed code.
More of a problem is that IDEs and LSPs often don't deal well with having to index and navigate very large codebases.
I've maintained both monorepo and polyrepo environments and there are pros and cons to both, and they vary _wildly_ based on the language being used.
Also I used bazel in my previous monorepo setup… so not really paying a huge CI cost.
Some of us have evaluated several approaches based on their merits and tried to makes decisions that are best for our specific organization.
I wonder though, is it worth it to setup a remote dev machine if you have an M1 Mac?
I think you can get good compile times for Rust etc. on an M1. For me I have an intel Mac at work and at home so the remote dev env is better for builds.
Since I see it mentioned in many threads here: for a huge repo one can always use `git clone --depth 1` to get that repo and later do a proper pull to retrieve all history.
M1s are a mixed bag for this way of working. Pre-M1, devs were running into local computing power issues. Post-M1, more compatibility and stability issues.
I definitely see the use case, but in saying that I find local development really valuable and default to it when I can. I do however run dev work almost exclusively inside a container so I'm flexible either way. I can see how some might not be.
That said Projector round tripped all the key strokes to get new draw commands (unlike VSCode) which also results in lag. The new IntelliJ remote architecture is much better, and it seems Uber is moving that way too.
I’ve used something identical and mosh makes this just work. Most devs at that company swear by remote builds and hate laptop builds
One funny aspect of Google moving development into cloud machines was that a decade before the process ran the other way. The desktop that was issued to most engineers was a production websearch machine turned sideways and stuck under your desk.
[1] https://chromium.googlesource.com/chromium/src/+/778a7e84f65...
nix solved my problem by and large for well modularized projects. Because, nix can provide nearly identical dev environment in practice.
This URL works without redirects: https://www.uber.com/en-US/blog/devpod-improving-developer-p...
From very high level it works as follow:
- developer logs in to AWS cli
- executes: dev/01-start-env.sh
- prepare infra services: dev/02-base-platform.sh
- code can be changed and run locally. But if they need to test in the bigger system: dev/03-deploy-code.sh
There is dev/99-delete-env.sh
An environment is a personal ec2 spot instance that auto shutsdown if there’s no developer activity for over an hour.
The idea is that all developers work (program/code) locally as much as possible. But deploys changes to their own private environment in a much heavier VM’s that runs all services in containers.
Also there is a dev/tests.sh that executes the exact same test cases as in continuous integration. In fact, we try to bring all checks enforced during CI to be available to developers in their semi-remote-dev area environments for quick feedbacks.
In seriousness, we historically used microrepos and just getting to language-specific monorepos was already a monumental effort.
I do not notice that i develop remotely. VS code also has some great quality of life features, for example if you run a command that exposes a port in your remote machine, it automatically forwards it to your local. Same for jupyter notebooks.
VSCode finally enabled the features that command line folks enjoyed for decades.
I've wondered for quite a while where all this code and complexity is at. My guess is a lot of it is in their self-driving cars projects and stuff like this, but I don't really know.
... say what?
I think it's hampered by licensing though, maybe that's why it's not used or mentioned in this post.
We have support for AWS-based Mac VMs on DevZero but we don’t find our customers having their biggest issues related to iOS dev yet (we also target enterprise cos that have a vast diversity of tools, mostly backend and front end)
The biggest services/apps at Uber are not developed in devpods. Speculative, but these IDE’s were developed first as local first environments, there’s lots assumptions they make, adding up to terrible latency.
Also does it automatically install dependencies? Feel like running npm install each time (or the equivalent) would be slow.
Say net cost to company for an engineer is $100k-$200k+. Even a net 10% savings over a year means $10k-$20k+ vs a $600-1k/yr investment (in worst case). Security posture is also significantly improved, which admittedly is harder to assign a $ value to
this can be as simple as an ephemeral ec2 spot machine that reacts every time files on it’s filesystem change. it then does stuff, like building and shipping.
your local setup needs to rsync files from local to remote every time you save a file.
i’m on an upload constrained setup right now, and this[1] significantly speeds up my iterations uploading lambda zips.
fancier setups probably are similarly advantageous, but add tradeoffs proportional to their complexity.
1. https://github.com/nathants/aws-gocljs/blob/258ea5bb72d06a50...
There's actually an elementary litmus test on migrating to monorepos. It is a questionnaire with only one question: is your company Google? If the answer is "no" — you don't need a monorepo.
You are welcome.
Anything over ~40ms is very noticeable when typing and editing.
Then you're screwed.
I'm a hacker dammit. I'm supposed to be dangerous if you let me within whistling distance of a payphone.
The idea of slowly becoming useless without an internet connection and access to my devpod makes my skin crawl.
Plus super frustrating to have somebody preconfigure your environment because they will inevitably get it wrong.
To do this well it takes manpower. There are some off the shelf options out there for remote/prod like Kubernetes based development that are smarter choices than building your own. The table stakes for doing this is being a big corp that can afford to spend millions on one custom tool with a hope for positive ROI. For the rest of us, find something that works that is open source or easily licensed.
also curious about the terminal access. does it maintain state on the remote (like screen/tmux) or are the sessions subject to reset under network cuts.
i'd also be curious about settings since the pods are ephemeral. i suppose people would have scripts to grab their dot files, but some things like say, the android avd tool, can store settings in myriad ways.
that said, looks very cool!
I am a founder of DevZero (devzero.io) where we are taking the theme of "remote compute with local tools", but built specifically to serve engineers in enterprise companies - still pretty early, but would love for people to check it out and provide feedback!
How we're looking at the space: - IDEs need to stay local but thankfully, VS Code, Jetbrains etc all now allow connecting to remote VMs and containers - the main issue is around not have enough of your dependencies present. So outside of standalone dev environment (VM/container), we also let companies "bring their k8s/serverless config" and let each engineer have their ephemeral full-stack to code against.
The standalone environments offer various perf boosts (super simple onboarding, switching project, we're seeing it reduces net time-to-deploy from start to deployed as well) for engineers coding in monoliths, which is true in many large companies still. We're already seeing really good traction here.
For the stuff where we're trying to take the engineer's IDE to an "ephemeral and hermetic env" that is built off of however "production workload management" works: an engineer can connect their local IDE to a remote "devpod" and do all their normal coding activities (w/ the relevant boosts in speed etc). When they want to do some form of end-to-end testing, the engineer can hit downstream pods/serverless stacks etc (their own copy, i.e., not shared tenancy). We're currently figuring out if we can enable engineers to test a full end-to-end call chain and connect live debuggers to arbitrary pods in that call chain -- I think this will make debugging amazing cause so far its been pretty hard to repro end-to-end call chains in "dev-mode". Our platform approach is basically making cloud dev environments (CDEs) even more awesome by putting them within a production-like environments for every dev.
Lots of info here (not shared widely yet, not even on the website). Please let us know if you want to use it, or consider working with us (we're actively hiring).
Re: workload management, we added support for k8s* generally but are now expanding out across the various clouds. For serverless, we started with AWS lambda and now have to tackle the other clouds. Then, we also need to do the default container mgmt for each of the cloud providers. (also looking at hashicorp nomad etc).
*Say you have helm to deploy containers/pods to prod. We look at that (and w/ a little more config re: dbs etc), give every engineer their copy of prod in their namespace alongside a devpod. Similar vibes for other workload mgmt systems.
so devpod "production OS" - the top level devpod - contains ALL running services ? at Uber scale it is what 64 GB RAM ?
if ur already using kubernetes ... why do it this way rather than have kubernetes namespaces with many containers in a dev cluster ? trivially this is a docker compose stack right ?
second question - has there been a ROI recovery in terms of laptop hardware for devs ? like - u only 8 GB ram laptops and not more.
I don't work there, but I bet their full stack takes more than 64 GiB RAM.
> trivially this is a docker compose stack right ?
In the way that docker is the same thing as Kubernetes, yes. However there are differences that become material when you zoom in a bit closer.
second question - has there been a ROI recovery in terms of laptop hardware for devs ? like - u only 8 GB ram laptops and not more.
> second question - has there been a ROI recovery in terms of laptop hardware for devs ? like - u only 8 GB ram laptops and not more.
I don't think it's an optimization for dev laptop specs. At the end of the day, it's cheap for Uber to just max out the ram on dev laptops.
I am a founder of DevZero (devzero.io) where we are taking the theme of "remote compute with local tools", but built specifically to serve engineers in enterprise companies - still pretty early, but would love for people to check it out and provide feedback!
How we're looking at the space: - IDEs need to stay local but thankfully, VS Code, Jetbrains etc all now allow connecting to remote VMs and containers - the main issue is around not have enough of your dependencies present. So outside of standalone dev environment (VM/container), we also let companies "bring their k8s/serverless config" and let each engineer have their ephemeral full-stack to code against.
The standalone environments offer various perf boosts (super simple onboarding, switching project, we're seeing it reduces net time-to-deploy from start to deployed as well) for engineers coding in monoliths, which is true in many large companies still. We're already seeing really good traction here.
For the stuff where we're trying to take the engineer's IDE to an "ephemeral and hermetic env" that is built off of however "production workload management*" works: an engineer can connect their local IDE to a remote "devpod" and do all their normal coding activities (w/ the relevant boosts in speed etc). When they want to do some form of end-to-end testing, the engineer can hit downstream pods/serverless stacks etc (their own copy, i.e., not shared tenancy). We're currently figuring out if we can enable engineers to test a full end-to-end call chain and connect live debuggers to arbitrary pods in that call chain -- I think this will make debugging amazing cause so far its been pretty hard to repro end-to-end call chains in "dev-mode". Our platform approach is basically making cloud dev environments (CDEs) even more awesome by putting them within a production-like environments for every dev.
Lots of info here (not shared widely yet, not even on the website). Please let us know if you want to use it, or consider working with us (we're actively hiring).
*Re: workload management, we added support for k8s** generally but are now expanding out across the various clouds. For serverless, we started with AWS lambda and now have to tackle the other clouds. Then, we also need to do the default container mgmt for each of the cloud providers. (also looking at hashicorp nomad etc).
**Say you have helm to deploy containers/pods to prod. We take those charts (and w/ a little more config re: dbs etc), give every engineer their copy of prod in their namespace alongside a devpod. Similar vibes for other workload mgmt systems.
> Unfortunately, here we are limited by the cloud provider availability that’s capped at 99.5%.
Over at https://GitHub.com/coder/coder you’ll find our source code btw. Essentially we provision software development environments using terraform for Linux,windows,Mac,arm,amd64 and soon FreeBSD…