Tracking developer build times to decide if the M3 MacBook is worth upgrading
incident.io
incident.io
That said, it would have been much easier and more accurate to simply put each laptop side by side and run some timed compilations on the exact same scenarios: A full build, incremental build of a recent change set, incremental build impacting a module that must be rebuilt, and a couple more scenarios.
Or write a script that steps through the last 100 git commits, applies them incrementally, and does a timed incremental build to get a representation of incremental build times for actual code. It could be done in a day.
Collecting company-wide stats leaves the door open to significant biases. The first that comes to mind is that newer employees will have M3 laptops while the oldest employees will be on M1 laptops. While not a strict ordering, newer employees (with their new M3 laptops) are more likely to be working on smaller changes while the more tenured employees might be deeper in the code or working in more complicated areas, doing things that require longer build times.
This is just one example of how the sampling isn’t truly as random and representative as it may seem.
So cool analysis and fun to see the way they’ve used various tools to analyze the data, but due to inherent biases in the sample set (older employees have older laptops, notably) I think anyone looking to answer these questions should start with the simpler method of benchmarking recent commits on each laptop before they spend a lot of time architecting company-wide data collection
We ended up collecting all this data partly to compare machine-to-machine, but also because we want historical data on developer build times and a continual measure of how the builds are performing so we can catch regressions. We quite frequently tweak the architecture of our codebase to make builds more performant when we see the build times go up.
Glad you enjoyed the post, though!
Sometimes these wandering paths to the solution have multiple knock-on effects in individual contributor growth that are hard to measure but are (subjectively, in my experience) valuable in moving the overall ability of the org forward.
So sure, an M3 might make my build 30% faster than my M1 build, but the network build is 15x faster. Is it possible instead of giving the developers M3s they should have invested in some kind of network build?
If you have a massive, monolithic, single-executable-producing codebase that can't be built on a developer machine, then you need network builds. But if you aren't Google, building on laptops gives developers better experience, even if it's slower.
It's like telling an indie hacker to adopt a complicated kubernetes setup for his app.
> [..] due to inherent biases in the sample set [..]
But that is an analysis methods issue. This serves as a reminder that one cannot depend on AI-assistants when they are not themselves enough knowledgeable on a topic. At least for the time being.
For once, as you point, they conducted a t-test on data that are not independently sampled, as multiple data points were sampled by different people, and there are very valid reasons to believe that different people would have different tasks that may be more or less compute-demanding, which confound the data. This violates one of the very fundamental assumptions of the t-test, which was not pointed out by the code interpreter. In contrast, they could have modeled their data with what is called "linear mixed effects model" where stuff like person (who the laptop belongs to) as well as possibly other stuff like seniority etc could be put into the model as "random effects".
Nevertheless it is all quite interesting data. What I found most interesting is the RAM-related part: caching data can be very powerful, and higher RAM brings more benefits than people usually realise. Any laptop (or at least macbook) with more RAM than it usually needs has most of the time its extra RAM filled by cache.
Also, they don't care since any incremental savings aren't shared with the employees. Misaligned incentives. In that mentally, it's best to take while you can.
* Repo started at this commit
* With this diff applied
* Build was run with this command
Capture that for a week. Now you have a cross section of real workloads, but you can repeat the builds on each hardware tier (and even new hardware down the road)
* They drew beautiful graphs!
* They used chatgpt to automate their analysis super-fast!
* ChatGPT punched out a reasonably sensible t test!
But:
* They had variation across memory and chip type, but they never thought of using a linear regression.
* They drew histograms, which are hard to compare. They could have supplemented them with simple means and error bars. (Or used cumulative distribution functions, where you can see if they overlap or one is shifted.)
Most of the time they deal with data the way their tools generally present data, which correlate closely to most analytics, perf analysis and observability software suites.
Expecting the average software eng to know what a CDF is the same as expecting them to know 3d graphics basics like quaternions and writing shaders.
Does your experience truly say that the average SWE is so ignorant? If so, why do you think that is?
> graphics is a very common and popular elective
I find these statements to be extremely doubtful. Why would a CS program cover statistics? Wouldn't that be the math department? If there any required courses, it's most likely Calc 1/2, Linear Algebra, and Discrete Math.
Also, out of the hundreds of programmers I've met, I don't know any that has done graphics programming. I consider that super niche.
I did write shaders and used quaternions back in the day. I also worked on microcontrollers, did some system programming, developed mobile and desktop apps. Now I am working on a rather large microservice based app.
I think the distribution is decidedly non normal here and the difference in the medians may well have also been of substantial interest -- I'd go for a Wilcox test here to first order... Or even some type of quantile regression. Honestly the famous Jonckheere–Terpstra test for ordered medians would be _perfect_ for this bit of pseudoanalysis -- have the hypothesis that M3 > M2 > M1 and you're good to go, right?!
(Disclaimers apply!)
Note that in some places they used boxplots, which offer clearer comparisons. It would have been more effective to present all the data using boxplots.
Like you, I'd suggest empirical CDF plots for comparisons like these. Each distribution results in a curve, and the curves can be plotted together on the same graph for easy comparison. As an example, see the final plot on this page:
Most people hates nuances when reading data report.
Why would this be preferred over a PDF? I've rarely seen CDF plots after high school so I would have to convert the CDF into a PDF inside my head to check if the two distributions overlap or are shifted. CDFs are not a native representation for most people
Ideally, of all components are the same, there is no jitter, and if you feed in a test signal from a generator with exactly the same area per pulse, you should see a histogram where every count is in a single bin.
In real life, components have tolerances, and readouts have jitter, so the counts spread out and you might see, with the same input, one device with, say, 100 counts in bin 60, while a comparably performing device might have 33 each in bins 58, 59, and 60.
This can be hard to compare visually as a PDF, but if you compare CDF's, you see S-curves with rising edges that only differ slightly in slope and position, making the test more intuitive.
A word of warning from personal experience:
I am part of a medium-sized software company (2k employees). A few years ago, we wanted to improve dev productivity. Instead of going with new laptops, we decided to explore offloading the dev stack over to AWS boxes.
This turned out to be a multi-year project with a whole team of devs (~4) working on it full-time.
In hindsight, the tradeoff wasn't worth it. It's still way too difficult to remap a fully-local dev experience with one that's running in the cloud.
So yeah, upgrade your laptops instead.
Code sits on our laptops but live syncs to the remote services without requiring a Docker build or K8s deploy. It really does feel like local.
In particular it lets us do away with the commit-push-pray cycle because we can run integ tests and beyond as we code as opposed to waiting for CI.
We use Garden, (https://docs.garden.io) for this. (And yes I am afilliated :)).
But whether you use Garden or not, leveraging the power of the cloud for “inner loop” dev can be pretty amazing with right tooling.
I wrote a bit more about our experience here: https://thenewstack.io/one-year-of-remote-kubernetes-develop...
Similarly if another dev (or a CI runner) triggers a build of one of our services, I won’t have to next time I start my dev environment. And because it’s built in the cloud there’s no “works on my machine”.
Same applies to tests actually. They run in the cloud in an independent and trusted environment and the results are cached and stored centrally.
Garden knows all the files and config that belong to a given test suite. So the very first CI run may run tests for service A, service B, and service C. I then write code that only changes service B, open a PR and only the relevant tests get run in CI.
And because it’s all in prod-like environments, I can run integ and e2e tests from my laptop as I code, instead of only having that set up for CI.
Many CI systems would spin up a new box instead of using persistent so likely have to rebuild if no cache, etc.
So basically I would say most of the overhead is in not having a persistent box with knowledge of last build or ability to choose what to run in there, which pretty much just equals to local capabilities.
Admittedly, documentation and stability weren’t quite what we’d like and we’ve done a massive overhaul of the foundational pieces in the past 12 months.
If you want to share feedback I’m all ears, my email is in my profile.
There was an internal search provider (bunnylol) that had tools like putting @od in front of any FB URL to generate a redirect of that URL to your currently checked out On Demand server. Painless to work with! Nice side benefit of living on the same domain as the main sites is that the cookies are reused, so no need to log in again.
I work for a not-really-tech company (and I'm not a full-time dev either), so I've been issued a crappy "ultra-portable" laptop with an ultra-low-voltage CPU. I've looked into offloading my dev work to an AWS instance, but was quite surprised that it wasn't any faster than doing things locally for things like Rust compiles.
I’ve been integrating psrecord into our builds to track core utilisation during the built and see that a lot of time is spent in single threaded activities. Effort is required to compile modules in parallel but that is actually quite straightforward. Running all tests in parallel is harder.
We get the most out of the cloud machines by being able to provision a 16+ core machine to run more complicated (resilience) tests and benchmarks.
Also note that typically the cloud machines run on lower clocked CPUs than you would find in a workstation depending on which machine you provision.
However, I also feel like our setup was well suited to remote-first dev anyway (eg. syncing of auto-generated files being a pain for local dev).
When you need to have 200+ parts running to do anything, it can be hard to work in a single piece that touches a couple others.
With servers that have upwards of 128+ cores and 256+ threads, my opinion is swinging back in favor of monoliths for most software.
It turns out that hooking every system call with vendor crapware is bad for a unix-style toolchain that execs a million subprocesses.
At large tech companies like Google, Meta, etc the dev environment is entirely in the cloud for the vast majority of SWEs.
This is a much nicer dev experience than anything local.
This is honestly a bit overkill for a dev workstation (unless you compile Rust!), but since it’s a dedicated server it can also host any number of fully isolated services for homelab or saas. There’s nothing else like it in the wild, afaik.
We have interns coming in and fully ready within an hour or two of setup. Same way changing local machines is a breeze with very little downtime.
As you point out rebuilding small test datasets instead of just filtering the prod DB is an option, but those also need maintenance, and take a hell of time to make sure all the relevant cases are covered.
Basically, trying to flee from the bulk and complexity tends to bring a different set of hurdles and missing parts that have to be paid in time, maintenance and bugs only discovered in prod.
PS: the test DB is still reset everyday. Eorse thing happening is we need to do something else for a few hours until it's restored.
This sounds like the result of a company investing in tooling, rather than something specific to a remote dev env. Our local dev env takes 3 commands and less than 3 hours to go from a new laptop to a fully working dev env.
There’s moments where you try to do a thing that normal on a local PC and it’s impossible on remote. That cognitive dissonance is the worst.
While my team (platform & infra) much prefer remote devbox, the development teams are not.
It could be specific to my org because we have way too many restrictions on the local dev machine (eg: no linux on laptop but it's ok on server and my team much prefer linux over crippled Windows laptop).
Can’t really say anything for performance, but I don’t think it’ll beat my laptop unless maven can magically take decent advantage of 32 cores (which I unfortunately know it can’t).
"spend X time to automate this task vs not do the task at all".
"Spend X time to automate this task that takes Y×n time normally and get it down to Z×n time, vs spend Y×n time to do the task"Also not all minutes have equal value, spending a few hours to save 5 minutes in an emergency with automation can be well worth it.
One thing that jumps out at me is the assumption that compile time implies wasted time. The linked Martin Fowler article provides justification for this, saying that longer feedback loops provide an opportunity to get distracted or leave a flow state while ex. checking email or getting coffee. The thing is, you don't have to go work on a completely unrelated task. The code is still in front of you and you can still be thinking about it, realizing there's yet another corner case you need to write a test for. Maybe you're not getting instant gratification, but surely a 2-minute compile time doesn't imply 2 whole minutes of wasted time.
I really struggle with task switching, and two minutes is the danger zone. Just enough time to get distracted, by something else; too little time to start meaningful work on anything else...
Hour long compiles are okay, I plan them, and have something else to do while they are building.
30 second compiles are annoying, but don't affect my productivity much (except when doing minor tweaks to UI or copywriting).
2-10 minute compiles are the worst.
More time working doesn't translate to being more effective and more productive. If that were the case then why are a disproportionate percentage of my "Oh shit! I know what to do to solve that..." in the shower, on my morning run, etc.?
- M2 Pro is nice, but the improvement over 10 core (8 perf cores) M1 Pro is not that large (136 vs 120 s in Xcode benchmark: https://github.com/devMEremenko/XcodeBenchmark)
- M3 Pro is nerfed (only 6 perf cores) to better distinguish and sell M3 Max, basically on par with M2 Pro
So, in the end, I got a slightly used 10 core M1 Pro and am very happy, having spent less than half of what the base M3 Pro would cost, and got 85% of its power (and also, considering that you generally need to have at least 33 to 50 % faster CPU to even notice the difference :)).
That said, while the transistor count of the M2 Pro -> M3 Pro did decrease, it went up quite a bit from the M2 -> M3.
It seems most likely Apple is just looking to differentiate the tiers.
Die shots show the CPU cores take up so little space compared to GPUs on both the Pro and Max... I wonder why.
I could depend on the language or project, but in head-to-head benchmarks of identical compile commands I didn’t see any differences this big.
I guess not that surprising given the different compilation toolchains though, especially as even with the Go toolchain you can see how specific specs lend themselves to different parts of the build process (such as the additional memory helping linker performance).
You're not the only one to comment that the M3 is weirdly capped for performance. Hopefully not something they'll continue into the M4+ models.
Yep, there appears to be no reason for getting M3 Pro instead of M2 Pro, but my guess is that after this (unfortunate) adjustment, they got the separation they wanted (a clear hierarchy of Max > Pro > base chip for both CPU and GPU power), and can then improve all three chips by a similar amount in the future generations.
There is if you care about efficiency / battery life.
I am never bothered with build times. There is "interactive build" (incremental builds I use to rerun related unit tests as I work on code) and non-interactive build (one I launch and go get coffee/read email). I have never seen hardware refresh toggle non-interactive into interactive.
My personal hardware (that I use now and then to do some quick fix/code review) is 5+ year old Intel i7 with 16Gb of memory (had to add 16Gb when realized linking Node.js in WSL requires more memory).
My work laptop is Intel MacBook Pro with a touch bar. I do not think it has any impact on my productivity. What matters is the screen size and quality (e.g. resolution, contrast and sharpness) and storage speed. Build system (e.g. speed of incremental builds and support for distributed builds) has more impact than any CPU advances. I use Bazel for my personal projects.
Sure, release builds with whole program optimization and other fancy compiler techniques can take longer. That's fine. But the regular compile/debug/test loop can still be instant. For legacy reasons compilation in systems languages is unbelievably slow but it doesn't have to be this way.
With tcc the initial compilation of hostapd it takes about 0.7 seconds and incremental builds are roughly 50 milliseconds.
The only problem is that tcc's diagnostics aren't the best and sometimes there are mild compatibility issues (usually it is enough to tweak CFLAGS or add some macro definition)
1. Debug compilation was split in shared libraries so only a couple of them has to be rebuilt in your regular dev workflow. 2. They had some magical distributed build that "just worked" for me. I never had to dive into the details.
I was working on DevTools so in many cases my changes would touch both browser and renderer. Unit testing was helpful.
If you would not significantly benefit from upgrading, it's only because you already have more CPU performance than you need. Today's CPUs are significantly better than first-generation Zen in performance per clock and raw clock speed, and mainstream consumer desktop platforms can now match the top first-generation Threadripper in CPU core count and total DRAM bandwidth (and soon, DRAM capacity). There's no performance or power metric by which a Threadripper 1950X (not quite 6.5 years old) beats a Ryzen 7950X. And the 7950X also comes in a mobile package that only sacrifices a bit of performance (to fit into fairly chunky "laptops").
If I had 100% CPU consumption around the clock, I would upgrade in a heart beat. But I’m working interactively in spurts between hitting CPU walls and the spurts don’t justify the upgrade.
If I were to upgrade it would be for the sake of non-work CPU video encoding or to get PCIe 5.0 for faster model loading to GPU VRAM.
- I think it is much easier to just load the data into R, Stata etc and interrogate the data that way. The commands to do that will be shorter and more precise and most importantly more reproducible.
- the most difficult task in data analysis is understanding the data and the mechanisms that have generated it. For that you will need a causal model of the problem domain. Not sure that AI is capable of building useful causal models unless they were somehow first trained using other data from the domain.
- it is impossible to reasonably interpret the data without reference to that model. I wonder if current AI models are capable of doing that, e.g., can they detect confounding or oversized influence of outliers or interesting effect modifiers.
Perhaps someone who knows more than I do on the state of current technology can provide a better assessment of where we are in this effort
Except when I did it, it was python and pandas. You can ask it to show you the code it used to do it's analysis.
So you can load the data into R/Python and google "how do I do xyzzzy" and write the code yourself, or use ChatGPT.
This to me is the biggest accomplishment. I've never worked at a company (besides brief time helping out with some startups) where I have been able to run a dev/local instance of the whole company on a single machine.
There's always this thing, or that, or the other that is not accessible. There's always a gotcha.
People who haven’t lived in that world just cannot understand how much better it is, and will come up with all kinds of cope.
I've heard of largeish companies that still manage to do this well, but I'd love to learn how.
That said, yeah I agree this is the biggest accomplishment. Getting dev cycles down from hours or days to minutes is more important than getting them down from minutes to 25% fewer minutes.
https://news.ycombinator.com/item?id=38816135
Like if you have logic apps and azure Data pipelines, how do you create and more importantly keep current the local development equivalents for those?
I'm not saying if you are YouTube that all the videos on YouTube must fit on a developer's local machine but would be nice if you could run the whole instance locally or if not, at least be able to reproduce the whole set up on a different environment without six months worth of back and forth emails.
Lots of stuff in this from profiling Go compilations, building a hot-reloader, using AI to analyse the build dataset, etc.
We concluded that it was worth upgrading the M1s to an M3 Pro (the max didn’t make much of a difference in our tests) but the M2s are pretty close to the M3s, so not (for us) worth upgrading.
Happy to answer any questions if people have them.
Thanks for the detailed analysis. I’m wondering if you factored in the cost of engineering time invested in this analysis, and how that affects the payback time (if at all).
Thanks!
The first day was spent hacking together a new hot reloaded but this also fixed a lot of issues we’d had with the previous loader such as restarting into stale code, which was really harming people’s productivity. That was well worth even several days of effort really!
The second day I was just messing around with OpenAI to figure out how I’d do this analysis. We’re right now building an AI assistant for our actual product so you can ask it “how many times did I get paged last year? How many were out-of-hours? Is my incident workload increasing?” Etc and I wanted an excuse to learn the tech so I could better understand that feature. So for me, well worth investing a day to learn.
Then the article itself took about 4hrs to write up. That’s worth it for us given exposure for our brand and the way it benefits us for hiring/etc.
We trust the team to make good use of their team and allowing people to do this type of work if they think it’s valuable is just an example of that. Assuming I have a £1k/day rate (I do not) we’re still only in for £2.5k, so less than a single MacBook to turn this around.
Or the tasks maybe finished so fast that it didn’t make a difference in real world usage?
Logistical question: did management move some deliverables out of the way to give you room to do this? Or was it extra curricular?
I suspect there might be considerable difference in developer behavior which results in these differences. Such as people with different types of laptops typically working on different things.
And a few random observations after a very cursory reading (I might be missing something):
- Go compiler seems to take little advantage from additional cores
- They are pooling data in ways that makes me fundamentally uncomfortable
- They are not consistent in their comparisons, sometimes they use histograms, sometimes they use binned density plots (with different y axis ranges), it's real unclear what is going on here...
- Macs do not throttle CPU performance on battery. If the builds are really slower on battery (which I am not convinced about btw looking at graphs), it will be because of "low power" setting activated
M3s have a smaller memory bandwidth, they are effectively a downgrade for some use cases.
The Rust compiler routinely hits the 45-50Gb/sec ballpark of the intra-memory transfer speed on compiling a medium sized project, more if the code base large. Haskell (granted, a fringe yet revealing case) case just as routinely hits the 60-70 Gb/sec memory transfer speed at the compile time, and large to very large C++ codebases add a lot of stress on the memory at the optimisation step. If I am not mistaken, Go is also very memory bound.
Then there comes the linking and particularly the LTO that want all the memory bandwidth they can get to get the job done quickly, and the memory speed becomes a major bottleneck. Loading the entire codebase into memory, in fact, the major optimisation technique used in mold[0] that can vastly benefit from a) faster memory, b) a wider memory bus.
Regarding mold… maybe it can indeed saturate 150GB/s on a 6-core laptop. But they were not using mold. Also, the timing differences we observe here are larger than what would be expected with a linker like mold with a 25% reduction in bandwidth. I mean mold can link clang in under 3 seconds. Reducing bandwidth would increase this by a second at most. We see much larger variation in M2 vs. M3 results here.
The company I work for is now running 5+ background services on their developer laptops, both Mac and Windows. Endpoint management, priviledge escalation interception, TLS interception and inspection, anti-malware, and VPN clients.
This combination heavily impacts performance. You can see these services chewing up CPU and I/O performance while doing anything on the machines, and developers have complained about random lockups and hitches.
I understand security is necessary, especially with the increase in things like ransomware and IP theft, but have other companies found better ways to provide this security without impacting developer productivity as much?
> have other companies found better ways to provide this security without impacting developer productivity as much?
I don't see this at all... the peak for all 3 is at right under 20s. The long tail (i.e. infrequently) goes up to 2m, but for all 3. M2 looks slightly better than M1, but it's not clear to me there's an improvement from M2 to M3 at all from this data.
Performance per watt and rendering performance are both better in the M3, but I ultimately decided to wait for an M3 Ultra with more memory bandwidth before upgrading my daily driver M1 Max.
I came away feeling that:
- M1 is a solid baseline
- M2 improves performance by about 60% - M3 Pro is marginal on the M2, more like 10%
- M3 Max (for our use case) didn’t seem that much different on the M3 Pro, though we had less data on this than other models
I suspect Apple saw the M3 Pro as “maintain performance and improve efficiency” which is consistent with the reduction in P-cores from the M2.
The bit I’m interested about is that you say the M3 Pro is only a bit better than the M2 at LLM work, as I’d assumed there were improvements in the AI processing hardware between the M2 and M3. Not that we tested that, but I would’ve guessed it.
On LLMs, the issue is largely that memory bandwidth: M2 Ultra is 800GB/s, M3 Max is 400GB/s. Inference on larger models are simple math on what's in memory, so the performance is roughly double. Probably perf / watt suffers a little, but when you're trying to chew through 128GB of RAM and do math on all of it, you're generally maxing your thermal budget.
Also, note that it's absolutely incredible how cheap it is to run a model on an M2 Ultra vs an H100 -- Apple's integrated system memory makes a lot possible at much lower price points.
This makes a load of sense, thanks for explaining.
M2 Ultra at 800 GB/s is for the mac studio only. So it's not quite apples to apples when comparing against the M3 which is currently only offered for macbooks.
M2 Max has bandwidth at 400 GB/s. This is a better comparison to the current M3 macbook line. I believe it tops out at 96GB of memory.
M3 Max has a bandwidth of either 300 GB/s or 400 GB/s depending on the cpu/gpu you choose. There is a lower line cpu/gpu w/ a max memory size of 96GB, this has a bandwidth of 300 GB/s. There is a top of the line cpu/gpu with a max memory size of 128GB, this has the same bandwidth as the previous M2 chip at 400 GB/s.
The different bandwidths depending on the M3 max configuration chosen has led to a lot of confusion on this topic, and some criticism for the complexity of trade offs for the most recent generation of macbook (number of efficiency/performance cores being another source of criticism).
Sorry if this was already clear to you, just thought it might be helpful to you or others reading the thread who have had similar questions :)
This is the most shocking part of the article for me since the difference between M1 and M2 build times has been more marginal in my experience.
Are you sure the people with M1 and M2 machines were really doing similar work (and builds)? Is there a possibility that the non-random assignment of laptops (employees received M1, M2, or M3 based on when they were hired) is showing up in the results as different cohorts aren’t working on identical problems?
I took a selection of builds that were triggered by the same code module (one that frequently changes to provide enough data) and compared models on just that, finding the same results.
This feels as close as you could get for an apples-to-apples comparison, so I'm quite confident these figures are (within statistical bounds of the dataset) correct!
I really like my laptop. Spend a lot of time typing into it. It's limited to a 30W or similar power budget on thermal and battery constraints. Some of that is spent on a network chip which grants access to machines with much higher power and thermal budgets.
Current employer has really scary hardware behind a VPN to run code on. Previous one ran a machine room with lots of servers. Both expected engineer laptops to be mostly thin clients. That seems obviously the right answer to me.
Thus marginally faster dev laptops don't seem very exciting.
It's quite expensive to set up, regardless of whether we're talking about on-prem or cloud hardware. Your employer is already going to buy you a laptop; why not try to eke out what's possible from the laptop first?
The typical progression, I would think, is (a) laptop only, (b) compilation times get longer -> invest in a couple build cache servers (e.g. Bazel) to support dozens/hundreds of developers, (c) expand the build cache server installation to provide developer environments as well
Bringing in Windows or Linux has a set up cost and a maintenance cost that may exclude it from even being considered.
Edit: plus, Macs are ARM, other options are inevitably x86. So it’s also two CPU architectures to maintain support for, on top of OS specific quirks - and even if you use eg Docker, you still have a lot of OS specific quirks in play :/
Even in an ideal scenario where your app already works on ARM, you will be dealing with OS-specific quirks unless your production machine runs MacOS.
Eg at work we use M1/M2 macs and dev on those using docker - so that’s a Linux VM essentially with some nice tooling wrapped around it.
We certainly see differences - mostly around permissions (as docker for Mac doesn’t really enforce any access checks on files on the host), but we also mostly deploy to ARM Linux on AWS.
We went Mac only from a mix of Linux, Windows and Mac as we found the least overall friction there for our developers - Windows, even with WSL, had lots of problems, including performance issues. Linux we had issues finding nice laptops, and more support issues (developers are often not *nix experts!). Mac was a nice middle ground in the end.
This is the same issue as before. Laptops are shiny so people don’t even bother considering a regular desktop machine. And yet desktops can be so much more powerful simply because they don’t have the thermal and power delivery restrictions that desktops have.
From the article:
> All incident.io developers are given a MacBook which they use for their development work.
Non-MacBook machines are apparently not an option, for whatever reason. Comparing against other machines would be interesting, but irrelevant.
> Exporting your data to a CSV
> Create an ‘Assistant’ with a prompt explaining your purpose, and provide it the CSV file with your data.
Once MSFT build's the aforementioned process into Excel, it's going to be a major game changer.
*EDIT* I guess if you needed a macOS specific build with Go you would us macOS but I would have thought you'd use Linux too. Can you build a Go project in Linux and have it run on macOS? I suppose architecture would be an issue building on Linux x86 would not run on macOS Apple Silicon but the reverse is true too a build on Apple Silicon would not work on Linux x86 maybe not even Linux Arm.
With that out of the way, I maintain a very large site with over 50,000 posts in a static site setup. It is built on a Phoenix/Elixir setup I wrote from scratch and has been serving my client well without hiccups for the last 5 years. From time to time, some of the writers may mess up something and they would want us to run a sweep occasionally - back track to the last X number of posts and re-publish them. Usually about 100-500 posts which covers a few weeks/months worth of posts depending on the run.
So, for a 100 post sweep, the M3 pro was slightly faster, by an order of magnitude of over 1.5x. But, for other everyday tasks, like say opening Affinity Designer/Photo and editing files, I didn't notice much improvements. And Apple's website is notoriously deceptive about the improvements. Their graphs always showcase comparisons with a top specc'ed M3 machine with a base M1 variant or M2 which makes no sense to me as a developer. Disk (SSD) speeds were slightly slower on the M3 than on my old M1 variant. But, I am unable to attribute it to the processor as it could be a bad driver or software version. For those interested, it is one of those internal Samsung SSDs connected via a SATA-USBC converter.
Long story short, if you are on M2 - Not worth upgrading. If you are on an M1 or M1 Pro, you can still hold on to it a bit longer, the only reason you would want to upgrade is if you got a good deal like me. I sold my 21' M1 Pro for almost $1800 US (it still had Apple care) and got a student discount (I'm in the middle of some certification programs) on the new M3 Pro, so I got it. If I didn't have these financial incentives, probably wouldn't have been worth upgrading. Hope this helps someone.
One thing that really stood out to me was:
> People with the M1 laptops are frequently waiting almost 2m for their builds to complete.
And yet looking at the graph, 120s is off the scale on the right side, so that suggests almost literally no one is waiting 2m for a build, and most builds are happening within 40s with a long tail out to 1m50s.
I don't use VSCode but most of my team do and I frequently pair with them. Never noticed it to be anything other than very snappy. They all have M1s or up (I am the author of this post, so the detail about their hardware is in the link).
It's a bummer because one of them is also a 2018 fully loaded and I would have a hard time even selling it to someone because of how much better the M2/M3 is. It's wild when I see people building hackintoshes on like a Thinkpad T480 ... its like riding a pennyfarthing bicycle versus a ducati.
My M2 Air is my favorite laptop of all time. Keyboard is finally back to being epic (esp compared to 2018 era, which I had to replace myself and that was NOT fun). It has no fan so it never makes noise. I rarely plug it in for AC power. I can hack almost all day on it (using remote SSH vscode to my beefy workstation) without plugging in. The other night I worked for 4 hours straight refactoring a ton of vue components and it went from 100% battery to 91% battery.
I suspect one of the reasons why Apple silicon looks so good is the previous generations were at a dip of performance. Maybe they took the foot off the gas WRT updates as they knew the M series of chips was coming soon?
I do regularly try VSCode to see what I’m missing.
VS Code has great remote editing that I do use.
However, on my M1 Pro 16”, VS Code is noticeable laggier than sublime Text!
Just clicking on a file in the side bar and waiting for text to appear has lag. Many non-nerdy people might think it’s instant, but I can see the lag clearly! LOL
For my tastes the VS Code UI is cluttered and generally feels slow. The extensions are great but also a nightmare of updates and alerts of things broken.
If it’s your thing, you can use CoPilot in Sublime Text through a plugin that actually works really well!
I’m on the fence about CoPilots benefits though.
Sometimes I’ll flick over to VS Code to use the chat copilot if I need an answer for an API call etc….
If I had to stick with one, I’m sticking with Sublime Text.
On my M2 Max, all of that is ~fully resolved. There is still some slight lag, and I have to figure it’s just the Electron tax, but never enough to really bother me, certainly not enough to defer restarting anything. And I can count the times I’ve even heard the fans on one hand… and even so, never for more than a few seconds (though each time has been a little alarming, just because it’s now so rare).
I'm a professional and I make a living writing software. Investing ~4k every 3-4 years on fast computer is a no brainer to me.
I couldn't believe it.
Absolutely get an apple silicon machine, no contest the best hardware on the market right now.
Chiefly, I think the problem is that the CTO solved the wrong problem: the right problem to solve includes a combination of assessing why company public opinion is generating mass movements of people wanting a new MacBook literally every year, if this is even worth responding to at all (it isn't), and keeping employees happy.
Most employees are reasonsble enough to not be bothered if they don't get a new MacBook every year.
Employers should already be addressing outdated equipment concerns.
Wasting developer time on a problem that is easily solvable in one minute isn't worthwhile. You upgrade the people 2-3 real generations behind. That should already have been in the pipeline, resources notwhistanding.
I just dislike this whole exercise because it feels like a perfect storm of technocratic performativity, short sighted "metric" based management, rash consumerism, etc.
If it’s useful: Pete wasn’t really being combative with me on this. I suggested we should check if the M3 really was faster so we could upgrade if it was, we agreed and then I did the analysis. The game aspect of this was more for a bit of fun in the article than how things actually work.
And in terms of why we didn’t have a process for this: the company itself is about two years old, so this was the first hardware refresh we’d ever needed to schedule. So we haven’t a formal process in place yet and probably won’t until the next one either!
I also worked at a company whose CIO put out an email noting how "amazing" a coincidence it was just how many company iPhones got broken/water damaged/lost or stolen in the 2 months following a new iPhone announcement event.
If you compare a jump between Intel and M1 and then numbers for newer generation I think they validate the desire to have a newer machine. It's not about having a new shiny object, but a tool that improves work. You only have one life and those seconds and minutes wasted on waiting on things add up. This is especially important for neurodivergent people, with conditions like ADD. Even waiting few seconds less for something to finish can have an impact - it may get the result just before one completely moved their train on thought on different tracks. I wouldn't like to work for an employer with such a patronising and nonchalant view of equipment. Your comment is actually toxic.
but we were all young once, I remember thinking the only thing holding me back was 4.77MHz
Being fast doesn't mean you do everything at maximum speed. Usain Bolt doesn't sprint everywhere he goes, but he can sprint quite fast when he needs to do so.
How can I tell how much I would benefit from a memory bump?
5 min on my m3 mbp pro.
Geekbench score differences were quite remarkable.
I am still wondering if I should return it, though
We’re super happy with 30s as an average build time right now. Assume we spent about £50k for the entire teams laptops, how much time would an engineer need to break our monolith into microservices? How much extra investment would we need to make it work as well as our monolith, or debug production issues that are now much more complex?
I’d wager it’s much more than half an engineers time continually, at which point you’ve blown the money you saved on a solution that works far less well for dev productivity and debugging simplicity in production.
5950X, 64GB of RAM, Windows 11 + WSL.
An AMD based ThinkBook for travel.
Also would it hurt the author if they had kept the same max y-value on all plots?
The best way (imo) to measure this is setting a threshold that we agree is bad (in our case, this is 1m+ builds) and continually measuring it so we catch when that starts to happen more frequently. And listening to devs when they say things have got slow/have become irritating.
https://pixelesque.net/blog/2022/08/basic-apple-m1-pro-cpu-b...
I wonder what the difference is... Maybe my tests were too short to show the difference between mains and battery?
Edit: 310fps vs 240fps
If you have a massive, monolithic, single-executable-producing codebase that can't be built on a developer machine, then you need network builds. But if you aren't Google, building on laptops gives developers better experience, even if it's slower.
I benchmarked several test suites to compare the machines. The test suites for two different Rails applications were around 10% faster on the M3 Max. A test suite for a Go web application that does significant integration testing in parallel (i.e. multiple complete stacks of database, application server, and browser) improved by around 20%. The extra P-cores help. Miscellaneous other tests and benchmarks were around 7-10% faster.
Didn't notice any difference in normal usage.
So not likely worth the upgrade unless your workload can really make use of the extra cores.
Cinebench and Geekbench tell a different story. M2 is a small upgrade over M1 while M3 is a bigger upgrade over M2.
It's worth noting they're using the binned Max and quite frankly are undercutting the actual results the Max produces, it appears to be substantial.
https://incident.io/_next/image?url=https%3A%2F%2Fcdn.sanity...
2. If you are doing traditional heavyweight software development, or are concerned with perception in an interview, promotional context or just impressing others at a coffee shop, get a 32GB 16” MBP system with as large a built-in SSD as you can afford (it gets cheaper per GB as you buy more) and go for an M2 Pro SoC, which is faster in many respects than an M3 Pro due to core count and memory bandwidth. Full Stop. (You could instead go 64GB on an M1 Max if you keep several VMs open, which isn’t really a thing anymore (use VPS), or if you are keeping a 7-15B parameter LLM open (locally) for some reason, but again, if you are doing much with local LLMs, as opposed to being always connectable to the 1.3T+ parameter hosted ChatGPT, then you should have stopped at #1.)
3. If you are nursing mature apps along, maybe even adding ML, adjusting UX, creating forks to test new features, etc.. then your concern is with INCREMENTAL COMPILATION and the much bigger systems like M3 Max will be slower (bc they need time to ramp up multiple cores and that’s not happening with bursty incremental builds), so might as well go for a 16GB M1 MBA (add stickers or whatever if you’re ashamed of looking like a school kid) and maybe invest the savings in a nice monitor like the 28” LG DualUp (bearing in mind you can only use a single native-speed external monitor on non-Pro/Max SoCs at a time). (In many cases, can even use an 8GB M1 MBA for incremental builds because, after loading the project, the MacOS memory compressor is really good and the SSD is really fast and you can use a real device instead of a Simulator. But do you want any M2 MBA? No, it has inferior thermals, is heavier, larger, fingerprints easy, lack’s respect and the price performance doesn’t make sense given the other options. Same goes for 13” M1/M2 Pro and all M3 Pro.)
Also, make sure you keep hourly (or better) backups on all Apple laptops WHILE CODING. There is a common failure scenario where the buck converter that drops voltage for the SSD fails, sending 13VDC into the SSD for long enough to permanently destroy the data on it. https://youtu.be/F6d58HIe01A
I thought that general consensus was that 8GB Macs were hammering the life of the SSDs? Yeah, they're fast, but people were talking about dozens of GB a day of swapping happening. And these aren't enterprise class SSDs, despite what Apple charges for them.
Is there some microbenchmark that illustrates the problem, or is this just wild extrapolation from some deep misunderstanding (maybe a bad car analogy about turbo lag?)
- 1 day for hot-reload (I don’t think this should count tbh)
- 1 day for messing with data
- 0.5 day for write-up
Assuming my day rate is £1k/day (it is not) we land at £1.5k, less than a laptop and less than I’d pay for the impressions I got from my post sharing this on LinkedIn, so feels worthwhile!
It's pretty cool how it works, too: the OpenAI Assistant uses the LLM to take your human instructions like "how many builds is in the dataset?" and translate that into Python code which is run in a sandbox on OpenAI compute with access to the dataset you've uploaded.
Under the hood everything is just numpy, pandas and gnuplot, you're just using a human interface to a Python interpreter.
We've been building an AI feature into our product recently that behaves like this and it's crazy how good it can get. I've done a lot of data analysis in my past and using these tools blew me away, it's so much easier to jump into complex analysis without tedious setup.
And a tip I figured out halfway through: if you want to, you can ask the chat for an iPython notebook of it's calculations. So you can 'disable autopilot' and jump into manual if you ever want finer control over the analysis it runs. Pretty wild.
Never had even remotely similar experience to Intel laptops.
I am hoping that companies that are getting rid of their M1 stock, put them on a second hand market rather than through disposal service (where they get physically destroyed).
Made me very curious for my own stack
In my experience the difference of low power mode is bigger when doing something else CPU intensive such as video calling.
Is anyone else doing this or why would anyone else do this?
This means devs have less system memory available for their host because it’s in the VM, battery is often harmed because the VM box is constantly churning, and as we run our Postgres database in docker the FS latency makes for a noticeably slower dev environment.
Docker is great for standardising the dev env but honestly, running Postgres locally isn’t hard, and we don’t have many other deps. If the quality of experience is better at the cost of the occasional “let me help you out with this pg_upgrade” then I’d consider that a win.
I guess M1 laptops are here to stay until M5 laptops come out.
The distributed build systems only really benefit from aggressively caching the modules that are built, right? But the majority of the builds we do are almost fully cached, having changed just one module that needs recompiling then the linker sticks everything back together, which the machines would then need to download from the distributed builder and at 300MB a binary that’s gonna take a while.
I may have this totally wrong though. Would distributed builds actually get us a new binary faster to the local machine?
I suspect we wouldn’t want this anyway (lots of our company work on the go, train WiFi wouldn’t cut it for this!) but interested nonetheless.
While I had no power over changing the laptops, I was co-administrator of the dev datacenter located 20 meters away and we had our own budget for it. Long story short, that dev datacenter soon had a new, very beefy server dedicated for CI jobs "and extras".
One of said extras was providing Docker containers to the team for running the product during development, which also happened to be perfectly suitable for remote development.
When I open the page.
Edit: Nvm didnt see the quantity 10
Moreover, if your whole stack (plus your test suite) doesn't fit in memory, what's the point of buying an extremely expensive laptop? Not to mention constantly replacing them just because a newer, shinier model is released? If you're just going to test one small service, that shouldn't require the fastest MacBook.
To test an entire product suite - especially one that has high demands on CPU and RAM, and a large test suite - it's much more efficient and cost effective to have a small set of remote servers to run everything on. It's also great for keeping dev and prod in parity.
Businesses buy MacBooks not because they're necessary, but because developers just want shiny toys. They're status symbols.