Hacker News Reader

Comment by danvittegleo | Hacker News Reader

CalyxOS is an awesome project. I have worked with the lead developer a bit over the past few years and it's been such a pleasure. We share some bits of code between our projects here: https://github.com/AOSPAlliance.

If anyone is interested in building their own custom android OS in the cloud (AWS) with same ability to lock your bootloader like CalyxOS, you can checkout my project I've been maintaining for a few years now called RattlesnakeOS: https://github.com/dan-v/rattlesnakeos-stack.

And if you prefer to not build in the cloud, there is also a really great project called robotnix (https://github.com/danielfullmer/robotnix) which provides a way to build many flavors of OS (AOSP, GrapheneOS, LineageOS, etc).

View on HN

Could you explain why you would build in the cloud? Based on a sibling comment, it sounds like it might be because it’s crazy resource-intensive? I’ve honestly not heard of cloud building before. Is it common for large projects like operating systems?

Yes, building AOSP requires a fairly powerful machine (at least to do it quickly): https://source.android.com/setup/build/requirements. It's definitely possible to do on a local machine with decent specs though.

By my reading, my not-really-a-gaming-desktop could do it in 3 hours, that doesn't seem bad at all.

Now granted, those were heavyweight specs when Android came out in 2007, but I'd figure about half of us probably have a similar box sitting around today, and the other half would just need to beef one up with some additional RAM.

I haven't used a desktop since 2010 and my laptop is old. Building AOSP is the sort of huge compute-task that sticks out with its rarity.

How expensive is it to build android in the cloud? And how are the build times? Is it possible to do it on a local Mac mini instead or will that take too long to build?

From memory, I think LineageOS 17 took roughly 8-12 hours for an initial build and 3.5 hours for subsequent (ccache) builds on an Intel i5-3570K and spinning hard drive. That's not including the initial git clone.

The idea might seem daunting, but assuming midrange hardware and a decent net connection, it's very much doable in under a day without resorting to cloud services.

How would it scale with the number of cores? 3950x should make relatively short work of it, or wouldn't it?

I would expect it to scale pretty well, at least until you reach the limits of your disk and buffer RAM.

The build process supports the -j option just like make. You can use -j N+1 if you want to keep all your cores busy, or -j N-1 to keep your machine more responsive during the build, or nice and -j 1 if you're in no hurry and your machine has more important tasks. (Actually, I think reasonable defaults for these might already be part of the build scripts, but it has been a while since I looked.)

It scales amazingly well with the number of cores you throw at it. Definitely the 3950x would work great.

that should tear through it. the linux kernel for instance compiles much better with higher number of cores

With rattlesnakeos-stack, it uses spot instances and defaults to a c5.4xlarge which takes about 7-8 hours to build AOSP and Chromium (for an up to date webview) and equates to about ~$1 a build. I typically build on a c5.24xlarge instance which takes about 2.5 hours and costs about ~$2 per build. Unfortunately both AOSP and Chromium are massive projects that require a ton of computing power to build quickly. It's definitely still possible to do on less powerful machines, but it's just going to take a lot longer to do builds.

Is any of the build cached?

In order to limit costs, everything is pulled from source on each build and there is nothing cached. This strategy takes advantage of the fact that AWS doesn't charge for ingress traffic and unfortunately puts additional load on Google's servers. I've attempted a few different strategies on caching AOSP and Chromium source trees, but since you have to incur the storage costs on an ongoing basis, it's just not very economical.

You could certainly do it. No point having multiple EBS volumes lying around - just create a snapshot of the volume with the git checkout / build cache after each new build is done.

When you want to build again, create the instance and then recreate the EBS volume from the snapshot and attach it to the new instance. Pull the latest set of changes from the git repo and build with the old cache!

Obviously there are cache purging considerations (e.g. starting from scratch once per week/month) you could optimise as well.

I investigated EBS snapshot as an option, but there were two problems. 1) cost as i mentioned initially - for just AOSP source tree alone you are looking at > 250GB and at a cost of $0.05 per GB you are already at > $10/month and 2) EBS snapshots lazy load from S3 which gives TERRIBLE performance which means you end up with far far slower builds. AWS released a feature "EBS Fast Snapshot Restore" to workaround this issue, but it's extremely expensive.

Are you doing partial clone?

https://source.android.com/setup/build/downloading

Although, an answer at https://stackoverflow.com/questions/33053615/how-to-download... from October 2020 says that even a partial clone is still 73 GB in size!? That’s insane! How the heck come it’s so big?

That's likely the size of the entire checked out tree, which would include all of the files.

I just ran the command here and my AOSP 11.0 checkout is 54GB, minus any git history, since I clone from a local mirror and use '--reference' to avoid having to copy objects.

A lot of the size here is from the various prebuilts, AOSP build is quite self-contained (jdk, clang, etc) and barely uses anything from the host.

Reply on news.ycombinator.com