Comment by fidgewidge | Hacker News Reader

Comment by fidgewidge | Hacker News Reader

Parent Full thread

Congrats on hitting the front page. As someone perhaps one step away from your target market, can you please explain how the value prop here really works? You charge $49/month for 50GB of disk cache, 32GB of RAM and 16 vCPUs along with a bit of custom tooling on top. For 44 EUR/month I can get a dedicated Hetzner machine with a terabyte of disk, 64 GB of RAM and 6 dedicated cores (12 with SMT), and no user or project limits. Because it has so much more disk and RAM, and because builds tend to be disk/ram/bandwidth limited, performance is probably competitive.

You say that this is meant to solve problems with CI being ephemeral. Maybe I'm old fashioned, but my own CI cluster uses dedicated hardware and nothing is ephemeral. It could also use dedicated VMs and the same thing would apply. We run TeamCity and the agents are persistent, builds run in dedicated workspaces so caching is natural and easy. This doesn't cost very much.

When you add more features then I can see there's some value there (SBOM etc) but then again, surely such features are more easily done by standalone tools rather than having to rent infrastructure from you.

So we have the concept of a "project", which in retrospect isn't the best name and is way too vague. :) But on our end, a "project" equates to one or two cache SSDs + one or two EC2 instances we're running, depending on whether you've asked for a single platform build or an Intel+Arm multi-platform build.

We do charge $0.05 per minute of build time used, but in theory that $49/mo plan gives you access to up to 20 build machines, if you're building 10 projects at once.

That said, if you already have your own dedicated build cluster / CI setup, you may prefer to just use that! Depot is effectively doing that kind of thing for you if you don't already have your own hosted CI system or would prefer not to orchestrate Docker layer cache.

We will be expanding to more things like SBOMs, container signing, insights and analytics about what's happening inside the builds, but hopefully in more integrated ways, since we control the execution environment itself.

As someone who is not a user yet 49 sounds like a rounding error in our dev budget and if all I have to do is change a line in my CI script that does sound very tempting. Standing up another machine and registering it as runner sounds like way more work. Also incurs a recurring task to run updates there or cycle the VM to get a new base image. An hour of dev time is really very expensive

I see. So it's $50/month plus .05 per minute of build time after 60 minutes is up in that month. Let's say you need 10 hours of building per week, so that'd be another $120/month on top. I'm not sure how fast these builds would go with your setup, maybe 10 hours a week is a lot. But we're still talking like $170/month for something with user limits and fairly restricted resources. For less than that I can get a 16 core AMD Rome machine with nearly 8 TB of flash split across two drives, which should eat image builds for breakfast. The extra cost is a bit of Linux sys admin which can be fully automated (apt-get install unattended-upgrades and a little more on first install).

Clearly from other responses in this thread there are people who feel this is a good deal, so best of luck to you. But I'm kinda reminded here of 37signals saying they can save $7M over 5 years by leaving the cloud. It seems the goal here is to dig people out of performance problems they get by using one type of cloud service, by selling them another type of cloud service!

>But we're still talking like $170/month

Rounding error for most companies.

>The extra cost is a bit of Linux sys admin which can be fully automated

You are overweighting hard dollar costs and underweighting the value of engineering time. Maybe you're the worlds greatest devops/platform engineer/sysadmin and once you wire up everything in under 5 minutes it will never need maintenance ever again but for most everyone else speeding up image builds by using a service that someone else thinks about and does maintenance on is absolutely worth it for $170/mo.

Yes, maybe. I do know Linux pretty well and don't consider sysadmin costs a big drain on my own company or time. I can see that it'd be much more expensive if you hire people who don't have much UNIX experience.

On the other hand, I've experienced first hand how cloud costs can explode uncontrollably in absurd ways. One company I worked at had a cloud cost crisis and they weren't even serving online services, just shovelling money into Azure for generic dev services like VMs for load tests, DBs for testing, super-slow CI agents, etc. They never managed to properly fix this because of the mentality you express here: a few hundred bucks a month here, a few hundred there, everyone gets access to spin up resources and it's all worth it because we're all soooo valuable. Then one day you realize you're inexplicably burning millions on subscription services and cloud spend, yet nobody can identify quite why or on what, or how to push costs down. Death by a thousand cuts, it was quite the revelation. Free cloud credits are murder, because they embed a culture of profligacy and "my time is too valuable to optimize this". By the time the startup credits run out it's too late.

I think the problem is that once you get in that mindset, you start forgetting that you need to optimize/fix your application.

Having network issues due to slow async calls? Just increase instance size until the machine is so fast it completes everything before it becomes a problem. Now you are paying 10x more for something that’s a few hours of dev time.

I think you're giving the worst-case scenario when citing large-scale cloud spend. A couple of hundred dollars a month for very fast builds is a good deal, and they can build on Intel and ARM, which is useful.

Worst case scenario, I mean, maybe? I don't have many data points. It does feel like cloud costs come up more often lately. I don't think there was anything particularly special about that company though. It felt like almost the default outcome of using modern development practices and giving everyone who "needed" it access to the Azure console.

I have accidentally spent far too much on Azure logging before, so I am a firm believer in giving as much (read-only) access as possible!

You should probably include the cost of your own salary in your calculations

I'm running a bootstrapped startup so a special case, my salary is nearly nothing :) :( :)

At a bigger company you'd just ask a junior to set it up or maybe a sysadmin. You can probably contract to get part timers too. I guess I spent half a day or so setting up the CI cluster at the start and have barely touched it since. That wasn't much cost even if I was earning a big salary. Our builds can use caches to speed them up (not docker, other types of cache) and when we turned that on it was like a 3x speed win, so having persistent disks is definitely worth it for many types of program. Especially if you're brave and trust your build system to cache unit test results between builds!

The value prop is for people who do not want to maintain billing, infra operations, custom software, security, custom features, multiple platforms, and Docker expertise, when they don't need to. All of that is time consuming, difficult, and expensive to develop.

Just paying extra for all the above to magically give you faster builds is easy, fast, predictable, guaranteed benefits. Totally worth it.

If you need a lot of milk every year, you could buy a cow, or you could just buy milk at the store. Most people agree the extra cost is worth it.

Billing? Custom software? Docker expertise? I don't quite understand. You have to pay for something no matter what, right, and you still have to write and test your Dockerfiles locally? Where's the custom software?

The alternative here is having CI workers that use regular disks that aren't wiped between builds, and constraining the jobs that benefit from caching to run on them. If you have CI already set up it shouldn't be that hard? People are acting like you need to pay a L7 Staff SWE $20k to set this stuff up, is that really a cost problem many companies are facing? It'd have been considered junior level stuff not so long ago.

As for most people feeling it's worth it, I dunno man. Yes if I read the comments people seem keen, but my post is voted to the top of this thread. It feels like a lot of people have doubled down on functional-programming like approach to server management and now have really slow infrastructure that spends lots of time doing redundant work and then throwing it away. Yes, you can do that and it even has some advantages, like FP sometimes does, but you can also just accept that computers are under the hood stateful mutable machines and set up things to lean into that. That's what this startup is basically doing, right? Just seems like a 21st century problem somehow.

I think you're missing some of the perspective. It is actually much harder to make a working version of what this company is providing than you describe.

Just the act of running a single VM - to do it right - requires technical expertise (just because you find it easy doesn't mean it is, or that most people would do it right), in addition to maintenance tasks, operational overhead, etc. Deal with the extra infra costs in your corp cloud budget, write the extra software to handle advanced caching on auto scaling instances on multiple platforms, understand how Docker works under the hood (far fewer people know that than you assume)... It is extra work someone has to do that has no bearing on what someone actually wants to be doing, which is just running a Docker build faster.

I would have to assign two engineers to build and maintain this for a medium sized company, at $120K per employee, plus infra cost, plus maintenance, plus the lead time to build it, etc. And they'd probably do a crap job.

So, pay $50 a month for a working solution right now? To increase velocity of sw development, with no other changes? Sign my ass up. It's a tragedy that people don't understand the value here.

> I think you're missing some of the perspective. I

Yeah that's why I'm asking. It's genuine curiosity so thanks for your answers.

Yes if you wanted to make the same product as these guys you'd have to spend the same amount or more, so sure, that'd be a poor use of money. No disagreement there. Productization is a lot of work.

But you wouldn't need a full product to solve this for your own use case! I guess what I'm struggling with is the apparent reluctance to fix this problem by just running ordinary computers. We're hackers, we're software developers, this is our bread and butter right? How can we as an industry apparently be forgetting how to set up and run computers? That's the message that seems to be coming through here - it's too hard, too much work, the people who can do it are too expensive. That'd be like chemists forgetting how to use a bunsen burner and needing to outsource it!? Computers are cheap, they're fast, they can basically maintain themselves if told to! To make your Docker builds faster you can just run them on a plain vanilla Linux machine that just sits around running builds in the normal way, the same way a laptop would run them, with permanently connected disk and cache directory and stuff.

I totally get it that maybe a new generation has learned programming with NodeJS on a Mac and AWS, maybe they haven't ever installed Linux before, in the way we all seemed to learn how to run servers a couple of decades ago. Times change, sure, I get that. Still, the results are kind of mind boggling.

Well not really. I’ve literally spent the past few weeks banging my head on these things.

Especially if you want/need multi-arch. That basically requires buildx which doesn’t cache locally by default. There’s a half dozen types of caching to figure out. Then buildx is very buggy and needs qemu setup even when building natively otherwise you run into decade old bugs doing things like running sudo in a dockerfile.

It took a couple of weeks of on and off tinkering to get a stable arm builder running on a Mac m1. To get the GitHub action server to run stably and not time out was a PITA. It required IT tuning cpu limits and page caching. Not fun.

We run native machines but I would’ve much preferred a cloud solution so I could do my actual job.

I wonder how much of this is specifically Docker related pain? (I try to avoid it) It's super fascinating to see how much ARM support is coming up in this thread. I guess ARM servers are finally happening, huh. Our CI cluster has an M1 Mac in it, it took about an hour to set up. But that's doing JVM stuff with no Docker or qemu involved, so multi-arch just isn't something we need to think about and it's no different to any other machine beyond being much faster.

For servers I'd have thought you'd make a decision up front about whether to use classical x64 machines or ARM, based on load tests or cost/benefit analysis. Then you'd build one or the other. It sounds like a lot of people are putting a lot of effort into the optionality of having both, and then they are using languages and tools that can't cross-compile or JIT compile. Are you using Rust or Go or something? Hmmm.

Reply on news.ycombinator.com