Hacker News Reader

Comment by tomalaci | Hacker News Reader

I know a lot of places tend to use things like AWS spot instances for their CI runners but they obviously provide inconsistent performance. As others have noted you can always measure performance through other metrics like certain counters, not just the absolute runtime duration.

I can recommend getting your company a dedicated instance from OVH or Hetzner since they are dirt-cheap compared to cloud offerings. Setup some simple runner containers with properly constrained CPU and memory that would be similar to production environment, hook them up to your Gitlab or Github and you are good to go. You don't really need high availability for development things like CI runners.

View on HN

> but they obviously provide inconsistent performance

No. Unless you're using a "burstable" (overcommitted) instance family, this shouldn't be the case. The performance will be consistent, but you may get preempted before you finish your work. I would keep those concepts separate.

That's true, however a lot of spot usage ends up being heavily diversified across instance types in order to avoid momentary supply issues and optimize cost.

Across different families, CPU performance can vary by a decent amount.

Sure, but you should still keep the concepts disjoint. "I took whatever I could get" isn't the same as "spot instances are inconsistent".

I’ve wondered about something like that. Is it possible to get the number of cpu operations some test takes? I get with modern CPUs it won’t be as exact as when I was counting 8080 operations, but it would surely tell you something.

Or at higher level abstractions, things like how many Python byte codes.

Yesn't, you can by, e.g., using qemu[0]. This however has the obvious downside of not accounting for pipelining and cache as well as the subset of instructions encoded in microcode vs. hardware. All of those are very important but CPU (microarchitecture, cache) specific.

[0]: For example: https://www.qemu.org/docs/master/devel/replay.html#instructi... more can be found searching the docs.

Iai^1 uses CacheGrind^2 to count instructions

[1] - https://github.com/bheisler/iai [2] - https://valgrind.org/docs/manual/cg-manual.html

On Intel CPUs yes you can, using perf. See https://www.brendangregg.com/perf.html ; getting started: perf stat command

CI/CD runners are a great example of something that doesn’t require a dedicated 24/7 server. Spot instances are great for this purpose unless you’re looking for ultra-stable benchmarking performance.

If you do need precise performance measurements for customer experience, it’s usually better to run the code on the same hardware you’d be using in production anyway. For client workloads this generally involves getting the actual hardware your customers would be using: Laptops, Macs, and other things you’re not getting from cloud providers.

Also, the performance differences of cloud instances aren’t so significant that it warrants going out of your way to set up something dedicated. If you need more performance, just get larger spot instances.

Reply on news.ycombinator.com