Show HN: Cloud Benchmarker: See how fast your cloud instances are for real
github.com
The problem is that these providers-- depending on the particular data center, or the time of day, or their aggregate customer demand-- have a habit of "overselling" their physical capacity. Of course, they don't generally announce this to you-- instead, you have to discover it by your machine acting far, far slower than it should be given its stated specs, which often causes your applications to fail in bizarre ways (this is doubly true if you're making a distributed/decentralized application!).
After having this happen to me one too many times, I decided to make this tool to systematically track the real world performance of a fleet of cloud instances. I didn't want to reinvent the wheel, so I just used sysbench and its battery of tests for things like memory, CPU, disk, etc. performance. What makes my project so convenient is that it uses ansible for everything: you just specify an inventory file containing your machine IP addresses and SSH key, and it will automatically connect to each machine, install the required software, run the benchmarks, extract the key values from the reports, and send them back to the control machine.
It schedules another benchmark every 6 hours by default, but you can easily change this in the .env file to whatever you want. This lets you get a good sense of how fast your machines are in practice over time, without placing undue load on them. I believe that you can't really know how your machines are performing without putting an actual big load on them!
Anyway, the control machine (the one running my python project) then assembles all of these benchmark reports into a single file, and processes it to arrive at a single normalized overall score for each machine between 0 and 100 (this is a relative score, with the most powerful instance getting a score of 100). All of this data is stored in SQLite, where the data points are enriched with the IP address and machine name. This data is then exposed via a FastAPI REST API with a Swagger page so you can easily get the historical benchmark data and use it in your other applications. You can also download the historical data as a csv file. Most usefully of all, it provides a nice web page with dynamic charts showing the historical data in various ways using Plotly.
This has already been very helpful for me, and hopefully it will be for you, too! Let me know if you have any questions.