Hetzner Introduces ARM64 Cloud Servers
hetzner.com
hetzner.com
So what's the point of all that? Seriously, one server with 16 cores and the memory bandwidth and storage speed of SSD drives should be able to serve many thousands of simultaneous requests and millions of users per month. I can't help but feel that the cloud infrastructure and microservice movement of the 2010s was.. a scam.
I just need a place to run Docker, similar to what we had with Linode for running a shell 20 years ago. And don't tell me how to set it all up. Just give me a turnkey Terraform (or Ansible-inspired declarative configuration management tool) setup that has stuff like load balancing and some degree of more advanced features like denial of service protection out of the box. What we used to think of as managed hosting, but open source, and with sane defaults for running standard suites like Laravel, Rails, etc.
I have to assume that I'm just terribly out of it and something like this already exists. Otherwise I can't understand why someone doesn't just offer this and provide a way for us to pay them the millions of dollars we lose to spinning our wheels on cloud infrastructure with 1% utilization.
The cloud almost always ends up costing a lot more at scale and increasingly at the beginning.
In some part this is because the cloud is someone else’s computers that you have to help pay for.
The current incarnation of cloud was designed in part to work around scaling issues with Linux networking, which have long since been solved. What a $5 Linode/VPS can do today vs 10 years ago is much different from a linux standpoint alone before additional horsepower.
At the same time tooling to host (docker, ansible, terraform) is making traditional devops heavier at times.
And yet it’s got basically perfect uptime. Whereas the impenetrable cloud goes down every week and costs billions a year.
Some monitors indicate that hackernews has a 99.84% uptime in the past 30 days. Hardly "perfect".
It's also about simplicity. Many services come pre-configured on.AWS, with clustering, failover, backups, etc already present.
When you are small, you can sysadmin your server fine, and you don't yet need the cloud.
When you are colossal, you want everything custom, and the cloud would cost you a huge amount, so you go for your own servers (and possibly datacenters).
But for the midrange, the expense of hiring several more SREs to handle dedicated servers is usually higher than the entire AWS bill, and the cloud looks very reasonable.
We'd likely have to have some reserve just in case our marketing and product do so well that the next peak of the load rises higher in absolute terms.
Most of the time we just removed GHz limits, sometimes adding vCPUs in advance, they fared well.
NB: IaaS provider with a guarenteered resources
Turns out, developers are cheaper than DevOps engineers and AWS budgets. So if a linear spike in servers necessitates a superlinear cost in operational complexity, maybe working highly performant web app backends is actually cheaper?
- database connection limits
- caches
- state changes
- handling graceful shutdown
- speed of provisioning new instances
... the list goes on ...
For $400 you can buy an SSD with 1.5 million sustained IoPS.
It is no big deal to build a 1TiB RAM box these days.
I think you inadvertently pointed out one of the advantages of microservices: the ability to bootstrap ideas without having to be hindered by artificial barriers to entry. If you have an idea and want to show it off, you just implement it and make it available to the public. Call it "aspirational goals", but I don't see any advantage in barring people from providing value without having to wrangle bureaucracies, red tape, and office politics.
If you keep the infrastructure simple, it’s also much easier to switch cloud provider and to set up testing systems.
Having some sort of vpc where you can safely put your db server that only listens to requests from servers within the same vpc (e.g., web server) sounds like good practice to me.
All the Linux distributions I got to know use sensible defaults so that critical services don't bind to a public-facing interface / bind only to localhost, e.g. mariadb and mysql on Debian.
Besides that, Hetzner's "Robot" interface allows to configure which ports/IP addresses you allow access to your Hetzner server.
Are you kidding me? Prices, sure, but comparing the configuration burden using AWS or GCP alphabet soup to a dead-simple Heroku setup is ridiculous. You set up something like hirefire.io for autoscaling, you connect your Github repo and that's it. You're done. Compared to weeks worth of headaches setting up Terraform Cloud and CI actions and massaging your state file and spending days tearing your headout over insane obtuse NAT configuration parameters? It's literally night and day.
I'm glad to have learned AWS, although I don't know that I'll build another cloud server. I think the biggest problem I found is that services tend to require each other. So I ended up needing an IAM for everything, a security group for everything, a network setting for everything.. you get the idea. A la carte ends up being a fantasy. Because of that interdependency, I don't see how it would be possible to maintain a cloud server without Terraform. And if that's the case, then the services just becomes modules in a larger monolith. Which to me, looks like cloud servers are focusing on the wrong level of abstraction. Which is why preconfigured servers like Heroku exist, it sounds like.
An open source implementation of Heroku running on Terraform on AWS/GCP could be compelling. Also a server that emulates AWS/GCP services so that an existing Terraform setup could be ported straight to something like Hetzner or Linode with little modification. With a migration path to start dropping services, perhaps by running under a single identity with web server permissions (instead of superuser). And no security groups or network settings, just key-based authentication like how the web works. More like Tailscale, so remote services would appear local with little or no configuration.
Also this is a little off-topic, but I'd get rid of regions and have the hosting provider maintain edge servers internally using something like RAFT combined with an old-school cache like Varnish to emulate a CDN. The customer should be free from worrying about all static files, and only have to pay a little extra for ingress for the upload portion of HTTP requests for POST/PUT/PATCH and request headers and payloads.
Oh and the database should be self-scaling, as long as the customer uses deterministic columns, so no NOW() or RAND(), although it should still handle CURRENT_TIMESTAMP for created_at and updated_at columns so that Laravel/Rails work out of the box. So at least as good as rqlite, if we're dreaming!
Edit: I don't mean to be so hard on AWS here. I think the concept of sharing datacenter resources is absolutely brilliant. I just wish they offered a ~$30/mo server preconfigured with whatever's needed to run a basic WordPress site, for example.
Your data is not encrypted in rest.
Their security is not bad but someone can easily plugin whatever they like.
Try this at Google. Google not just has much stricter physical access control but also FULL control over the hardware. They know the firmware of the Mainboard. The server don't even unencrypt and start if you move them out of their Datacenter.
Google doesn't just has ingress egress they have undersea cable.
There is so much difference between hetzner and google it's not the same thing.
It's up to you to set up "encryption in rest": E.g setup LUKS on a Hetzner machine.
> Their security is not bad but someone can easily plugin whatever they like.
That's not how it is. Check e.g. this six-year old security video from Hetzner: Hetzner is great but it's not a good choice for companies which really care. Your data is not encrypted in rest.
Their security is not bad but someone can easily plugin whatever they like.
But it's just levels different to Google.
I have a small startup at hetzner.
But my main job has everything at the big cloud providers because of audit logs, high slash, global network etc. And no the hetzner video shows already that hetzner can't fullfil all required security certifications.
They have rows and rows of desktop PCs there.
With 'really' care I mean if you process millions or billions you just don't go to hetzner.
And doing encryption on rest with what decryption method? Manual unlock through a remote console?
At Google you literally have encryption on rest by default with key rotation build in and you can use your own keys.
Dimensions of difference
If you only compare the cloud products. How do you know that they aren't encrypting their drives?
And BTW, they own undersea cables as well ;) https://www.hetzner.com/presse-berichte/2016/01/156818
Please educate yourself before you make such wild claims.
They invested in one and it's not stated how much.
And yes I'm comparing google with hetzner as answer to the op to tell him that cloud is more than just cheap compute.
I'm trying to explain why people pay a premium for cloud services and what the difference is.
And I use and like hetzner a lot. I'm happy to use hetzner for plenty of things just not for everything.
Unfathomably based.
I think the general consensus is that the most-root cause for why cloud infrastructure and microservice architectures matter is more People, less Technology. Its hard to hire highly experienced people, especially at the hundreds-to-thousands of engineers scale that many companies operate at.
That, in isolation, is reasonable. The actually uncomfortable question that no one wants to address is: was that a scam. Put another way, how much of that was a zero-interest rate phenomena, and how much "state of art best way to engineer" things was influenced from that, and has root causes in US monetary policy.
Between AI and high interest rates: if you're not ready to spend the 2020s questioning every single thing you think you know about software engineering; you won't be long for the industry.
Computers are so powerful now this may be the wrong instinct. You can mind boggling amounts of processing for say... $30,000 in hardware, and even with redundancy it would be much simpler to manage.
Indeed the main selling point of microservices is People, more specifically how to set hard boundaries on responsibilities and accountability while ensuring independent decision-making.
Another important selling point of microservices is flexibility. It's trivial to put together an entirely new service from scratch, written in whatever technology crosses your mind. With microservices you don't even need access to a repo.
This isn't really true as most of the time the services interact heavily and need to be aligned with each other not to break and work properly.
And all what you're speaking of can be achieved by properly modularizing the code of a monolithic application. And microservices if you want them can be run on a single machine, you don't need "the cloud" or multiple computers in general to run a bunch of REST apps if your single instance is powerful enough.
And for most web apps it is with the currently available hardware.
I'm not sure you got the point. The need to provide interfaces and establish contracts and pledge SLAs is irrelevant and a non-issue. Naturally, your goal is to have a perfect uptime and never break interfaces, just like your goal when working on a monolith is to not add broken code that crashes the service.
The whole point of microservices is that you are entirely free to make any decision regarding the things you own, and your only constraint is the interfaces you provide and consume, and SLAs. From those interfaces inward, you are totally free to make any decision you think is best without having to bother about anyone else's opinions and technical choices and tastes and permissions. You can run one service behind your interface or hundreds. As long as your interfaces are up, you can do whatever you like. That greatly simplifies your problem domain.
> And all what you're speaking of can be achieved by properly modularizing the code of a monolithic application.
No, no you can't. Think about it for a second: can you pick your software stack? Can you pick what dependencies you adopt? Can you rewrite your code in any other language? Etc etc etc. Your hands are tied left and right.
Nonsense. Each team owns it's services, and they know best what works and doesn't work. It makes no sense to claim organizational inertia as a valid reason to prevent you from picking the best tool for the job.
To add to that, microservices allow you to gradually upgrade services without having to be blocked by the idea of having to go through major rewrites. You just peel off a concern, get it up and running, and direct traffic to it. Done.
This is orthogonal issue.
You can pick any stack, but you don't have to. There still can be general company wide rules about what is acceptable and what is not. It's just that this is informed and enforced by the actual needs (not being too diverse in the technologies deployed), that can change in time independently from contents of any code repository.
Plus you can still make strategic choices here and there, that don't comply with general rules, and have, say 95% of teams working under stricter limits of what's allowed, and this couple of teams with special needs, where you were free to decide otherwise.
Microservices aren’t a silver bullet for this, of course. They can be part of a winning recipe though.
You're just missing the whole point of why you need n>1 instances. You're focusing on what might be the least compelling reason for the majority of applications, which is scalability.
Reliability and fault tolerance is a major selling point, and you can't simply expect to have adequate performance in more than one region if you're providing your services out of a single box in a single data center. The laws of physics ensure your quality of service will suck.
Operations also compell you to have multiple instances, ranging from blue/green deployments to one-box deployment strategies. And are you going to allow your business to be down for significant amounts of time whenever you need to update your OS? How would you explain all that downtime to your boss/shareholders? Are the tens of dollars you save on infrastructure a good tradeoff?
Also, security is also a concern. You're better off isolating your backend from frontend services, and software-defined networks aren't a silver bullet.
And in the end, are the tradeoffs of not designing your web services the right way really worth it?
Similar with security. A simple monolith is more secure than isolated front- and backend.
Even if they don't, there is usually a huge tax on developer productivity with these architectures, which often outweighs any improved stability over the long term.
Let's keep in mind that modern hardware and software is very stable, generally.
Not at any significant scale. dimm will fail, power will be down, disks will need replacement.
It is all about risks after all. If you are ok to have couple of hours of downtime if one of the memory sticks stops working - good for you. But generally any large enough business won't tolerate such risks.
I'm not saying that the cloud is the answer, but I don't see any future for single instance solution. And if you design your system like this, you are taking much more risks than necessary.
...except when your single piece breaks and you get your whole service down. Then what?
Ultimately it's the CTO/Technical Founders fault that this goes on. Avoid cloud services like the plague unless you really actually need them. They will slow you down and eat your wallet in the long run.
The required either PayPal or passport. I have no PayPal account, and their 3rd party verification system only allows passport from your country of residence (signup requires providing a contact address and they pre fill using this address; you can’t change the passport country). I am a British citizen living in Japan, and therefore hold a British passport; there was no way for me to provide a Japanese passport. I asked what I should do to comply, and they banned my account 6 hours later.
I can’t be the only one to experience this, can I?
They are remarkably well known for having draconian anti-fraud
We ban these people — they're violating our ToU by engaging in illegal activities. But they come back. With different names, different IPs, different browsers, different credit cards. They have complete identities to burn. (We spot the correlations anyway, along other dimensions I won't disclose here, and so can keep them out pretty effectively.)
And guess what? Very often, their requests are coming from Hetzner IP blocks.
I don't think the scammers have a direct business relationship with Hetzner, mind you. I think Hetzner tries just as hard as we do to stop these people from making use of their services. But I believe that these Hetzner boxes are either set up as exit nodes of one or more common VPN providers; or they're being registered for other purposes by other parties, and then resold on the secondary market on dark-web forums.
If I were a VPS provider, and I didn't want to support illegal activity, I'd probably just give up on providing service to individuals altogether, only taking corporate customers; and even then, requiring a DUNS number or something as an additional proof-of-work for that corporation, so that people can't just keep spinning up corporations in places where that's essentially free.
Hetzner hasn't gone that far; but it makes sense to me that if a user account is flagged as needing extended verification, and the ops person responsible for verifying the account takes a look at the user-lifecycle activity logs for the user, and sees that this user has: their IP coming from multiple places during registration vs login, their browser locale and timezone bouncing around between requests and set for settings uncommon to the country their IP is originating from, etc. — that the answer would be "ban" rather than "ask the user why the heck that's happening."
One time out of ten, the user is a real person doing something weird. The other nine times out of ten, the user is a scammer and is going to make up some story about being a real person doing something weird. Every scammer has their very own pool of man-hours, and if you're in the critical path for their scam, they can burn a number of those man-hours being really insistent that they're authentic. Until you let them in, and see that they immediately start up the same dumb phishing-scam bot script that all the other scammers purchased.
But I’d be willing to bet you’re seeing hacked servers, not necessarily Hetzner’s fault. Hell, they didn’t even have ipv6 firewalls until recently (like the last six months).
Translated quote:
> To do this, we go here: [link to carding forum] and create a topic in the section "verified Hetzner accounts."
> Offer price — no more than 400 rubles is needed. The priority is people from Ukraine, as they have benefits. GEO of the person who verifies the account - any, excluding Russia due to sanctions.
> Another important detail: the seller must register a fresh GMail account, use that account to create an account on Hetzner, and verify it themselves.
> After verification, we wait 3 days before the creation of the new server — otherwise the likelihood of the account being blocked for abuse increases.
> After purchasing the account credentials, we change the password, both on the Gmail account, and on the Hetzner account.
I’ve got streaming replication of my core data going from one cloud company to other company as that way if one has some antifraud system go rogue on me I still have access.
As somebody who used to spend a lot of time thinking about drives breaking it’s an interesting shift.
I do NOT want to be making that post!
Is this something new? I (Norwegian) have been using Hetzner for 10+ years, and never had a problem, and never had to attest my identity. CurrentlyI have a four servers running there. The last one was set up approx. a year ago, IIRC.
That may be the difference. Some nationalities get KYC'd easier than others and they seem to take it very seriously
(But I believe the KYC rules aren't only because of this.)
I've run into this with a few different German businesses, I don't know if I'm right or they are, but it seems like they don't _have_ to charge me VAT. Which is underlined by the huge "import VAT" bill I'm dumped with when I have physical goods delivered to me in the UK.
I had similar struggles with some non-IT service providers in Germany. They couldn't fathom why I have non-German nationality, German address and driving license from third country. Passport, German address and driving license all have different address on them (all three being EU addresses). It is apparently huge red flag in EU in 21st century. Incredible
I had that issue with German companies before, especially those who rarely deal with international customers. For example in Austria only a few people own an ID card (most people have a passport, but not everyone). So the german companies were very surprised when I gave them a copy of the Austrian ID card, that also doesn’t have an address on it.
https://www.consilium.europa.eu/prado/en/search-by-document-...
In the USA the driver's licenses are ID cards and have addresses. Or if you don't have a license you can get a regular ID card, which has an address. The laws vary from state to state about how long you can go after you move before it gets updated. I'm not aware of a legal requirement to have one though, unless you're driving.
Providing a recent utility bill or bank statement is a much more common ask.
Is it? From a quick search it looks like a very large number of countries have compulsory ID cards:
https://en.wikipedia.org/wiki/List_of_national_identity_card...
We have issues with voter registration because it’s been politicized. Poor people are disproportionately less likely to have ID cards (because it costs money, takes time, and most people get it to drive which is expensive), less likely to have it up to date. It’s disenfranchising to mandate IDs. Whether or not it’s a problem falls down party lines and your favorite statistician’s analysis. But that’s why it’s “an issue”.
Morally, even if it works out OK, it ought to bug you if you don’t have a cost-free way to vote.
Especially since it's usually white people using POCs as almost "noble savages" who can't figure out how to vote, get an ID, or have a drivers license.
But in every community there is a range of willingness to deal with bureaucratic annoyances to vote. Adding more hurdles bumps some people from the voter to non-voter bucket. The reason it is a big issue in the US is that we have a well documented history of adding those hurdles selectively in order to suppress votes from particular communities. This is part of a really dark chapter in our history so people have a visceral reaction to it. I mean, since you are Canadian—I guess people would be a little skeptical if someone tried to start a conversation like “Well lots of countries have boarding schools so here’s my plan for education in some underserved communities…,” right?
But in theory you could also bring two or more „identity witnesses“ that either have an ID or are known to the people at the voting station, that can vouch for your identity.
You need to be on their list though, but you get added automatically by the city to the list. When you move you need to register your new address, so they know that you live there.
Individual voter fraud is a silly way to influence an election, you have to get away with it thousands of times to make a dent.
Part of the issue is that you don't need an ID in America for most of your daily life. Most people get it to drive - if you don't have a car (which is actually expensive) then you may not get an ID at all. Yea most people will get an ID, but it's not something people need.
When it comes to the cost of the ID, part of the cost is taking time out of work to sit in a crummy office and fill out paperwork. They require certain proof of identity paperwork that can be hard to get for certain walks of life. Its quite an edge case in society that can't produce a small amount of paperwork to self-identify, of course.
Its a small cohort that truly don't have the resources to get an ID, but there is almost no observed downside to not requiring IDs to vote - American elections are and have been perfectly legitimate (or until 2020, depending on who you ask...). Why would we put up extra barriers to vote when we could just... not?
This is not true. You must have an Identitätsnachweis, but if you have a valid passport you do not need a Personalausweis.
Side note: It seems like many African ID cards list the profession and I've had to explain to several suspicious border guards why my passport doesn't.
EDIT: I've always wondered about this and apparently the body that sets passport specifications is ICAO, the International Civil Aviation Organization, as mandated by the UN. Specifically through ICAO document 9303.[1]
[1]https://www.icao.int/publications/documents/9303_p4_cons_en....
You can see the field in this official example passport (field 11): https://www.gesetze-im-internet.de/normengrafiken/bgbl1_2017...
Another thing to consider: cloud providers are not very interested in individuals as customers. They usually want companies as customers, that also buy more than a 3$ vserver. A solution for this problem could be a sign-up fee (50 or 100$), to pay for an extended manual vetting of customers, that is then added to the account balance.
A key theme in the "cloud vs data center" story is that most public cloud providers (AWS, etc...) were really easy to sign up, requiring a CC and nothing else.
Meanwhile, hardware vendors wouldn't even talk to you as an individual / small business.
Meanwhile, my small server (not Hetzner) has been running for many years without any issues, never had to send them anything after I signed up...
This is actually exactly what happened with us. After creating a new account with the intention of exploring the ARM64 services, our account was unexpectedly suspended. I contacted them to have information on the specific concerns regarding my customer information and the reason for deactivating my account. Unfortunately, we did not receive any response to our inquiries.
Sounds like a good thing to do?
My current hoster in Germany made a surprise call and asked me what the name of the hotel near the address I provided was. This after I submitted the order and before they accepted.
I ran into the same thing as well, maybe my real name sounds a bit funny to their system? It was very discouraging to move forward after being instantly banned. I reached out to them how I could verify and the only way was sending them an unencrypted mail with my Passport copy. Upon request they suggested I could simply send them a fax.
Note, this has been some years ago and I've never gave Hetzner a new try. As long as I can see from professional experience, you will have a lot of back and forth with Hetzner support, which becomes quite bad the moment your team is international because they'll always manage to sneak in some sort of German text. It really feels antiquated having to go through support for basic server hardware debugging. Eventually they'll often resort to replacing your instance.
But according to my Minecraft benchmarks in Oracle's instances, its better than old Skylake cloud instances... by how much is hard to say since other tenants generate so much variation, but there are proper reviews of the whole SoC floating around out there.
We've moved on from lakes!
See: https://www.anandtech.com/show/15631/intels-cooper-lake-plan...
It's limited everywhere.
I work at one of those customers; we use loads of them.
Nevermind Ice Lake / Alder Lake (2021) / Raptor Lake (2022) -- Intel hasn't abandoned the "Lake" naming scheme yet.
i.e. Skylake+++++
Hetzner offering them right now is curious. Maybe its a pipecleaner for the next Ampere gen?
Meanwhile AMD and Intel both do have a later offering so they are "old".
On my workload, using Intel as a baseline. AMD under performs relative to standard benchmarks. ARM over performs relative to standard benchmarks. I have seen good performance from Ampere, particularly for the price.
However, ARM architecture complicates my workflow significantly. I've found that they are not worth using it right now due to compatibility issues, easier to just stick with x86 for most things. Dealing with required missing packages just kills the value.
My system isn't generally compute limited anyway. IOPS and egress dominate rather than MIPS.
Which begs the question: what is Hetzner pushing on these? SUSE maybe?
You can mount your own ISOs though.
I strive for simple infrastructure.
ARM doesn't support the x86 docker eco system. That's a significant strike but not totally unworkable for me.
My primary database isn't prepackaged with the ARM packages nor does the originating org build one. I could build it myself, which I have done plenty over the years. I did try it, but the dependencies aren't up to date on ARM. So now I'm building dependencies and the software package. Building software from scratch can be a rabbit hole that eats untold amount of time. After a one-day effort I gave up and abandoned it. I'll revisit again if Hetzner brings Ampere to the US and I can verify the instances have much higher IOPS ceilings.
Enter another distro and I have to rebuild all of my provisioning and management scripts for all my services with all the testing entailed in the new dependency chains. I'd rather try to build my db from source again, frankly.
maybe things have been fixed now, but about a year ago I was having issues with ruby bugs that were due to differences between muslc and gnulibc
"Missing packages" may be packages missing important features or bugfixes.
Arch and SUSE are ARM friendly though.
There is a kind of after market ARM community that occasionally builds these items. But they are generally pretty out of date.
I don't have an issue with the base distro and what it tries to do, but there are so many random pot holes and road blocks that just don't show up for x86.
I have ARM servers running in production now. But I have workloads that aren't reasonable portable to the uArch and I don't force them there.
Minecraft is limited by a single thread, but some stuff does run on other threads. In addition the JDK will do compilation and GC on other threads now, which is especially apparent if you run GraalVM like you should.
Minecraft wont scale to 16 cores. But if you buy a 2C/4T instance, you are going to get less performance than you would get on a wider one.
Hmm, not so outdated. if you’re talking about JVM gc then sure but pretty much everything is built around a single thread ticking the mod. I know some mods can do chunk pregen on another thread but those are rarely used.
I have no interest in subjecting myself to this treatment.
I'm attempting to soften that friction when I launch my PaaS
My concern over bots and misuse is a concern. I'm thinking of crippling the free service to allow full functionality but limiting it's use to really tiny files.
In the early days it was slightly amusing to get calls from older folks trying to understand what a virtual server was.
What I think you’re suggesting is that my trouble is with the machine—some version of frustrated human fighting with inanimate objects. In general this is a framing I oppose; automated machines are created by people, and people carry the responsibility for their creations. “The computer did it” is never an excuse.
So in a way, yes; I tend to view “error messages” as you put it as an extension of their creators, who very well may be worthy of indignation.
https://hastebin.com/share/biyejiriyi.bash
https://hastebin.com/share/unocorikef.bash
https://gist.githubusercontent.com/12932/8ba27254846072a43b0...
You can increase the IOPS, and thus the throughput for 4k operations, but at a cost, of course:
https://gist.githubusercontent.com/Q726kbXuN/7f03a9c11cab514...
https://gist.githubusercontent.com/12932/8b47af122ba79c79b82...
Host wasn't my choice...just sharing the links
Ran the kernel build benchmark (result is seconds, lower is better):
AMD64:
272.916
273.128
270.477
ARM64:
1011.799
1004.713
1015.261
Also ran sysbench cpu --time=60 --threads=4 run
AMD64: events per second: 14681.70
ARM64: events per second: 13455.11
Personally, if I were to do it, and I had some sort of load-balanced application that uses multiple backends/frontends/whatever, is balance across the two and compare over time.
And to be 'fair' you'd want to compare similar pricing.
Or if you had a consistent group maybe try a Minecraft server and move it between the two every week and see what people "feel"?
That depends on how many vCPUs are they running on each physical core for both x86 and ARM.
This will never be 100% scientific or correct since there are many factors that come into play, stuff like noisy neighbour for example.
What I'm trying to get is a ballpark idea of how their arm vCPU compares to the amd equivalent.
sysbench memory --time=60 --threads=4 run
AMD64: 5859951.00 per second
ARM64: 6052749.14 per second
Next up I timed compiling nodejs. time make -j4. I ran the test two times and took the faster result for each.
AMD64:
real 28m46.385s
user 107m48.971s
sys 5m12.994s
ARM64:
real 39m18.443s
user 146m25.801s
sys 7m53.271s
Rescaled the ARM VM to 8 vCPUs:
real 22m20.624s
user 162m30.176s
sys 8m50.104s
What I learn from these benchmarks is that you might get some really good value out of these ARM instances if your usecase is not impacted all too much in terms of performance.
sysbench cpu --threads=2 run
Intel events per second: 1864.20
Arm events per second: 6687.05
Coremark
Intel iterations/sec: 20077.633516
Arm iterations/sec: 23625.767837
rsa 512 bits 0.000070s 0.000006s 14268.2 159885.6
rsa 1024 bits 0.000405s 0.000021s 2468.4 47757.7
rsa 2048 bits 0.002847s 0.000078s 351.3 12830.1
Hetzner x86: sign verify sign/s verify/s
rsa 512 bits 0.000067s 0.000004s 14893.9 240902.8
rsa 1024 bits 0.000127s 0.000009s 7845.9 114300.9
rsa 2048 bits 0.000874s 0.000027s 1144.6 36800.6
dd if=/dev/urandom of=1GB.bin bs=64M count=16 iflag=fullblock
Intel:
Compressor Compress. Decompress. Compr. size Ratio Filename
memcpy 5062 MB/s 5013 MB/s 1073741824 100.00 1GB.bin
zstd 1.5.2 -2 2083 MB/s 4743 MB/s 1073766410 100.00 1GB.bin
zstd 1.5.2 -5 210 MB/s 4775 MB/s 1073766410 100.00 1GB.bin
zstd 1.5.2 -9 85.5 MB/s 4774 MB/s 1073766410 100.00 1GB.bin
Arm:
Compressor Compress. Decompress. Compr. size Ratio Filename
memcpy 10876 MB/s 10950 MB/s 1073741824 100.00 1GB.bin
zstd 1.5.2 -2 3175 MB/s 11168 MB/s 1073766410 100.00 1GB.bin
zstd 1.5.2 -5 192 MB/s 10967 MB/s 1073766410 100.00 1GB.bin
zstd 1.5.2 -9 146 MB/s 10909 MB/s 1073766410 100.00 1GB.bin
To offer a somewhat more varied example I've tested the POV-Ray benchmarker on Debian 11 on Oracle's Ampere Altra servers ("A1.Flex") versus an identical setup/build running on a 2 GHz EPYC 7281-based x86-64 VPS, and on that single-threaded test the Arm VPS handily outpaced the EPYC with almost 2x the performance.
On my local machine with 4 threads I get ~10k events per second on cores 0+2+4+6, but on cores 8-11 I get ~13k. Does this mean that Gracemont Atom is 30% faster than Golden Cove Core? No, it is only measuring the fact that the efficiency cores happen to share an L2 cache.
e.g. average price in August was over 400% higher than this March
https://www.nordpoolgroup.com/en/Market-data1/Dayahead/Area-...
The big peak (up to 10x) is over, but the energy prices already increased a lot in autumn of 2021, when Russia first reduced the gas deliveries.
ARM design had to be adjusted/developed for a cell phones often cost 10% of a normal computers (main use of x86). I can buy a cell phone a display, battery, memory and ARM cpu for $100 total (no idea how much ARM chip alone cost). The phone will have a semi-acceptable performance.
Even cheapest x86 CPU with emi acceptable performance probably cost more than $100 and cheapest x86 cpu more than $50.
Nvidia claims to be working on getting in the server game later this year so maybe ARM will get to x86 levels of competition.
16 cores doesn't say much. It depends how many DMIPS those cores will yield.
AWS has been trying really hard with Graviton 3.
Building your own based on them isn’t really difficult at all; docker buildx works remarkably well and build tools such as maven, sbt, etc. seem to have rather decent support for building multiarch images.
Even better if you have decent build automation, implement cross building for amd64 and arm64 once and just make sure your FROM images support those.
I actually registered the org oci-base (https://github.com/oci-base), when Docker Inc. announced the registry fees.
In part as I've been annoyed with multiarch and docker/registry in the past.
Then to top it off the official registry has/is taking ages to support signing/attestation.
Plan is for it to be contributor driven, and RFC style requirements following best practices.
Not had much time to work on it lately, kids, Easter, etc.
It's not just the building of it but the testing/verification of what's built.
1 major problem with a lot of use cases is there isn't enough test coverage and hence confidence in the change. Some people / companies find it easier to just pay the difference.
Quick Googling shows:
- Intel W9-3495X has 56 cores for $5889
- AMD EPYC 64 has 64 cores for $4299
- Ampera Altra M128-30 has 128 cores for $5800
Hetzer offers a 16 core AMD (v)CPU machine at 225% of the price, so that roughly works out.
Research costs are also very expensive for x86, with 2 vendors (Intel and AMD) competing savagely in many different markets such as high-end gaming, servers, ultrabooks and low cost PCs. However, ARM (the company) focuses on licensing the designs and instruction sets. So their research costs are split among many licensees while ARM focused on what it does best: designing chips.
That in turn translates into many vendors who can focus on improving manufacturing efficiency, procurement, logistics and sales. x86 vendors have a wild range of manufacturing lines to keep up.
Finally, the laws of supply and demand play a huge part. The ARM market is 2 orders of magnitude (!!) greater than x86 as far as raw number of processors shipped, ~370M vs ~32B chips in 2022.
> the number of transistors tend to be similar between the two platforms
sorry, come again?
So, given a x86 and an ARM chip with the same number of transistors, the ARM one will still be a simpler, low-power chip with probably a higher number of cores.
Also, x86 backwards compatibility and SMT don't cost much in terms of power or transistors but both have large costs in of verification work.
Unless I'm blind, the price clearly says 4.52 EUR/mo with IPv4 at https://www.hetzner.com/cloud for a CAX11 instance, the smallest listed there.
Edit: nevermind, I didn't deselect Germany's 19% VAT. It is indeed 3.79 euros before sales tax.
Also haven't been sued yet. But let's see.
I use my oracle free servers (and one paid vps on hetzner) as automatic cloud fallbacks if my homelab is inaccessible over the internet for some reason. Which means I fall in the underutilized category as most of the time workload is under 2%. So I "fake" work to prevent reclaimation by oracle and its worked well so far.
``` # Oracle Cloud reclaims idle resources # https://docs.oracle.com/en-us/iaas/Content/FreeTier/freetier...
- name: Create cronjob for minimum CPU utilization
ansible.builtin.cron:
name: fake work to prevent re-claimation
special_time: hourly
job: timeout 30s sha1sum /dev/zero
But I don't really want to do business with Oracle anyway, maybe it's better so.
---
Intel 1-core VPS single: 719
(2 GB RAM)
---
Intel 2-core VPS single: 729
Intel 2-core VPS multi: 1402
(4 GB RAM)
---
AMD 2-core VPS single: 1141
AMD 2-core VPS multi: 2214
(2 GB RAM)
---
Ampere 2-core VPS single: 872
Ampere 2-core VPS multi: 1716
(4 GB RAM)
---
Ampere 16-core VPS single: 880
Ampere 16-core VPS multi: 12143
(32 GB RAM)
---
And for some fun, my iPad Pro from 2018, single: 1135
If Android CPUs are 4 years behind Apple, then Arm on servers is maybe 8 years.
With all the AI hype I'd assume they'll try to offer GPU instances again with precautions put in place.
It's been a wonderful ride. It's cheap and reliable.
I wish our team was big enough to afford having to deal with dedicated hardware.
One thing I personally like with OVH is that they have fixed prices for things like bandwidth, not pay-per-GB like most others.
elastic aws cloud billed by the second is for great good. glad to see this continuing to mature.
neither of these is great when they are idle 90% of the time. one of them is really not great.
why are there so much idle yet paid for resources? it’s complicated, and not for technical reasons.
empty housing is similar.
* Oregon
* Virginia
Been using a couple VPSes in Oregon, pretty happy with them.
The latest Xeon and EPYC offerings are very performance competitive, I doubt we need to overhaul and entire processor ISA paradigm for continued improvements.
Well if they're spending less to power my ARM machine, then they should damn well be charging me less to host it.
That's the best, and only, long term path towards more efficient compute.
I don't mind eating the cost to save the planet if there is a cost.
What I am objecting to is businesses making a larger profit off people who want to save the planet.
They're literally taking your goodwill and turning it into profit. If they aren't interested in saving the planet, then why are you giving them money, instead of giving it to their competitors?
I guess the question is, why not pay less to save the planet? Your extra money doesn't go towards saving the planet, it goes towards making you feel like you're saving the planet.
That doesn't seem likely, as the release of AWS Graviton based instances predates the Apple M1 release by about two years
So, if you want to compete, you'll have to go ARM. Amazon graviton and Apples M series have only made it more appealing.
The main reason to switch is cost.
Currently anyone can make an x86-64 processor.
Nice to see that things seem to start moving a bit now. Energy consumption and cooling are major concerns for data centers. ARM is just so much better when it comes to energy consumption. So the potential should be there.
We have seen performance improvements in PHP, Java, and various CPU-bound tasks such as video transcoding.
This goes for either EC2 m7g vs m6i and also for Fargate x64 (from my experience, perfomance seems comparable to m5 instances) vs ARM (Graviton2).
I'm currently running Ubuntu but am open to alternatives. My local Raspberry Pi uses a fun, but little maintained OS called Hypriot which is just this: Start Docker and not much more.
Maintained, light on logging unless you ask for it (saves your SD card from wear and tear), and focused on low host-os resource usage while still being Debian-based.
If you're serious about running _everything_ in containers, then maybe RHEL Edge? RHEL + microshift? Fedora Silverblue? openSUSE MicroOS?
But their hardware design could improve. See picture below.
CAX11 - 3.95eur/m
CAX21 - 7.19/m
CAX31 - 14.39/m
CAX41 - 28.79/m