Show HN: VPS Comparison – Automated tests to compare VPS by yourself
github.com
github.com
You know, this is the kind of of knowledge that we all like to have, but even then, all we have is your word (I don't have any reason to doubt it), what I tried to share here is a way to back all of that (at least performance, reliability is a different kind of monster) with numbers.
I just hope they both get block storage deployed on all their regions.
You can find the numbers at https://gitsense.com/blog/benchmarking-march-14-2016.html
I probably should run the tests again to see how things have changed though.
DO - $10 | Vultr - $5 | Vultr - $10
-----------------------------------
00:38:59 | 00:31:58 | 00:29:43
Edit:
To put the numbers into perspective, I ran another test in a virtual machine with VirtualBox on my local desktop and finished indexing in 00:21:29
I allocated 1 virtual core and 1 GB of RAM for the virtual machine. The desktop processor is an i5-4460 @3.20GHz
As Vultr and Linode provide double the memory and other benefits for same price as DO, unless DO provides better plans, it's becoming a bit of a joke concerning raw performance / $.
That's one of the reasons why I started this, all the reviews and comparisons along the web have some kind of bias -even when is positive and truthful by grateful users- and is very hard to know what is right from wrong. I wanted to offer some way to have transparency and reproducibility all the way.
Most providers will artificially limit you below the published limits if you use "too much" CPU or internal network bandwidth. You think that 1Gbps link between your machines isn't shared? See what happens when you try to use it full tilt for a couple days.
Did you mean "overselling," selling more than your infrastructure could support if every customer used all you sold them?
I didn't make up that sentiment, by the way; an account representative said it to me verbatim. It seemed mostly true in practice, and it's a big driver of why it's tough for firms like Linode and DigitalOcean to compete in the same arena.
Performance is mildly all over the shop - eg, some days packages might install in under a second, other days pacman stalls for 30s+ "resolving dependencies", and then a further 2-3+ minutes regenerating the manpage cache or whatever it's doing - but, you can peg both cores, and they don't seem to care. So there's that.
It's in Germany, and ramp-up time to US servers takes a couple seconds (eg, pulling data from the US will take a moment or two to wind up to ~12MB/s). Similarly, latency is mildly high.
A big plus: it's just QEMU and they give you access to the KVM VNC console. You can kill iptables when it goes rogue! You can disable SSH!
Oh - a tidbit to remember: they do mention that if you use >60Mbps for 6 hours+ they kill your link, but I learned from some friends what that means in practice: your connection goes to 10Mbps for the rest of the month.
Not a shill, just mentioning what I learned from using a friend's instance for a couple months. Would be interested to hear about any negative experiences people have had so I can file those away too.
---
Another option: online.net have dediboxes based on Atom C2350s with 4GB RAM, 1TB and 1Gbps networking, for €8.99/mo (+€20.00 setup).
So on the one hand you're dealing with this sadness: http://ark.intel.com/products/77977/Intel-Atom-Processor-C23...
But on the other hand it's a dedicated machine. The CPU has virtualization support (despite being 1.7GHz :D hahaha). That's gonna translate to 100% usage sooner or later... so... that may be a potential alternative.
I haven't tested this option, it would be very interesting to pit it against a Contabo VM.
Anecdotal, of course, but I have an application on OVH (on the USD $3.50/month plan) that pushes/pulls >10TB/month - coming close to saturating the 100/100 connection at times - and I've not heard anything from OVH about it being an issue.
Your assumption makes sense to me, and probably is close to the truth.
It's like being at their mercy (or other clients) or like if the rules were written in the shore's sand, you're well until the tide rises. I would prefer clear lines into this kind of things.
Another thing completely different is the permissibility about certain illegal activities or loopholes in their servers. That's not my concern, that's only a question of moral/ethics/legal between the client, the provider and the authorities.
As these graphs suggest, their disk speed is not great compared to my Vultr servers.
Scaleway is actually quite good when you factor in the performance / price ratio with it. (VC1S) But that's not something I would recommend for 'production use'. Yet, it's very nice for hobby projects.
All systems are under constant monitoring. CPU, RAM, Disk I/O & Networking. Unfortunately the data collected isn't something I can publish.
Btw. OVH SSD VPS 1 does provide IPv6, yet it's not being tested. Why? - Yes, you'll need to manually add the routing once to enable it.
Yes, I know about the OVH plan and IPv6, but in the hurry and in the middle of all the tasks to accomplish I left behind a few things. I'll keep improving it. Thanks!
In a perfect world, you wouldn't care so much about network reachability and care a lot more about, say, mTLS or at-rest crypto. But as a young company, managing that probably isn't the best thing to spend your resources on right now. Much like Google's BeyondCorp: just because you can expose some apps to the internet with years of work and millions in resources doesn't mean that a VPN or a SG isn't a pretty good idea for now -- if nothing else, it'll do a pretty decent job protecting you from the background radiation of the Internet :)
Having to pay with a credit card before you can use paypal didn't earn them any points either. Suffice to say I did not go with Linode.
But stressing instances regularly to test them to find a better one is something that you could probably do on your own, but as a service I have serious doubts if the providers would like that from a given scale. I'm not sure, but I don't think that would be something rare to have you account suspended. Perhaps, if you arrive a some type of agreement with them and their "business model" is compatible wit that, maybe. But, don't give me too much credit in this, I'm only speculating.
Thanks for you feedback. I'll add this as an issue into the repository to improve the comparison.
Well, I would like to include more, in fact I would include Ligthsail for sure if they offered a datacenter in Europe. But to start something like this you have to start small, or the costs will bury you pretty son. As a pet and personal project, the way to limit the cost was setting a clear seat of boundaries, like plans under $5 and available in Europe. And being one-man band to make this affordable in terms of time and effort I had to keep it small at the beginning. I worked on this in my spare time for more than two months. If I find a way to sustain the costs, or the providers itself offer me a way to test their machines, I have no problem to expand the comparison. But in the meantime, I share the tools because I knew from the beginning that I only have limited resources. And after all, reproducibility, independence and transparency are the keys IMHO to have a trustful comparison.
Sometimes they'll reboot my instance due to host errors they claim, seems to be an automated email.
Last one was January 17, before that was Dec 16, before that was Oct 29, then Oct 22
Each time the instance in question was down for less than a minute, no credit was issued.
I have instances in Sydney, Los Angeles and New Jersey
In order to even come close to 100%, you need a multi-region distributed system and even then, so many things can go wrong.
Look at their outage credits (https://www.vultr.com/sla/) if you want to get realistical percentages.
Even Google Cloud Platform - which has multiple layers of redundancy at every level - does not claim 100% uptime: https://cloud.google.com/compute/sla
Disclaimer: I work for a competitor who would never advertise 100% uptime - it's a known fact that shit happens
Recently they had downtime in Amsterdam for at least an hour.
I like them, but if reliability is important they're not my first option.
Performance is nice, but if you are already on (a) blacklist (s) from your start.. :) since most of those providers, have a lovely reputation in regards of compromised machines being abused in botnet and other malicious activities... i think it might be an interesting index.
I'll have to think about that, and how collect that information. Thanks!
The only concern is that they only have 2 datacenters and they're both in the Netherlands.
[0]: https://tilaa.com
And I've never had a node go down, that helps.
I'm inclined to expand the review, but only if I find a way to get free credits from providers or another way to finance the tests. But right now, the best thing I can offer you is the same tools that I used here in the repository itself.
I understand you, but as I said in another comments, I would like to expand the comparison, but in the end is a question of time and costs. If I find a way to sustain the costs, or the providers itself offer me a way to test their machines, I have no problem to expand the comparison. But in the meantime, I share the tools because I knew from the beginning that I only have limited resources. And after all, reproducibility, independence and transparency are the keys IMHO to have a trustful comparison.
I'll add this suggestion as an issue to the project. Thanks!
>Send root password by email Yes No[1] No
>Notes:
>1. DigitalOcean sends you the passwords only if you don’t use SSH keys, in plain text.
Otherwise, great comparisons.
Yeah, I agree, that would be nice, but as I automate the creation of those tables and cut/paste them every time that I run a set of new tests, to do that would require a lot of manual job to me. That's the reason why I don't do that, to keep it manageable.
What about including other popular frameworks? Ruby on Rails, Laravel, ASP.NET, symfony, express.js
Yes, there is always that possibility, is a question only of time, more tests implies more time to complete the full batch. And of course, more time to implement it.
I have the intention to keep improving it, and thanks to the Ansible tags, the tests can be run individually and independently (I tried from the beginning to make them as atomic as possible for this same purpose). Thus is very easy to extend.
In the end you have to know what you are doing, but it will point you to very useful tips and show some warnings that you can look to improve the security of you server. But in this, I still don't know two SysAdmins that concur on the best way to harden a server. There is a lot of things to consider and there are very strong opinions on how to do certain things.
But without doubt, it's a very useful tool, even if you know what you are doing you can use it like a checklist to know if you are forgetting something. It's not perfect (nothing is, and less in this field) but I can only recommend it.
Well, I would like to include more, in fact I would include Ligthsail for sure if they offered a datacenter in Europe. But to start something like this you have to start small, or the costs will bury you pretty son. As a pet and personal project, the way to limit the cost was setting a clear seat of boundaries, like plans under $5 and available in Europe. And being one-man band to make this affordable in terms of time and effort I had to keep it small at the beginning.
Maybe, If providers offer me free credits to make the tests or if I found a way to assume the costs, them I could think to expand the project.
I've used all the service providers available and these opinions include my historical experience with them (I've used all providers for many years, my account with DO is since 2013). Most of my service uses were focused in those service provider's North America locations, Asia locations, and Oceania locations. Note I did do a short comparison between DigitalOcean and Vultr back in 2014 here: https://vpsboard.com/threads/digitalocean-vs-vultr-the-asian...
Most service providers here are fairly decent.
From my experience, OVH's KVM VPSes weren't worth the time, but their VMWare VMs were absolutely top notch.
Linode has had two major security events over the last few years, one of which was in early 2016 which included release of credit card information. However, they also offer paypal payments now and are fairly top notch in terms of performance. (They've taken approaches to fix their problems from 2016 with this blog post here: https://blog.linode.com/2016/02/19/security-investigation-re... ).
DigitalOcean is fairly reliable. Their Singapore location early on had regular packet loss events, but they've been shaping up pretty nicely over time.
Vultr has been a major problem for me. Vultr for a while (on their storage instances in Japan) had terrible I/O which took a solid two hours just to install Ubuntu 14.04. When contacted support, they stated they "stopped a noisy neighbor and to retry" but no real improvements were seen.
We had a service running on a VC2 that irregularly required a large amount of CPU power (higher than 0.5 load on a 1 CPU instance for a period of 1 hour a night). The same task was completed faster with VMs from DigitalOcean and Linode. But the biggest problem I've experienced is their network.
For the longest time their Japan location was single-homed and routing was funky. We'd regularly see any traffic going outside of Japan being routed back through their Los Angeles location, so adding 300+ ms latency for any of our users in Korea, China, and anywhere in South-East Asia. Today it's much better with 40 ms latency from their Japan location to our Korea client, and a solid 80 ms latency to our Singapore client.
A colleague of ours had very important clients who required 100% uptime (which Vultr agreed to) but over a period of a month had regular packet loss events or network loss (regularly 40 minutes per event and around one event every few days monitored via external monitoring systems such as uptimerobot, statuscake, and a GCP server running Icinga2). Didn't receive much help from Vultr's engineers nor could they find the issue. They ended up moving to Google Cloud Platform. They did end up getting SLA credit from them but no reason to have SLA credit when you're not on their platform anymore.
I regularly return to Vultr every few months to every year or so to see how it's going and they've come a long way with the services they're offering. I'd recommend them for a dev environment and testing over a period of an hour a day or so. However, I would caution with production systems. Your mileage may vary, but I'd suggest reviewing the quality of the product in addition to the price tag per resource.
I had a shitshow in SK just even getting service
Singapore is a big problem. The network there is so fragmented. Last I checked, Singtel owns a large portion of Singapore's networks but won't peer with certain ISPs or anyone peering with certain ISPs. This meant a Singapore user trying to communicate with a Singapore server sometimes needs to be routed through Hong Kong or another exchange before returning back to Singapore.
Don't forget SoftLayer's network in Asia. It's a real hit or miss on residential ISPs. One time a client of ours in Seoul on SK was being routed through Japan to hit SoftLayer Hong Kong. What is usually 40ms latency from Seoul to Hong Kong turned into 90ms of latency with a higher chance of packet loss during peak hours (for those who aren't familiar with Asia network, all countries are connected via submarine cables which really makes it expensive to expand bandwidth between countries. During peak internet usage you really can sometimes experience the packet loss unless you pay more. Expect higher operation costs in Asia especially since many central IX countries are currently experiencing power shortages).
And thank you very much to share your experience with us. I think that the same provider can offer you a very different experience in different regions. Even AWS is not immune to that, they had datacenters with some major outages in some US regions while in another countries almost none big problems.
In the end, what matters is how they respond to these kinds of events and issues and what actions they take to prevent those issues in the future. While I can't say for certain what issues were avoided through proactive measures, I will state that Linode has been great with setting up fixes that hopefully prevents problems in the future. DigitalOcean recently had a problem twice with their billing and control panel system going down (in addition to their API being down). They've also taken action to prevent this in the future and have been fairly open about it (one was their SFO datacenter being down if I recall correctly).
My only gripe about Vultr is that their support responses are very basic. Limited to "try again now" and "we've turned off a neighbor". Honestly I was hoping a solution that would work long-term and communication on how they're working to ensure that. DO and Linode focus on relaying that information and I feel really works with me. This is why I'm really fine with buying through them even if they're technically "twice" as expensive as Vultr. That peace of mind is really worth it.
I'm really interested in your experience with OVH - how long ago did you last use it, and what problems did you have?
I've used OVH successfully for various unimportant things for a while now. I'm about to setup a prod environment on OVH for a side project that we anticipate getting some good initial traction. So, not trying to spend a ton, but willing to forego the conveniences of AWS the best cost to performance ratio.
OVH has made a big Openstack push, it seems. "VPS Cloud n" series is KVM (Openstack), with NVMe drives, using Ceph (Openstack) storage, and a moderate 99.99% uptime (same as DO).
It's quite attractive at their prices - so I'm trying to gather as much relevant intel as possible on others' experience with them.
My experience with OVH's KVM Openstack line was when it was still maturing as a RunAbove product. I remember they had problems out the gate and at one point had regular service restarts and I think at one point had a storage problem becoming read-only. I stopped using them since then and just chalked it up as another problem with using OVH's cheapest cloud platform. They've never worked out for me.
However, for those who stuck around and also bought KVMs from OVH after they "graduated" from RunAbove, I believe it works to an extent. It's not the best, it's obviously focused on minimizing costs and maximizing possible resources and reliability, and (this is my own speculation now) I think their server density is fairly high (VMs per host node). I haven't heard of any major problems lately with their KVM platform but again, I wouldn't put anything intensive on there either.
If cost is a major factor for you, then go with OVH's Openstack KVM line. If you still need fairly solid reliability, I think going with AWS or GCP might be worth another look. If you're fine with a VPS starting out without Ceph, NVMe, etc, then I'd really suggest you hit up Linode. Linode offers some of the best resources per cost ratio while still providing a decent reliability factor. If you really do need reliable storage then I'd look at DigitalOcean's Block-Storage (also known as network storage). Once you've grown your operation a bit more and have more resources and funds, then I think it'd be worth re-visiting the idea of moving to AWS or GCP or any other cloud platforms.
YMMV as I am just one person.
If you want a more detailed response feel free to let me know.
Best of luck!
Occasionally a server will hard lock and when you got into the management web app and ask for a reboot it will say that the system is not responding and they have to send a tech to go physically power cycle it. When that happens (and it will, it's happened to ALL of my dedis there at some point) you're down for 30-90 minutes.
They provide remote reboot and a java-based ILO which can solve most weird issues, but the hard lock stuff continues to confound us.
We currently have multiple dedicated servers out of OVH BHS Canada (bare metal servers) and I have never ran into that problem. Most of the time I believe OVH gives you all the tools necessary. If I recall correctly, I believe their power-cycle feature is sending a SNMP command to the UPS, so their online control panel should technically work, but mileage may vary.
We've started moving on to Colocated hardware though. Unless you specifically need OVH's VAC system or their network (or free 256 IPs with setup fees), I've found OVH's availability of hardware and configurations to be fairly lacking. Glad it works out for you guys though!
The CEO is actually really nice and accessible, and their customers seem to really vouch for them.
gene.gaddy@ibm.com
I lost a lot of time waiting for the benchmark tests to finish ;-)
For instance, I can install VMWare and get virtualisation going on my laptop, but I wouldn't consider that the cloud. Similar, you can get bare metal computers on the cloud which aren't virtualised.
The difference is only surrounding automated deployment (cloudformation) or services that are offered (RDS)
But, they do not provide any sort of "cloud app runtime" comparable to AWS or GCE.
All of those plans that I compare there use the same technology in the the end, KVM virtualization. The differences are in the "flexibility" and "services" that they offer in order to act more as a cloud or more as a classic VPS. But the lines are very blurred these days. You can almost achieve the same things with any of them in these kind of plans. IMHO I think that you can really start talking about clouds when you're playing in the same league as AWS, Google Cloud and Azure.
I'm sorry for that, was a huge mistake, I hope that nobody have been offended by that.