OnMetal: The Right Way To Scale
rackspace.com
rackspace.com
1. I want to know how much these things cost. Even a line like, "Under $x/hour." would be useful.
2. The blog post says that the memory and compute types have no disk. This confused me at first, but then I saw on http://www.rackspace.com/cloud/servers/onmetal/ that they have 32GB boot drives. You might want to add that to the comparison table in the blog post.
3. Has any thought been given to letting users provision OnMetal servers in specific cabinets? Sometimes users want to have servers next to each other for better network connectivity. Other times, they want them in different cabinets for increased reliability.
Lastly, kudos to Rackspace for building this. Most people don't realize how virtualization can hurt performance. Even without noisy neighbors, a modern Xeon still takes >500 cycles to switch between VM and host.
That said you can pull a lot of performance out of a modern 2U server, 24 cores and 256 GB of RAM can be had pretty cheaply these days, pair it with 8 SSDs and you've got a real workhorse.
Even a traditional MySQL/Postgresql DB can maintain upwards of 140,000 tps on hardware like that.
2) They do have 32GB SATADOMs, but these are intended for boot media and configuration only. They are not intended for anything more than boot media. Think something like an SD Card in a solid-state machine.
3) Yes, I want this. No, we don't do it currently.
FWIW, we didn't build all this. Ironic existed long before we came around, we added a deploy driver and set it all up and exposed it for customers. We'll have a blog up shortly with exactly how we're running this in production.
The coolest part of all this is something nobody has mentioned here yet -- this is all on Open Compute hardware. When I managed servers, I always wanted access to the cool hardware and scale that companies like Facebook and Google had. Hopefully with OnMetal, we're going to give our customers the cool hardware I always wanted.
Edit: As a note, I'm a Racker and I work on the team that built this.
It's extremely easy. Insert 512GB of data into postgres, there, done.
Ah, that's still way better than no disk. I was initially worried about having to deal with PXE. I wonder if logging will cause trouble for the SATADOMs. I'm guessing they don't do much in the way of wear-leveling.
I didn't recognize your nick at first, but we worked in the same office. I was part of the Cloudkick acquisition. I left Rackspace in August of 2012 to start Floobits. Small world. :)
You should've left off the last paragraph; I assumed you knew who I was :).
- Total cost
- Cost per core
- Clock speed (this actually matters a lot for, say, a Python webapp)
We'll be gathering feedback on the resource balance, and iterating as we go.
Virtualization is interesting for many problem sets but it does add overhead and it has it’s costs. All of these technologies are cost -vs- reward and for too long we’ve been drinking the virtualization cool aide not realizing the real costs of those configurations.
Last year I did a super low-latency high qps project on a cloud box and I was lucky to get 37k qps out of the box. I had the same exact box reloaded as bare metal and was able to get to 165k qps. Mind you this is latency in the neighborhood of 10ms and really extreme qps loads. That said the virtualization layer is not free, it’s useful as are many technologies and it’s an awesome tool for the right problem set.
I’m happy to see Rackspace coming out with this, I asked them for something like this years ago and am excited to see they’ve figured out a way to do it..
Like many have said.. Let’s see the pricing. But before you compare make sure you are really comparing apples to apples.
1) It's true that VMs generally provide bad performance for high i/o applications, particularly databases. There are various ways to mitigate this such as using high IOPS drives like AWS EBS PIOPS or SSD backed storage from Google/Digital Ocean.
2) This is using OpenCompute and is all open source and going to be released, so they can take advantage of the efficiencies of that hardware architecture.
3) It gives you quick access to physical servers connected to your cloud environment. Benefits of flexibility/scalability with the benefits of hardware. This seems to be how they're differentiating against Softlayer who have bare metal servers available via API, but they're billed monthly and take a few hours to provision (which is still pretty impressive).
However, there is no pricing announced - this will be key.
Also, will this start to eat into their managed services perhaps? Anticipating disruption?
Rackspace is definitely disrupting themselves with their cloud product and today we can see that they're not doing it half way.
The one area that this announcement doesn't address, however, is contended networking. Rack local servers with a 2G (or ideally 10G) full port-to-port simultaneous switch did amazing things for our app's performance. When you're looking at whole-app performance, you're sensitive to contention on all of cpu, io, and networking. Hadoop is particularly sensitive to contended i/o, but secondarily sensitive to networking.
edit: s/big data/large data/ -- people who say bigdata are tools
I hope a Racker will correct me if I'm wrong, but:
- With the exception of the boot device, you're dependent on non-local storage over the network.
- You're not going to get dedicated networking gear
- OnMetal is still dependent on the virtual networking layer (OVS?)
I can appreciate having more bang for the buck, but there are enough shared components here that I would be very hesitant to assume that that MTBF crosses the imaginary line where you can eschew fault-tolerant architecture patterns.
The other option is PCIe Flash, that's a solid state "disk" plugged into the motherboard expansion slot, basically. Again since that flat out state PCIe, they are saying it is local.
We're working to hook this up to our existing block storage offering as well, which will offer more flexibility to attach SSD or spinning disk backed block devices to any flavor.
Reliability is similar to an SSD. The underlying storage tech is basically the same: some flash chips, with a controller that handles wear leveling and so on. Just that instead of running operations through a SATA controller it uses the PCIe bus, which offers higher bandwidth and lower latency.
Interestingly, Apple is shipping PCIe flash in all (I think) of their latest laptops.
Our first iteration, you'll get two networks: "ServiceNet" is our intra-DC network, which is unmetered (ie, free bandwidth) and "PublicNet" which is the internet and will be billed just like we bill for other cloud server bandwidth. Collectively you get 10Gb/s across these, which you can break down however you want. To you these just show up as VLANs which will be configured on build.
We're working on hooking these boxes up to our "Cloud Networks", which are per-tenant software defined networks. If this sounds like a fun project, hit me up, contact info is in my profile.
Even better if it's standardized across competing hosting providers, or at least multiple physical locations.
OpenStack can do this -- makes a lot of sense.
I personally like provisioning hardware with a hardware-virtualized hypervisor anyway, and running admin/keys/monitoring in another VM or the dom0 while running the application in a domU, but I hate multi-tenancy, especially across organizations, but often even across functions on the same server from the same organization.
Pretty excited by this; would love to set it up in a colo cage.
I'm not even considering EC2 because I don't trust the reliability and I know that its not a good value compared to the others. EC2 only makes sense to me if you really need to take advantage of features they have like the private networking or DynamoDB etc.
I think there's a market opportunity for simplifying the implementation of online multiplayer games, especially on the back end. Right now, there's a lot of painful barriers and some apparent pent up demand.
And would it good to have a HighRam/HighIo server ?