asharp

I don't believe you understand what the salt is for.

You use a salt so that time/space tradeoff attacks are no longer viable. All you need to do is provide a few bits (say 20) of entropy to make them infeasable.

I greatly doubt that it is faster to compute on the fly then to use a rainbow table.

Sure it matters.

If you don't have a reasonable salt then you're vulnerable to time/space tradeoff attacks like rainbow tables/etc.

Also, in case you haven't used it. Scrypt is pretty damn awesome. It's designed to be memory hard, which makes it very hard to crack even with GPUs.

Everybody wants us to sell it. We currently arn't aiming to, though.

We are wholesaling this though, and we are looking at massively stepping it up. If you're interested, come talk to us.

Isilon has damn awesome tech, albiet stupidly expensive. Atmos is absolutely horrific in most senses of the word.

iirc. Ninefold has a bunch of "cheapie" Dell EMC units, hence their troubles awhile back. Don't quote me on it though.

CloudCentral did in fact build a ZFS San, which they used to great effect.

Yep. Best path through is via SeMeWe3 to singapore which is fairly fast. Most people end up with bw through PPC1 or SX which both end up in the US. People mostly do this because SeMeWe3 is stupidly congested and expencive compared to say PPC1, and most traffic goes through to the us rather then europe/asia.

Also remember that PPC1 when it stops over in Guam can end up coming back into asia via Asia-America gateway/etc. I'm not sure if any of the major providers actually backhaul through there.

I agree.

It would be nice to see one or two more innovative startups around the place though.

Sigh. Some things never change.

Ninefold is cloud.com on top of some EMC sans. CloudCentral is very modified cloud.com running on top of a damn cool san. Most of the others are all Vmware and such, all on fairly standard hardware.

And then people wonder why their support costs are so high (hint: Cloud.com isn't exactly quality), and why they can't maintain profitability (Those $1m sans....)

But yeah, we're a little different. Think of Orion this way: - Take a standard HPC cluster - Make its DSAN tankier and more customised to deal with virtual disks - Run (very heavily customised) Xen on the (very horribly customized) nodes - Build on a full cloud orch stack on top.

This lets us have a better experience then any other stack, whilst being orders of magnitude cheaper.

So you can have cloud servers with hard disks faster then dedicated servers, that are HA under hardware failure, and that have proper layer 2 private networks between each other and the outside world. All for cheaper then the standard dell server + EMC san arrangement most people end up with.

We have a thriving VC community in Aus, with lots of awesome people and stupid amounts of money.

It's for Mining/Agribusiness.

Now trying to get a Tech VC community bootstrapped is going to be interesting, mainly because we seem to be stuck in a: Vcs are an oligopoly -> Nobody wants to go through them -> No new companies are formed -> No new VCs start here -> .... Cycle. The influx of a whole new environment to nurture startups recently seems interesting. Things like Fishburners, Silicon Beach, etc. Are doing good things incubating new startups adn getting them networked, but it stands to time if this recent upswing will get us out of the cycle.

I'd give you that.

Our VC market is very good for farms and for mines and for not much else. It's changing, but slowly.

Going to SV makes this so much easier, but it means that you have to basically expand out of Aus, which is harder said then done.

Here's to hoping things get better in the future :)

Not exactly innovative? Interesting.

Would you mind telling me what gave you that impression?

The most common reaction we get is that we should sell off our tech and not bother being a service provider. Mostly because we are, at the end of the day, a tech company.

Just to take a moment to explain.

From the point of view of the consumer:

- I want to make a virtual server on demand. It must be HA, must have hard disks that are reasonable and I need out of band management in case I lock myself out. - I need a proper virtual network on demand. I need to be able to set my own ip addresses, use multicast (say for db replication for rack), etc. I want to be able to have multiple virtual networks, attach multiple networks to the same machine. I want to be able to connect up over MPLS or a crossover into one of these virtual networks.

Basically, people don't want randomly crippled servers. They want a virtualised datacenter. Can any other provider do this, at scale? We call this the virtual datacenter paradigm and we think that this is the future for cloud.

It's also basically impossible to build, which is surprisingly convenient.

Disclaimer -> I'm Alex Sharp, VP Development at OrionVM.

You just need to do rolling deployments to make simple things like that go away.

Roll out a deployment to N machines (like say 10), run self checks (you have those, right?), if everything passes give them standard load. Over the next K period of time, periodically check up on them. After that, roll over to N*2 or N^2 nodes, continue until you have rolled out to your entire cluster.

What are you made of?

Either way, if you are using renewables, you are going to be using a baseload renewable power source (geothermal/hydro) that isn't going to care about temperature.

What you want is access to cheap electricity and cheap cooling. Whatever minimises TCO is where you want to be.

Very few workloads need to scale out past the limits of any reasonable cloud provider.

Properly built clouds can give you IO faster then a dedicated server, and give you CPUS on demand that far exceed what you would otherwise afford (due to you only needing it occasionally).

And any reasonable cloud provider can discount things such that is unreasonable for you to move off. They have a much lower bottom line then a comparable dedicated server provider.

Is interesting that this doesn't seem to be the case currently.

Just a question.

Who in the US provides cloud with disks comparable to dedicated servers? (or otherwise 'good')?

I say this because I run a cloud company called orion in Australia, we produce cloud VMs with faster then dedicated disk performance. When we were last in SV pitching, nobody knew of anybody in a similar sort of space.

I'm just asking because you seemed to emphasise 'normally' and 'good'.

Remember that most loads are 1:10 inbound:outbound, so unless you have a strange sort of workload it really doesn't help you much.

If you do happen to have a lot of inbound traffic though, it's a nice little arrangement as the bandwidth may be almost free to the provider (Symetricity requirements) and still very useful to you.

Your point about having multiple VM's forming additional seeks is correct, but it misses the bigger picture.

Basically, on any reasonably sophisticated hosting infrastructure those arn't a problem.

If your problem is just seq->random conversion due to additional VMs then bcache/flashcache does a surprisingly good join at making that just plain go away with little additional cost.

On any reasonably sophisticated host you have a DSAN, which has a cool little stats trick, which basically means that as you add additonal vm's together the variance on the total IO load drops, and the load pattern itself becomes more and more normal, the larger and more uncorrelated you get.

That gives each vm more 'burst' capacity when required with many fewer failures (ie. the vm asks for more IO then the current capacity of the system).

This leads to a bunch of interesting stuff when you try to apply it in real world systems, either in HPC or in clouds.

Not for most situations.

I work over at orion, and our standard cloud VM's dominate dedicated hardware in IO. There really isn't much competition.

You find most of the 'virtualisation is bad for I/O' is either 'oversold I/O is bad for io' or 'Trying to push all IO for every VM on a box as well as all WAN traffic through one gigE port is bad for io'.

Interesting.

Also interesting what is simply an artifact from the fact that none of the current "clouds" out there were built to deal with, well, actual loads.

Some things are small, but seem rather strange. Why does no cloud give out 95/5 billing? Why isn't there more resource limiting/etc?

I see a bunch of things leaking out of EC2. People forget that EC2 was designed to deal with large numbers of stateless servers and it's not good for much else. They take the limitations of that and the rest of the AWS platform and apply it to the 'cloud' overall.

Two examples would be from the 'variability' section. CPU limiting under XEN (the hypervisor used by both Amazon and Rackspace) is trivial. The fact that CPU is so variable, especially for smaller tiers, is thus rather interesting.

Similarly with IO. With Rackspace, you are on local disks. As such, unlike Amazon, Rackspace has no defendable reason for being able to starve other users of disk IO.

Also, just as a general data point. There is no real reason why a cloud should be in the same order of magnitude of cost as anything you could touch. Fairly simple reasoning, everything they buy is at massive scale, and there is a very minimal fixedish management cost to deal with all the hardware. What you can work out is that even given almost list prices you are still looking at thousands of percent ROI on cloud servers. What that then says about the market is that there is a current monopoly due to a lack of cloudsmithing knowhow which is the cause of the current situation. Over time, I would expect cloud products to simply dominate standard dedicated servers/colocated servers for most applications.

Do any American clouds give you 95/5 billing?

Amazon.com was only recently transitioned, but AWS was used to host ecommerce apps for partners, iirc. before it's public debeau. This is what it was built for.

Two points.

One, datacenter failure happens with any hosted application that runs within a single datacenter. Most other types of hosting also have a litany of other failure modes that go away with cloud, to be replaced by other modes. For example, you no longer have to deal with failures of individual hard drives, but now you have to deal with failures of EBS clusters.

Two, cloud provides a standard system that makes HA easy. Rather then having to do dual dc failover in hardware and having to deal with bgp, anycast ips/dns failover, dealing with splitbrains and hw stonith/etc. All of this is just taken care of for you and/or made a lot easier.

I know of this. The interesting question it brings up is how do you keep a cloud like this patched and up to date without dropping SLA?

Interesting choice, and I'd imagine that it sidesteps all of the balance issues. On the other hand, what now separates you from a VPS provider with an API? (like say Linode)

Interesting. I agree with your approach. When I was looking at KVM, I noticed its rather insane surface area (Everything is in the kernel as a kernel module) and that almost all of the vulnerabilities found seem to stem from that arch decision. As an example have a look at this: nelhage.com/talks/kvm-defcon-2011.pdf

I was just wondering if you knew of any way to secure that down, or any way to patch quickly enough that you don't break SLA by having to forever reboot people.

Just generally given that we've had CVE-2011-2212 CVE-2011-2527 CVE-2011-1751 CVE-2011-0011 CVE-2011-1750 for kvm itself in the last few months, so say you have 5 critical bugs in kvm a year. If you need to restart everybody to patch and it takes a minute or so to restart per vm and you have say 40 vms/box (random numbers for the sake of argument), that's 3 hours of downtime/year not counting kernel upgrades/etc. That means the best you can do is 99.9% uptime not even considering transit failure/dc power failure/etc.

So my question is how do you deal with that?

> persistent disks How do you then deal with people who create an instance and then don't run it. Unless i'm mistaken, you'd be forced to either unbalance for storage or VM usage.

> cgroups currently How do you protect the kernel from something like CVE-2011-2212?

Quite cool otherwise :)

> No Sans

Ephemeral disks? Or persistent local?

> Our own stack

Very cool ^_^ How do you deal with geographic zones? Are they silod?

> Bandwidth

Ok, makes perfect sense. Thanks.

> Securing KVM

Do you use cfgroups/selinux to deal with compromise of a kvm domain? I've seen quite a few vulnerabilities coming out on the debian/etc. security mailing lists.

Amazon has some pretty damn amazing uptime overall. Given the number of regions that are sitting at 100% uptime (like say, all of s3) it's hard to argue that they have uptime problems.

They really do need to work on getting ebs to cold boot properly though. It seems to have a habit of biting them. (But there's a good chance that anybody else at that sort of scale would have the same sort of issues)

Interesting offering, looks damn cool ^_^.

There are a few interesting questions raised, though.

Are you guys using local disks or SAN?

Are you guys using Eucalyptus?

Also, it's interesting that you charge less for incoming data then outgoing. I understand that standard asym links are cheaper upload then download, due to them being basically "slack" cap from Adsl tails/etc. Why is your outbound data more expensive then?

How are you securing KVM?

(assuming somebody from there is around, or that anybody else would have answers....)