Hacker News Reader

Comment by VanL | Hacker News Reader

Van Lindberg here from Rackspace. If you have any questions, I am around to answer.

* The ZeroVM site makes a big deal about application execution being completely deterministic. How does this interact with applications that require random numbers, such as crypto?

* Is ZeroVM capable of running unmodified Linux binaries? If not, what compiler toolchain is required to get it working? The main advantage of other lightweight virtualization solutions (OpenVZ, LXC) is that it's very easy to take regular binaries (e.g. postgresql) and drop them in a sandbox with minimal fuss.

* You will need to supply it with random seed. * No, you will need to recompile. We use modified gcc/glibc toolchain.

- It is deterministic based on the inputs. You would need to pass in a seed or read from an external source of randomness to get different values out of a PNRG.

- Binaries need to be recompiled. There are two toolchains, a GCC-based one and an LLVM-based one. We can also compile within the ZeroVM container itself.

We expect that a lot of people will use existing language runtimes (Python, Lua, JS) To avoid compilation.

Over the long term, though, a lot of the power comes from composability. Think Unix pipes, in parallel, across the cloud.

Does the hypervisor use multiple communicating CPUs? If so, how do the races inherent in concurrency not destroy the determinism? Is this a single CPU/thread/fiber hypervisor?

Each container has a single process. Each Individual part is deterministic, so the entire system is composable deterministically.

That's not true if the processes communicate, or contain communicating threads. In one run, an input queue looks like {A, B}. In another, {B, A}. The source of the non-determinism is just entropy bubbling up from the hardware.

EDIT: using completely synchronous I/O (mentioned below) is a very clever solution, but it requires a process to know its inputs ahead of time. This may also cause cluster scalability issues, as now each "round" of inputs is gated by the slowest of the source processes.

Assuming A and B are produced from separate input processes, they have to be submitted to a "gather" process that takes both of them. If the "gather" process uses select() or non-blocking I/O -- says basically "read from A or B, whichever becomes available first" -- you'll get them gathered in a nondeterminstic order. OTOH if the "gather" process uses synchronous blocking I/O -- "Read one message from A, then read one message from B" -- then you always get (A, B) order.

If the framework you're using requires all I/O to be synchronous, and there's no way for a program to tell time or tell when an action would cause a delay, then there's no way for nondeterminism to develop based on timing.

I don't have any idea if ZeroVM is like this, and a framework that only allows synchronous I/O would have its own problems (basically you'd have to worry a lot about deadlock).

EDIT: To expand on this, you might still be able to do a lot even if cyclic interprocess data flows are forbidden. This is particularly true of database style applications, which are where ZeroVM originated.

All reads from other sessions are blocking. There is no input queue, zerovm processes read and write directly to each other. This way determinism can be preserved even for clusters.

With the potential of starting a ZeroVM in 5ms, and running it for a very short amount of time, do you see yourselves starting to charge in smaller time increments (ms)?

I work a lot with smallish data, where queries/processing may take 20 minutes or a few hours. Often I can split this work up significantly, but when most places charge by the hour it's rarely worth it. PiCloud are excellent in this area, and Manta looks interesting (but possibly a bit more expensive) but I'm not aware of many others. I love the idea of being able to start, with little overhead, a large number of short lived jobs. Particularly if I can run them locally or my own cluster.

I look forward to seeing more on this.

The platform itself produces accounting data with 1 ms accuracy. How to do accounting in a real world public service is another question. But we aim for a very short-lived (10 seconds) but widely horizontally spread (1000 machines) workloads.

Thanks for the reply, this sounds like just the kind of thing I'm interested in, and I'm happy to see more competition in the field of renting machines for short periods. I look forward to seeing this released and giving it a go.

Hey Van!

How much did you guys pay for them? :p

Are there interpreters already available for ZeroVM? Or at the moment, must everything be compiled?

Lua, Python and C/C++ are available right now. C/C++ is compiled in run-time with LLVM. Porting an interpreter requires a minimal effort, although porting some of the interpreter libraries can be not that easy. I have compiled PHP just for fun, and it was working "out of the box".

I've read the architecture page and I am not clear how multithreading is dealt with if application demands it. If you can expand a bit on that.

If you have a legacy application that demands threads we can emulate threads by essentially using coroutine approach. http://en.wikipedia.org/wiki/Coroutine

What would be the impact of such a coroutine emulation if the threading is used to leverage multi-core hardware for high performance computing such as done by Atlas [1], OpenBLAS [2] or MKL [3]? These libraries are tuned to maximize CPU cache hits. It seems to me that executing each thread task sequentially using coroutines would probably break such optimizations.

[1] http://math-atlas.sourceforge.net/ [2] http://www.openblas.net/ [3] http://software.intel.com/en-us/intel-mkl

From the sounds of it, you'd want to split up your workload and run each subset in a separate zerovm

That's correct. And that's how most of parallel processing frameworks do it anyway.

Probably the most obvious if will work only in rackspace or can be used everywhere (and the cost)

It's open source. https://github.com/zerovm Can be used anywhere. Can be installed on top of Openstack installation.

We see this as bigger than just Rackspace. Anywhere your data is, we want ZeroVM to be there.

... and how much did you buy them for ? :)

How much did you acquire them for?

Reply on news.ycombinator.com