Fission: Serverless Functions as a Service for Kubernetes
blog.kubernetes.io
blog.kubernetes.io
> To optimize cold start overheads, Fission keeps a running pool of containers for each environment. When a request for a function comes in, Fission doesn't have to deploy a new container -- it just chooses one that's already running, copies the function into the container, loads it dynamically, and routes the request to that instance. The overhead of this process takes on the order of 100msec for NodeJS and Python functions.
100 ms of overhead is pretty steep. Google recommends a limit of twice that for the entire server response (https://developers.google.com/speed/docs/insights/Server#rec...):
> You should reduce your server response time under 200ms
Still, I've heard that AWS Lambda may run around 200 ms, so perhaps Fission is doing OK here.
But with modern progress in fast-starting compiled languages like Go and Rust, I'd love to have some easy way to supply small static binaries instead of source code. With compiled web server frameworks, it's feasible to handle over 250,000 (very simple) requests per second, or thousands of requests while Fission is trying to figure out where to send your source code. It doesn't always make sense to pay for interpreter startup overhead on every request (EDIT: this is not the case according to the Fission developers), especially not now that compiled code is so easy.
If you were getting even 1 request per second, the container last used would remain live, so the response time would then be in line with normal responses from a typical NodeJS server running the same code.
I agree though, supplying binaries in something like Go or Rust will be a very nice addition.
About that 100msec overhead -- there's a bunch of optimizations yet to be done, and also we think we can reduce the median at the expense of tail latency; but for now 100msec is good enough performance for a good enough set of use cases (but of course we'd love to hear from people for whom it really doesn't work at all).
Also, we could add knobs that let you tune how long a function stays "warm", so you can tune the latency overhead vs. resource consumption trade-off.
> But with modern progress in fast-starting compiled languages like Go and Rust, I'd love to have some easy way to supply small static binaries instead of source code.
We're very interested in this use case. We're redesigning the function environment stuff right now to better support compiled languages, and allowing users to bypass the source-code level stuff is quite possible.
BTW, to be clear, for a compiled language Fission wouldn't compile source code at runtime; there would be separate build stage that would run when you add the function to fission. I'm also not sure why you say we pay interpreter startup overhead on every request (we don't).
Isn't that just what "regular" Kubernetes is, except that you wrap a Linux image/container around your static ELF binary? The Stack build tool for Haskell has an option for building as a Docker image, wrapped around the static Linux ELF binary that it would otherwise produce. I assume something similar exists for Go and Rust.
Disclosure: I work at Google on Kubernetes.
While this probably keeps the majority of the requests warm, I would expect that there would still be some cold ones every now and then. Amazon doesn't provide a guarantee in how the containers get recycled.
This is just me speculating, though.
[1] - https://www.iopipe.com/2016/09/understanding-aws-lambda-cold...
So in the old days there was a way to supply a binary rather than source code. That binary could be fast starting. There was even an open standard for connecting those binaries to different web servers. The standard used unix patterns such as reading stdin and writing stdout.
Sometimes it seems like people have created an insanely high learning curve to run their CGI programs. Sometimes it seems like most of the magic and value add of some of these platforms is that they reduce some of the overhead they added.
We've been planning on supporting other DIY/non-AWS offerings in this space - particularly IBM's OpenWhisk[1]. If the Fission authors are here - can I ask how the two stack up? OpenWhisk's ability to write your own custom even sources seems like a leg up, as does their wider language support, but Fissions's Kubernetes foundation seems preferable. Curious how the two compare!
[0] https://github.com/Miserlou/Zappa [1] https://developer.ibm.com/openwhisk/
We're working on a detailed comparison with Openwhisk, but basically -- we bet on Kubernetes as a good foundation to build this on, and it seems to be going well so far. It's true that we're at an earlier project maturity stage than Openwhisk is. In particular, we're actively working on integrating event sources and adding support for more languages.
Can you talk more about Fission Router? This seems to be where most of the hand-waving is happening right now :) AWS API Gateway doesn't get very much love, but it is actually a very important part of the server-less ecosystem.
Why did you write your own Router and not make an nGinx plugin, for instance? How well does it scale? How do you deploy it? What features does it have?
Feature-wise, fission router is very simple right now. It's a service based on gorilla/mux. We had to write our own router because of cold-starts -- basically, the router needs to be able to talk to the pool manager and to get an instance when one doesn't exist, while holding on to the incoming request. (I'm not sure how much of this is possible to write as an nginx plugin.)
I don't have perf numbers just yet, but the router is stateless and you can just run more replicas when you reach its limit. It caches results from the rest of fission components, and most requests are on this fast path.
(We're also thinking about adding some existing feature-ful API proxy in front of the router, to get stuff like authentication, rate limiting, etc.)
So if one goes with statically compiled web services written in e.g. Go, Rust, Haskell or any other language supporting static binaries (most languages support this actually, but not all of them encourage you to do it), the only thing missing is a deployment framework that distributes your binaries to your (potentially virtual) machines. So am I correct that this is the point where Kubernetes comes in?
So how to use Docker in production is a controversial topic and the consensus seems to change rapidly, but in my opinion is mostly used as a weapon against dependency hell, but this problem is mostly solved using static binaries. Other use it for sandboxing, but I am not sure what security guarantees one can expect here.
If one compares static binaries with Docker containers how do they compare from a security point of view if one is hyperparanoid that the code executed may harm the host?
IMHO configuration management overlaps with dependency hell, or could be seen as a dependency. You do not have to bake the dependency into the binary, but just have to setup the runtime environment.
Of course, the perfect solution is to upstream the necessary changes into the binaries that you need, so that you can supply all configuration via args and env.
I have a couple of questions, maybe someone here has answers for me:
1) In particular I'm wondering about expensive one-time/setup operations (e.g. opening a database connection) that are usually done once at startup of a service. But that doesn't really apply here. How are these things handled?
2) As a Java guy I also wonder which of these tools support Java and how they deal with dependencies and JVM startup times etc.
3) And my last question: How do these integrate into development workflows? Continuous integration? Run this offline/local etc. Will I need a Kubernetes instance on my machine?
2. We don't support Java yet, but we'd like to. The JVM would already be running by the time there's a request, so there won't be a JVM startup in the request path.
3. Dev workflows: Lots to do on that front. A big part of our roadmap is improving the dev workflow, better versioning support, CI/CD.
For running locally, yes, the best way is to run a local Kubernetes instance on your machine. minikube makes it very easy to do that, see the instructions in the README: https://github.com/fission/fission#setup-kubernetes
While one cannot rely that all executions after a cold start are warm, it's possible to for the action code to check if it's warm and forgo internal initialization as needed. This is a convenient trick that's common to avoid expensive setup.
For Java actions running in OpenWhisk, this can be done with static fields.
A nice flow of the lifecycle can be seen on slide 24 here http://www.slideshare.net/psuter/openwhisk-deep-dive-the-act....
I've seen a couple of issues and I agree that it would be fabulous to have OpenWhisk integrated into OpenShift/Kubernetes.
Should you need help with OpenWhisk please reach out to us on http://slack.openwhisk.org/ or GitHub.
You manage dependencies by bundling them up in a JAR or zip file (handled by Maven or Gradle)
Static assets can be kept warm between invocations (nice for things like DB connections)
If I keep DB connections for example am I now relying on an implementation detail or is this documented behavior? I realize that this will be different for every system.
I'm sure they use your invocation history in deciding how long to keep them around.
(You watch this issue if you'd like to stay up to date on progress https://github.com/fission/fission/issues/24)
`fission function create --name hello --env python --code hello.py --route /hello`
The `function` function, doesn't seem support route, and should be broken up into two parts:
`fission function create --name hello --env python --code hello.py fission route add --function hello --url /hello`
Taking a look at the help menu for function it's not there, I'm running:
`fission version 0.0.0`
[0]http://blog.kubernetes.io/2017/01/fission-serverless-functio...
$ fission function create --name hello --env python --code hello.py --url /hello
On slightly different note, I am still looking for a good job pattern for my use case. I run jobs on very custom docker images (including deep learning and computer vision stuff). I currently use redis queue with todo items, each pod in deployment picks up items from the queue in the loop, but I dislike some things about it. Fission is not the good fit, but I am wondering if HN community would have some suggestions. I thought about kubernetes jobs, but they have some things I don't like.
On one hand, celery and python-rq do not work well big custom jobs (they are more suited for small tasks, but too big to run on the web server). On the other hand, using raw redis queue is finicky if I want to implement custom things like retries, timeouts, etc.
Disclosure: I work at Google on Kubernetes.
Alternatively, all Jobs matching this query must succeed before this job. Failures can be OK if a configurable threshold of success is met. Exceeding the thresh fails the dependent jobs before they get to run.
You can do all of this with supervisor pods, but since you asked...
- Retry control is course grained. IIUC the only option are "no retry at all" or "terminate if job have been active for X seconds". If my jobs tend to be long and failure happens early, second option can lead to hundreds of retries. Each job retry is expensive, as it creates a new pod. I remember ending up with 100s pods.
- As mentioned previously, retry count limit to job/pod would be useful.
- Minor nit: Deleting old job with the pod is cumbersome with current python API. Either delete_namespaced_job with orphan_dependencies: False, but I would need to check if job exists beforehand (deleting if job does not exist fails). On the other hand delete_collection_namespaced_job lets me delete job by label selector, but I also need to manually clean up the pod.
- Another nit: When exporting stackdriver logs collected by GKE from a job to big query, each log ends up as a separate logsource.
If you can model your data flow as pipelines that run when their inputs update, you might interested in Pachyderm (https://www.pachyderm.io/). This uses Kubernetes and Docker images to run multi-stage computations. It's still a bit rough around the edges. (I wouldn't recommend the current version if you work with multi-gigabyte binary input files, though they're currently working on fixing an issue affecting that use case.) They do have some examples (http://pachyderm.readthedocs.io/en/latest/examples/readme.ht...), including one using TensorFlow.
We're experimenting with Pachyderm at work, and the design feels "right" for our use case.
What you're suggesting [initially] is very similar to what the folks at iron.io did but I'd really want to ship containers all around, not pieces of code. Iron.io addressed that a little too late in my opinion and their feature set around docker leaves a lot to be desired.
IMO, AWS Lambda and Google Cloud Functions are going to support docker in the near future, they have to if they want their ecosystem to be adoptable by all.
Disclosure: I work at Google on Kubernetes
Still, it's part of the puzzle.
http://blog.alexellis.io/functions-as-a-service/
Long version:
> FaaS is a platform for building serverless functions on Docker Swarm with first class metrics. Any UNIX process can be packaged as a function.
> Enabling you to consume a range of web events without repetitive coding.
We've got better functionality, streams ( not a buffered context ), more language support, and better response times.
On the upside, this has led me to a good practice: When I do a website for one of my apps, the first few sentences will state in the utmost clarity what the software actually does.