Effe, a building block for an open source AWS lambda | Hacker News Reader

Effe, a building block for an open source AWS lambda

redbeardlab.github.io

42 pointssiscia23 comments

The power of Lambda comes from being able to use other AWS services. Lambdas on their own are not very interesting.

A true Lambda alternative, open-source or otherwise, would also have to offer alternatives to DynamoDB, S3, SNS/SQS etc. Without those you're basically reinventing CGI.

I've been giving some thought to an AWS Lambda equivalent on Joyent Triton (largely because it's open source). You'd have RethinkDB (DynamoDB), Manta (S3), your choice of queue, TensorFlow, etc. I'm generally a huge fan of the Joyent engineering ethos (not affiliated, just a fan).

Python 3.5 with DTrace probes in an AWS Lambda equivalent would be sweet.

Something running on Kubernetes is probably going to get more traction at this point than something running on Triton. 1.2 added a task API, probably not high performance enough yet for this, but I think it's only a matter of time till the ability to execute ad hoc tasks is a core platform expectation.

All of those services have very valid Open Source alternative, why be locked in a closed environment ?

So the point of Lambda is that you don't have to run any infrastructure.

If Lambda had an open source equivalent, the "other pieces" are also needed to make it useful. The "Open Sourceness" is only one factor in that environment - it is at least as important that you can somehow call them from a stateless service and get adequate performance.

That tends to push towards all the services being run in the same environment (although it doesn't have to be like that - it's just easier)

Because you now either have to run those open-source alternatives yourself or be at the mercy of another vendor to provide those for your as a service, and personally my trust in pretty much any other vendor would be much lower than my trust in AWS by default.

It's a godsend for me, and I don't use anything else on AWS. The value is that I can run arbitrary programs written in any language, and have it automatically scale up without having to think about VM instances. In my case there's a single (quite complex, cpu intensive) commandline C program that I need to run between zero and hundreds of on demand - I haven't found anything else that lets me do that.

What are you using to call your C program? I've been considering possibly a python wrapper?

Yes, did some research and found that the Python run time is the fastest to start up of the various options - all it really does is start up, call the C program, then shut down (<10ms to get to the subprocess.call() if memory serves).

Out of curiosity, what does your C program do? I don't imagine yours is a typical use case if you don't use any other services (not even S3?) and when you consider that you're limited to 300 seconds of running time and not more than 100 concurrent executions.

I'm going to be purposefully vague to avoid doxxing myself, but it's a lot like the "run" button that compiles the code on the Rust homepage, the difference being that the compiler I'm running is too performance critical to use Emscripten ATM.

As someone who's using Lambda in production today (successfully, at relatively high volume), I'd like to suggest that what most would-be Lambda alternatives miss is the infrastructure piece. Talk of lock-in notwithstanding, running your code on Lambda requires minor changes at most (assuming it's already structured as a callback). The real value is not having to manage EC2 boxes via any abstraction other than the Lambda service itself. You set one knob (CPU/RAM). It does the rest. That's the part that adds value for me as a developer and an operator.

This looks interesting but the screencast video kind of failed to make the point in my opinion. Take a look at serverless talks or apex intro https://medium.com/@tjholowaychuk/introducing-apex-800824ffa... for how to prepare a better illustration that proves your point.

> Isolation is not a real issues today, containers can provide complete isolation between runtimes and much more.

Most people will not necessarily agree. Containers don’t provide complete isolation and from a theoretical standpoint they never will (unless we rewrite the Linux kernel). There have been security vulnerabilities in the past about this and there will be, it’s just the way it is. Your argument here should probably be ”it's safe for single tenant use“ i.e. all the code runs on effe is your company's code.

Also coming back to the video again, I just can't fully understand what this project does, it just looks like a Go web server framework.

You could use the same logic to say virtualization isn't safe, but it doesn't stop cloud providers from putting multiple people's VMs on the same instance. Regardless of the technologies you use in your datacenter, there will be vulnerabilities that will need to be found, patched, and fixed.

The point is to have a single binary for every single endpoint you need.

Now you can deploy those binaries in whatever platform makes sense for you and scale those endpoint independently.

All being open source.

If you're tied to AWS then Lambda is great. If you don't want to be tied to AWS or if you need more control over resource allocation/longer running jobs then that's what all the emerging "container engine" services are for.

"Bring to you by RedBeardLab"

I would have someone proof read this for you. It's hard to go through this introduction as it is.

Since they are likely non-native English speakers (from Italy), a few grammatical mistakes are forgivable.

Is the "Bring to you by RedBeardLab" intentional or something?

> In November 2014 AWS Lambda was released and it was nothing short than a revolution.

still not good enough to be reliable in production, more expensive than running on micro instance or digitalocean.

Lambda needs a serious makeover soon or it risks developer abandonment. It's great to lock yourself in with a cloud provider. That's the only serious benefit you get. You still need to figure out how to scale properly, getting on AWS helps but it's not a plug and play scalable thing.

I've largely abandoned development on Lambda because you have to upload your code every time which is annoying. Not only that, they randomly take 5~10 seconds to respond. Also unhelpful are service error messages with no explanation or help from discussion boards. I was told to purchase their support license.

So it seems like AWS is locking in developers and companies with a non user friendly documentation and arcane hidden pitfalls then leaving them no choice but to purchase support since the cost is cheap (pay to play right?) until you add up the effort of integrating with AWS and the cost of lock-in....

AWS lambda is great for prototyping but it doesn't have the tools or a proper development guideline to utilize it. There doesn't seem to be much enthusiasm from AWS either, and if they pull the plug, well hope you didn't build any mission critical apps on lambda backends because you are going to have some considerable downtime.

Can you share some more details about what you're trying to do with us? There's a latency floor when using the API Gateway of about 200 ms, which is unacceptable for a lot of APIs, but other than that we largely haven't been running into the issues you're dealing with. And if Meetup is any indication, AWS seems really invested in Lambda. (In terms of cost, it really would depend on what you're using it for.)

well I built a user authentication and it takes 5~6 seconds for a real simple operation. it happens randomly and no way to debug it.

the opacity of being able to peer at what's going on in the backend is a real problem with AWS Lambda. I think that this lack of insight is going to bite down the road. Even now, I'm getting 'service error' with NO EXPLANATION what happened.

No offense, but this sounds exactly like the experience I had, and many friends had, before we actually learned how to use the service. If I had to guess, you're using the smallest possible 128MB config and not calling it very often. This presents a maximally-bad first experience for devs by Amazon but is not reflective of how it works once you get some actual utilization happening. All that said I moved to containers on Azure for unrelated reasons. But Lambda works fine (within its obvious architectural restrictions) and they'll hold your hand for a $49/month developer service plan fee.

Reply on news.ycombinator.com