Hacker News Reader

Comment by foxtr0t | Hacker News Reader

So, to summarize, you should:

1. not use the programming language that works best for your problem, but the programming language that works best with your orchestration system

2. lock yourself into managed services wherever possible

3. choose your api design style based on your orchestration system instead of your application.

4. Use a specific frontend rpc library because why not.

...

I've hacked a few lambdas together but never dug deep, so I have very little experience, but these points seem somewhat ridiculous.

Maybe I'm behind the times but I always thought these sort of decisions should be made based on your use case.

EDIT: line breaks.

View on HN

The way read above comment is - If you can live with following limitations, then use lambda/serverless, works great. I have got to a point where for any internal systems used by internal users, lambda is my defacto standard. Very low cost of operation and speed to market. For anything that is external facing I prefer not to use lambda, specially if growth of usage is unpredictable.

You are not wrong. But it is all about saving money on labor. The rest are just the constraints of the system you use. (Aka requirements) its like complaining about the need to use posix for Linux.

I see your point though POSIX imposes very few (if any) architecture decisions on application developers. The kind of design choices we’re talking about are very different from those of POSIX-like utilities so I’m not sure if that analogy is a good one.

I feel like we're just transferring the labour from ops to dev though. Where I work we still haven't got as good a development workflow with lambdas as we did with our monolith (Django).

Paying one guy is cheaper than two. So... You are not wrong :)

Unless your work doubles and you need to hire another dev.

No, that is always the way of work: reduce workers, increase load. We in IT were just spared from it so far. Now it arrives too.

> Paying one guy is cheaper than two.

Now that guy's workload doubled and needs another colleague to be able to deliver.

I think you're right about this.

Optimistically, it could represent a positive trade-off that replaces perpetual upkeep with upfront effort, and all-hours patching and on-call with 9-5 coding.

In practice, I think a lot of those fixed costs get paid too often to ever come out ahead, especially since ops effort is often per-server or per-cluster. The added dev effort is probably a fixed or scaling cost per feature, and if code changes fast enough then a slower development workflow is a far bigger cost than trickier upkeep.

Moving off-hours work into predictable, on-hours work is an improvement even at equal times, but I'm not sure how much it actually happens. Outages still happen, and I'm not sure serverless saves much less out-of-hours ops time compared to something like Kubernetes.

20 years ago. We had enterprise java, it’s still “there”, but running spring is very different from what it used to be.

You’d simply upload an ear or ear and the server and deployed would handle configuration like db etc.

It worked perfectly (ear/war, the persistence framework was too verbose and high level imo, but that was replaced by hibernate/jpa). There was too much configuration in xml, but that could easily be replaced my convention, annotations and some config

Again.. we are running in circles, and this industry will never learn, because most “senior” people haven’t been around long enough.

> Again.. we are running in circles, and this industry will never learn, because most “senior” people haven’t been around long enough.

And that likely won't change in our lifetime, given the rate of growth in demand for software: we literally can't create senior engineers fast enough for there to be enough to go around.

As an aside, I have the privilege of working with a couple of senior folks right now in my current gig, and it's pretty fucking fantastic.

The percentage of seasoned engineers is so low that 'senior' as a title often seems to stretch to "whoever is most experienced around here". That's probably fine, since people understand that experience is not reducible to that title. But this does bring to mind a metric for finding "objectively" senior engineers:

What's the biggest idea you've seen abandoned and then reinvented with a new name?

Cooperative multitasking => Node

That's so true and I'm happy people begin to realize that.

The worst for me is the vendor lock in, directly followed by the costs.

There are some HUGE benefits to this type of architecture (services + lambda where required) for large corporations, the main one being an insane reduction in a bunch of worthless crap that you no longer have to do:

- OS version patching, patch windows & outages, change mgmt relating to patching and reporting relating to OS patching

- antivirus, the same patching management and reporting as above

- intrusion detection / protection, host based firewalls, the same patching and management as above

- Other agents (instance health monitoring, CMDB, ...)

- Putting all this junk on a clean OS image any time something changes, re-baking and regression testing everything

This all adds up, and can be a significant cost to an organisation - team(s), licenses, management, etc.

That's basically why we're doing it, and we're seeing some really good costing implications from implementing various things using Azure Functions. Not perfect, but an extremely capable system, and there are things some of our teams are doing with Durable Functions that I was completely stunned by when they explained how it all works. Microsoft have some very good technology there.

The only thing I'm sad about is that you can only host it on Azure. I'd love an open standard for function-like programming and hosting.

Of course, not everything is suitable for this model, and it certainly won't be the most cost-effective or performant way to do it for everything either.

I think the comment is exactly opposite of what you are suggesting.

The comment is saying that Lambda has limitations and works best when considering those limitations. If those limitations don't fit your use case, you shouldn't be using Lambdas - or, at least, don't expect it to be an optimal solution.

Think about serverless as framework-as-a-service. It has a learning curve, but if you buy in, it is an amazing productivity boost.

(If Reddit’s video hosting being built and operated on a serverless stack by a single a engineer won’t convince you, I don’t know what will.)

Until he leaves and some poor other dev has to dig into his stack

90% of Reddit videos are unwatchable for me : they start OK then the quality is downgraded to something unwatchable and there's nothing I can do about it.

I even tried to download them with youtube-dl but it doesn't work.

100% agreed. The initial buffering time on them is ridiculous. I've started uploading to streamable and just posting that link rather than upload video straight to reddit.

Makes me wonder in how many other [recursive] ways progress is held back by a lack of it.

If Reddit video is the poster child then serverless is in big trouble.

That thing almost never works.

They've already ruined React!

In 2010 I was 24 yr old and built a myspace clone SaaS with music and video hosting with everything it implies: large uploads, background jobs, compiling an nginx patch to support range requests, ajax to have videos and music playing while browser, with Django on a 20 bucks/month server. If you're not convinced I don't know what will.

that sounds cool, what happened to it? that'd be a fun project to work on today :)

I just had no marketing, no partner for that, but I could rebuild that a similar time frame if only I had a plan to make profit out of it. Made it into a local newspapers but that's was the beginning and the end. It's from the times where I was finding musicians in the streets to pay me 50 bucks per website and hosted them on grsecurity hardened gentoo ... which I don't practice at that cost anymore of course. https://www.charentelibre.fr/2010/02/09/article-5-internet-a...

I think the point is, with your Django approach, you'll be stuck doing ops work 9-5 once you start getting customers, whereas with serverless you can spend the time to do more feature dev work.

Not quite sure about that, nowadays I require a couple of 20 bucks/month servers to have one CI/staging and one training/production deployments. I'm at the point where I practice CD as in "deploy each git-push to branchname.ci.example.com and run cypress on it" and still am able to deliver in half a day what the customer would expect to happen in two weeks.

And of course, baremetal provides a much better ROI than VMs/VPS/<insert glorified chroot/jail here>.

You seem to have gotten your deployments down and I really think that's good. In my own experience, though, managing your own infra always works well until it doesn't anymore. And when it stops working well, it crashes and burns and sucks up all the time. Going with managed services like serverless helps to get around that.

> I've hacked a few lambdas together but never dug deep

Then why comment? You clearly don't understand the use-case that AWS fits.

I've had jobs that took 18 hours to run on single machine finish in 12 minutes on Lambda. I could run that process 4 times a month and still stay within AWS's free tier limits.

For the right workloads it is 100% worth realigning your code to fit the stack.

>Then why comment?

Because the instruction list from above isn't backed with any solid reasoning and because commenting is what people do on HN.

>You clearly don't understand the use-case that AWS fits.

Pray tell, what is this enviable use case that I so clearly do not grasp?

>I've had jobs that took 18 hours to run on single machine finish in 12 minutes on Lambda. I could run that process 4 times a month and still stay within AWS's free tier limits.

Ok I'll bite. What takes 18 hours to run on a single machine but finishes in 12 minutes on Lambda.

An unoptimized query?

Why would the processing time differ? Would you have multiple lambdas running different subsets of the unoptimized query?

You could achieve the same with basically any decent concurrency model on a single machine.

Maybe one lambda is timing out to fallback, slowing sagas down. I don't know.

I worked on a service a year ago that would stream a video from a source and upload it to a video hosting service. A few concurrent transfers would saturate the NIC. Putting each transfer job in a separate lambda allowed running any number of them in parallel, much faster than queuing up jobs on standalone instances

If you throw more resources at a bottlenecked problem, it will go faster.

Right, but without the lambda infrastructure it would be infeasible from infrastructure and cost perspective to spin up, let’s say 10,000 instances, complete a 10 minute job on each of them, and then turn them off to save money, on a regular basis

Isn't that also possible with EC2? Just set the startup script to something that installs your software (or build an AMI with it). Dump videos to be processed into SQS, have your software pull videos from that.

You'd need some logic to shut down the instances once it's done, but the simplest logic would be to have the software do a self-destruct on the EC2 VM if it's unable to pull a video to process for X time, where X is something sensible like 5 minutes.

Yes but running multiple lambda jobs in parallel would still add upto more time than 12 minutes. What am I missing?

If I was running 10,000 transfer jobs in parallel and the longest of them took 12 minutes, the job would take 12 minutes

Yes but you are still charged for 18 hours of compute time?

That’s true, the cost needs to be factored into the model. But the near infinite bandwidth scalability allows the service to exist to begin with. If every job saturates your up and down bandwidth and takes 10 minutes, and you have 100 coming in a minute, you would need to design a ridiculous architecture that could spin up and down instances and handle queuing on the scale of thousands based on demand. Or you can write a simple lambda function that can be triggered from the AWS sdk and let their infrastructure handle the headache. I’m sure a home grown solution will become more cost effective at a massive scale but lambda fits the bill for a small/medium project

We developed a web-based tool that described water quality based on your location. We generated screenshots of every outcome so the results could be shared to FB/titter. It was something on the order of 40k screenshots. Our process used headless chrome to generate a screenshot then it was uploaded to S3 for hosting.

Doing that in a series took forever. It took something like 14 hours to generate the screenshots, then 4 hours to upload them all. Spreading that load across lambda functions allowed us to basically run the job in parallel. Each individual lambda process took longer to generate a screenshot than on our initial desktop process, but the overall process was dramatically faster.

The parallelism argument doesn’t pass muster because you can do the same thing with a cluster of free tier t2.micro machines with any good orchestration platform, not just lambda.

This argument is basically: no counterpoint to the original post, but you can do things that are also easy on any other comparable platform.

Tell me again what I don’t understand?

> Tell me again what I don’t understand?

As someone who has done both, it's far, far easier to stand up a lambda than it is to manage a cluster of servers.

This still doesn’t make sense. There are portable systems that do the same, and have fully managed options, such as kubernetes.

In my mind the thing that makes lambda “easier” is they make a bunch of decisions for you, for better or worse. For real applications probably for the worse. If you have the knowledge to make those decisions for yourself you’re probably better off doing that.

> This still doesn’t make sense. There are portable systems that do the same, and have fully managed options, such as kubernetes.

The whole value proposition behind AWS is that they can do it better than your business due (directly or indirectly) to economies of scale. I think Kubernetes is super cool, but rebuilding AWS on top of Kubernetes is not cost effective for most companies--they're better off using AWS-managed offerings. Of course, you can mix and match via EKS or similar, but there are lots of gotchas there as well (how do I integrate Kubernetes' permissions model with IAM? how do I get Kubernetes logs into CloudWatch? how do I use CloudWatch to monitor Kubernetes events? etc).

> you can do the same thing with a cluster of free tier t2.micro machines

Not if you already used your free tier. Lambda is X free per month. Free tier t2.micro is X free for first 12 months.

Reply on news.ycombinator.com