Hacker News Reader

Comment by herpderperator | Hacker News Reader

The irony in this is that with CloudFront at least, it can't strip the "/api" prefix before sending it to your backend, so you have to have your backend server understand /api/xyz rather than just /xyz calls like when it was on a dedicated api subdomain.

But don't worry! AWS thought of this! They invented Another Cloud Thing, namely Lambda@Edge, to solve this. Now you can run a JS function for every single request that hits your CloudFront Distribution so that you can run logic in there to strip the prefix before it gets passed to your origin. You read that right! Execute code every time a request hits your proxy! But wait, doesn't executing code for every single request sound insane AND doesn't it also add latency which this was trying to remove? Yes! Isn't cloud just lovely?

View on HN

> They invented Another Cloud Thing, namely Lambda@Edge, to solve this.

Yes, designed for usecases like those handled by Cloudflare Workers, which boil down to updating data cached in edge servers without requiring global redeployments or pinging a central server. We're talking about stuff like adding timestamps to images or adding headers to HTTP responses or pre-rendering some HTML or emit CDN-aware metrics.

> Now you can run a JS function for every single request that hits your CloudFront Distribution

Not exactly. Lambda@Edge are event handlers from CDN events. You use them when they suit your needs.

> so that you can run logic in there to strip the prefix before it gets passed to your origin.

Your strawman example doesn't even feature among the dozen examples provided by AWS regarding how to use Lambda@Edge.

Everyone is free to come up with silly ideas and absurd examples, but if you design systems around braindead ideas then that says a lot about you and nothing about the tools you chose to abuse

> You read that right! Execute code every time a request hits your proxy!

Yes, that's what web servers do. What exactly is your point?

> But wait, doesn't executing code for every single request sound insane

It doesn't. That's what a web server does. Moreso, Lambda@Edge (and Cloudflare Workers too) only do it if you explicitly decide to make them do it, to match precisely what you tell them to do.

What point are you trying to make, exactly?

> AND doesn't it also add latency which this was trying to remove?

It does. It adds tens of milliseconds when the alternatives can add hundreds of milliseconds. You're also expected to do basic engineering work and do basic performance work when seeking performance improvements, such as measuring things instead of mindlessly jumping on bandwagons without caring for the outcome.

> Isn't cloud just lovely?

I feel your comment manifests too much cinicism to cover too much ignorance on a topic you are not familiar nor understand the basic premise.

You shouldn't be so stern. The use case I provided is AWS's recommended way to solve this problem[0]. :)

> Yes, designed for usecases like those handled by Cloudflare Workers, which boil down to updating data cached in edge servers without requiring global redeployments or pinging a central server. We're talking about stuff like adding timestamps to images or adding headers to HTTP responses or pre-rendering some HTML or emit CDN-aware metrics.

> Not exactly. Lambda@Edge are event handlers from CDN events. You use them when they suit your needs

> Your strawman example doesn't even feature among the dozen examples provided by AWS regarding how to use Lambda@Edge.

For all of these counterpoints, see [0].

> Everyone is free to come up with silly ideas and absurd examples, but if you design systems around braindead ideas then that says a lot about you and nothing about the tools you chose to abuse

I take it you are directing this particular comment at AWS considering they are recommending this solution? [0]

> Yes, that's what web servers do. What exactly is your point?

Don't be daft. You know what I mean. Obviously web servers execute code. In this case the web server (CloudFront Distribution) is passing the request to yet another "thing" which happens to be a JS function that is invoked specifically to handle something web servers like nginx were built to do extremely efficiently on their own. CloudFront could easily support this without additional dependencies that add complexity to your system design and introduce additional failure modes. In this case, I am making your point for you: using Lambda@Edge for this IS ridiculous, but that's the AWS way.

> It doesn't. That's what a web server does. Moreso, Lambda@Edge (and Cloudflare Workers too) only do it if you explicitly decide to make them do it, to match precisely what you tell them to do.

> It does. It adds tens of milliseconds when the alternatives can add hundreds of milliseconds. You're also expected to do basic engineering work and do basic performance work when seeking performance improvements, such as measuring things instead of mindlessly jumping on bandwagons without caring for the outcome.

Sorry for not writing a detailed blog post to describe all of my findings. FYI, the latency added with Lambda@Edge caused my request time to double and added much more variance.

I'm surprised a lot of your counterpoints are just blaming me for doing the wrong thing, when this is actually what is recommended by AWS. Invoking a custom function to process the request. See [0]. Of course the right solution is to use nginx (but then you lose out on AWS scaling) or completely redesign your system to fit another solution like API Gateway.

From AWS's blog:

> In this scenario we can use Lambda@Edge to change the path pattern before forwarding a request to the origin and thus removing the context. For details on see this detailed re:Invent session. [0]

[0] https://aws.amazon.com/blogs/architecture/serving-content-us...

>But don't worry!

I don't, I can just teach my backend to process /api. The backend is running code under my control anyway. This seems like one of the least difficult "problems" to solve in any non-trivial codebase.

I don't want to argue that the feature should be built in (and most importantly free, because that's something you forgot to mention, those lamdba functions are paid for each run) but

> doesn't executing code for every single request sound insane

How would you strip some url path without executing code for every single request? It could be code you did not write, but some code is always needed to do something. If latency of lamdba@edge is an issue, you can try CloudFront functions, which should be better for small tasks like this (though I don't have any experience with them as we switched to a similar competitor)

CloudFront Distributions cannot pass the request to CloudFront Functions before sending to the origin. In other words, they cannot be used to modify origin request/responses. They can only modify the viewer request/responses. [0]

Only Lambda@Edge can help the scenario which I provided, which is also AWS's recommended solution. [1]

[0] https://docs.aws.amazon.com/AmazonCloudFront/latest/Develope...

[1] https://aws.amazon.com/blogs/architecture/serving-content-us...

From your link [0] "URL redirects or rewrites" is a supported use case.

For stripping /api/ from the path, I don't know what would be the benefit of doing it at the origin request, rather than at the viewer request?

Cloudfront functions are pretty new, so some older docs might still reference lamdba@edge only.

The origin request is your backend. The viewer request is the one you're requesting from the client. You want the viewer to have /api. You want the origin to not have /api.

Therefore stripping must be performed on the origin request, which is not supported by CloudFront Functions.

Both the viewer request and origin request hook both happen early enough that you can update the path

See https://docs.aws.amazon.com/AmazonCloudFront/latest/Develope...

I have used lamdba@edge and our primary use case was to rewrite url (to add languages information for example) so I'm sure it works in viewer request events.

No. Please see the many other responses I gave to other comments. It does not work because the viewer request is the request the viewer views. If you change the URL of the viewer request, CloudFront sends a 301 accordingly. In other words, changing /api/blah to /blah in the "Viewer Request" part results in CloudFront injecting a `301 location: /blah` and the client doesn't hit your backend because they got redirected. The client instead hits your frontend as a result of your URL rewrite. There is a very clear difference between viewer and origin. If you could rewrite origin requests at the viewer level there would not be any difference between the two. This is the difference. Viewer is what the client sees. Origin is what the backend sees.

My understanding is that you would need to rewrite the url and update the origin.

I could be wrong as I did this lambda@edge a while ago, but the main difference between the two request event is that viewer request is done in all cases, while origin request is done only in case of cache miss.

[EDIT] you are right, updating the origin is indeed something that cannot be done on the viewer side. My bad, I should have checked earlier.

Personally, I did it with templates in api gateway, as at least then I'm not paying for the cpu time to do these sorts of manipulation. But yes, stuff like this made me really shake my head at 'the cloud'

I don't know. This doesn't feel like something that should require edge logic. nginx handles this with a single rewrite directive, so I'd expect some part of the AWS routing layer to have a setting to rewrite the URL accordingly. OTOH, AWS can't and shouldn't account for every edge case, probably.

A rewrite directive is just writing code.

Rewrite directives handled by low-level routing logic of the web server are definitely different to a container image that gets spun up in a distributed network and queried for each request, don't you think?

Not really, no. In fact the only significant difference (that the Lambda@Edge function is run at the closest geolocation to the client, rather than on the origin server) is an advantage to Lambda@Edge.

What difference are you alluding to?

A HTTP URI path prefix mapping is done in nginx, in the same process, no context switching, no buffering, I think it is even done without any allocation.

Of course it doesn't matter, basically logging takes more resources than whatever routing is needed. (Especially because even if it requires running a Cloudflare worker it's just a NodeJS V8 isolate, very lightweight. As far as I know. But I have no idea of its costs.)

But AWS Lambdas are pretty pricey compared to a free nginx rewrite.

> But AWS Lambdas are pretty pricey

Kind of why CloudFront Functions we're invented for this use case, AWS Lambda@Edge is the more general purpose edge computing facility.

> (...) are definitely different to a container image that gets spun up (...)

I'm not sure you're making a informed observation. "Spinning up a container image" is just fancy jargon/handwaving to refer to launching a process, which is something nginx already does under the hood in a myriad usecases without warranting complains from users.

> and queried for each request, don't you think?

I'm not sure if either you framed your sentence poorly or you don't have a clear idea about what you're talking about. In AWS lambda, you don't "spin up containers" at each request. You launch your lambda once it's deployed or when it scales up, and it stays up as long as you see traffic.

Poor phrasing, then. Meant to say a container is started once, then queried on every subsequent request. And what I’m talking about is not the user-facing lambda, but what AWS does on the infrastructure layer below, which involves running a containerised process on firecracker VMs, communicating via HTTP. There was a great summary of how AWS works a while ago, going into way more detail, that I sadly can’t find anymore. But there’s definitely more work involved in calling a lambda function than forking a process in nginx.

> Rewrite directives handled by low-level routing logic of the web server are definitely different to a container image that gets spun up in a distributed network and queried for each request, don't you think?

I’m pretty sure Lambda@Edge is not a container image like regular Lambda (why it only supports JS and not the broader set of Lambda runtimes) and is relatively local to the edge environments serving CloudFront anyway (why it is support in CloudFront the way regular Lambda isn't, and what the @Edge part refers to.)

Crafty alternative: just.. don't strip it? It seems like a perfectly reasonable and simple route that avoids complexity and cost to just keep /api.

It's just one of the many things unexpectedly out of your control when using a cloud provider.

Well, we've all read this thread so it's not unexpected anymore.

>doesn't executing code for every single request sound insane AND doesn't it also add latency which this was trying to remove? Yes!

but surely executing code at the source of your web app is more efficient than having the client and server attempt to perform it? faster than 40-90ms per request at least

> But don't worry! AWS thought of this! They invented Another Cloud Thing, namely Lambda@Edge, to solve this.

No, CloudFront Functions is the Another Cloud Thing AWS invented more specifically for this use case. Lambda@Edge is the older and more general purpose edge computing facility.

> You read that right! Execute code every time a request hits your proxy!

Any rewrite rule in any platform, and heck, any other functionality provided by the distribution, is executing code every time it gets a request (whether or not you are writing code for the purpose.)

Only Lambda@Edge can help the scenario which I provided, which is also AWS's recommended solution. [1]

[0] https://docs.aws.amazon.com/AmazonCloudFront/latest/Develope...

[1] https://aws.amazon.com/blogs/architecture/serving-content-us...

Not sure why you cannot strip in your backend, and it's a huge deal. Code needs to be run somewhere though. While Lambda@Edge sucks, Cloudfront runs functions now which is not Lambda@Edge, and faster because it executes on the PoP location. https://aws.amazon.com/blogs/aws/introducing-cloudfront-func...

Only Lambda@Edge can help the scenario which I provided, which is also AWS's recommended solution. [1]

[0] https://docs.aws.amazon.com/AmazonCloudFront/latest/Develope...

[1] https://aws.amazon.com/blogs/architecture/serving-content-us...

I've just tried it. The following function works exactly as-is when the function is configured as Viewer request. I changed it to append some text to URL.

See: https://github.com/aws-samples/amazon-cloudfront-functions/b...

That MODIFIES a viewer request, resulting in a viewer redirect. We want to keep the viewer request the same, i.e. with /api, and strip it only from the backend origin request.

The viewer request is the one you make from the client, with /api. The origin request is the one CloudFront sends to your backend, which, without using Lambda@Edge via the origin request, will get sent as-is with the /api prefix. You have to strip it from the origin request.

If you strip /api from the viewer request, your `domain.com/api/users` request becomes a redirect to `domain.com/users` which results in a call to your frontend instead of your backend.

The example you referenced solves a completely different issue, not the one we are talking about.

CloudFront is a CDN, it should cache static resources, not so much manipulate incoming requests - that's more a job for a load balancer / proxy like Elastic Load Balancer.

Fully agree here: I don’t expect anything else but reliable and performant HTTP caching from a CDN like CloudFront.

Request manipulation is not the duty of a cache - even though other CDN providers mix request manipulation functionality with caching. In my opinion, they don’t need to be in the same product.

If you still need request manipulation, because you don’t control the origin or you don’t want to introduce another service between CloudFront and the origin, you would use CloudFront Functions, which is cheaper than Lambda@Edge and easy to set up.

Only Lambda@Edge can help the scenario which I provided, which is also AWS's recommended solution. [1]

[0] https://docs.aws.amazon.com/AmazonCloudFront/latest/Develope...

[1] https://aws.amazon.com/blogs/architecture/serving-content-us...

CloudFront is CDN with limited features, other CDNs (Akamai, Fastly etc) are more than capable of manipulating incoming requests

Which is why CloudFront and Lambda are separate products. Lambda@Edge simply combines them.

We ran into the same issue and we had to use API gateway to rewrite urls. but it had the added benefit of handling CORS there and the preflight requests never hits the application servers. Additionally preflight requests are cached at cloudfront and at browser with "Access-Control-Max-Age" and for the first request we do pre connect during the main domain load so the users never notice the small latency it may have.

I agree that AWS products are all over the place.

"AWS products are all over the place."

It's the customers. They run into issues using something against all advice in the docs and then request features and services or AWS sees all the clamor around a perceived issue and they create a "solution."

Developers are really averse to reading and understanding documentation. It's why they all jumped on the cloud. But it turns out it isn't all just magical infinite performance.

Is your lambda awake?

In asp.net we just use:

app.UsePathBase("/api");

And we have 0 issues.

Similar to Lambda@Edge, Cloudflare also offers Workers (https://workers.cloudflare.com/) which is the same thing, but only with a JavaScript runtime (no nodejs, they use V8), so I believe that it's significantly faster.

CloudFront has CloudFront Functions now which is similar; a very stripped-down JS environment that runs at the edge locations.

Only Lambda@Edge can help the scenario which I provided, which is also AWS's recommended solution. [1]

[0] https://docs.aws.amazon.com/AmazonCloudFront/latest/Develope...

[1] https://aws.amazon.com/blogs/architecture/serving-content-us...

A viewer request function can, of course, be used to modify the request before it is sent to the origin. The difference is only related to caching — viewer request happens before the cache lookup, origin request happens after the cache lookup when CF has decided to make an origin request.

For the stated purpose, either function is fine:

With Lambda@Edge you'd use origin request if you were caching these paths, because your function would be called less often so your costs would be lower.

With CF Functions you can only do the pre-cache-lookup modification, so it will be called for every request, but the cost is much lower than Lambda@Edge so it may not matter, and maybe you were not caching these paths anyway in which case it's virtually identical.

Modifying the URL in a viewer request results in the viewer request changing accordingly. If you look at my other responses to other comments, I explained that in detail.

Basically, using the Viewer Request trigger and modifying the URI there causes CloudFront to force a 301 to the user to the new URI, because it's for the viewer. Therefore, in the example of /api/users, if you modify the viewer to remove /api, CloudFront literally removes /api from your request URI, meaning the client accesses /api/users but the server instead sends you to /users (read: server returns 301 location: /users when you hit /api/users) because of your viewer rewrite. You end up hitting your frontend instead of your backend because in order for it to hit your backend, the viewer request has to have /api in it. Therefore you cannot strip it in the viewer request. You must do it in the origin request, which is not supported by CloudFront functions.

Reply on news.ycombinator.com