It would be nice if this came with reasonably priced NAT gateways. The current pricing is outrageous.
I was curious how they do this, so I set up a service on Google Cloud Run that just echo'd the user's public IP address. When curl'd over IPv4, it said I was coming from a unique local (i.e. private) IPv6 address. The private IPv4 address of my server was embedded in the address, along with some other random-looking bits that probably identified my VPC somehow. So they must have been doing some sort of stateless IPv4 to IPv6 translation behind the scenes.
It was a clever solution that takes advantage of the fact that all of Google's API endpoints are dual-stack, even though (at the time) they didn't support IPv6 on customer VMs. The problem AWS currently has is not all of their internal endpoints are dual-stack, so even using IPv6 can't save you from cloud NAT costs when accessing AWS services.
As the first hyperscaler, they gotta pay the legacy tax.
Have quite a bit of experience with AWS and Azure, and only recently learning about GCP, it’s very clear that Google nailed Some of GCP’s core cloud engineering concepts and got them exactly right.
Although unfortunately they will never reach the size of AWS or (maybe Azure? It’s hard to tell Azure’s market size as they don’t disclose it.)
GCP especially takes a lot more trial and error building systems that compose a bunch of different primitives. That the API is awful doesn't help either.
I would find it rather surprising if the actual cost to Amazon of connecting a VPC to S3 were substantially lower than the cost of connecting a VPC to any other AWS service.
At that point, you might as well be running a Layer 7 Firewall or an Intrusion Protection System.
Instead of putting the fragmentation in L4 (like QUIC now does) and including a UDP header on every fragmented packet in a datagram, UDP only includes the header on the first packet. With fragmentation happening; firewalls, NATs, and end-hosts have to buffer and coalesce IP packets based on IP IDs, before the destination can be identified. It's a real nuisance. A lot of CGNAT "stateless" implementations can't handle this and you get very hard to debug issues when there are fragmentation and MTU mismatches.
It has a legitimate purpose in old-timey systems which have bespoke MTUs on each link, but now the usual thing is to use 1500 bytes for WAN traffic, which is the generic Ethernet MTU, and reserve larger sizes for intra-datacenter communications.
1500 is absolutely not a pervasively usable WAN MTU, you're going to need pMTUd if you're sending 1500 byte packets broadly. Plenty of WAN links won't tolerate it. If you don't want to deal with fragmentation at all ... 500 is the minimum guaranteed MTU, but in practice it's exceptionally rare to see anything below about 1200 require fragmentation. But you can always only control what you send, not what others are sending you.
Edited to extend: this is why TCP has a "Maximum Segment Size", and why Path MTU Discovery information has to be passed into the TCP state machine. It is TCP that takes responsibility for carving up the data into the packets, not IP.
One of the goals of UDP was to avoid needing this kind of state, which is why the IP layer handles fragmentation for it instead. This is allowed on a hop-by-hop basis, unless the DF bit is set; so when a "too big" packet gets to a node with a smaller MTU, it can just split it and send on the fragments. No PMTUD needed.
The design could have been for the fragmenting node to also add a UDP header as part of that process, but was not. It would have been a simple change at the time. It's had a lot of consequences since and is responsible for a decent amount of complexity in hardware and software packet pipelines.
TCP, UDP, ICMP and IP were all designed contemporaneously; UDP fragmentation could also easily have just been specified for. It's just an odd regrettable quirk.
If you go down this path consider using Transit Gateway so you can route multiple VPC traffic to a central security VPC in a region. I’ve done this a Palo Alto VM and it seems to work well.
so that's: source ip, dest ip, protocol, source port, dest port, connection state (say 16 bytes total)
doing NAT too is what, 3 more bytes per connection (8 bits for an offset into an IP table and 16 bits for the translated port)
I hear such takes all the time and its really frustrating; usually in threads regarding IPv6, incidentally it is usually programmers who think they understand everything about networks because they know how tcp operates.
> I hear such takes all the time and its really frustrating
maybe you'd be less frustrated if you understood what people were saying, because I didn't say that
AWS already do 1:1 NAT and there's additionally a stateful firewall, which necessitates connection state tracking
adding the extra few bytes to do port translation shouldn't vastly increase the memory required
> incidentally it is usually programmers who think they understand everything about networks because they know how tcp operates.
from someone who has written a commercial packet filter: in terms of complexity, TCP blows the preceding layers of the stack out of the water
Is that really conceptually so different from a stateful firewall allowing inbound packets only for established connections/flows?
"NATs are good because otherwise people wouldn't have any firewalls" is a tired take, yes, but I don't see the point being needlessly pedantic about the semantics of NAT vs. stateful firewalls when in this case, the effect is the same: No inbound packets without prior outbound packets (or a connection establishment for TCP).
$0.045 per GB is nuts. That’s $20.25/hour or $14580/mo for 1 Gbps. One can buy a cheap gadget using very little power that can NAT 1 Gbps at line rate for maybe $200 (being generous). One can buy a perfectly nice low power server that can NAT 10Gbps line rate for $1k with some compute to spare. One can operate one of these systems, complete with a rack and far more power than needed, plus the Internet connection, for a lot less money than $14580/mo. (Never mind that your $14580 doesn’t actually cover the egress fee on AWS.)
A company with a couple full time employees could easily operate quite a few of these out of any normal datacenter, charge AWS-like fees, and make a killing, without breaking a sweat. But they wouldn’t get many clients because most datacenter customers already have a NAT-capable router and don’t need this service to begin with.
In other words, the OpEx associated with a service like this, including the sysadmin time, is simply not in the ballpark of what AWS charges.
+ Run extra for failover, HA etc + manage security + Monitor performance + ...
https://docs.aws.amazon.com/vpc/latest/userguide/VPC_NAT_Ins...
I say sorta because it's built on an old version of Amazon Linux and is headed towards EOL with no replacement except "go build your own" as you suggest.
https://www.lastweekinaws.com/blog/an-alternat-future-we-now...
AlterNAT uses managed NAT Gateways as a fallback when the NAT Instance is out of service, but again you will have to make your own NAT AMI.
This is not to excuse AWS' frankly absurd NATGW pricing, but to point out other ways around it.
https://github.com/somleng/somleng-project/blob/main/infrast...
If it's the former... oh sweet jesus, what? Probably way cheaper to just run an a1.large or something with Linux on it, plus a very short shell script to set up NAT. That's assuming well more than half of the traffic going through it is ingress from the internet. If it's 50/50 ingress and egress, then it's basically the same pricing as NAT gateway.
> You also incur standard AWS data transfer charges for all data transferred via the NAT gateway.
Yes, the $0.045/GB “data processing” charge is in addition to the usual $0.09/GB egress charge. You are paying an effective $0.135/GB for all of your egress, in addition to the $0.045/hr just to keep the NAT gateway running.
And yes, your ingress and even internal-to-AWS traffic is also billed at the $0.045/GB rate. (An example given on the aforementioned page is traffic from an EC2 instance to a same-region S3 bucket, which they note doesn’t generate an egress charge but does generate a NAT processing charge.) As far as I can tell, the only traffic which isn’t billed is traffic routed with internal VPC private IP addresses, which don’t hit the NAT gateway and thus aren’t counted.
There are highly paid AWS consultants who shave literal millions of dollars off of many company’s AWS bills by just setting it up a cheap EC2 box to handle their NAT instead of using the built-in solution. Doing that instantly wipes out the ingress charges and effectively halves the egress charges, and it’s probably a lower hourly cost than they’re already paying: an a1.large is $0.051/hr on-demand but that immediately drops to just $0.032/hr with a 1 year no upfront reserved plan. If you’re willing to pay upfront and/or sign a longer contract, you can get it as low as $0.019/hr.
LOL. Not Metronet. They are doubling down on CGNAT. They've acquired ISPs with IPv6 and killed it in favor of CGNAT.
There is no possible use case in no possible universe where AWS is cost effective.
Renting the same compute resources wholesale will cost you 20 times less. (Not a typo.)
A) Promising scale (and delivering to a certain extent)
B) being significantly more convenient than contemporary solutions
C) becoming trendy
D) hoodwinking CxO’s into the belief that not owning your data is better for you, actually. (CapEx vs OpEx)
E) unfathomable amounts of DevRel.
Nobody has ever claimed AWS was cost effective, they have said that “it’s worth the cost” though.
> it’s worth the cost
Sounds about the same.
AWS is a boutique retail reseller for compute. It's okay for very tiny projects, or for vanity purposes.
The issue tends to be that people do not actually stay on top of their spend- they claim to need less headcount but then spend more than a few salaries worth on their cloud spend.
They claim they do not need headcount but then spend the same headcount in infra people anyway, or finops people in the best case.
people have lost touch with how much compute actually costs, because its little by little and claims to scale to zero or you only pay what you want. - yet every installation I’ve ever seen has had a base cost higher than the largest colo installation cost we would have needed times 2.
Its not cost effective, because its on average 11x more expensive than a fully managed colocation installation. - your packets dont care that you spent 11x more on half the performance.
For a provisioned IOPS volume, you can get up to 256k IOPS (so still a fraction of a single drive) at a cost of $25600/month (plus per-GB storage costs). For that price, you could buy 8x of these: https://www.newegg.com/micron-30-72-tb-9400/p/N82E1682036315... giving you 240 TB of raw SSD storage.
Right now, this is done on AWS, with lambda + S3, and costs under $0.02/month.
Can you point me to something more cost effective that that? Don't forget I also need backup for data, automatic failover in case of machine failure or crash, amd no maintenance (like OS upgrades) for 5+ years.
I have no doubt that there are plenty of cases when local hardware is cheaper, but gp said "There is no possible use case in no possible universe where AWS is cost effective."... and I claim there are many use cases where AWS is cheaper.
You mean like S3 object storage? That costs less outside AWS because you are usually getting less. That's if you can get it at all.
Enterprise workloads need compliance. AWS and GCP provide that. They are very few hosting companies out there who are better at security and compliance than those two.
AWS has notoriously high egress fees.
For a little traffic $40 is outrageous.
But I think the point is more that it's outrageous compared to the marginal costs.
I would expect them to reduce NAT pricing in the long run, but who knows.
I always love this comments. We pay them literally hundreds of thousands a year.
Almost all of my use cases I could easily ride out to the internet through a shared pipe (apt updates and such) and don't care whatsoever what IP that exits the AWS network from, since I'm not applying firewall rules or anything.
You'll start to care pretty quickly if it's the same IP as a bad actor that's blocked everywhere.
Edit: I see from another post that NAT gateway costs $0.045/hr + $0.045/GB of transfer. That seems... not terrible? An a1.large on EC2 is $0.051/hr + $0.09/GB transfer to the internet (which I assume this type of box would be doing a lot of).
AWS used to maintain a AMI to do just that, nowadays you have to do it yourself, but it's honestly not much more than adding 2/3 iptables rules.
I find this trade-off to be exactly the reason why AWS is so good even for small startups. You can bootstrap something quickly, though it will be a tad expensive.
And if you need to down your costs later on, you start chasing the quickwins like maintaining your own NAT gateway. The same could apply for all managed services.
Maintaining your own OpenVPN VS AWS VPN. Maintaining your own Postgres VS RDS. etc
If we have ended up at a place where it’s cheaper to run them yourself on an EC2 box then something has gone awry.
I think my team's use is kind of high, with 16 TB going through NAT last month. The bill for that came to ~1300, which is higher than I'd like, but that's only about 1.5% of our AWS spend. Tbh I never really looked at the spend for NAT before, but this doesn't alarm me.