bscanlan

Fun article, the phenomenon is interesting to see in practice, I've seen it regularly with newer instance types as it can take time for people to add them to their configurations.

We're heavy users of spot here in Intercom. I spot-checked our biggest workload, and this week we could have paid around 10% less if we were able to get the cheapest spot host possible in us-east-1 that is suitable for our workload (all 16xlarge Gravitons). However that would be at the cost of fleet stability, I think that to run relatively large production services used in realtime on spot you need to prioritise fleet stability, so choosing the "Capacity Optimized" strategy. We've seen incessant fleet churn when trying out cost optimised strategies.

> Well I can only guess so much of the underlying egress internet routing of AWS.

> At worst, if no explicit region is specified, it will reach the global aws endpoint through internet which is likely in a complete different part of the world than where you are, redirect to the local endpoint, and back.

There's no need to guess:

From https://aws.amazon.com/vpc/faqs/#Peering_Connections

"When using public IP addresses, all communication between instances and services hosted in AWS use AWS's private network. Packets that originate from the AWS network with a destination on the AWS network stay on the AWS global network, except traffic to or from AWS China Regions."

In practice there is not much risk from accessing AWS services using public endpoints, you just need to take AWS at their word.

Intercom is pretty similar. We use EC2 hosts and no containers (other than for development/test environments and some niche third-party software that is distributed as Docker containers). Autoscaling groups are our unit of scalability, pretty much one per workload, and we treat the EC2 hosts as immutable cattle. We do a scheduled AMI build every week and replace every host. We use an internally developed software tool to deploy buildpacks to hosts - buildpacks are pre-Docker technology from Heroku that solves most of the problems containers do.

I wouldn't necessarily recommend building this from scratch today, it was largely put in place around 8 years ago, and there are few compelling reasons for us to switch.

> Under the EU Internet Forum, the Commission has launched an expert process with industry to map and preliminarily assess, by the end of 2020, possible technical solutions to detect and report child sexual abuse in end-to-end encrypted electronic communications, and to address regulatory and operational challenges and opportunities in the fight against these crimes.

It is a spectacular overreaction to equate this to "EU wants to ban encryption". This will never happen.

On one hand, this type of service merely solves business intent with unfriendly or un-automatable interfaces. Why should this exist and be so successful? On the other, I guess it's like Segment or Tray.io, gluing together various services to improve business outcomes, and taking their slice of the efficiency improvements they make possible. I guess the most commonly integrated services should wake up and see the potential revenue they're losing in having crumby interfaces, and in the meantime UIPath will provide a good path for efficient IT services. Fair play to them in executing so well in this space.

It looks like some sort of dispute or contractual issue between Notion and and the .so registry, or a complete messup by the registry.

From a whois of notion.so:

Updated Date: 2021-02-12T12:54:35.982Z

Creation Date: 2015-03-31T00:00:00.0Z

Registry Expiry Date: 2021-03-31T00:00:00.0Z

Registrar Registration Expiration Date: 2021-03-31T00:00:00.0Z

Domain Status: clientHold https://icann.org/epp#clientHold

It looks like some sort of dispute or contractual issue between Notion and and the .so registry, or a complete messup by the registry.

From a whois of notion.so:

Updated Date: 2021-02-12T12:54:35.982Z

Creation Date: 2015-03-31T00:00:00.0Z

Registry Expiry Date: 2021-03-31T00:00:00.0Z

Registrar Registration Expiration Date: 2021-03-31T00:00:00.0Z

Domain Status: clientHold https://icann.org/epp#clientHold

I miss #wtf. I hope it lives on in Slack.

This is a response to this AWS Lambda launch, over a year later: https://aws.amazon.com/blogs/compute/new-for-aws-lambda-pred...

Segment's blog posts on cost optimisation have plenty of detail and tips on this topic:

https://segment.com/blog/the-million-dollar-eng-problem/ https://segment.com/blog/spotting-a-million-dollars-in-your-... https://segment.com/blog/the-10m-engineering-problem/

Similarly this Honeycomb writeup is also excellent: https://www.honeycomb.io/blog/treading-in-haunted-graveyards...

By the sounds of it, you need to take drastic action. It sounds like you will not be able to just optimise your AWS spend to get more runway, though you should definitely do some bill optimisation. You will need to optimise your product itself and maybe even getting rid of unprofitable customers.

If you are not sure exactly who or what is driving the AWS cost, take a look at Honeycomb to get the ability to dive deep into what is eating up resources.

You are forgetting to price in some minor features that Aurora provides: - Aurora's storage is spread across three availability zones. - Backups. - Automatic failover. - No need to configure anything, it just works.

If your time is free, and you don't actually need anything resembling high availability for the data in the database, then that's a good price comparison. I'm not arguing that managed databases makes sense for everybody, but if you're doing a price comparison then at least factor in multi-site redundancy for the data?

It's clearly missing storing data uuencoded in Route53 TXT records.

Letsrun is unfortunately a very toxic forum, but also has loads of knowledgeable folks. It's like reading a wikipedia article with commentary from 4chan on the side.

I work for Intercom, and we also did not implement these features for the same reasons.

Note that AWS offer convertible reservations, and this is what Stripe say they use in the article.

In fairness, the question could be interpreted as "would anybody, who is not facebook, ever run this in production?".

This is an interesting question the answer to which is definitely not certain :)

Dealing with the backscatter from CSV misunderstandings can be fairly challenging - for a lot of us, the customer experience is improved by being as accommodating as possible instead of correct. We at Intercom released a Ruby CSV parser that "is a ridiculously tolerant and liberal parser which aims to yield as much usable data as possible out of such real-world CSVs".

https://github.com/intercom/hippie_csv

Yeh, how hard could it be? Pesky regulation.

Two sentence summary: Slow clients strangled our service. The slow clients were caused by an external AWS networking issue.

Ah, so just like SDB. I wonder why AWS moved on from that model.

Enterprise Support customer here :)

It looks like the bad .io servers aren't responding to requests anymore, at least from my location. I have flushed a bunch of my own records on public Google DNS: https://developers.google.com/speed/public-dns/cache

Response from AWS Support:

We have investigated this issue and are seeing incorrect responses from two of the .io nameservers: ns-a4.io and ns-a2.io.

These nameservers are returning NXDOMAIN intermittently for domains that do exist. As a result, once a resolver receives the erroneous response, it will cache the non-existence for the negative TTL, which for .io is set to 3600 seconds (1 hour).

Doubtful. Almost certainly the same storage backend, but IA requests are more aggressively throttled, so there's more chance of requests being rejected. S3's capacity planning then requires less request processing infrastructure to serve IA, so it's cheaper to provide than regular S3.

Open source is definitely important part. The "trust" I was referring to was deliberately ambiguous. There's not just leap of faith that potential users have to make to ensure that there are no deliberate backdoors or flaws in this software - there's also trust that this project will be cared and maintained for over time: bugs will be acknowledged (and even fixed :) ), builds for new platforms will work, security issues will be dealt with...

I'm not suggesting that bugs won't be fixed or there are backdoors, and this is definitely a really interesting project. However there's just very little to convince me that I should trust this project for anything more than a throw-away experiment right now.

"the project was not initially meant to be OpenSourced"

Sounds like an interesting backstory - what's the motivation to open source it so?

"Not being famous does not necessarily means being suspicious."

It's not being famous that matters here. Publishing software anonymously is unusual (I'm sure there are exceptions here, however they're definitely exceptions). Whenever I read of a new web service or software I generally dig into who is behind it and why they're doing it. Code on its own does not convince you to use it.

Definitely looks interesting, but difficult to trust based on how it's been pseudo-anonymously dumped onto github in one commit.

New Relic.

AWS have posted an update about related upcoming EC2 maintenance: https://aws.amazon.com/premiumsupport/maintenance-2015-03/

"We’ve received a Xen Security Advisory that requires us to update a portion of our Amazon EC2 fleet. Fewer than 10% of EC2 customer instances will need to be rebooted. We’ve started notifying affected customers when their reboots will take place. These updates must be completed by March 10, 2015 before the underlying issues we are addressing are made public. Following security best practices, the details behind these issues will be withheld until they are made public on March 10."

It's good that this was published, some tough lessons in there. It's missing a timeline though, which is important to learn from - when was it realised that user data was accessed, how long did it take to figure out that keys were stolen, exactly when & what actions were taken to protect customer information, etc.

Other interesting details not covered was whether an incident response plan existed or was followed, how many unpatched Internet-facing servers were sitting around and whether any existing security measures did actually work in slowing down the attacker.

The performance data looks interesting, but this is work based on a pretty old kernel (originally released in 2010 or so). There have been many changes and improvements added to the 3.x kernel that may overlap with this work. Publishing the code and details on github is great, but working with the kernel community and merging into the mainstream kernel is the only way for work like this to have a long-term meaningful existence - Google in particular have been doing a great job getting networking improvements in.

That said, it's interesting to have this kind of thing come out of large-scale production web environments in China.

"West Europe" is located in the Amsterdam area. "North Europe" is in Dublin.