Hacker News Reader

Comment by silverbax88 | Hacker News Reader

The idea of cloud storage being down is less of an issue - I don't like it, but I understand it. What bothers me about this is:

1. I was never notified of the outage. I noticed it myself when attempting to log into one of my VMs and then started looking for status updates. Sadly, the best status updates I got were here on Hacker News.

2. When my servers did come back up, at least one of my IP addresses had changed, which meant I had to update all of the relevant DNS entries (which, as everyone here no doubt knows, can take up to 48 hours to propagate). I was never notified of this change in any way.

View on HN

2. Azure does not guarantee that you keep your ip address by default. You should configure a cname if you use Azure Websites or get a reserved ip address, available with Cloud Services

Also as I found out, set the TTL on your CNAME very low.

Also this is a PITA if you use the @ entry in your DNS.

Why should you have to keep the CNAME low if it always resolves to the same domain on Azure?

Only because some caching DNS resolvers are broken and keep the referral cached for the TTL of the CNAME record.

Actually, I am supposed to have a fixed IP. And I have CNAMES configured. I will review to determine if there is some other way I can set this up, but my issue is that I would never allow my products to be offline for hours without notifying my customers.

First, I am really sorry about the impact the incident had on your service. Specifically, for your changing IP, are you currently using the reserved IP feature. You can reserve both your external IP and your internal IP.

External IP: http://msdn.microsoft.com/en-us/library/azure/dn690120.aspx

Internal IP: http://msdn.microsoft.com/en-us/library/azure/dn630228.aspx

Do you having monitoring setup on your cloud vm, because you should have alerts triggering emails to notify you when this is happening.

Secondly, you are using an IP address and expecting that to be static? The recommended approach is to use a CNAME so you don't hit that issue, alternatively, you can have up to 5 Reserved-IPs per subscription and attach that Reserved-IP to your VM : New-AzureReservedIP from powershell

Edit : see http://azure.microsoft.com/blog/2014/05/14/reserved-ip-addre...

Yes, I do. That's why I was checking my VMs. This still doesn't absolve my cloud server provider from notifying me when they have a global system failure.

> which, as everyone here no doubt knows, can take up to 48 hours to propagate

I think that's been largely dispelled.

I can't find the link right now unfortunately but I remember a post looking into DNS propagation realities from either this or last year, and they found that overwhelming majority of DNS servers they tried (99%+) respected the TTLs set exactly as they should. *

My personal rule of thumb is, if it hasn't propagated within an hour, I need to look at it again because I messed up.

Tools like this [1]are invaluable when you're paranoid about whether your new record has propagated.

[1] https://www.whatsmydns.net

* ugh. Does anyone know which post I'm talking about? My google-fu is failing me hard.

Perhaps not the link you are looking for - but there was some discussion on here sometime ago - https://news.ycombinator.com/item?id=3397253

Sadly this is a common situation. They appear to hold off making any public notice (including often on their own "service dashboards") until support forums are screaming with upset users.

Google App Engine has had numerous outages like this, the only one I can find any public documentation for being a 6 hour outage in 2012: http://googleappengine.blogspot.co.uk/2012/10/about-todays-a... (and let's not forget the old-style Datastore corruption incident, where every App Engine user got to manually merge split-brain database tables after a messed up failover)

Hi hosay123,

The App Engine team has a proactive policy about posting about downtime:

https://groups.google.com/forum/#!forum/google-appengine-dow...

Since the team highlights basically anything that looks like it is impacting customers, the issues don't always warrant a stand-alone blog post, but you'll notice that generally speaking the last post in each thread is a full public post-mortem with diagnosis and remediation.

Let me know if there's more you think might be useful for you as a GAE customer. Thanks!

I'm not trying to be evil but I used to migrate a small website away from GAE because of continuing issues and very limited communication from Google.

Azure isn't perfect but their support is way more responsive, at least to us eventhough we don't have any fancy support contract. Eventhough my GAE experience was 2-3 yrs ago, I have to say that Azure has way less issues.

Reply on news.ycombinator.com