bscanlan

"an initial verification led us to believe the changes had been rolled out successfully"

I would love more detail in the type II error in this validation step, and is worth exploring deeper. What was the verification step? Why did it not detect the issue? What review process was used for the verification step?

While the failed verification step is not the root cause, having good safety checks are the most important part of planning good changes, whether they're DNS reconfigurations, network changes or software deployments.

Agreed - the assumption is that the MTBF of a single network device is a relevant statistic is undoubtedly incorrect in anywhere but a hobbyist's network.

"The original Dynamo design was based on a core set of strong distributed systems principles resulting in an ultra-scalable and highly reliable database system. Amazon DynamoDB, which is a new service, continues to build on these principles, and also builds on our years of experience with running non-relational databases and cloud services, such as Amazon SimpleDB and Amazon S3, at scale."

http://www.allthingsdistributed.com/2012/01/amazon-dynamodb....

There are EU laws per se.

http://en.wikipedia.org/wiki/Regulation_(European_Union)

"BTW the EU isn't really a law maker as such. Our contract with being in the EU means that we implement the EU directives, but in theory we could refuse -- the EU would fine us and/or kick us out if we did this."

Incorrect. Directives do exist, and they are transcribed by members into national law. However, Regulations have immediate effect across all member states, no transposition required: http://en.wikipedia.org/wiki/Regulation_(European_Union)

Also, there is no mechanism in the current treaties to throw out a member from the EU - all members must agree for a member to leave the EU.

Fines do exist for non-enforcement of EU law though.

...and it's been repaired.

$ dig +trace www.google.ie ... google.ie. 172800 IN NS ns1.farahatz.net.

google.ie. 172800 IN NS ns2.farahatz.net.

;; Received 79 bytes from 193.1.142.2#53(193.1.142.2) in 4 ms

www.google.ie. 14400 IN CNAME google.ie.

google.ie. 14400 IN A 119.235.27.219

google.ie. 86400 IN NS ns2.farahatz.net.

google.ie. 86400 IN NS ns1.farahatz.net.

$ whois 119.235.27.219

...

route: 119.235.16.0/20

descr: Route object of PT Inet Global Indo

descr: ISP

descr: Jakarta Barat

country: ID

origin: AS18351

mnt-by: MAINT-ID-INET

changed: hostmaster@idnic.net 20090211

source: APNIC

person: Santoso Halim

address: Pluit Permai 8 No.3A

address: Jakarta-Utara

address: Indonesia

country: ID

phone: +62-21-30047799

fax-no: +62-21-30047798

e-mail: hostmaster@inet.net.id

nic-hdl: SH1061-AP

mnt-by: MAINT-ID-INET

changed: halim@inet.net.id 20061020

source: APNIC

Not to mention redundancy - Google certainly do not store your docs on one hard drive.