Tell HN: Domain migrated from Google to Squarespace went down after 1 month
After more than 1 month, it randomly went down, at the worst time possible (the weekend, when it's the busiest time for my platform and when all the engineers are off). My company's main domain and email have been down for more than 12 hours and customer support is now asking for us to wait even more.
Not very hopeful posting here will help, but thought people should know.
Here's the timeline:
(1) Saturday, 6PM UTC - our customers start reaching out that they cannot load our platform.
DNS requests are returning `ns1.unknown-nameservers.com` and `ns2.unknown-nameservers.com` as the name servers.
I send an email to from my personal inbox to my work inbox. It doesn't arrive. So not only my platform is down, most of our customer emails don't arrive too!
Trying to access Squarespace account just shows a message that says "This account is currently unavailable. Please contact Customer Care."
Panic ensues and me and my co-founder start trying to reach out to customer support via email, their helpdesk, Twitter, Facebook, Linkedin. Even have someone we know who knows a PM at Squarespace escalate our case.
Squarespace says they have 24/7 support but we cannot get anyone to respond. Various Facebook and Twitter messages indicate that this happens to others too.
(2) Saturday, 11:00 PM UTC - it's been 5 hours. Whatever users were able to access the platform will likely have had their DNS caches updated by now. My co-founder and I decide it's time to migrate to a different domain. So I spend the next 30 minutes deploying and updating third party services with our new domain name. I set up our voicemail to tell people to use the new domain while the main one is down. Hopefully they will find our number even if the main domain is down.
(3) Saturday, 11:30 PM UTC - Squarespace still has not responded on any of the channels. But now I notice they changed the name server to `ns-cloud-a1.googledomains.com.`. That's almost correct, but before they were randomly changed by Squarespace, they actually were `ns-cloud-e1.googledomains.com.` (with e instead of a)
Next 30 minutes I spend playing the lottery: I keep creating new Cloud DNS zones on GCP hoping that it would be assigned `ns-cloud-a1.googledomains.com.` as the name server (instead of b1, c1, e1 or d1). No luck. Seems GCP is now reserving `ns-cloud-a1.googledomains.com.` so I will need Squarespace to undo whatever they did for this to work.
I then send GCP an email asking them to create us a zone with `ns-cloud-a1.googledomains.com.` name server. It seems like their API for creating zones has a parameter called nameServerSets which may help (but the parameter documentation says to contact account manager if we want to use it).
(4) Sunday, 01:10 AM UTC - Squarespace finally responds on Facebook and Twitter. They suggest `ns-cloud-a1.googledomains.com.` is a default Squarespace name server and that it shouldn't matter whether it's e1 or a1.
They say it doesn't matter, but they refuse to change it back.
What's even more frustrating is that I can run `dig @ns-cloud-a1.googledomains.com mydomain.com` and `dig @ns-cloud-e1.googledomains.com mydomain.com` and see that the former doesn't work and the latter does work.
They send me a form to submit for account recovery. I submit the form.
(5) Sunday, 02:05 AM UTC - Squarespace responds via email to the form I submitted to gain access to my account. They say it will take 1 business day to get a response.
(6) Sunday, 10:00 AM UTC - I managed to get some sleep. Squarespace is not responding. All the while thousands of customers are not able to access our platform and missing out on their sales. At least now that I know I've exhausted all the options, I can write this rant.