All Xero services facing major outage | Hacker News Reader

All Xero services facing major outage

status.xero.com

43 pointsJ_tt26 comments

As an ex-Xero employee, I just had to blink twice seeing this trending on Hacker News. For context, Xero is primarily huge in New Zealand and Australia with a large UK chunk but not much presence in the US. While I shouldn't say too much, I'm sure the incident history speaks for itself that this isn't unheard of :) https://status.xero.com/history Arguably underreported I might think considering the userbase however

Admittedly, this incident is probably a pretty big one since it was acknowledged on their Twitter account. They'll mention it in response to customers but you only see an actual broadcast to followers when something is really on fire. I hope the responders at least get a bit of sleep (it's currently 11pm here in NZ and the HQ is in Wellington, New Zealand)

Can you shed any light on what the codebase is like?

Using it feels like there are two seperate templating libraries and apps that are serving different routes. Is there any truth to this?

Ahh dang, I wrote a whole post and then I accidentally closed the tab :)

As background, Xero is in the same boat as a whole host of different businesses. It's nothing unique to them at all. They started with a single C# monolith and that scaled until it started running into trouble. At this point, you start splitting out your monolith into different services and Xero is quite a ways down that track.

At present, there's still a decent chunk of code that still lives in the original monolithic codebase. The initial home screen (dashboards etc) live in there among other things as an example.

When you navigate between services however (ie; invoicing or payroll), those are actually entirely different sites run by other teams that you're switching between, even though the UI is the same (the navigation bar is itself a separate service too for example which is why sometimes you get incidents where the nav bar disappears, if that service runs into trouble)

There is an in-house CSS library that is standard to keep everything consistent but there may still be older versions of it between services or even different JS libraries depending on how old or new a given service is.

So I guess to answer your question, the answer is yes, there are multiple templating libraries and things like that under the hood.

Beyond that, I probably shouldn't go into too much detail! Most of the above I think you can infer just by navigating around the application and seeing how various cookies, stylesheets and what not change so there's nothing particular unique.

Interesting that their competitor Sage is having almost exactly the same outage, which started at roughly the same time: https://status.sage.com/

Although, I prefer Xero's status update to Sage's at this point: I don't think it's appropriate to claim all your other services are "operational" if you can't log in to any of them!

In Sage's case, they're having an Identity outage and Xero have their own identity setup internally (ie; not dependent on any external services) so it'll be purely coincidental.

3 hours down so far and no notification on the login page, no explanation in the status to say they have diagnosed the issue and no eta for a fix is very very poor for a service of this type. Particularly annoying as I have an invoice to create.

Xero aren’t exactly well known for their customer support. :-)

We use Xero for all of our accounting but what has always shocked/surprised me is that their backup/export facilities are so poor. We have all the source data so it's not a disaster if Xero loses data, but we would pay double or triple to have a monthly data dump a la Slack.

Apparently, Xero is an online accouting SaaS. Never heard of it before.

I'm in the UK and 100% of my clients and probably around 50% of the small business owners I meet at networking events use it.

Xero being down is a big deal for a lot of people.

I use it because my accountant makes me

That's not all he makes you do

Very common in Australia also.

One of the first global accounting saas providers. Forced Intuit to rapidly improve their cloud services but have failed to gain significant market share in the US.

Their presence in the USA is small compared to Quick Books but outside of the USA they are massive.

Xero is more popular in oceania. Where as quickbooks is popular in NA

Saw the headline a few times before noticing it wasn't Xerox.

Canada checking in.

Xero has about 60% of the SME market in Oz.

Xero doesn’t let you export or backup your data. One Kiwi company does auto Xero backup (Control-C) but their interface is super clunky - I know they are trying to fix their stuff though but the backup service provided is for use in emergency only.

Each time I see an outage at Xero, and they are getting more and more lately, my heart stops.

I fear one day I’ll wake up and my business won’t work.

Still - for SME’s, the alternatives are sparse.

I wish they offered offline and backup.

I wish they had a competitor with similar pricing and features that offered backup.

What’s a backup going to do? It’s not as if a database backup would enable business to continue the next day. For a data model the size of Xero, the logic of the application is inherently tied to the data.

In the event of catastrophic data loss at Xero, you could at least take your debits and credits, transactions, invoices and other basic accounting data with you to another vendor. Sure, Xero adds plenty of application logic, but at the end of the day, it's just double-entry accounting.

Xero is miles away superior to qbo but only because qbo is such an atrocious product.

Qbo does seem to have reachable customer service but only does so because their platform is so unreliable and riddled with errors.

That being said this is the 1st time i hear of a major xero disruption.

There's nothing else out there to compete with either of them unfortunately.

I was an early adopter in the US. I've been relying on Xero and it's APIs for my business to operate since 2009! This is the first time I can ever remember an outage like this. There are definitely some things that annoy me about the product, but I understand that when you deal with a companies general ledger, you can't afford to "Move Fast and Break Things." I do not envy the sysadmins getting woken from their sleep right now.

Double take moment, used to work for Xero during uni ~5 years ago in Auckland.

Great people and company to work for, since then they've continued to grow massively and everyone I worked with then is now team/country/region lead.

Fingers crossed they're back up and running soon.

Direct link to issue, but doesn't show affected services:

https://status.xero.com/incidents/h2cmcdfcpxgl

Reply on news.ycombinator.com