Hacker News Reader

Comment by Nextgrid | Hacker News Reader

This doesn’t sound that bad. Unless I’m misunderstanding something, the ERP was used as a source of truth for DNS records, and that mechanism broke. There’s no indication of actual data loss so it should still be in the ERP DB, and it’s a matter of reverse-engineering how it was all put together and then extract the data into a zone file (or into a managed DNS service such as Route 53) at least for the core domains which should at least bring their internal services back online and allow them to proceed further.

View on HN

From a reply the poster made in that thread:

"So, we have backups on an encrypted drive, which we can't access (authentication is done via our custom ERP software, which is down as well - in hindsight, a very bad idea). And the backups don't work. We can rebuild the DNS servers from scratch, and the records are probably still safely stored somewhere inone of the 3 different databases our ERP software uses, but we don't have an authentication key or the scripts used to pull the DNS domain / record information from the ERP system anymore. The API's are undocumented, and there's only one key that can access those records."

Elsewhere, they say that two years ago, they found they were unable to restore from backups...

> Elsewhere, they say that two years ago, they found they were unable to restore from backups...

So their business actually failed a couple years ago, and they have only noticed this month.

They noticed 2 years ago, and the ship sank below the water line this month.

They were already dead 2 years ago, the body just didn't realize it and kept moving until last Friday...

That isn't the failure of the business; backup failure is a screwup of the IT, which I think is not that uncommon.

But the fact that after this event the management did not get it fixed (and requested that IT demo it, three times, with no failures) is an existential failure.

The technical people noticed, left, and they had to hire new ones who didn't know yet. Now the new ones also know.

Basically Wile E. Coyote running in the air.

I'd say if it was such a house of cards, my best bet is that this security chain is full of holes. Maybe the ERP used a fixed password, which can be guessed/bruteforced.

honestly, "password" would be my first guess.

Oh come now, there are password complexity policies in place in any modern software! Try: password123.

it doesn't sound like modern software. At my current employer (to remain unnamed) when I started there were a bunch of MS-SQL servers with an SA password of <blank> (i.e. no password). It took a long time to fix that because we had hundreds of local installs of the client app that had SA/<blank> in the app config file (yes, they were setup to access as system administrator), so fixing it meant reconfiguring hundreds of local installs. (at the time we didn't have a mechanism for automating it.) We still have "enterprise" software that does no password rules, and would happily allow a blank password.

Oh please, even such a crappy place knows better.

Its Password123. Adding capital letters makes it more secure.

But hackers might be expecting capital letters, so disallowing them from passwords is more secure!

Need capital letters and symbols:

Password@123

We don’t know that the records are still in the database. This could be a matter of ON DELETE CASCADE followed by a delete of what looked like innocent data and then boom it’s all gone. Then the integration updates bind which helpfully erases all records and then once there are no domain names to talk to everything is down.

I would be looking at the Internet Archive and Whois too figure out some basic domains and what the registrars are and go from there but of course depends on what business they are in and what their domain names were doing.

Yeah, the poster didn't make totally clear whether all of their business records are totally gone, but it does sound like that's the case. The ERP was very likely the source of truth for everything- including employee information that the poster says is gone (can't send SMS to remote employees).

So, it probably is that bad...

Nothing in his post suggests the ERP's DB is gone, merely that the mechanism which populates DNS records from the ERP has broken down, which probably has ripple effects on everything else as their internal DNS is broken and services can't talk to each other.

But the data should still be there, and it's a matter of reverse-engineering how to go from "ERP DB" to "DNS zone file".

Okay, fair enough. I probably skimmed the post too quickly.

> Skimmed through the MySQL and Postgres databases, no signs of anything domain-related. I'm betting it's all stored in the custom integrated [and encrypted] KV database..

-- https://old.reddit.com/r/sysadmin/comments/ma4mwl/the_absolu...

If the system is old (going by other replies, introduced in a deal 20 years ago), maybe they'll be able to brute-force the encryption...

Encryption being 20 years old doesn't mean it's easy to break, given that AES is 23 years old :)

It doesn't, but in my experience, the ciphers and key sizes commonly used were often not future-proof :). This rests on the assumption that nobody updated the keys/encryption components in 20 years - which I think is quite possible with such internal enterprise deployment.

I did something similar for a customer of ours who was a major UK cinema chain - they had had a falling out with some small company that had built an online booking system.

Highlights:

1 Bookings where made at individual cinemas

2 The network was a packet radio system that blocked :-) only Transaction could be in flight (Leicester square had two channels)

3 Core of system was Perl running on three windows NT machines which manipulated a shonky screen based system (Pacer Cats)

4 Trying to sell tickets online for Phantom menace system crashed at midnight (when tickets went on sale) we made 1 transaction at 6am

% And no credentials so I had to research how to crack NT - I did consider setting up or fleet of a dozen or so of our suns as a cluster to crack NT, but not sure if the internal security would have liked that (aka secret squirrels and the bit that Bruce Shenier worked for

I'd love to read a detailed write-up about this. Why packet radio - what was the real-time requirement and why couldn't it be fulfilled by an internet connection (I understand internet was primitive back then, but ISDN should've been able to give you enough bandwidth?).

We never understood why the where using packet radio as you say in 1999 ip via ISDN would have been fine.

Its complicated by at the time the total lock on ticket booking systems cinema and theatre by a particular company - who wanted 1/4 mill plus to update the system.

It was an interesting project I had to get cover from a v senior manger to investigate how to break into NT.

I agree. My first reaction was, "I think I could fix this". Worst case this may be be proceeded by a forensic recovery of the data. This doesn't seem impossible at all unless there is some other factor at play like ransomware/physical damage.

> > All domain information was wiped out and records became null.

> > Every person who knew how the custom DNS system worked has left the company years ago. Our backups are fucked. Our records are wiped from all domain servers out there.

Sure sounds like the data is lost to me.

To me it sounds like those people who left can command a very high day rate to help with the immediate aftermath. A high enough rate that even if they are doing hours elsewhere some will be tempted to sign in after their work hours to help out.

A company does not just stop existing or being able to function, deleting DNS records didn't empty the bank account or the ability of the accountants to be directed to pay someone... there are lots of options.

To me it sounds like those people who left can command a very high day rate to help with the immediate aftermath

Those people are probably employed somewhere else and would need very significant incentives. More than likely their current employers would take a dim view of them freelancing to help a competitor, so: enough to take a year off or even to retire on. Basically the idea of hiring back former employees as consultants is a non-starter.

I guess if you are in the business of selling services that include custom domains that isn't quite as crazy as it initially sounds...

You are, of course, assuming they have a backup and the backup can actually be restored and that the restore contains the required information.

It does say in the original post that:

> Our backups are fucked.

So I assume that they have already tried and they didn't work.

Early in my career (>30 years) ago I nuked a companies salary database (a missing $ in a shell script moved everything to the file i)

No problem they said - we have backups. They had three tapes.

First tape failed.

Second tape failed.

Third tape worked.

In retrospect that was a useful, if rather stressful, lesson.

> nuked a companies salary database

> a missing $

Bravo!

>a missing $ in a shell script moved everything to the file i

I've always had anxiety when running a shell script, this is a perfect example of why

I always wrap the actual cp/mv/rm etc in echo first before actually running it e.g. running echo "mv $src i" first would have immediately highlighted the problem visually.

It is also helpful to have at least two tape drives, and check the backup tapes against both.

Early in my career, we had a storm that caused an electrical surge. In spite of the power protection, it damaged the tape drive on the main server (unknown to us at the time). The entire rest of the system (CPU, RAM, hard drives (RLL!)) were fine.

Listening to the backups being run, I noticed it sounded a little different. Eventually I determined that the tape drive (old QIC, maybe 40MB capacity?) worked, but it would not switch tracks when reading / writing. So the backup ran, but it just overwrite the same track rather than write to subsequent tracks.

A short backup / restore wouldn't have caught the problem, it was necessary to try to read the whole backup to see. This wasn't caught for a while, so all the backups were mostly useless.

Reply on news.ycombinator.com