QuickBooks Cloud Hosting Firm iNSYNQ Hit in Ransomware Attack
krebsonsecurity.com
krebsonsecurity.com
There is discussion on twitter that the company said the backups were on the same network as the data. Hopefully there is an offsite backup available.
https://twitter.com/ConleyU/status/1151862278909825024
https://twitter.com/MRasconCPA/status/1151894366291734533
https://twitter.com/hockeygirlPDX/status/1151945932935585792
Ouch. This is the sort of stuff that can kill a company.
Does Quickbooks with the cloud option offer local backups?
Backups are so simple, and yet the only times people seem to realize their true value is when they don't have any.
At the very least, if you have backups, you might have the ability to rebuild/redeploy, but for any non-trivial system, without a comprehensive DR plan that is drilled regularly, even having backups won't save the business if the tooling to get those backups into a production ready state.
If a Cloud/SaaS provider is going to claim to be your hassle-free one-stop shop, shouldn't they provide, as a matter of course, a feature to dump your data daily/weekly/monthly to a designated off-cloud site of your choosing? But they don't, because they want to pretend that they are impenetrable and 100% available.
The goal to hopefully influence companies into providing such a fundamental ability.
It gets more interesting when you're trying to backup a distributed database. When your durability strategy is to spread your risk across a lot of nodes in a cluster, and you're federating your cluster across multiple sites, traditional backups may not be a practical option. However, as I write this, I'm realizing that there's nothing simple about distributed anything, and saying "it's hard to..." may not be all that interesting.
Keep your backups on-site but off-line - you already have many locations. You should practice a location (and it's backups) going offline at the same time as a code bug screws with data in the active system.
We restore our MySQL and Cassandra data every day in our production DC from backups in the DR DC. It's the only way to ensure things are working. It's about 100TB of data, but things zip along nicely now that 100gig networks are (more) common.
Disagree. A high-confidence backup plan that includes continuous restoration is of middling difficulty to design. Many get backups wrong and only find out when their archives turn out to be incomplete.
(Which works reasonably well as long as the damages are quantifiable and the other company is actually able to pay them....)
There's a third-party or two that can use the API to do a backup/restore, but also, still, not all of the QBO data has an API. Eg, IIRC, recurring transaction tasks.
Just a matter of time for ransomware to replace data via APIs.
I asked about this in a cybersecurity meeting at an investment bank a few years ago - how much security is there around Bloomberg's data feed? All of the algorithms that make million & billion dollar decisions rely on that data so I asked if someone stepped in the middle and fed the algorithms bad data or even slightly delayed data would we even know it had happened? They looked at me like I was a crazy person. I'm actually kind of surprised it hasn't happened yet. Or maybe no one has noticed yet.
I wonder how many other feeds/sources like that in which so many businesses rely on in at least some level of autonomous fashion.
Reminds of the story of Google wiping out all of their CDN servers by Diskerase because of a single detected value, IIRC.
How do you create a backup server that is reachable by production servers (so that they can back up to it) without then being vulnerable to the same kind of ransomware attacks that infect the production servers? You can't exactly make them read-only, or else they can't accept the "legitimate" writes that might occur during the normal backup process.
This gives me the heebie-jeebies.
It seems better to me to have one on Amazon, and one on another cloud provider entirely. Plus regular sneakernet archives, if your data is of a size that would permit that sort of thing.
I'm fortunate that this kind of thing is handled by other people where I work. It's my understanding that we have production data on one service on the east coast, a mirror with a different company in the south, and weekly backup archives stored offline on the west coast.
10TB is not even the largest single HDD money can buy... and I wouldn't use a single drive anyways. What I'd do... There's the new Silverstone CS381 chsasis with is <30L (so it easily fits anywhere) and houses eight 3.5" hot swap disks plus 4 2.5" for boot and a micro ATX PC... Looking at https://diskprices.com/ I'd go with the Seagate 8TB drives at 120 bucks by shucking them from external and then you need a cheap uATX board and an LSI-9207-8i card.
That case does look very nice for a storage box though.
My home computers push backups regularly throughout the day, and every day I create snapshots of each volume (how long to keep the snapshots is another question). This snapshot can only be accessed or managed on the NAS itself.
This effectively creates an append-only backup NAS thanks to the periodic snapshots.
https://docs.oracle.com/cd/E23824_01/html/821-1448/gbciq.htm...
You just go back to the last good version.
Production has no access to backup.
Backup has read only access to production.
Backup writes are append and not overwrites.
Deletes/archival are governed by a retention process.
It's nice to have a restore step in there too, so one has both a validation that the backup is usable, and gives a "playground" where one can have a safe day old environment for testing/training/whatnot against production data.
That is essential. If you do not regularly test restoring, you do not know if you have backups. I normally overstate this just a little for effect, "If you don't test your backups you don't have backups", but that isn't precisely true.
And yes, it frequently makes sense to base test/fixture data and whatnot on backup data flows, but unless you don't deal with any PII, you're hopefully not using that data without a sanitization step.
To anyone viewing my answer as a guide: a backup by itself is not a continuity plan. There are other important areas such as setting recovery metrics (RPO/RTO) and testing restores against those metrics. The world is rife with stories of unreadable backups and/or restorations that took days instead of hours.
1. Gain access to change the code on the front-end web servers (usually PHP)
2. Change the database access layer to transparently encrypt data being written to the database, and decrypt data being read from the database. The key would be loaded into memory by curl'ing an attacker-controlled website at startup.
3. Wait 30 days
4. Notify the company that they're compromised, turn off the attacker-controlled key service, and restart the web front end
Now step (3) ensures that most data in the database has been re-written, and if your backups are dumps of the production database, you now have a month of encrypted backups that you can't read... If you're lucky, you may have a month-old backup to restore from; if you're unlucky, you rotate every 30 days.
Implementing zero downtime transparent encryption on any moderately complex codebase above the database layer sounds like a big engineering challenge. Even moreso if you're trying to do it without the sysadmin noticing. Even moreso if there is no persistence for the encryption key so every user request ends up having to fire a request to the attackers server - that'll kill performance. I doubt many attackers would attempt it.
Read through any modern multi stage remote exploit. The amount of engineering required to pull these stunts off - reliably no less - is truly staggering, and yet attackers continue to innovate and develop new techniques and even more complex exploits.
And you overestimate the competency of the defenders: sysadmin? What small business has a dedicated sysadmin? It’s not like we are talking about running a global hosted chat service like slack here. (Oh wait...)
BTW Persistence for the key in memory is trivial: create a shared memory segment and store the key there, with a ttl.
Was it a site in maintenance mode, where no one was working on anymore?
Oh, and there must not be a DBA at the company either, since absolutely nobody ran a SQL or analytics query in an entire month and noticed that some of the rows were full of unreadable garbage..
This story must be about someone's single server, single dev LAMP blog from ten years ago if it ever actually happened.
Yes, we are talking about the long tail of little LAMP servers handling small business needs with some custom code written by a contracting shop. What’s a devops? DBA? Once the site is developed and working, there’s little reason to pay to change it.
Edit : thinking about it, this story doesn’t add up. Doesn’t this mean the client must have a copy of the decryption key, meaning any cached client would render the ransom demand worthless? It’s just also so much easier, if someone has that level of access, to make a copy of the database, encrypt it, then blow away the old one. Doing a silent deploy of client code with no one noticing seems way harder.
Honestly, it used to be Eastern Europe, until we started realizing they've got some serious programming talent and contracted some work out. Now it's less bad, though still a bit sketchy. Of course, this doesn't help nation-state attacks.
Where should I send the resume?
Where it breaks down is assuming only the web server that is compromised has access to the database.
Of course, it's also possible it's not, and I wonder if that's more likely. Every industry has such stories; maybe this is the white whale of forensics.
The network is segregated to limit impact if they do get hacked.
To keep everyone familiar, Data Recovery processes are practiced regularly.
...
Of course IANAL so idk how this jives with various EU laws.
As usually with security, the principles of least privs and segregating as much as possible are important.
Cronjob to an (S)FTP server and an upload script trigger to chown/chmod all incoming files making the whole thing WORM (Write Once Read Many).
Once its submitted the same user account can't alter it. Even if the malware is clever and scans for .netrc and .id_rsa and manages to create its own connection to the backup server it doesn't have access to anything anyway.
- I have a UnRaid machine, and a backup machine. The backup machine is a small itx board, and has a single HDD attached.
- A NodeRED instance has a so-called "Flow" on the UnRaid machine that is waking up the backup machine every 7 days.
- Thanks to anachron, with a 10 minute delay, rsnapshot connects to the UnRaid machine, pulls the data, and then issues a shutdown to the backup machine.
This setup let me sleep pretty well.
EC2 -> S3 bucket with only write access and versioning enabled. EC2 -> EFS and it's a rotating set of 7 with 7 different security groups that rotate.
This does not affect non-iNSYNQ QuickBooks instances, such as those operated by Intuit (the creator of QuickBooks).
I was concerned because one of my clients' customers rely heavily on QuickBooks Online and her app integrates heavily with it.
Also, what kind of hacky backup system takes this much time to sort through to identify issues. They should have a clean image, and a clean way to backup/restore data for the application being hosted as a pull from production/active deployments.
In the end, this will or maybe even should kill the company in question. Beyond this, it is an opportunity for others. For that matter, really surprised Intuit doesn't have this as a cloud service at this point.
They do have a cloud offering, QuickBooks Online:
https://quickbooks.intuit.com/online/
But it does not have all the same features as the Desktop version, giving rise to a number of third party offerings, like Right Networks' "QuickBooks Desktop Cloud": https://www.rightnetworks.com/cloud-solutions/accounting-sol...
It's been my outspoken opinion that this was an inevitable outcome for as long as I've been familiar with their product.
I was randomly one day looking at dentist new patient forms and one even wanted to know your relationship status, not sure how that's relevant if a single or married guy gets a cleaning... I know home alone when the internet went out, so called the local cable company to see if an outage and the lady wanted the social security number on the account before continuing, which I didn't know. Just insane how many things use the same number, it's like single sign on for real life.
Same issue with bank account numbers. To pay someone with direct deposit, they can use the same number to withdraw from your account. I'm surprised banks haven't figured out a way to offer deposit only option... Just create a new account number but linked to another account, where deposits to account 4321 goes to account 1234 instead, but can't ever withdraw from 4321.
I got a feeling Facebook's account system is probably more secure than my local bank. Pretty sad when someone's hobby blockchain project has more technology in it than banks with billions of dollars of assets under management.
This is for spousal rights - i.e. if your spouse is allowed to request access your data.
> I'm surprised banks haven't figured out a way to offer deposit only option
They have, some German banks assign an IBAN also for "Sparbücher" (saving plans). These cannot be withdrawn from.
For withdrawal security, under SEPA rules you have 8 weeks to (instantly!) reverse a transaction. If you misuse this, you can get your account closed and criminal proceedings filed so that is a relatively effective fraud prevention.
I know I heard in Europe there's some law called PSD2 that banks would provide standard APIs too, but haven't been following that space since not in Europe. I know there's budgeting and other apps but they login to your bank account and scrape the data. I was using a app that categorize your spending for a little while but got sick of it making me relog my accounts over and over. I think one of my credit cards was thinking their servers were trying to hack my account. So a actual official API sounds like the move in the right direction.
It basically means things like YANB (budgeting tool) or freeagent ( accountancy tool for small business) can get access to data you want them to, without you giving them your credentials.
Historically, most banks offered all or nothing access, and the aggregators would screen scrap using credentials that can steal all your money, and the links usually broke, or needed you to contioniously re-auth.
Its actually working. Modern banking apps will likely start to pull in all your financial services in the future.
It's like a Wordpress hosting provider getting attacked. Wordpress is just software hosted there.
There was much discussion of the poor state of Baltimore's IT infra (see https://www.baltimorebrew.com/2019/05/21/baltimores-out-of-d...), and at the same time Microsoft released a patch for an RDS flaw (CVE-2019-0708, https://msrc-blog.microsoft.com/2019/05/14/prevent-a-worm-by...) which says "Vulnerable in-support systems include Windows 7, Windows Server 2008 R2, and Windows Server 2008. Downloads for in-support versions of Windows can be found in the Microsoft Security Update Guide. Customers who use an in-support version of Windows and have automatic updates enabled are automatically protected. "
So unpatched servers are where I'm laying my money.
It seems to me that it's time the OS providers start providing a very easy way to restore the state of data. We all know that backups are the answer but as long as people, have to think about it, there will always be some that don't do them. And now that you can get a 1TB HD for less that $100 then it's a no-brainer.
Virus protection is now automatic with Windows when will backups become automatic on all OSs?
Wat?
How are you imagining this would be implemented? Right now you can have a cronjob that runs rsync which allows exactly that but doesn’t protect you from ransomeware. Are you thinking if it were part of the file system in the kernel it would be better protected? I guess that’s partially true but in practice probably will do very little (also, that already exists on Linux in the form of things like ZFS.)
Once something like that infects the machine there really isn’t a whole lot the OS can do to prevent the damage, all it’s doing is handling user requests for operations on things like the file system and the malware is operating as a proxy for the user.
What a mess, though. Worst part of a business to be crippled is its core - financials. A part often overlooked by techies. If you can’t invoice, you can’t pay the wages.
Like suggesting DotNetNuke when WPEngine goes down.