Comment by jacquesm | Hacker News Reader

Comment by jacquesm | Hacker News Reader

Parent Full thread

I wouldn't be so quick. Everybody gets hacked, sooner or later. Whether they'll own up to it or not is what makes the difference and I've seen far, far worse than this response by Checkout.com, it seems to be one of the better responses to such an event that I've seen to date.

> Like, how many other deprecated third party systems were identified handling a significant portion of your customer data after this hack?

The problem with that is that you'll never know. Because you'd have to audit each and every service provider and I think only Ebay does that. And they're not exactly a paragon of virtue either.

> Who declined to allocate the necessary budget to keep systems updated?

See: prevention paradox. Until this sinks in it will happen over and over again.

> But mere words like these are absolutely meaningless in today's world. People are right to dismiss them.

Again, yes, but: they are at least attempting to use the right words. Now they need to follow them up with the right actions.

> Everybody gets hacked, sooner or later.

Right! But, wouldn't a more appropriate approach be to mitigate the damage from being hacked as much as possible in the first place? Perhaps this starts by simplifying bloated systems, reducing data collection to data that which is only absolutely legally necessary for KYC and financial transactions in whatever respective country(ies) the service operates in, hammer-testing databases for old tricks that seem to have been forgotten about in a landscape of hacks with ever-increasingly complexity, etc.

Maybe it's the dad in me, years of telling me son to not apologize, but to avoid the behavior that causes the problem in the first place. Bad things happen, and we all screw up from time to time, that is a fact of life, but a little forethought and consideration about the best or safest way to do a thing is a great way to shrink the blast area of any surprise bombs that go off.

> Maybe it's the dad in me, years of telling me son to not apologize, but to avoid the behavior that causes the problem in the first place.

What an odd thing to teach a child. If you've wronged someone, avoiding the behavior in future is something that'll help you, but does sweet fuck all for the person you just wronged. They still deserve an apology.

I think people this approach is overcompensating for over-apologizing (or, similarly, over thanking, both in excess are off-putting). I have a child who just says "sorry" and doesn't actually care about changing the underlying behavior.

But yes, even if you try to make a healthy balance, there are still plenty of times when an apology are appropriate and will go a long way, for the giver and receiver, in my opinion anyway.

Sorry, I should have worded that as "stop apologizing so much, especially when you keep making the same mistake/error/disruption/etc."

I did not mean to come off as teaching my kid to never apologize.

"Sorry - this is my fault" is such an effective response, if followed up with "how do we make this right?" or "stop this from happening again?"

Not a weird thing to teach a child.

It’s 5-why’s style root cause analysis, which will build a person that causes less harm to others.

I am willing to believe that the same parent also teaches when and why it is sometimes right to apologize.

Thanks, this is where I was coming from. I suppose I could have made that more clear in my original comment. The idea behind my style of parenting is self-reflecting and our ability to analyze the impact of our choices before we make them.

But of course, apologizing when you have definitely wronged a person is important, too. I didn't mean to come off as teaching my kid to never apologize, just think before you act. But you get the idea.

Yea, plus, anyone with kids knows that a lot of them just treat "sorry" as some sort of magic spell that you casually invoke right after you mess up, and then continue on with your ways. I teach my kid to both apologize and then consider corrective action, too.

> a little forethought and consideration about the best or safest way to do a thing is a great way to shrink the blast area of any surprise bombs that go off

I don’t think I agree with this at all. Screwing up is, by far, the most impactful thing that can minimize the future blast radius.

Common sense, wisdom, and pain cannot be communicated very well. Much more effective if experienced. Like trying to explain “white as snow” to someone who’s never seen snow. You might say “white as coconut” but that doesn’t help them know about snow. Understanding this opens up a lot more grace and patience with kids.

Most often when we tell our kids, ”you know better”, it’s not true. We know better, only because we screwed it up 100 times before and felt the pain.

No amount of “think about the consequences of your actions” is going to prevent them from slipping on the ice, when they’ve never walked on the ice before.

I don’t see how any of what you’re suggesting would have prevented this hack though (which involved an old storage account that hadn’t been used since 2020 getting hacked).

You don't see how preventative maintenance such as implementing a policy to remove old accounts after N days could have prevented this? Preventative maintenance is part of the forethought that should take place about the best or safest way to do a thing. This is something that could be easily learned by looking an problems others have had in the past.

As a controls tech, I provide a lot of documentation and teach to our customers about how to deploy, operate and maintain a machine for best possible results with lowest risk to production or human safety. Some clients follow my instruction, some do not. Guess which ones end up getting billed most for my time after they've implemented a product we make.

Too often, we want to just do without thinking. This often causes us to overlook critical points of failure.

For the app I maintain, we have a policy of deleting inactive accounts, after a year. We delete approved signups that have not been “consummated,” after thirty days.

Even so, we still need to keep an eye out. A couple of days ago, an old account (not quite a year), started spewing connection requests to all the app users. It had been a legit account, so I have to assume it was pwned. We deleted it quickly.

A lot of our monitoring is done manually, and carefully. We have extremely strict privacy rules, and that actually makes security monitoring a bit more difficult.

These are excellent practices.

Such data is a liability, not an asset and if you dispose of it as soon as you reasonably can that's good. If this is a communications service consider saving a hash of the ID and refusing new sign ups with that same ID because if the data gets deleted then someone could re-sign up with someone else's old account. But if you keep a copy of the hash around you can check if an account has ever existed and refuse registration if that's the case.

It would violate our privacy policy.

It's important that "delete all my information" also deletes everything after the user logs in for the first time.

Also, I'm not sure that Apple would allow it. They insist that deletion remove all traces of the user. As far as I know, there's no legal mandate to retain anything, and the nature of our demographic, means that folks could be hurt badly by leaks.

So we retain as little information as possible -even if that makes it more difficult for us to adminster, and destroy everything, when we delete.

I think you misunderstood my comment and/or fail to properly appreciate the subtle points of what I suggest you keep.

The risk you have here is one of account re-use, and the method I'm suggesting allows you to close that hole in your armor which could in turn be used to impersonate people whose accounts have been removed at their request. This is comparable to not being able to re-use a phone number once it is returned to the pool (and these are usually re-allocated after a while because they are a scarce resource, which ordinary user ids are not).

> I think you misunderstood my comment and/or fail to properly appreciate the subtle points of what I suggest you keep.

Nah, but I understand the error. Not a big deal.

We. Just. Plain. Don't. Keep. Any. Data. Not. Immediately. Relevant. To. The. App.

Any bad actor can easily register a throwaway, and there's no way to prevent that, without storing some seriously dangerous data, so we don't even try.

It hasn't been an issue. The incident that I mentioned, is the only one we've ever had, and I nuked it in five minutes. Even if a baddie gets in, they won't be able to do much, because we store so little data. This person would have found all those connections to be next to useless, even if I hadn't stopped them.

I'm a really cynical bastard, and I have spent my entire adult life, rubbing elbows with some of the nastiest folks on Earth. I have a fairly good handle on "thinking like a baddie."

It's very important that people who may even be somewhat inimical to our community, be allowed to register accounts. It's a way of accessing extremely important resources.

> I provide a lot of documentation

> Some clients follow my instruction, some do not.

So you’re telling me you design a non-foolproof system?!? Why isn’t it fully automated to prevent any potential pitfalls?

lmao you taught your son to not apologize and if he can help it not do anything that gets him caught. maybe this is how we get politicians that never admit they were wrong and weasel out of everything

The prevention paradox only really applies when the bad event has significant costs. It seems to me that getting hacked has at worst mild consequences. Cisco for example is still doing well despite numerous embarrassing backdoors.

Well said, ideally action comes first and then these actions can be communicated.

But in the real world, you have words ie. commitment before actions and a conclusion.

Best of luck to them.

I like this post. No matter how/when/where/why someone apologizes for a mistake on the Internet, there will always be an "Armchair Quarterback" (on HN) that says: "Oh, that's not a _real_ apology; if I were CEO/CTO/CIO, I would do X/Y/Z to prevent this issue." It feels like a version of "No True Scotsman".

<rolls eyes>

I feel like most of these people will never be senior managers at a tech company because they will "go broke" trying to prevent every last mistake, instead of creating a beautiful product that customers are desperate to buy! My father once said to me as a young person: "Don't insure yourself 'to death' (bankruptcy)." To say: You need to take some risk in life as a person, especially in business. To be clear: I am not advocating that business people be lazy about computer security. Rather, there is a reasonable limit to their efforts.

You wrote:

    > Everybody gets hacked, sooner or later.

I mostly agree. However, I do not understand how GMail is not hacked more often. Literally, I have not changed my Google password in ~10 years, and my GMail is still untouched. (Falls on sword...) How do they do it? Honestly: No trolling with my question! Does Google get hacked but they keep it a secret? They must be the target of near-constant "nation state"-level hacking programmes.

> Literally, I have not changed my Google password in ~10 years, and my GMail is still untouched.

The flip side of this is how many people are wrongly locked out of their gmail. I bet there's quite a few of them that failed to satisfy whatever filters Google put in place.

> How do they do it?

To begin with, they have a culture of not following "industry standards".

(For the reason that the industry never had this scale yet)

There are millions of companies even century or decade old ones without a hacking incident with data extraction. The whole everyone gets hacked is copium for a lack of security standards or here the lack of deprecation and having unmantained systems online with legacy client data. Announcing it proudly would be concerning if I had business with them. It's not even a lack of competence... it's a lack of hygiene.

>There are millions of companies even century or decade old ones without a hacking incident with data extraction.

Name five.

The pedantic answer is to point to a bunch of shell companies without any electronic presence. However in terms of actual businesses there’s decent odds the closest dry cleaners, independent restaurant, car wash, etc has not had its data extracted by a hacking incident.

Having a minimal attack surface and not being actively targeted is a meaningful advantage here.

>there’s decent odds the closest dry cleaners, independent restaurant, car wash, etc has not had its data extracted by a hacking incident.

And there's also a decent chance they have. Did we not just have a years long spate of ransomware targeting small businesses?

Most ransomeware isn’t exfiltrating data. For small business you can automate the ‘pay to unencrypt your HDD’ model easy without care for what’s on the disk.

There are definitely companies who have never been breached and it's not that hard. Defense in depth is all you need

Isn't defense in depth's whole point that some of your defenses will get breached?

Take the OP. What defenses were breached? An old abandoned system running unmantained in the background with old user data still attached. There is no excuse.

Not everyone gets hacked. Companies not hacked include e.g.

- Google

- Amazon

- Meta

Amazonian here. My views are my own; I do not represent my company/corporate.

That said...

We do our very best. But I don't know anyone here who would say "it can never happen". Security is never an absolute. The best processes and technology will lower the likelihood and impact towards 0, but never to 0. Viewed from that angle, it's not if Amazon will be hacked, it's when and to what extent. It is my sincere hope that if we have an incident, we rise up to the moment with transparency and humility. I believe that's what most of us are looking for during and after an incident has occurred.

To our customers: Do your best, but have a plan for what you're going to do when it happens. Incidents like this one here from checkout.com can show examples of some positive actions that can be taken.

> But I don't know anyone here who would say "it can never happen". Security is never an absolute.

Exactly. I think it is great for people like you to inject some more realistic expectations into discussions like these.

An entity like Amazon is not - in the longer term - going to escape fate, but they have more budget and (usually) much better internal practices which rule out the kind of thing that would bring down a lesser org. But in the end it is all about the budget, as long as Amazon's budget is significantly larger than the attackers they will probably manage to stay ahead. But if they ever get complacent or start economizing on security then the odds change very rapidly. Your very realistic stance is one of the reasons it hasn't happened yet, you are acutely aware you are in spite of all of your efforts still at risk.

Blast radius reduction by removing data you no longer need (and that includes the marketing department, who more often than not are the real culprit) is a good first step towards more realistic expectations for any org.

Facebook was hacked in 2013. Attacker used a Java browser exploit to take over employees' computers:

https://www.reuters.com/article/technology/exclusive-apple-m...

Facebook was also hacked in 2018. A vulnerability in the website allowed attackers to steal the API keys for 50 million accounts:

https://news.ycombinator.com/item?id=18094823

Nah.

The Chinese got into gmail (Google) essentially on a whim to get David Petraeus' emails to his mistress. Ended his career, basically.

I'd bet my hat that all 3 are definitely penetrated and have been off and on for a while -- they just don't disclose it.

source: in security at big orgs

Do you have a source that the Google hack was related to David Petraeus? This page doesn't mention it[1]. Does the timeline line up? Google was hacked in 2009[2]. The Petraeus stuff seems to have happened later.

Disclosure: I work at Google but have no internal knowledge about whether Petraeus was related to Operation Aurora.

[1] https://en.wikipedia.org/wiki/Petraeus_scandal

[2] https://en.wikipedia.org/wiki/Operation_Aurora

> I'd bet my hat that all 3 are definitely penetrated and have been off and on for a while -- they just don't disclose it.

Considering the number of Chinese nationals who work for them at various levels... of course they're all penetrated. How could that possibly fail to be true?

The relevant difference here is that these companies have actual security standards on the level that you would only find in the FAA or similar organisations were lives are in danger. For every incident in Google cloud for example, they don't just apologise, but they state exactly what happened and how they responded (down to the minute) and you can read up exactly how they plan to prevent this from happening again: https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1S...

This is what incident handling by a trustworthy provider looks like.

Google just got hacked in June:

https://cloud.google.com/blog/topics/threat-intelligence/voi...

https://www.forbes.com/sites/daveywinder/2025/08/09/google-c...

That was a Salesforce instance with largely public data, rather than something owned and operated by Google itself. It's a bit like saying you stole from me, but instead of my apartment you broke into my off-site storage with Uhaul. Technically correct, but different implications on the integrity of my apartment security.

It was a social engineering attack that leveraged the device OAuth flow, where the device gaining access to the resource server (in this case the Salesforce API) is separate from the device that grants the authorization.

The hackers called employees/contractors at Google (& lots of other large companies) with user access to the company's Salesforce instance and tricked them into authorizing API access for the hackers' machine.

It's the same as loading Apple TV on your Roku despite not having a subscription and then calling your neighbor who does have an account and tricking them into entering the 5 digit code at link.apple.com

Continuing with your analogy, they didn't break into the off-site storage unit so much as they tricked someone into giving them a key.

There's no security vulnerability in Google/Salesforce or your apartment/storage per se, but a lapse in security training for employees/contractors can be the functional equivalent to a zero-day vulnerability.

There's no vulnerability per se, but I think the Salesforce UI is pretty confusing in this case. It looks like a login page, but actually if you fill it in, you're granting an attacker access.

Disclosure: I work at Google, but don't have much knowledge about this case.

Google got hacked back in 2010, lookup Operation Aurora. It wasn't a full own, but it shows that even the big guys can get hacked.

They also have plenty of domestic and foreign intelligence agents literally working with sensitive systems at the company.

Didnt Edward Snowden release documents that the NSA had fully compromised googles internal systems?

Yup. The NSA has every single major US tech company tapped at their server level and are harvesting all their data. Issues them NSLs and there is zero way these companies can refuse the taps.

You are joking right?

All of these companies have been hacked by nation states like Russia and China.

fair or not, if their customers get hacked it's still on them to mitigate and reduce the damage. Ex: cloud providers that provide billing alerts but not hard cut-offs are not doing a good job.

Everybody includes Google, Amazon and Meta.

They too will get hacked, if it hasn't happened already.

... that we know of. Perhaps some of those "outages" were compromised systems.

"shit it's compromised. pull the plug ASAP"

Meta once misconfigured the web servers and exposed the source. https://techcrunch.com/2007/08/11/facebook-source-code-leake...

I like your stance.

We also have to remember that we have collectively decided to use Windows and AD, QA tested software etc (some examples) over correct software, hardened by default settings etc.

Reply on news.ycombinator.com