Agree completely that it's absolute wild to run such a system without backups. But at this point no government should keep critical data on foreign cloud storage.
https://kakaocloud.com/ https://www.nhncloud.com/ https://cloud.kt.com/
To name a few.
Korea is great at a lot of engineering disciplines. Sadly, software is not one of them, though it's slowly changing. There was a similar issue a couple years ago where the government's internal intranet was down a couple days because someone deployed a switch in front of outbound connections without anyone noticing.
It's not a talent problem but a management problem - similar to Japan's issues, which is unsurprising as Korean institutions and organizations are heavily based on Japanese ones from back in the JETRO era.
> South Korean corporate management is stuffed full of ex-military officers
Absolutely not. It was very common in Germany to deny military service and instead do a year of civil service as a replacement. Also, there were several exceptions from the """mandatory""" military service. I have two brothers who had served, so all I did was tick a checkbox and I was done with the topic of military service.
As somebody that has also done work in Korea (with on of their banks), my observation was that almost all decision making was top-down, and people were forced to do a ton of monotonous work based on the whims of upper management, and people below could not talk back. I literally stood and watched a director walk in after racking a bunch of equipment and commented that the disk arrays should be higher up. When I asked why (they were at the bottom for weight and centre of gravity reasons), he looked shocked that I even asked and tersely said that the blinking lights of the disks at eye level show the value of the purchase better.
I can't imagine writing software in that kind of environment. It'd be almost impossible to do clean work, and even if you did it'd get interfered with. On top of that nobody could go home before the boss.
I did enjoy the fact that the younger Koreans we were working with asked me and my colleague how old we were, because my colleague was 10 years older than me and they were flabbergasted that I was not deferring to him in every conversation, even though we were both equals professionally.
This was circa 2010, so maybe things are better, but oh my god I'm glad it was business trips and I was happy to be flying home each time (though my mouth still waters at the marinaded beef at the bbq restaurants I went to...).
The big problem from my point of view is management. Everyone pushes responsibility and work all the way down to the developera so that they do basically everything themselves from negotiating with the customer, writing the requirements (or not) to designing the architecture, writing the code and testing the system.
If they're late,they just stay and work longer and on the weekends and sleep at the desk.
This is the only part that sounds bad? Negotiating with customers may require some help as well but it's better than having many layers in between.
> Korea is great at a lot of engineering disciplines. Sadly, software is not one of them
NHN Cloud is in fact being used more and more in the government [1], as well as playing a big part in the recovery effort of this fire. [2]
No, unlike what you're suggesting, Korea has plenty of independent domestic cloud and the government has been adopting it more and more. It's not on the level of China, Russia or obviously the US, but it's very much there and accelerating quickly. Incomparable to places like the EU which still have almost nothing.
[1] https://www.ajunews.com/view/20221017140755363 - 2022, will have grown a lot now [2] https://www.mt.co.kr/policy/2025/10/01/2025100110371768374
My head hurts
Yes, but your backup DC's can have diesel generators and a few weeks of on-site fuel. It has some quakes - but quake-resistant DC's exist, and SK is big enough to site 3 DC's at the corners of an equilateral triangle with 250km edges. Similar for typhoons. Invading NK armies and nuclear missiles are tougher problems - but having more geography would be of pretty limited use against those.
Primary? No. Back-up?
These guys couldn’t provision a back-up for their on-site data. Why do you think it was competently encrypted?
Doesn't have to be an American provider (Though anyone else probably increases Seoul's security cross section. America is already its security guarantor, with tens of thousands of troops stationed in Korea.)
And doesn't have to be permanent. Ship encrypted copies to S3 while you get your hardenede-bunker domestic option constructed. Still beats the mess that's about to come for South Korea's population.
Using a commercial provider is not a guarantee.
Permanently losing data at a given store point isn't relevant to losing data overall. Data store failures are assumed or else there'd be no point in backups. What matters is whether failures in multiple points happen at the same time, which means a major issue is whether "independent" repositories are actually truly independent or whether (and to what extent) they have some degree of correlation. Using one or more completely unique systems done by someone else entirely is a pretty darn good way to bury accidental correlations with your own stuff, including human factors like the same tech people making the same sorts of mistakes or reusing the same components (software, hardware or both). For government that also includes political factors (like any push towards using purely domestic components).
>They simply should have made local and off-line backups
FWIW there's no "simply" about that though at large scale. I'm not saying it's undoable at all but it's not trivial. As is literally the subject here.
I can't reveal any details but it was a lot more than just a given storage point. The interesting thing is that there were multiple points along the way where the damage would have been recoverable but their absolute incompetence made matters much worse to the point where there were no options left.
> FWIW there's no "simply" about that though at large scale. I'm not saying it's undoable at all but it's not trivial. As is literally the subject here.
If you can't do the job you should get out of the kitchen.
(If IBM was also running the local storage then we're talking about a very different risk profile from "run your own storage, back up to a cloud" and the anecdote is worth noting but not directly relevant.)
A quick search reveals IBM does still sell backup solutions, including ones that backup from multiple cloud locations and can restore to multiple distinct cloud locations while maintaining high availability.
So, if the claims are true, then IBM screwed up badly.
Sorry, not brain not really clicking tonight and used lazy imprecise terminology here, been a long one. But what I meant by "store point" was any single data repository that can be interacted with as a unit, regardless of implementation details, that's part of a holistic data storage strategy. So in this case the entirety of IBM would be a "storage point", and then your own self-hosted system would be another, and if you also had data replicated to AWS etc those would be others. IBM (or any other cloud storage provider operating in this role) effectively might as well simply be another hard drive. A very big, complex and pricey magic hard drive that can scale its own storage and performance on demand granted, but still a "hard drive".
And hard drives fail, and that's ok. Regardless of the internal details of how the IBM-HDD ended up failing, the only way it'd affect the overall data is if that failure happened simultaneously with enough other failures at local-HDD and AWD-HDD and rsync.net-HDD and GC-HDD etc etc that it exceeded available parity to rebuild. If these are all mirrors, then only simultaneous failure of every single last one of them would do it. It's fine for every single last one of them to fail... just separately, with enough of a time delta between each one that the data can be rebuilt on another.
>If you can't do the job you should get out of the kitchen.
Isn't that precisely what bringing in external entities as part of your infrastructure strategy is? You're not cooking in their kitchen.
>But someone thought they were exceedingly clever and they were going to fix this without any downtime and that's when a small problem became a much larger one
The sentence "and that's when a small problem became a big problem" comes up depressingly frequently in these sorts of post mortems :(. Sometimes sort of feels like, along all the checklists and training and practice and so on, there should also simply be the old Hitchhiker's Guide "Don't Panic!" sprinkled liberally around along with a dabbing of red/orange "...and Don't Be Clever" right after it. We're operating in alternate/direct law here folks, regular assumptions may not hold. Hit the emergency stop button and take a breath.
But of course management and incentive structures play a role in that too.
Until you need them - like with the article here ;) - then the FSB says "only if you do these specific favours for us first...".
For those there's sure no problem
Like, don't store it in the cloud of an enemy country of course.
But if it's encrypted and you're keeping a live backup in a second country with a second company, ideally with a different geopolitical alignment, I don't see the problem.
https://www.huffpost.com/entry/south-korea-human-rights-inve...
How Trump’s ICE Raid Triggered Nationwide Outrage in South Korea
https://www.newsweek.com/trump-ice-raid-hyundai-outrage-sout...
'The raid "will do lasting damage to America's credibility," John Delury, a senior fellow at the Asia Society think tank, told Bloomberg. "How can a government that treats Koreans this way be relied upon as an 'ironclad' ally in a crisis?"'
I'm sure cryptographers would love to know what makes it possible for you to assume that say AES-256 or AES-512 can be broken in practice for you to include it in your risk assessment.
But you could also go with something like OTP and then it's actually fundamentally unbreakable. If the data truly is that important, surely double the storage cost would also be worth it.
Exfiltrating terabytes of data is difficult, exfiltrating 32 bytes is much less so.
Huh? An enemy country will shut off your access. Friendly countries don't.
> Even if it's encrypted data, all encryption can be broken, and so we must assume it will be broken.
This is a very, very hot take.
you are seeing the local storage decision under the lens of security, that is not the real reason for this type of decision.
While it may have been sold that way, reality is more likely the local DC companies just lobbied for it to be kept local and cut as many corners as they needed. Both the fire and architecture show they did cut deeply.
Now why would a local company voluntary cut down its share of the pie by suggesting to backup store in a foreign country. They are going to suggest keep in country or worse as was done here literally the same facility and save/make even more !
The civil service would also prefer everything local either for nationalistic /economic reasons or if corrupt then for all kick backs each step of the way, first for the contract, next for the building permits, utilities and so on.
But while countries go from unfriendly to attacking you overnight, they don't generally go from friendly to attacking you overnight.
If the US wants its state-puppet corporations to be used for integral infrastructure by foreign governments, it's going to need to provide some better legal assurances than 'trust me bro'.
(Some laws on the books, and a congress and a SCOTUS that has demonstrated a willingness to enforce those laws against a rogue executive would be a good start.)
America is not a stable ally, and has a history of spying on friends.
So unless the whole of your backup is encrypted offline, and you trust the NSA to never break the encryption you chose, its a national security risk.
The lack of a backdoor can be proven by choosing parameters according to straightforward reasons that do not allow the possibility for the chooser to insert a backdoor. The curve25519 parameters have good reasons why they are chosen. By contrast, Dual_EC_DRBG contains two random-looking numbers, which the NSA pinky-swears were completely random, but actually they generated them using a private key that only the NSA knows. Since the NSA got to choose any numbers to fit there, they could do that. When something is, like, "the greatest prime number less than 2^255" you can't just insert the public key of your private key into that slot because the chance the NSA can generate a private key whose public key just happens to match the greatest prime number less than 2^255 is zero. These are called "nothing up my sleeve numbers".
This doesn't prove the algorithm isn't just plain old weak, but nobody's been able to break it, either. Or find any reason why it would be breakable. Elliptic curves being unbreakable rests on the discrete logarithm of a random-looking permutation being impossible to efficiently solve, in a similar way to how RSA being unbreakable relies on nobody being able to efficiently factorize very big numbers. The best known algorithms for solving discrete logarithm require O(sqrt(n)) time, so you get half the bits of security as the length of the numbers involved; a 256-bit curve offers 128 bits of security, which is generally considered sufficient.
(Unlike RSA, you can't just arbitrarily increase the bit length but have to choose a completely new curve for each bit length, unfortunately. ed25519 will always be 255 bits, and if a different length is needed, it'll be similar but called something else. On the other hand, that makes it very easy to standardize.)
Absence of evidence is not evidence of absence. It could well be that someone has been able to break it but that they or that organization did not publish.
I agree on the evidence/absence of conjecture. However, the impact of the secret feels impossible to keep.
Time will, of course, tell; it wouldn't be the first occasion where that has embarrassed me.
America is a shitty ally for many reasons. But spying on allies isn’t one of them. Allies spy on allies to verify they’re still allies. This has been done throughout history and is basic competency in statecraft.
No nation should trust the USA, especially not with their state secrets, if they can help it. Not that other countries are inherently more trustworthy, but the US is a known bad actor.
We also know this is also true for Russia, China and India. Being spied on is part of the cost of relying on external security guarantees.
> Not that other countries are inherently more trustworthy, but the US is a known bad actor
All regional and global powers are known bad actors. That said, Seoul is already in bed with Washington. Sending encrypted back-ups to an American company probably doesn't increase its threat cross section materially.
Case in point: https://en.wikipedia.org/wiki/2025_Georgia_Hyundai_plant_imm...
> The raid led to a diplomatic dispute between the United States and South Korea, with over 300 Koreans detained, and increased concerns about foreign companies investing in the United States.
That they are. Americans tend to view themselves as "the good guys" however, which is a wrong observation and thus needs pointing out in particular.
> That said, Seoul is already in bed with Washington. Sending encrypted back-ups to an American company probably doesn't increase its threat cross section materially.
If they have any secrets they attempt to keep even from Washington, they are contained in these backups. If that is the case, storing them (even encrypted) with an American company absolutely compromises security, even if there is no known threat vector at this time. The moment you give up control of your data, it will forever be subject to new threats discovered afterward. And that may just be something like observing the data volume after an event occurs that might give something away.
You really have no evidence to back up your assertion, because you’d have to be an insider.
Perhaps is doing a lot of work here. They do, and they are. That is what the Snowden leaks proved.
> You really have no evidence to back up your assertion, because you’d have to be an insider.
I don't, because the possibility alone warrants the additional caution.
> Bernard Squarcini, head of the Direction Centrale du Renseignement Intérieur (DCRI) intelligence service until last year, told French daily Le Figaro he was “astonished” when Prime Minister Jean-Marc Ayrault said he was "deeply shocked" by the claims.
> “I am amazed by such disconcerting naiveté,” he said in the interview. “You’d almost think our politicians don’t bother to read the reports they get from the intelligence services.”
> “The French intelligence services know full well that all countries, whether or not they are allies in the fight against terrorism, spy on each other all the time,” he said.
> “The Americans spy on French commercial and industrial interests, and we do the same to them because it’s in the national interest to protect our companies.”
> “There was nothing of any real surprise in this report,” he added. “No one is fooled.”
I always thought it was a little unusual that the state of France owns over 25% of the defense and cyber security company Thales.
Unusual from an American perspective, maybe. The French state has stakes in many companies, particularly in critical markets that affect national sovereignty and security, such as defence or energy. There is a government agency to manage this: https://en.wikipedia.org/wiki/Agence_des_participations_de_l... .
People who don’t know history think spying on allies is bad.
The data is never safe thanks to the US Cloud Act.
1. future cryptography attacks that do not exist today
2. Availability of data
3. The legal environment of the data
Encryption is not a panacea that solves every problem
Has there been any interruption in service?
Not that they should just go all in on it, but an encrypted copy on S3 or GCS would seem really useful right about now.
Cost wise probably having even a Korean different data center backup would not have been huge effort, but not doing it exposed them to a huge risk.
...in an ideal world anyway, in practice I've never seen a disaster recovery training. I've had fire drills plenty of times though.