Hacker News Reader

Comment by FigurativeVoid | Hacker News Reader

At my first gig, I had "god" level access to our production database.

All I learned is that nobody should have this level of access unless it is some sort of temporary break glass situation. It is extremely dangerous and even experienced engineers can cause irreparable data loss or some other bad outcome. In our case, some engineer accidentally sent around 10,000 invoices to customers that shouldn't have gotten them.

There are far better data access patterns. In the case of US gov data, I don't see why the DOGE team would need anything more than a read replica to query. It could even be obfuscated in some way to protect citizens' identities.

View on HN

Ah, I remember a time 30 years ago when I logged accidentally into the PROD database (forgot to add the suffix "1" to the connection ID), thinking it was a Dev instance, and then issued a "truncate table CUSTOMERS"... the reaction came within 75 seconds - and restore from backing took several hours.

That's not bad. I wonder how many companies would not be able to recover from such an event in days / weeks / ever.

That's when you pray that the engineering team didn't fix the replication lag.

I've worked with older governmental systems, and chances are they are running a wide variety of systems, some of which, the oldest and most critical, are probably written in COBOL running on IBM mainframe hardware. In those environments, there is no real distinction between "database" and "application". COBOL systems are very file- and batch-oriented, and are "monolithic" in the extremist sense. The technology itself makes it impossible to give read only access to such systems.

You can absolutely give read only access in COBOL systems. That's just lazy administration and IT security on a shoestring budget.

> The technology itself makes it impossible to give read only access to such systems.

This isn't true. Mainframe COBOL systems commonly store data in VSAM files, or DB2, or IMS, or sometimes some more obscure non-IBM database (e.g. CA/Broadcom's Datacom/DB or IDMS, or Software AG's ADABAS). But whichever one they use, there are multiple ways of granting read-only access.

For example, if it is VSAM, you can configure RACF (or TopSecret or ACF2) to allow an account read (but not write) permission to those VSAM datasets. Or, you can stick DB2 in front of VSAM (on DB2 for z/OS, CREATE TABLE can refer to a pre-existing VSAM file, and make it look like a database table), and then you can have a readonly account in DB2 to give you access to that database schema. Or, there's a lot of other ways to "skin this cat", depending on exactly how the legacy app is designed, and exactly how it stores data. But, probably this is already implemented – most of these apps have read-only access for export into BI systems or whatever – and if it happens for whatever reason not to be, setting it up should only be a modest amount of work, not some multiyear megaproject.

>Or, there's a lot of other ways to "skin this cat", depending on exactly how the legacy app is designed, and exactly how it stores data. But, probably this is already implemented

Given that neither of us knows the actual systems in question, what is more likely, that it's a well-designed system or one that has organically accreted over time? It seems like you tend to believe the former, and I the latter. I suppose my view is based on the fact that, like in statmech, you enumerate all possible systems that can do a particular job, the vast majority of those solutions will not have any organizing principle and will not be amenable to surgical analysis or change.

I think the difference is that I know that getting data out of mainframe COBOL systems is a long-known and long-solved problem, and I can list lots of different ways to do it (I mentioned a few, there's several more I didn't mention). Without knowing the details of the exact system, I'm not sure which one would be the best one to use, but the odds that you'd have a system for which none of these existing solutions is suitable is rather low – and indeed, likely most of these systems are already using one or another – there are whole teams of sales people who have spent the last 20-30 years convincing government agencies (inter alia) to buy these solutions.

Whereas, you don't seem to know anything about that topic, and are speculating based on parallels with completely different disciplines (such as statistical mechanics).

We both are speculating due to lack of details about the specific systems under discussion, but wouldn't you expect the person whose speculations are based on greater relevant knowledge to be more likely to be correct?

I'm sorry, but just because I didn't pepper my post with shibolleths like z/OS or VSAM or the vagaries of ACCEPT and DISPLAY keywords, doesn't mean I don't know what I'm talking about. I worked specifically on connecting COBOL system to a DB/2 database, and one thing was for certain: understanding the data format was the hardest part of the problem. Those definitions, in our system, were tightly coupled to the user interface code, AND the batch processing code.

No, it's not my specialty and didn't work with this system for long, but my overall impression was that COBOL programmers get (understandably) low-level abstractions, and therefore had to build higher level abstractions themselves. This is not like modern software development where you have an embarrasment of riches from any level of abstraction you want, and a large system where every part of the stack is a custom solution is generally going to be more chaotic. To put some numbers on it, to add a column of data to the system I worked on required on average about 20k hours of coding work. No doubt some of this was sand-bagging, but I'd say 80% of it was legitimate.

> I worked specifically on connecting COBOL system to a DB/2 database, and one thing was for certain: understanding the data format was the hardest part of the problem.

But now you are shifting the goalposts: from getting readonly access to the data, to understanding what it actually means. Yes, I totally agree, a lot of legacy COBOL systems, it can be very hard to work out what the data actually means - even though you probably have a COBOL copybook telling you what the columns/fields are, they can be full of things like single letter codes where the documentation telling you what the codes mean is incorrect. And likewise, you are right that seemingly simple tasks like adding a field can be monumental work given the number of different transaction screens, reports, batch jobs, etc, that need to be updated, and the fact that many mainframe programmers don’t know what “DRY” stands for

But simply getting read-only access to data? Most mainframe COBOL systems would already support that. Could there be some really badly maintained ones in which it was never configured properly and they just give DOGE read-write access because DOGE refuses to wait for it to be done properly? I doubt that’s the norm but it might be a rare exception. Such a system would likely violate security standards for federal IT systems, but agencies can get exemptions.

> To put some numbers on it, to add a column of data to the system I worked on required on average about 20k hours of coding work.

20,000 hours is 10 years of full-time work for a single person. If you "didn't work with this system for long," it is quite simply statistically impossible that you could have witnessed enough projects to have anything resembling an accurate "average".

>20,000 hours is 10 years of full-time work for a single person.

Or, while we're mythical man-monthing it, 6 months of work for 20 people? Or merely a single sprint for 240 people!

This implies good faith actor, which is not the case.

These old systems need to be upgraded but Congress never approves the financing or execution of it because they're too divided and won't increase taxes on anyone especially the rich and giant corporations to do it.

Never mind the direct risks, if you have "god mode" to basically any government thing, you instantly become the target of foreign intel/military operations. You can bet good money that there are entire teams, if not divisions, working around the clock to exploit this situation.

I can imagine Chinese and Russian hackers laughing at the DOGE l33t hackers.

And if I was advising the Ukranians I'd tell them to try to exploit it too, hey, if you're fighting 2 superpowers with another 1 quietly backing the fight against you, you need all the help you can get.

  > It is extremely dangerous and even experienced engineers can cause irreparable data loss or some other bad outcome

It is literally why we never log in as root.

  HERE BE DRAGONS

I don't know an admin who hasn't, on multiple occasions, unintentionally caused irreparable damage. It is easy to do even with the best of intentions and with extreme levels of care. Any one trying to rush through a dragon's den is only going to get burned. Considering how many dragons' dens they are running into, I do not question "if" damage has been done, but "what".

I remember having some kind of C programming bug where output filenames got scrambled (string memory error probably). And output files in the same folder as the source code.

That seems innocuous, but remember then some of the output files might have the character "?" or even "*". So imagine trying to remove these files and going an asterisk too far. All gone!

I've had a company give me full admin access to their cloud account. Thankfully, I learned the lesson earlier in my career and immediately created myself of more mundane user. Break glass access is important, but definitely not as the usual level of access.

> I don't see why the DOGE team would need anything more than a read replica to query.

They shouldn't need more than limited read access. The fact that they have more access, very likely demanded and not accidentally given, is due to their intent to do more than simply query data.

I loathe working places where they just give you all the permissions because it's "easier". One risk is if something does happen, and they don't have exceptional tracing and logging, (and let's be honest, at an organization sloppy enough to hand out privileges like candy, what's the chance of that?) it's difficult or impossible to pin down the source to any individual. As a result, both responsibility and suspicion is diffuse.

I have so many horror stories from there.

When they did decide to lock down the database, the DB admin only locked in down in the sql server client most people used. If you used some other client, you still had access. _sigh_

What DB system operates that way, that's nuts.

My favorite security anti-pattern! Locking the main doors while leaving all the windows wide open.

A second thought. It leads to lazy application development. Whenever you have production intervention that happens more than a few times, you should just make a feature that does it safely via application code.

I've definitely worked in places where "Move fast and break things" tended to focus on breaking things. There would be bugs that we didn't fix because "We can just fix the database when it happens." It would take 2hours to fix a bug that would cause of 10's of hours of weekly support request, but the focus would always be on building new features, of which 10% got any real usage.

The appropriate restrictions are relative to the size and momentum of the organization. It's easy to spend months setting up safeguards rather than working on product development that won't proportionally return.

Of course, this involves being honest with yourself about risk and reward, and we all have implicit incentives to disregard the risk until we get burned and learn to factor that in.

It's not just about the risk. It signifies that you're not dealing with an experienced database administration staff. (At a startup that might just mean one guy, but that's better than zero.

I agree. Good access controls and being prevented from accessing things that I don't need access to protect me as an employee just as much as the data itself.

Meta completely restricted graph data access to requiring a specific business purpose and managerial approval tied an articulable, concrete task need.

Why should they even have read access? They're not a legal government institution, and they're being led by a private citizen that's not been elected or appointed by Congress to access our data in agencies that were made by Congress under particular rules to keep these kinds of snoops out.

Ultimately someone has root permissions. Re: federal agencies, in the United States, that someone is clearly, constitutionally, the President. Article II of the constitution vests all power of the executive in the person of the President. The President has authority to appoint agents. That same article _does also_ say the President has to "take Care that the Laws be faithfully executed", but the "Care" there is highly debated. But the idea that the President doesn't have the right to appoint Musk to get root access to federal agencies seems legally incorrect.

I'm not make a value judgement on this, it's just how it is. At a startup, the founder ultimately has root access to the database, no matter what the technical controls.

Now, maybe it's stupid, and maybe it should be some other way, but to my mind the other way is that Congress gets together and writes a law saying "the executive cannot get root access to X, Y, Z". In absence of that law, the executive can do whatever they want.

Not to be THAT GUY, but "an append-only database which cannot be modified by anyone" is something HN has spent the past 10 years saying is completely useless...

And Trump can launch the nukes to blow up the world too... but building a system where he can just click a button to do so would be idiotic. Same idea with giving godmode to the guy who thinks carrying a sink and saying "Let that sink in" is hilariously clever.

The power rests with the office. There is an important but nuanced distinction there.

There's a good balance between preventing accidents and reducing friction.

One person having "god-mode" access isn't usually that terrible.

Reply on news.ycombinator.com