Hacker News Reader

Comment by pc86 | Hacker News Reader

Don't underestimate the absolutely jaw-dropping incompetence from even senior "tech" folks employed by governments.

Within the last couple years, I've overheard senior, so-called technical government employees 1) complain about Git and wonder aloud why we weren't using Visual Source Safe; 2) insist that rotating through a list of 12 hard-coded passwords, in code, checked into Git, was totally fine; 3) refuse to believe that automated deployments were possible (not hard, or against norms, but physically impossible); 4) try to explain to another so-called technical gov't employee the difference between CSS and JavaScript, and get it wrong; 5) stand up in the middle of a conversation and walk out the door because it's 2:30 PM and their day is over; 6) even more nonsense you wouldn't believe if I showed you a video of it.

I would hope Federal is a little better than State, but I'm not convinced.

View on HN

Number 5 sounds kind of nice, actually.

I worked with Federal briefly. I was at a startup at the time, and we had this integration with the Dept. Homeland Security. I was young and stoked. I was the one who was going to wire everything together. It seemed really big and important. I thought it'd be a gold star on my resume.

Reality was super disappointing. So much "we're working on a document for when we can give a date for a date on another document" kind of bureaucracy. I'm sure down in the deep cores of these agencies there are super bright and talented people. The ones at the edges, though.. the problem seems to be that you have to be more of a politician than an engineer to actually get anything done.

>seems to be that you have to be more of a politician than an engineer to actually get anything done.

In Red Mars, they try to colonize Mars. At first, the challenges are completely technical, and the engineers prevail. But the success is chased by increasing bureaucracy and involvement of stakeholders, and it quickly becomes all politics. c:

> But the success is chased by increasing bureaucracy and involvement of stakeholders, and it quickly becomes all politics.

And then quickly becomes colonial revolution.

To be fair I know of people in private industry who have said some of the exact same things. It isn't limited to just government employees.

If you work for a modern software company (not a service company that sells software), and you honestly suggest Visual Source Safe, you'll get fired. There's just no reasonably intelligent reason to suggest it.

You practically have to assault someone in the break room to get fired from a government job.

But if your org is using $oldtech and some new person comes in recommending $newtech, you'll be in great company if you argue that $newtech sucks and you wish you could just keep using $oldtech.

Organizational inertia is definitely a thing, and technology trends do not move as quickly in all sectors.

But how good is $newtech really? Is it someone trying to screw over other older employees in the name of his job security? Is Mr. $newtech in touch with the salespeople hawking this product? Is he being incentivized to advocate for it? How many secondary changes will $newtech result in, and will they deliver so much more value as to justify the expense, inconvenience, retraining, and tears that will come from replacing $oldtech?

My guess is, in most circumstances, probably not

"New tech" lately feels more and more like whatever the next breed of inexperienced geeks decided to reinvent

Well, to move back from the abstraction, Visual Source Safe is truly awful, and you're never safe from it losing your code :)

Whatever your perspective on fad-chasing in software development, some things just suck down through the ages...

I worked for a startup with ex-Microsoft management in the 1990s, so we used VSS. All joking aside, we actually did lose about a day's work to that pile of fun.

I'm told that at this point sales will quietly try to talk you out of buying it.

Would you get fired, or would people just assume you were being sarcastic?

I'd ask whether to set up the build pipeline for VB6 or ASMX Services, lol

You would not get fired. Probably everyone would forget you even said it.

If you want people in a company to be able to produce good ideas, they must have reasonable leeway to disagree or be contrarian.

> > honestly

The question isn't whether you honestly suggest it; the question is whether they believe you when you express sincerity.

> 5) stand up in the middle of a conversation and walk out the door because it's 2:30 PM and their day is over;

My dad would do the same stuff. He worked at Samsung. When he got into his 50s, it was like a switch turned on and he just wouldn't stay beyond a standard work day. Got a meeting you scheduled late in the day? Too bad, he was there at 6am and was going home. He was always home at the same time.

In comparison, during this period of time my mom would go into the office at 8am and stay as late as 1am. She would even pick up 2nd dinner for me on her way home.

There's a difference between saying "I saw you scheduled a meeting for 2-3, just so you know my day is over at 2:30, can we reschedule?" (which I have done, and will always do), and cutting someone off mid-sentence to say your day is over because your clock beeped, and being out the door and in your car 15 seconds later.

Sure, but the latter is completely resonable when the person scheduling the meeting is a pointer-haired twit who would respond to the former with "sure, sure, sounds good" <later> "okay, we rescheduled for 2:00-3:00", which from what I've seen/heard seems to be typical of government work.

Often times government contractors aren't authorized to work overtime. It's illegal to do unpaid work.

Some people that work early-shifted days do so because they have child care obligations. Many day cares charge by the minute for late pick-ups. So an abrupt departure may not be 100% attributable to laziness. Failure to communicate those constrains ahead of the meeting very well might be.

On the flip side, does your employer care if you clock out at 2:25 because you're bored? I bet some employers (like large bureaucracies) don't like that. We lack context.

I've seen worse in private companies. For example I once did some consulting for a company that manages an industrial production line using software written by a single "rock star" programmer leading a team of imbeciles. Dude is adamantly opposed to version control and his "process" for deploying new code to production hardware is to manually rename the active version to .BAK and copy over the new one. He keeps track of versions by remembering which computer has which version installed. This is in a company where if the line goes down, it's hundreds of thousands of dollars an hour till it comes back up.

>Don't underestimate the absolutely jaw-dropping incompetence from even senior "tech" folks employed by governments.

When I was in training with the RCAF, I was in lecture where the Information Systems Security Officer (i.e. the senior most infosec person at this base of 6-8000 people) told us about a time where "two guys were emailing back and forth and just picked up a virus".

I made the mistake of pointing out that that's not really how computers work, i.e. an email can't just pick up a virus in transit like a dog picks up a tick.

I learned two lessons that day:

1. Militaries don't like it when people stand out

2. Governments are fundamentally incompetent

Sounds like those two guys were man-in-the-middled

When you hear hoofbeats, think horses, not zebras.

Or, you know, the guy caught some malware through an entirely separate vector.

My firewall protects you, and your firewall protects me.

Ain't the CF grand?

Nope. It's a joke of a military that can't possibly defend Canada. The people and the technology are all obsolete.

Regardless of how incompetent they are. Incompetence should not be the benchmark for which terms like "Reasonable Burden" are applied against.

Each part of the government tends to give a great deal of deference to the other parts of the government.

This reminds me of when I worked on a public policy project for a biotech company. Our product didn't fit in the current Medicare policies, so we decided to approach Medicare and suggest a solution.

Medicare spent months just digging into the laws and talking to their lawyers just to prove they actually had the jurisdiction to change the policy. That really blew me away. One of their bigger concerns was "this might not be our job".

CMS, as a regulatory agency, does have a high priority in making sure it’s got the legal right to make regulations pertaining to a thing. Because otherwise those regs are illegal, which causes lawsuits and chaos.

You make it sound like they spent months trying to finagle a way out of working.

Throwaway account because I still work this job.

> Don't underestimate the absolutely jaw-dropping incompetence from even senior "tech" folks employed by governments.

100% agree. Except each agency can significantly change on this. Some agencies (DEA, USCIS) can and are completely competent. I've seen then turn around on certain moderately complex tasks within hours.

Then you have the idiots at HISN and VA that takes a year for even basic one-way integration. Or, they will argue with you how SAML works (or doesnt) for weeks at a time.

> 1) complain about Git and wonder aloud why we weren't using Visual Source Safe;

Ah so you were talking with their developers. Sounds DoD'ish as that's what they use, along with Firebird.

> 2) insist that rotating through a list of 12 hard-coded passwords, in code, checked into Git, was totally fine;

Ive dealt with network infrastructure entities that did similar for the commercial side of things. I wish it were the "OMG" exception. There's more terrible out there than this, admittedly.

> 3) refuse to believe that automated deployments were possible (not hard, or against norms, but physically impossible);

Well, they ARE impossible by policy. FedRAMP and FISMA requires manual and intentional deployments. Of course we all know that this is the norm for a software SaaS company - but its not the norm when you have to go through a VPN with a CAC or PIV card, and then another VPN to either the testnet or prodnet... And the only software support for both are Windows. (And the VPNs will also kick you out if you're not running windows.)

> 4) try to explain to another so-called technical gov't employee the difference between CSS and JavaScript, and get it wrong;

I've had screaming fights break out in calls when technical issues came up like this. And usually its the single real employee screaming at the legions of 3rd party contractors. And also, Snowden was correct in that 95% of the "government employees" aren't. They are contractors working for Accenture or the host of other Gino (govt in name only) employees. They don't get govt benefits, nor do they have whistleblower protections. Snowden in Hawaii was such a 3rd party govt employee.

I have only ran across 1 fed employee (contractor or real) that was in any way and shape competent.

> 5) stand up in the middle of a conversation and walk out the door because it's 2:30 PM and their day is over;

I get your point. But it's usually 4p. Ive also seen government employees (employed by the local/state/fed) get in trouble for working non-critical issues outside their hours.

> 6) even more nonsense you wouldn't believe if I showed you a video of it.

One thing I've learned is that there is almost always a reason (and a decent one too) of why there is a rule. It's because someone previously abused something, or did something that broke stuff, and the new rule is the usual overcompensation so that anybody in power can claim "They didnt follow our rules, which protect against that." And remember, it's not far up the foodchain to an elected official.

But yeah, there's absolutely crazy shit out there, especially in the fed side of things. And much of it is for complete show. One such tempest in a teacup issue is the new (as of 2017) NIST password guidelines: not a single fed has implemented them. They still enforce the old and terrible ruleset.

Loved your insights here. I never contracted for DOD, this was for a handful of state agencies in Pennsylvania.

This most likely explains why I didn't get a dime of my unemployment money. Thanks PA.

Where do FedRAMP/FISMA require manual deployments?

Totally made up answer ahead, but based on lived experience in a different country.

It probably happened like this: In 1995 people started planning for this Y2K bug. Some developer said "Nah it's not an issue I'll just change this field" so they changed it like they change everything else, and then 6 months later people got paid the wrong amount because a date calculation was wrong.

So they went looking for someone to blame. They fire the developer of course, but they want someone responsible, who signed off on the release. There was no one, so and with the biggest software fixing exercise in their history coming up they figured they should make sure this problem didn't happen again.

So they introduced sign-off procedures, where someone above a certain level had to sign off on things.

And that is how manual deployments are "banned". They probably aren't exactly, but some poor person will have to put their job on the line to sign off on automating them.

'sign off' procedures, and 'automated deployments' are not different ends of the spectrum.

Nobody who gives a shit about their product/service is deploying from a developer directly into production without any kind of, you know, code review, pipeline testing, etc.

It's 'automated' in that you git push. The pipeline handles code reviews, sign-offs, etc., and then automatically deploys the approved code change.

This isn't banned at all.

Just randomly pushing shit to production is banned (separation of concerns, amongst other controls). You can still have everything be fully automated, just not fully autonomous. Someone reviewing a change and then clicking 'LGTM' is still an automated, continuously integrated and deployed system.

Because someone, somewhere, sometime years/decades ago, auto-deployed buggy/incomplete code to production, and it destroyed something important, and the 'solution' they came up with was "all deployments must be conducted manually by team lead^W^Wan approved engineer^W^W^Wsome random shmuck".

'Where', not 'why'.

I know 'why' people might think that, and make assumptions. The actual controls are pretty level headed though.

> Don't underestimate the absolutely jaw-dropping incompetence from even senior "tech" folks employed by governments.

How do we replace them with folks that know what they're doing? It almost sounds like it would be better for the government to outsource to a tech firm.

These people have huge salaries and pensions, too, right?

You let government fire incompetent people, which is very hard to do with unions like AFSCME in place, in part because "incompetent" in this context is extremely subjective. I'm not saying unions are bad but they do make it objectively harder to fire these borderline folks.

Huge salaries? No. The agencies I contracted for topped out for senior folks in the $85-90k range while the same title in the private sector is going to be about 40-50% higher than that. They have pensions but unless you're already 15 years in you really need to work the system to get anything out of it. New hires (rightly) have to put a lot of money into the system so it's not much better than a private retirement plan in terms of returns, however it is guaranteed by the taxpayers.

There are exceptions, though. If you're retiring from any of the agencies I contracted for, and you worked there your entire career, you're probably getting a lump sum $80-100k vacation time payout and anywhere from $60-100k/yr for the rest of your life depending on your exact job. This is in addition to whatever personal retirement accounts you might have. Obviously no 401(k) with the pension in place, but it was not unheard to have some of the higher ups cashing in just enough vacation every year to fully fund their Roth.

So yeah you can set up a nice retirement nest egg if you're willing to commit to the state (at least where I live). But for short term stuff it's terrible. You don't even vest any pension benefits until I think 5 years now, and then you're getting something like 5-10% from age 70 on or something.

Does that mean these people never take vacations? Or do they get a ton of vacation time?

Based on my observations, too much vacation to take.

State employees with 25 years get 54 days of leave a year[1] and 10 federal holidays. Additionally many agencies work 9 hour days, adding every other friday off, for an extra 26 days. All told, that is 90 days off a year, or more than 1 in 3 work days.

https://www.calhr.ca.gov/employees/Pages/vacation-vs-annual-...

I am incredibly jealous and somewhat outraged.

$80k/yr with pension benefits and a third of your work days off is better than a lot of private sector comp. I'd happily take that last benefit.

I wanted to correct myself, the vacation and annual leave I linked to are not additive. Annual leave is the greater of the two and 30 days opposed to 54. This would be a total of 66 vacation days.

Just to clarify #5 - the hour difference here is suspicious, were you working with an EST person from the west coast? If so scheduling a meeting that overruns 5:30 PM is a pretty unfortunate arrangement.

Suspicious how?

The agencies I worked at routinely had employees' shifts ending at various times, anywhere from 2:30 until 5.

You just described standard govt competency levels in any field.

Reply on news.ycombinator.com