CISA broke into a US federal agency, and no one noticed for a full 5 months
theregister.com
theregister.com
Your typical hands-on-keyboard blue team engineer in federal government is a GS-12 getting paid around $68,000 per year (or $99k in very high cost of living areas like DC). They have expensive health benefits, 13 days of PTO a year, put a huge chunk of their paycheck (almost 5%) into a mandatory pension plan that consistently underperforms the market, and can literally go to jail for making mistakes at work depending on the statutory context they work in.
The best people in these jobs burn out fast and quit or they end up having to abandon IC work for GS-14/15 jobs (max pay is around $190 for those) in order to keep up with cost-of-living and justify their careers.
As a result, you have almost zero genuinely capable principal/senior engineers in government who have the authority to architect complex IT systems for security. Instead you get contractors who charge the taxpayers enormous overhead costs and cut corners wherever possible.
If there's one letter to write your congress person to improve government - my vote would be for civil service reform to attract and retain actual top tech talent. They've done it for doctors and lawyers (both of whom can get paid well above the $190k GS pay ceiling), but engineering is still not treated as a comparably skilled professional trade.
A friend of mine, who is a lawyer and does HR for the federal gov't, spent about a week helping me get my fed resume tightened up and I still got nothing. I don't even care about the pay cut. It just seems like interesting work.
A nice balance might be working somewhere as a civilian contractor for those government projects.
They can absolutely make adjustments if congress needed or wanted to. The DoD as of 2019 does direct officer commissions for cybersecurity roles bringing people in as majors IIRC (it's still not as good pay as civilian cybersecurity roles but the gap is smaller and it has the prestige and lifetime benefits of being an officer).
There is less inherent risk for public sector jobs than for companies that can go bankrupt. Hence for the contract value to be the same, the pay needs to be a bit lower.
One valid sounding concern that I’ve heard is that the WASHINGTON-BALTIMORE-ARLINGTON, DC-MD-VA-WV-PA GS Locality Area underpays folks in DC by including farflung areas like PA and WV that skew the cost-of-living analysis. Whether that’s an intentional cost-cutting move or bureaucratic incompetence I’m not sure, but in the end the DC-area federal government pay ranges I’ve seen have struck me as quite low.
I do largely agree with your post but I'm also suspicious that stratospheric civilian tech compensation is a bubble.
But I'd need to be paid more to suffer though any enormous bureaucracy, so it tends to balance out to needing market rates.
https://www.cbo.gov/publication/52637
Since the Obama era, this has gotten worse because there were a ton of people trying to score political points saying they were cutting waste by freezing civil servants’ salaries and that really got ugly in tech jobs because salaries were booming once things like the Silicon Valley wage collusion lawsuit and high demand for security, DevOps, etc. started raising the ceiling for the private sector. In 2010 the top end of the GS scale was competitive once you factored in benefits, hours, etc. but a decade later that just wasn’t the case. I knew multiple people who were trying to stay in the public sector but it was literally 2-3 times more money if they went private even though their skills were considered mission critical for their agencies.
This sabotages contract work, too, because there isn’t anyone qualified to guide or review the work and that tends to burn orders of magnitude more money than simply paying more directly would.
Tech could drop salaries to 40K/year and get just as many resumes discarded in the trash.
> They have expensive health benefits
Hmm… maybe more expensive when compared to private tech industry jobs, but cheap compared to owning your own business.
> 13 days of PTO a year
Starts at 13 days for first 2 years, then is 20 days from 3-15, then 26 days from 15 on.
Plus medical leave.
Plus it’s usually easy to get people to donate leave in the event of a medical emergency.
> put a huge chunk of their paycheck (almost 5%) into a mandatory pension plan
Not mandatory at all. The government puts in 1% for folks automatically. They match up to 5% total.
> that consistently underperforms the market
It literally is the market. They have funds for S&P 500 and Dow total market, plus a few others, all at super low fees.
None of these funds are speculative other than the total market that the fund represents.
* Federal Pay (GS-12): $100,000 * Startup Pay: $150 base + $25 k bonus + equity
* Federal Health Insurance (United mid-tier plan, no family): $2,500/year * Startup Insurance (United mid-tier plan, no family): $0/year
* Federal Leave: 20 days (after 4 years in federal government) * Startup Leave: Unlimited
* Federal Sick Leave: 13 days * Startup Sick Leave: Unlimited
The pension I'm talking about actually isn't the TSP (which is fine, but slightly more expensive than comparable Vanguard funds).
All federal employees must contribute 4.4% of their salary to the FERS now which is taken out of their base pay just like their health/dental/fegli. It used to be 0.8% but congress gutted it a few years ago.
FERS takes decades before it's more than pocket change and the same money invested in the market would yield higher expected returns without requiring you to work 20 years in gov to benefit from it.
The frustrating thing for people in fed jobs is that if you hit your 13 days that's it (during your first 3 years in government). It can be impossible to get PTO until you build up hours again. You have to either quit, negotiate LWOP (often seen as a performance adverse metric on your record), or work. So if you land a sweet concert ticket, see a flight deal, have a friend get married, etc. you better hope you've banked up the leave for it. Since you gain hours every 2 weeks (4, 6 or 8 depending on service) you also start out in government with virtually no leave and can't actually take a 2 week trip until you've been there almost a full year.
I’m not sure if this is your actual experience, or if you’re just reading the docs, but…
Most supervisors totally understand the limited leave for folks in their first two years, and they will frequently grant advance leave (basically leave that gets repaid when earned) for folks who are performing at an acceptable level.
It’s not a shit show unless someone wants to take a lot of leave before earning it.
Weddings, concerts, even helping family for health stuff… all that’s usually covered under advanced leave when necessary.
I would say that the leave situation as a fed is much easier than in an “unlimited leave” situation.
The real shitty part, imho, is “time and attendance”. Kicking out early for your kids ball game, for example, will cost leave. As a business owner, I like that I can just stop working and do whatever.
I guarantee that someone in the org saw a password file and said “yo? wtf? Let’s get a proper secrets vault going we can do it ov…..” *punched in the clit, thrown out a window*
https://www.justice.gov/usao-dc/pr/former-federal-government...
Yes, he shouldn’t have accepted bribes, but in the private sector this would have been extremely unlikely to result in jail time.
Even if jail time isn’t a common thing, it’s far closer to happening to the average person working in the government than it is to those working in the private sector. The private sector simply fires bad employees. The government seeks to be made whole.
Show me someone going to jail for bringing down prod or making the wrong architecture call or choosing the wrong platform/backend/language or even just getting burnt out and spending a week on the clock re-watching all of Star Trek: Voyager. I want to go, "Holy shit, that could have been me!", not "Well no shit he went to jail."
That's a Tuesday for some GS folks. Not all, some of the best folks I ever worked with were GS11-12's. Some of the worst also. Very few in the middle.
Their hesitation leads me to believe these legal repercussions happen more often than not. Would be interesting to see some data on the claims. My guess is the people being held responsible for these things aren't your average developer taking down prod.
Sounds easy. Just keep a list in front of me. Maybe a book. Throw it into a RAG on local ollama. Keep a Teams chat open with the compliance folks.
Right now if a government agency wants to do something like make a webform where you can apply for a passport, they have zero web developers on staff who can do it. Instead they must pay a team of non-technical officials and lawyers to make and adjudicate an RFP. Then pay a contracting firm to put a developer behind a government computer to do the actual work. Putting this contractor in a seat can easily cost the taxpayer $500k a year despite the contractor only receiving $130k of that money. The rest goes to the HR department, IT Department, C-Suite, lawyers, lobbyists, and shareholders at the contracting firm. The government has their own HR/Lawyers/IT too, but the contractor can't use those so the tax payer ends up double-paying overhead and missing out on economies of scale on every contract.
This is one of the many reasons government websites are always $50 million dollar boondoggles that an intern could have done better. The government ends up spending millions of dollars feeding leeching middle-men before they can hand that money to a mediocre dev deep in the bowels of Accenture's cheapest subcontractor.
If an agency just could hire a few strong web developers directly and then assign them to whatever task is needed during a particular sprint, we'd see a massive reduction in cost and increase in the quality of engineers working on our country's most important work. But most agencies are literally not allowed to spend more than $120k on an in-house engineer, while no one bats an eye on them spending 5 times that on an Accenture contract placement.
Isn’t that what usds [0] is for? I think there’s always an alignment challenge for service needs that are outside an organization’s primary knowledge domain. Without knowledge of what the “strong web devs” can and can’t do then the results are often not great [1].
The federal government is an enterprise with 4 million employees (more than half in DoD as military or civilian). So the handful of people at USDS are basically only sufficient to swoop to fix the most dire of dumpster fires.
But how to fix ? Most US Gov agencies are underfunded, it is either beef up security or provide services. Really a tough choice, and the outlook looks like they may lose even more funding.
Doesn't the US outspend in terms of dollars almost every single developed country on the planet at absolutely everything, even in per capita statistics, from military, police to education and healthcare? How could it be underfunded?
It’s not all or nothing here. We spend too much and get too little out of it.
Currency counterfeiting is a different set of laws than interstate financial fraud, which is a different set of laws than throwing a Snickers wrapper on the ground in Yosemite.
Which is somewhat irrevant. Teams dealing with each can still share headquarters, IT resources, support stuff, cafeterias, and lots of other things.
As for IT functions, all of that should be centralized under the GSA with proper security controls. There's no benefit to having every agency maintain its own IT infrastructure. Most of those staff are redundant and could be laid off.
This works for email. Would you work at a company where you had to build and deploy apps on infrastructure controlled by someone in a different department and location, whose boss gave them the mandate to standardize as much as possible to reduce costs? (Hope you like Oracle…)
This argument doesn't hold.
If you’re not in the military, the fact that someone else has a big budget doesn’t help you any more than your neighbor having a Mercedes helps pay your internet bill.
There are general budgets and people build in support costs, of course, but it’s terribly easy to find people who have been asking for budget to replace something years before its end of life but keep getting turned down in the congressional budgeting process. Politicians want to fund things their constituents like, but the unloved internal support app is just as much of a risk to have on your network.
Every other agency and branch of the US government? Absolutely not.
We'll start with the fact that M1s aren't built on a "base", they're built at the Joint Systems Manufacturing Center in Lima. Government owned, contractor (GDLS) run, not a base.
What they do do is refurbish older tanks, which one I suppose could distort into "disassembling", if one wanted to make a rather distorted claim.
The waste contention for that base comes from an Army proposal to temporarily shut down the factory in 2013, which was supposed to save ~$1B. GDLS explained that, sure, can do, but spinning up production again is going to cost ~$1.5B, and restarting production in 2017/18 was always planned. It's not as simple as "the politicians always allocate money to the military".
While centrally managed economies can just mandate that state-owned factories continue to exist, private markets won't do this. If you don't order tanks and missiles, the factories that make tanks and missiles will cease to exist, and the market will reallocate resources elsewhere.
That sounds very implausible, bordering on conspiracy theory.
We of course run these procedures while writing and formally verifying them, before handing them to the customer, not running them would be folly.
Whatever this story is sounds like the worst kind: a small nugget of truth surrounded in a giant ball of shit.
Even more than that, a customer doesn’t just hand us a pile of money and say “talk to you again in 2 years” far from it. They literally “status the status” once or more weekly. If the answer to the question of “what is the plan this week” has the answer of “we’re out of work so we’re just charging you pad our profits” won’t play.
Also, building tanks is fucking hard, I sincerely doubt any company in the industry isn’t using every penny they have to solve problems and deliver a good product.
For those that scoff at the last bit: companies are in the business of making money, and if you deliver an inferior product, you won’t get another contract.
Our state legislature recently voted down adding a new Data Analyst position to one of their departments. That department cannot function without that position, so instead it has to use it's funding to buy that same position from a 3rd party contractor for 3x the price or more.
The result is that we pay more than anyone else for basically everything we do.
They might also generally still drug test? I don't even do drugs, but I'm not going to pee in a cup for someone to effectively do charity lol. Good luck recruiting a professional with decades of engineering experience when you treat them like they're a 16 year old working at Taco Bell. Even someone with 0 years doesn't have to deal with that kind of treatment in industry.
Regarding pay, it's actually pretty bad. A typical IT worker will be a GS-11 to GS-13 depending on location and degree (possibly lower in some locations, maybe higher in some high COL areas). GS-13 in many places is restricted to management and SMEs, though they're bumping up a lot of the "working level" grades because they realize they can't compete in hiring.
To pick a high COL area where you might find GS-13 working level IT folks, San Diego GS-13's max out at $153k. If they're actually GS and not another pay system (has a different pay raise method but usually maps to some GS grades, like Acqdemo) then it takes 18 years to go from GS-13 Step 1 to GS-13 Step 10. Most likely they aren't starting at Step 1 in any grade, let's say they start at Step 4, then it's 12 years to max. Once maxed, they only get the general pay increase every year. There are few technical GS-14 positions (this is changing, but not rapidly) even in high COL areas so the only "promotion" option for many is to go from a GS-13 technical role to a GS-13 management role (same pay) and then leverage that into a GS-14 management or technical role, if someone dies and a position opens up. GS-15 technical roles are pretty rare.
Based on my experience, every federal organization requires at least a public trust clearance for every IT contractor.
I wish to believe there are still people that don’t care about making Yet Another few hundred thousand and just want to actually contribute to society instead of working on ad tech or whatever bullshit.
Equality is good for equality sake. This is a lesson contemporary North Americans seem to have forgotten in record time.
Additional requirements not common in the private sector, such as rigorous drug testing, ethics codes, requirements on gift reporting, increased surveillance, etc., should come with additional benefits to compensate. Instead, government workers submit to these requirements and a substantial pay cut.
That's mostly because conservatives 1) desire tax cuts at any cost and 2) want to demolish the entire administrative state. The stability and consistency that comes with a well-funded civil servant class are an obstruction to their stated goals.
However, I think you're wrong, at least in part, in your third paragraph. I mean, I think the word "mostly" is wrong in that paragraph. Politicians from all political factions are (quite reasonably) under pressure to lower the cost of doing the work of government, and (quite reasonably) to raise the integrity of the process. Combined with some of the dysfunction inherent in agent-principal problems, I think that's more than enough to cause the problem you're talking about. I experience this firsthand in a jurisdiction that has much less of the "demolish the entire administrative state" that afflicts the American right wing (which I'm guessing is your point of reference).
Mind you, I am not claiming that the problem is not badly worsened by American right-wing politics. I wouldn't know. I'm just claiming that the problem is semi-intrinsic to the situation, and I strongly doubt that it's "mostly" caused by those particular political issues.
Background: You make an argument that at least some people should consider putting contributions to society ahead of "making yet another few hundred thousand". I agree with you, at least broadly, and I think the up-thread poster is not disagreeing.
Summary: We're discussing the act of taking a personal financial hit, for the good of society.
The word for that is "charity". That's what that word means.
---------
I also am sympathetic to the GP's point, about which you are so "disgusted", but I think there's room to disagree there.
I am sympathetic because professionally I do work that many people think is "good for society", I currently earn approximately median income (below mean) for my age/gender/nationality, far far below software engineer pay, and I am treated with unbelievable disrespect by my employer, the government. If I was not trapped in this job by personal circumstance (for now), the disrespect part would definitely factor into my decision making about staying in this allegedly-virtuous job. If you're gonna pay people below market, and you treat them badly, that's not a combination that gets you quality employees. Even if there's some social purpose.
The word for that is "charity". That's what that word means.
Calling it "charity" impies it's done out of pity/compassion.
The parent implies it should be seen as a duty / contribution to the country instead.
The wages offered are hardly poverty - just not competitive with the private sector.
Besides, "doing something out of a sense of duty", when duty meant something, has also often meant doing it for free, or even doing it on one's own dime, and it absolutely meant accepting a pay cut.
If we push it lower how are we not expecting that to require poverty? What legion of people in the US do you reckon even have "their own dimes" to spend on being full time volunteer public servants and can afford to serve from a sense of duty? Retirees?
Not all tech jobs are ads. I work in networking equipment and it pays much, much better.
Anyway, my point was they don't even give respect to the people who do that, and still treat you like their property. Same with the vaccine mandates (especially for remote workers): whether you got it isn't the point. My employers have never asked because it was never any of their business.
If you aren’t, your budget will go to someone who is.
I don't buy it thought. I think the reason why the spend is less in Europe is due to higher salary equality - good people take job in government because the salary is only 50% higher in private sector (for tech, even less for other areas).
The areas where the vast majority of Americans actually live are fairly high density.
Some cost might come down to density, but not much.
Arguable, Denmark has a super low density also if you count in Greenland.
As also written, I don't buy that argument. I think the core is inequality.
If we're considering contiguous US, then no.
Europe is 3.93 million square miles.
Contiguous US is 3.15 million square miles.
As a non-contiguous later addition, with a small population, where statistically nobody lives there per sq mile, and is not pertinent to the discussion of population density as related to infrastructure problems?
Except in what's holding US bureucratic efficiency down (what we were discussing), and requires spending inflated federal budgets for little returns, Alaska is a big factor relative to its size...
Lesson learned: The organization did not effectively or efficiently collect, retain, and analyze logs.
Lesson learned: Bureaucratic processes and decentralized teams hindered the organization’s network defenders.
Lesson learned: A “known-bad” detection approach hampered detection of alternate TTPs.
https://www.cisa.gov/news-events/cybersecurity-advisories/aa...
The actual issue is probably that these people are (a) ineffectual at communicating and prioritizing concerns clearly; or (b) good at communicating, but are not being listened to; or (c) they are listened to, but the organization has no practical means to fix this - no money, unable to recruit talent, etc.
Most techies often assume (b), but (a) is at least as common. The last issue - (c) - might be superficially true, although it's usually not correct in a deeper sense: there is plenty of discretionary and wasteful spending in any sufficiently large bureaucracy. Central resource allocation is just a hard problem.
Anyway, my point is that the problems that need fixing are almost never just technical. Recommendations such as "implement sufficient controls to detect malicious activity" seldom get to the root cause. They are still useful in temporarily overcoming organizational obstacles, but it usually doesn't last.
Pretty much everyone gets breached.
The only ones I don't think get breached deep are the really big software engineering companies where most of the company are also software engineers... like Google.
Software is too complex to be secure without a massive team IMO.
We could do better as an industry though. Modern operating system design makes it far too easy to shoot yourself in the foot.
Imagine a world where all we all use memory-safe/null-safe/type-safe languages, applications and data are strictly sandboxed, access to data is only granted using capabilities-based security, application-level security patches are automatically applied by the OS, data was always encrypted while at rest and while in transit, and passwords are completely replaced with passkeys / smartcards (for users) and X.509 certificates (for servers). While this isn't a panacea, it would solve a great number of the most common security vulnerabilities.
Each of these pieces exist individually. There's no reason why we can't have all of these things today, other than support for legacy applications and retraining engineers. However, it's nearly impossible to get away from legacy software needs.
But if you want low hanging fruit... stop writing C/C++, and get rid of passwords. These are the biggest flaws in the stack.
If it was popular amongst the voters to hold corporations seriously responsible, you would see politicians campaign on it and win. It's not nearly as popular as virtually anything else based on empirical data.
Another way to say this is if 80% of all voters, regardless of party prioritized this as the #1, #2, and #3 issue, politicians would pass laws. It makes the politician look good and solidifies their reelection. Likewise, politicians that vote against those laws would almost certainly not be reelected.
Assuming software can be secure (and hence not doing proper defense in depth, limiting the types and nature of information processed, etc). is the bigger issue IMO.
I remember one time Satya said the red teams reported to him which Microsoft services they were currently in. He would then ask the heads of those services if they had detected any breaches. Sometimes there would be services that had been breached for years, undetected. Must have been hard for Satya to keep a straight face.
One phrase that struck with me from their security training: "Assume Breach".
The goal the the red team should be to increase security. Waiting years before telling them means there's a years-long delay before security can be improved. That goes against the goal.
> hired to find flaws in space station
> find that exhaust port has flaw
> single blast to it would lead to reactor
> would blow whole station
> tell my boss
> he asks engineers if there's flaw
> they say no
> he snickers behind their back
> "we know better, don't we anon?"
> doesn't tell engineers
> mfw blown up by Luke Skywalker
He is so incompetent that I initially thought that we have communication problems (I am French). But no - he simply has absolutely no idea about cybersecurity and the teams he "oversees" from that perspective are losing their minds.
I had no idea that you could work for years in such sensitive US environments and have completely no knowledge.
He is good at saying generalities, though, with complicated words.
It's hard to establish constructive dialogue after that, allot of bad feelings and burnt bridges - and sometimes HR. It's tough because there's generally allot of dynamics at play, but I'm sure the impact of this testing was felt by people within the targeted org.
You're missing a few steps between the pivot to partner organizations and the report. The public report was made on 11 July 2024. They revealed the breach to the target last year, June 2023, and then began a collaboration effort at that point, running through September 2023. They also don't name and shame in this public report, we don't know what the target organization was.
I generally find that working incrementally alongside the teams provides better outcomes. This is not to say you can't use black-box testing or perform unscoped testing, but collaborating throughout the process gives you additional visibility that can save a lot of time, especially in large-scale and diverse environments. People generally respond in a more positive and collaborative manner as well.
Did the author mistype Password-Spraying or is there a seperate type of attack known as Password-Praying? Googling doesn't reveal any other hits on this term.
There are many federal agencies. One of them will fuck up.
Same with private companies.
If you have 100 people doing the same thing, at least one of them is going to fuck it up.