Hoax email blast abused poor coding in FBI website | Hacker News Reader

Hoax email blast abused poor coding in FBI website

krebsonsecurity.com

431 pointstodsacerdoti100 comments

Previous ongoing thread:

Email from FBI Looks Odd - https://news.ycombinator.com/item?id=29208276 - Nov 2021 (150 comments)

I suppose it could have been worse. I appreciate that they made the emails very clearly fake. The fact it passed DKIM checks was definitely alarming, though. A lot of people thought there was a flaw in DKIM somehow, but unsurprisingly it's a flaw with the FBI's site.

All in good fun, I suppose.

I'll never forgive the Twitter large accounts hacker for not choosing literally anything funny & instead tweeting a dumb "gibs me Bitcoin"

Could've been the funniest thing the internet had seen in years, & will likely not happen at that scale again for quite some time.

I'm surprised that the FBI hacker didn't use it for some for-profit fraud scheme.

Maybe he doesn't want to be also charged with fraud when they catch him.

One crime at a time

A ton of people don’t ever rotate their DKIM keys. If you’re not using email tooling that handles it automatically on a regular basis it’s an easy thing to do since it doesn’t expire like an SSL cert.

Anybody gets their hands on the private key, ex employee, compromised via hack, etc then everything will sail through.

You may enjoy this (I believe it was on HN last year): "DKIM: Show Your Privates" [0].

[0] https://rya.nc/dkim-privates.html

I remember it! Solid points all around.

Well if you're going to wake up Deus Ex Machina and make it look stupid, it's probably your survival instincts suggesting you add just a dash of "task failed successfully" lest you find out what happens if you're successful, or worse, very successful.

Letterhead and perfect graphics and absolutely perfect text and whatnot? You could send mail literally anywhere. Media. White House. Obscure government... stuff/facilities. International contacts...? FVEY? Infinite rabbithole much. SO MANY social engineering possibilities, like this is absolutely mad.

And then... and then you're on the run for the rest of your life - not only against someone who can have you added to all the nonexistent facial recognition databases, but against unimpressed individuals who will specially go out of their way to find you regardless of where you are.

Alternatively, you can pop the balloon in a way that's very obviously stupid, make absolutely no demonstrative points about social engineering in the process, and leverage everyone's collective panic attack to ensure there's a widespread search for the sending email address that would be much more far-reaching than a news article ever would.

Genius.

> And then... and then you're on the run for the rest of your life

US government entities, like any other entity, aren’t superhuman. Taking basic steps to protect and anonymise yourself would be sufficient.

But what can close the loop there is a patriotic sense of not on my lawn, for want of a better way to put it. "Vendetta" almost seems too strong a word, but maybe in some situations it wouldn't be. Basically the kind of mindset that can drive long-term focus/fixation. That's scary, tbh.

They have a lot of resources though and it's very hard to not leave any trace behind.

If it was that easy they wouldn't be trying to push backdoors in companies, going as far as leveraging Five Eyes nations to legislate backdoors and carry out surveillance on their behalf, and eventually blaming "Russia" for every attack based on trivial IP geolocation.

Somehow every single one of the US' enemies manages to get around it -- from OBL through to the entire ISIS network.

The reality is that the FBI can't beat maths and has no leverage over services from foreign companies. i.e, a Russian VPN on a clean VM is probably enough to skirt the whole of the FBI.

US digital intelligence is, for all intents and purposes, a paper tiger in 2021. The whole thing is a farce to give the appearance of sophistication to act as a deterrent.

>If it was that easy they wouldn't be trying to push backdoors in companies, going as far as leveraging Five Eyes nations to legislate backdoors and carry out surveillance on their behalf,

Just because this would be easier for them to have official backdoors doesn't mean they can't ever do anything given enough interest and funding behind it.

>eventually blaming "Russia" for every attack based on trivial IP geolocation

There's plenty of political reasons to state that (even if they were to know otherwise) and if they actually do have more accurate information on a different entity it could avoid showing their hand if they just attribute it incorrectly.

>The reality is that the FBI can't beat maths and has no leverage over services from foreign companies. i.e, a Russian VPN on a clean VM is probably enough to skirt the whole of the FBI.

Probably can't beat math, but if the FBI is running the supposed "Russian VPN" that gives them lots of information. You only have to make one mistake and you could potentially out yourself. (Assuming you don't have further layers to fall back on)

>US digital intelligence is, for all intents and purposes, a paper tiger in 2021. The whole thing is a farce to give the appearance of sophistication to act as a deterrent.

Are you willing to bet your life that this is the case?

> Just because this would be easier for them to have official backdoors doesn't mean they can't ever do anything given enough interest and funding behind it.

Considering there are random "mom and pop" scam agencies across South Asia and the Caribbean stealing billions of dollars from Americans annually while posing as the FBI and IRS and nothing whatsoever happens to them I'd wager that it's quite unlikely the FBI has the ability to do much of anything regardless of interest and funding.

> There's plenty of political reasons to state that (even if they were to know otherwise) and if they actually do have more accurate information on a different entity it could avoid showing their hand if they just attribute it incorrectly.

This is suggesting there is a 4D chess move at play, which is a straying a bit too far for me. The only other possibility here is if [Russia, China, NK, Iran] know the US is not able or willing to enforce a deterrence and they don't even bother hiding.

> but if the FBI is running the supposed "Russian VPN" that gives them lots of information

That's extremely unlikely to the point where if there was a complex covert operation like this they wouldn't burn exposing it on outing an independent malicious actor.

> Are you willing to bet your life that this is the case?

There are entire groups that have literally bet their lives on this and are still very much alive. If the US government was as sophisticated as you're suggesting, I don't think they'd still be driving patrols around the levantine desert trying to find ISIS members or have a giant fentanyl issue plaguing the country.

I really can't make sense of the idea that the US has all this power at their disposal but completely refuses to use it against actual organised groups targeting the US and Americans, but will somehow put all their cards on the table when Johnny from Idaho exploits a mail server.

"when you requested the confirmation code [it] was generated client-side, then sent to you via a POST Request"

That is shocking. What must the internal culture be like for such an idea to even be a possibility?

I spent two weeks or so working at a government contractor. I put my two week notice in almost as soon as I’d started, and I permanently swore off the industry.

There were some genuinely good, smart people working there, but the culture was such that I’d be very surprised if they shipped a working product in 5 years time. If they did ship it, it would be an awful thing to behold.

I don’t know how to fix this issue. But from what I’ve seen and heard, the best minds (other than the rare altruist) stay in the private sector. It pays better, and there’s just so much less BS to deal with.

I know another government contractor whose idea of client responsiveness is to download the entire server database to the client browser on initial load

That's completely the right strategy for the scale most web apps operate at.

How much more expensive is 100KB of JSON compared to last year?

Moreover, most mobile apps, web or native, could suck A LOT less if they did just that, load all the data at start. I have taken that to the extreme of serving all the code and data in one file. Definitely will do that again if I ever make another app with similar and fairly common constraints. Never waiting for unstable network and zero bugs with some part of HTML/CSS/JS/data missing or out of sync is pure joy.

This works great until you have something that doesn't fit, and it can block new features as a result of that.

I worked on an application a number of years ago where it was trying to load all the comments and details about an internal bug tracker into memory. It must have worked fine at first, but after time it was a POS.

There is no "one size fits all" solution.

If the database fits onto client hard drive and the modifications are rare, preloading everything is almost always better.

If you have a dynamically changing system such as bug tracker, it is still possible to go fully local, but that would require considerable cooperation from server side. When the back-end does not have a fast, efficient API for sending diffs, you may get stuck waiting for it to be implemented. But that's a purely organizational problem.

Of course, all of above applies to actually saving data to permanent storage. Storing everything in memory is a sin by itself.

Until someone doesn't think twice and adds some column to the DB that contains PII.

For a sufficiently small database (that doesn't require more granular access control), that seems to be the right choice. A 1 MB initial load is cheap nowadays.

Are we using the term "database" loosely here to mean some limited subset of data? Because I think of the term as referring to an app's primary datastore, and I can't recall a single meaningful modern app I've seen for which the database is anywhere close to that small.

Think of small, in-house applications managing an inventory of a couple hundred to a couple thousand items.

"Hard discount" stores like Aldi are supposed to have <1500 SKUs, for example.

Got it. Thanks for clarifying. Thought there might be some kind of clever sharding or tooling I was missing.

No really clever tooling. But even where the database gets too big, you can still stay in a similar operating mode. For example, when the user requests the first page of items or performs a search (that you have to do server side due to the size of the whole data set) - why not send the first 100 actual items instead of just the visible data from the 10 rows forming the first page? Or send the first 10 immediately followed by prefetching the next 90?

Now the user will immediately get to see the full item and will be able to page through the results much more quickly.

I've definitely had cases where I had to process the data before sending it to the client, but I've also sent absurd amounts of data and rendered it client side. In fact, I think sending data embedded in HTML to the client is rarely a good idea, and once you've adopted that mindset, apps can look very different.

I made it a few years on the gov side of this equation and had to quit. Like you say, there are certainly lots of bright people…but just as many (if not more) lazy and willfully ignorant people punching the clock and keeping the money faucet turned on regardless of the quality of output, if only to pad their performance reviews (executed x dollars managing project of y size) and keep their budgets from being cut. This attitude is insidious too, as I found myself becoming guilty of the same apathy and laziness around the time I decided to break free and throw myself back into more challenging and meaningful work.

The single largest employer in North America, responsible for untold millions of jobs both direct-federal and government contracting, and you pegged the culture, nationwide, inter-agency, on day one. Pretty amazing insight from not much data. Perhaps you'd like to generalize in a more targeted manner?

Having worked in government (although not in US), top fixes for me would be:

* Make it easier to fire incompetent people. A job that's in the service of the people should not be a cozy "I'm now set for life" type of gig.

* Pay semi-market rates.

* Stop going for the lowest bidder for contractors.

Why semi market rates? The whole problem is politicians win elections by promising low taxes and less spending, so the government pays less, and to offset the lower pay, they compensate by not firing people. Inevitably, this will attract an undesirable amount of people who want to coast, with no ability to get rid of them.

> Inevitably, this will attract an undesirable amount of people who want to coast, with no ability to get rid of them.

Yes, my point exactly. I think it's cheaper to pay people a reasonable salary and less job security. Otherwise it just ends up being a lot of dead weight throughout the organization. This dead weight leads to low productivity, which in turn e.g. leads the management to bring on expensive consultants to try and fix it.

My point was that you need to pay market rates, not semi market rates for that. The trade off of compensation for job security should not be occurring.

Aha, sure. My intention with "semi" was to indicate that it would still not be anywhere near FAANG-levels; but rather closer to a median "good" salary.

If it needs to be FAANG level to attract the workers you need, then it needs to be FAANG level. I do not see why there would be any arbitrary limit.

I remember when healthcare.gov was launched and the clusterfuck it was, and then a much of FAANG level employees had to quickly go and clean it up as charity.

Low wages make for more bribeable employees

Same reasons judges are paid a lot

US municipal is plagued by:

Private sector unions (making them impossible to fire)

Nepotism

Layers and layers of bureaucracy

That's not something unique to the US, I saw the exact same thing across the pond.

There are a lot of highly terrible government contracting agencies out there who charge enormous fees and who knows who actually does the work given the likelihood no one even validates any of the work. Suck our taxpayer money out of the system and leave trash in the wake.

You'd think an agency as important as the FBI would verify who is working on their systems, but probably no one did.

> You'd think an agency as important as the FBI would verify who is working on their systems

Why?

The upper echelons of these federal law enforcement and intelligence agencies are universally political animals with names suffixed by III and IV that instinctually perceive anything as even vaguely technical as far beneath them. The only time something like the security of a network becomes a priority for these people is when it causes them embarrassment. At all other times the operation of these systems is a budget item that gets farmed out according to the prevailing political prerogatives of the day; actual competence being well down on the list of priorities.

This wouldn't be the first time the FBI mismanaged software contractors: https://www.centreforpublicimpact.org/case-study/fbi-virtual...

Or the second: https://www.newsweek.com/fbis-expensive-sentinel-computer-sy...

It's the government.

Lowest bidder.

Not low bidder, most politically connected.

I have been trying to get US government contracts for years through my company, including offering $0, $1 and other guaranteed low price bids to try to get the work. We exceed every requirement in the RFPs. We are recognized as the best in the nation in our service area and have 200+ full time employees. Crickets.

What does this actually mean? Why does it need to be sent to you (presumably referring to the client) when it's generated client side.

Agree this wording is confusing, but from the article I think we can infer that the client makes s POST request to the backend which sends the email.

From the government work I've done. The most important thing is the paperwork and paper-trail, everything else is secondary.

A comment elsewhere is that the site was made with the 'IBM Form Experience', and that this issue is possibly just a part of how that product works.

In which case I could totally see how it's part of the culture, having worked in orgs like this:

- A person brings up this concern

- "Uh sweetie, I think IBM wouldn't make a mistake like that"

Being able to scapegoat IBM of all companies would just show how far people in the org are out of the loop.

How incredibly stupid. A twelve year old who spent two seconds thinking about what confirmation codes are for would realize the error of generating them in the browser. What do they think conf codes do? Just be there for show? Don't programmers have to pass some kind of minimal literacy test to work for the government?

"Just be there for show?" Yes. In the same way that many law enforcement organizations (not so much the FBI, but definitely US Marshals Service) will carry around badges and take extreme umbrage at anyone actually attempting to confirm their identity.

An important highlight in this article is the Internet Explorer requirement: The site is old.

The security focused mindset we have today in web development just wasn't developed to that level whenever this thing was written. It's kinda a case in point for replacing websites entirely from time to time.

This is so funny. I've seen websites that leak one-time code through client code so the verification could be automated, but this is another level. Generating a code client-side and allowing the client to decide what the email content could be!

It makes me chuckle to think of the contractor’s thought process on how you implement email verification. How could you not even Google something so simple before you reinvented it yourself in the worst way?

It takes time to google things, and the engineer is paid to write code not google stuff.

A department manager at a previous job once blocked stackoverflow. It went about as well as you'd expect and he unlocked it two weeks later.

Probably was written before google came out ;)

I have used the LEEP portal. Honestly - people are making a huge deal about this, but the verification code could be completely removed and it wouldn't matter. You can start the same process by just emailing the helpdesk.

> I am contacting you today because we located a botnet being hosted on your forehead.

Brutal.

> “Members of the RaidForums hacking community have a long standing feud with Troia, and commonly deface websites and perform minor hacks where they blame it on the security researcher,” Ionut Illascu wrote for BleepingComputer.

I appreciate that krebs give an explicit source to the claim - it shows journalistic integrity.

Ironically, by targeting this guy, they give him more clout, and with more clout comes more work opportunities and potential resources to identify these groups.

It's good that no harm came of it.

Also, on a side note.. this is our gov, this is how they operate. I worked for a short period on a project with the state government and it was miserable. The culture is truly suffocating. I've warned many, gov jobs is where your career goes to die; there is a stigma whenever you go anywhere else even if no one says it

Miserable as in nothing ever got done, even after requesting creds (once i got certified) they dragged their feet for 3 months. It was the worst gig.

Uhhh, there was definitely harm...

The FBI's helpdesk # reportedly got swamped and this probably wasted hundreds if not thousands of man-hours of agents getting panicked calls from organizations they actually work with.

I'm guessing this wasted hundreds of thousands of man-hours of time at organizations around the globe as people tried to figure out WTF was going on. I'd bet a lot of people told their bosses it was obvious bullshit and were told to call a local FBI office to confirm anyway "just in case."

The person who exploited this could have done a proper vulnerability disclosure.

Or sent a genuinely funny/clever message along the lines of "We were lying about the aliens all along, press conference to be held at DoJ HQ this Sunday, 7:15AM" to a couple of news stations.

Whoever did this came across the vulnerability and decided to be an asshole about it.

> The person who exploited this could have done a proper vulnerability disclosure.

If that proper vulnerability disclosure happens to land on the desk of some irrational apparatchik at the FBI that doesn't like your brand of facebook posts or doesn't want to be exposed as an incompetent they won't hesitate to open a file on you and dispatch a cadre of life ruining agents. And before you say "but if it's done properly..." I say hire a good Beltway lawyer before you say a mumbling word because you don't know what 'properly' is or if it even exists.

I wish they'd done a self-referential bulletin along the lines of "We have found a vulnerability in one of our security bulletin systems which allows attackers to craft and deliver notifications which seem legitimate..."

> Members of the RaidForums hacking community have a long standing feud with Troia

Anyone know what the beef is? Do they think he's incompetent?

He’s cybersecurity expert/researcher, and has been trying to identify TDO for a long time now.

Not only he identified some of them to the authorities, he also literally wrote a book on it (https://www.amazon.com/OSINT-Toolkit-Intelligence-Gathering-...), exposing them and the methods to the general public.

I guess we must disable view-source in all browsers now! Welcome back to the 70s where you could only use a phone made by AT&T and got to rent it at $10/month.

Chrome is actually adding a feature that lets school admins disable view source because school test pages had the answers in the source.

Enterprise-level Chromium removes view-source to prevent students from cheating [0].

[0] https://news.ycombinator.com/item?id=29211611

Can you still write something like javascript:alert(document.body.parentNode.innerHTML) in the address bar and press enter?

document.documentElement.innerHTML might be better than document.body.parentNode.innerHTML. The latter fails if there is no body element, e.g. when the page is an SVG document.

If you want a closer approximation of the full source, outerHTML might also be better.

I've also seen "new XMLSerializer().serializeToString(document)" suggested. That seems to give the most complete source, but I've also read that it might have problems with things that need escaping. I have no experience with that approach because for what I needed the first thing I found when Googling, document.documentElement.innerHTML, gave me what I needed.

One more thing to consider. All of the above I'd expect give you source that would produce the currently displayed page including any modifications that were made after loading by JavaScript (which is probably what you'd want for cheating on a test so is fine). I'm not sure that is the same as what "view source" gives--does it give the current page or the page as it came over the wire?

I worked in DSL support for one of the baby bells back in 2002 and talked to a customer who had a line item in her bill for telephone rental that went back as far as the billing system could show me. It jumped out at me since I'd seen thousands of bill plans and never saw that line item before.

She said she had the phone installed on the kitchen wall around the time her grandson was born. He was 27. I told her she could replace it for less than 20 bucks and she said no thank you she liked this phone very much.

She wasn't calling about the bill, she just wanted to get her e-mail working.

She spent upwards of $3,000 on that phone in rental fees alone. She might be still paying it today for all I know.

Using a system designed to warn of a cyberattack as part of your actual attack. Hopefully the Department of Redundancy Department does a full security review.

mildly ironic that the FBI employs so many elite hackers but can barely keep its own properties safe

I don't buy it when news hits that the FBI took down some ransomware gang or seized bitcoin or what have you. I've never heard of a single former FBI hacker, I don't know of anyone who would want to work for them (who wouldn't pick another agency first). Their pay is terrible and they disqualify almost everyone who has the background of a hacker. They must have some other agency do the deed and then they take the credit out of legal necessity.

I think this quite accurately explains it: https://xkcd.com/538/

Not saying the method is violence per se, but rather that there are a lot of alternatives to finding vulnerabilities and backdoors.

Plausable deniability for any email FBI sends out?

Good sports, well played.

I refuse to believe this. How in anyone’s mind this idea (email sending part on FBI site) was ok?

We should not blame "the gov" or "organizations". This kind of crap was coded by someone, maybe more than one person. These are the people to blame. Our profession will never be a truly profession until we (developers) are not held accountable for the crap code that we wrote.

Our profession has a relationship with complexity that no other engineering discipline has to deal with.

Other engineers can reasonably design around known variability in the environment. You can engineer a 4x safety margin in a bridge. No such concreteness exists for programs.

When we make a product, we really have no idea what the landscape of computing will look like in the future. Even the projects that are less than five years old that I’ve worked on have had so much grafted onto them that I barely recognize what I had originally wrote for it. My hunch is that the email system is basically a “legacy app” that had more and more jammed into it as time went on. The prudent thing to do would have been to go with a new provider, but that is extremely expensive compared to jamming new features like that script into it.

In this scenario, how would a developer be held accountable? Would telling a judge “I really didn’t want to write this code but the client demanded these changes” be a viable defense?

If the dev has that in writing, sure.

and now you just described every dev ever becoming more of a lawyer than a dev. why would we want that?

It's called job security... imagine if anything else was built like modern software...

> This kind of crap was coded by someone, maybe more than one person. These are the people to blame.

I'd say it depends on why the actual problem is there.

Did a developer get strong armed into ignoring any potential problems by the management because it was necessary to ship software to meet some made up deadline? I don't think the blame lies with the developer, perhaps more so with the management.

Did a junior developer get tasked with getting something done with ancient technologies that just refuse to cooperate with them properly, without any processes being in place to catch these sorts of issues? I don't think the blame lies entirely with the developer, perhaps more so with the overall environment and the lack of testing, QA and other processes.

Did some developer just not care? Then the blame probably lies with the developer, but if that's the case, why are they even employed in the org, and why wasn't their work caught in one way or another before hitting prod?

Honestly, if we introduce full criminal responsibility for the code that individual contributors write, we'll end up with the same situation that happens in countries that choose to make their doctors have criminal responsibility for procedures gone wrong - they'll simply choose to work in other countries where they're not faced with such circumstances.

Christopher Wray (FBI director) needs to resign. We live in age where internet is very very critical and this could cause huge damage.

Right, right.

Also, the next zero day in Windows means Nadella should be tossed out of Redmond. It is very critical, no?

I don't agree with the GP's call for resignation but this is way worse incompetence wise than an average windows zero day.

Zero day is the wrong analogy - perhaps: "The next time microsoft.com dns expires and is renewed by a good samaritan"

And, yes, it would be time for some senior folks to reflect on their continued helmsmanship.

I think there are a lot of reasons he should resign, but this isn't one of them.

Dupe: https://news.ycombinator.com/item?id=29208276

EDIT: All right, “Dupe” might be the wrong word; “Related to the same newsworthy event” might be better.

I wouldn’t classify this as a dupe. This is much clearer about what happened and how than the scattered comments in the reddit threads.

Yeah, this is an update if anything.

Your link is to some one reporting the phishing email where it wasn't known to be phishing at that time. This link is to an acknowledgement from Krebs and the FBI that it is indeed, a phishing email.

Not dupes.

Reddit isn't a primary source?

Reply on news.ycombinator.com