Hacker News Reader

Comment by lenerdenator | Hacker News Reader

I think you could interpret this as "you need to know, personally, the party you're handing this off to, and make reasonable judgments as to whether or not they could be easily compromised by bad actors".

Like, meeting someone at several dev conferences should be a requirement at the very least.

View on HN

> Like, meeting someone at several dev conferences should be a requirement at the very least.

This is utterly and completely unfeasible. Most open source maintainers, especially those that are struggling and are pressured to hand-off maintenance, don't have the time, means and will to travel to meet up with prospective co-maintainers, not just once but multiple times.

In practice it would just result in projects getting abandoned, the prospective co-maintainer starting a fork, and everyone switching to use the fork.

Intelligence agencies mastered fooling people about their bona fides in person a long time ago. Meeting someone in person will stop casual individuals who just want to crash the project for the lols or some other personal-level reason, but it would have been barely a bump in the road for this clearly-nation-state-level attack.

It adds another layer of complexity, though, and when someone trips up (and eventually, they will), it lets us know who is doing what and why. It also adds another layer of expense and vulnerability. Part of the beauty of cyberattacks for intelligence agencies is that they are very light on tradecraft. This helps to reduce that advantage.

We actually know who poisoned Alexander Litvinenko and what they're up to today, for example.[0]

[0]https://www.bbc.com/news/uk-35370621

You could probably get about 80% of the job done with just 20% of the work. I’m not in OSS but I hire technical people remotely, and I know many people that do. Some consultant friends have caught blatant scams within the first couple of rounds of interviews on Zoom.

>I’m not in OSS but I hire technical people remotely, [...] interviews on Zoom.

Different situations with different incentives and psychology:

- potential to receive money : job candidates are willing to get on Zoom calls or meet in person because they want a paycheck.

- no money involved & volunteer for free : potential open source contributors are not interested in getting on Zoom calls for $0 pay.

Also, while I'm not a lawyer, and in general the bar is very high to criminally prosecute an employee for doing "a bad job," I wouldn't be surprised if there are major jurisdictions where intentionally backdooring your employer's code could land you in prison.

That's when you dangle a grant in front of them.

Interviewing is a different situation though, because you start having essentially no relationship with the interviewee, and you haven't seen their work. OSS projects don't just add everyone that asks as a co-maintainer. Usually it's someone that has contributed to the project for a while already, and through that has shown they understand the code, are capable of improving it, and can make sensible decisions about the road to take.

Yea, I get that, from what I see in the details of this case the contributor was very competent. What I'm questioning here is whether in OSS projects there is enough face-to-face communication, probing about values, etc.

I don't think it's crazy for a maintainer to Google the person a bit, and if there is no positive match, ask the other person for at least a little bit of detail about themselves, like where they live (country/city), who they work for, etc. Maybe hop on a phone call or something.

In this case, Jia Tan just doesn't seem to match any real person we can find online. It's not like there's an elaborate online persona that they have really built.

While I don't want to put Lasse Collin on trial since he's a victim too, I do think he owes the community an update and explanation of what went down. It's not because we want to point fingers at him, but to learn from the experience.

Really, it starts before things get bad. This thing where - in the famous XKCD example - a single guy is thanklessly maintaining a project for 20 years in Nebraska needs to stop. Software libraries like these are no longer a one-person job. They can't be for the bus factor alone. Major projects like Linux distros or bigger foundations like Apache or Mozilla need to start harping on people hard to contribute to important libraries. We'll get to whatever the buzzword of the day is once we do the important work first.

Find a way to make it happen. Sorry, "I just can't" isn't going to cut it after this.

Is the argument that well known software should be taken over by professionals? There are many motivated software maintainers, including single guys in Nebraska, who have better operational security than well funded companies.

Remember the recent incident with the signing keys at Microsoft? Or the one before that? And these are the biggest, most well funded, companies on Earth we are talking about.

Organizations such as Let's Encrypt work well because they are staffed with motivated and competent people, not because they are well funded. This is not a problem that can be solved with funding alone.

I agree we need to stop depending on the 20-year hobby project of the guy in Nebraska, but adding barriers (which requiring travel and in-person meetings is) to sharing the load is not the solution. What these projects need is the necessary resources (mostly money) for multiple people to work on it professionally.

What I don't understand is, where are all the code and security contributions from Big N and other multi-billion-dollar international scale and users? Do they all have their own internal fork of every major library? If not, you would think that they would be their own financial interest to keep somebody on payroll to maintain fundamental libraries like this.

I think this is a really important point. Every commercial contract I've been involved in has clauses that are intended to mitigate supplier risk, eg that they go out of business, and the contracts people do due diligence on suppliers to vet that they are who they say, try to eliminate the one-person operations, and generally mitigate risk if they really need the code but the only supplier is a tiny startup.

Perhaps large corpos need to apply their standard risk mitigation lens to their supply chain. Their stack or their security depends on these 390 packages. 27 of them have less than 3 maintainers. Recommendation: find alternatives.

> bigger foundations like Apache or Mozilla

What bigger foundations? Apache foundation has yearly revenue $2.1 million. Why do you think they reacted as they reacted to log4j? There are no resources.

Open source is running on fumes.

pretty much this.

That's why for whatever anyone thinks of Theo's antics, I appreciated the OpenSSL/LibreSSL Valhalla blogs and overall effort to do something about it.

TBH I'm amazed in it's current state that Apache took in Pekko(FKA JVM Akka...), part of me is guessing it's because some of their other infra is dependent on it...

Foundation based OSS is on fumes. Open core... I am still hopeful for on many levels.

1) Not everyone (current and potential future maintainers) has the time to go to dev conferences.

2) Simply meeting IRL is a terrible proxy for credibility.

> 2) Simply meeting IRL is a terrible proxy for credibility.

Disagree; trust your intuition, but you can never do that if you never meet IRL.

Also, it's not racist or xenophobic to recognize that some countries exercise nearly complete control over their citizens (and sometimes indirectly over non-citizens), and that those people could be putting themselves at extreme personal risk by disobeying those dictates (assuming they did disagree, which doesn't seem to be a given)

> WRT 2, no, it's not.

> Trust your intuition, but you can never do that if you never meet IRL.

I'm sure Edward Snowden also met up with colleagues in the office at least a few times. May have even passed a security clearance.

> Also, it's not racist or xenophobic to recognize that some countries exercise nearly complete control over their citizens (and sometimes indirectly over non-citizens), and that those people could be putting themselves at extreme personal risk by disobeying those dictates, if they even disagreed with them.

Hold up, where did I make this claim about national origin/external pressure?

I'm only suggesting if you have pets, a kid, or a project at work, conferences take a non-zero amount of time to plan to attend.

Plus, what conference options even exist if you're finding other people for the xz library? Searching for #CompressionConf2024 isn't turning up much.

> I'm sure Edward Snowden also met up with colleagues in the office at least a few times. May have even passed a security clearance.

And that's why we know who Edward Snowden is. That's more than we can say about Jia Tan.

Say what you will about what he did and why, it is going to be very, very hard for someone to explain to a contract's security auditor why, in the year 2024, a commit from an account known to belong to Edward Snowden is in the source code of security-critical software.

And that's what FOSS-based companies and orgs need to start doing after this. If I'm working for Debian/Mozilla/Apache/wherever, I'm going to start asking project maintainers more about who they are. "Hey man, we've got an all-expenses-paid trip to one of the major conferences this year, which one can we put you down for?" needs to come out of someone's mouth at some point, and excluding some very good reasons and evidence for why they can't appear at one of these events in-person (think health or long-term family obligation reasons, confirmed by multiple people who know the maintainer), they need to be at one or more meetings within a reasonable amount of time. Randomly-timed remote video meetings could work in a pinch.

If they can't after a couple of years, then these projects need to inform the maintainers that they'll be forking the project and putting it under a maintainer who can be verified as a living, breathing, single person.

Repeat until there's at least some idea of who's working on most of these projects that make up critical systems that society is built upon.

Let's use the current theory that this is a state sponsored attack. If that's the case, another Jia Tan will be recruited. The identity of a single person simply doesn't matter. All that matters is that the attack was attempted.

Consider the issue of candidates who lie in the interviewing process by hiring other people to interview on their behalf. Now replace "interview" with "attend conference". This is just adding another vector of blind trust waiting to be abused.

It raises the bar quite a bit.

Especially when you've met Jia Tan and the new Jia Tan is obviously not the same person.

Meeting in person is quite literally the opposite of blind trust. Blind trust would be assuming that the person physically sitting on the other end of the internet connection and controlling Jia Tan's keys is the same Jia Tan you had lunch with a few months ago.

You cannot prove the person behind the keyboard is the person who is meeting up with people.

This is blind trust because of the assumption that the person is the same.

This is true even when they are no longer in that country. Some governments are known to threaten the family of expatriates. "Do this for us or mom and dad are going to spend the rest of their soon to be short lives doing hard labor" is a pretty tough threat to ignore.

What would be better?

It's naive to believe that any form of physical presence means someone isn't going to do something nefarious in the eyes of the project.

This problem can only be solved by more skilled eyes on the projects that we rely on. How do we get there? shrug.gif.

Anything less is trying to find a cheap and ineffective shortcut in this trust model.

> It's naive to believe that any form of physical presence means someone isn't going to do something nefarious in the eyes of the project.

It's not the only thing, but it is something.

There's a lot of social engineering that went into the xz backdoor[0]. This started years ago; Jia Tan was posting in projects and suddenly someone appeared to pressure projects to accept their code. Who's Jia Tan? Who's Jigar Kumar, the person who is pressuring others to accept patches from Jia Tan? We don't know. Probably some person or group sponsored by a state APT, but we don't know for sure, because they're currently just text on a screen.

Having this person or group of people have to continually commit to the bit of publicly-known open-source maintainer who attends conferences, has an actual face, and is on security camera footage at multiple hotels and airports is far, far harder than just talking a vulnerable person into allowing maintainer access on a repository. Making them show up to different places a few times adds a layer of identity. Otherwise these "skilled eyes" could be anyone with a wide variety of motivations.

[0]https://boehs.org/node/everything-i-know-about-the-xz-backdo...

> Having this person or group of people have to continually commit to the bit of publicly-known open-source maintainer who attends conferences,

This is assuming maintainers even care/want to go.

> has an actual face, and is on security camera footage at multiple hotels and airports

The same footage that'll get wiped a few weeks after the conference ends, and quickly becomes not useful.

This is wonderful posturing in the name of security theater but doesn't solve anything.

> This is assuming maintainers even care/want to go.

If they don't want to go, don't use their project. Sorry, these aren't the TI-83 games you passed around at your high school with programming cables; they're the code libraries our society is built on. If my project relies on your project, I need to know who you are. If I can't figure that out, I'll try to find another one.

> The same footage that'll get wiped a few weeks after the conference ends, and quickly becomes not useful.

This is wonderful posturing in the name of security theater but doesn't solve anything.

Along with receipts, eyewitnesses, plane tickets, etc. that put a person at a place at a time. Doesn't all have to be digital evidence.

You have a good point, but there's also a reason why companies like people to come into work and don't hire remotely as much as they should (or could). There's a reason why interviews often include a meal together. Meeting people IRL is good for building trust, on both sides.

This is really not the responsibility of unpaid developers.

Big vendors should pay to get to know them, because they're the ones making the money off of the developers' work, but "I don't want to meet anybody and want to just manage the project" is the FOSS version of "just trust me bro".

This is almost impossible for remote OSS maintainers. Do you want people to upload passports? And what if a three agency can easily produce whatever material you want?

Sounds like it's time for someone to either pay a few visits to the remote maintainer or give them a scholarship for attending a few conferences.

The big companies can do that. But then there is also the question of -- how many of these critical OS libraries are there in the wilderness?

I feel a census coming on.

There needs to be a reckoning of who is doing what where on this sort of thing. After this whole fiasco you'll probably see more contracts wanting to know who's working on these things, and that will, in turn, have people auditing their software's packages.

The reality of such criteria is that it will be a ladder-pull for any new entrants.

Not a workable option, full stop.

That's not what the above commenter said. This may be your interpretation but the above commenter is essentially saying "don't work with Chinese-sounding developers" and is the completely wrong take here. Jia Tan may or may not be Chinese but the core issue is the lack of basic vetting to make sure he/she/they are a real person.

Reply on news.ycombinator.com