Hacker News Reader

I'm not saying you, as an engineer for those companies, should be the one to donate your time and energy toward the problem. We all have competing priorities, as do the maintainers of those FLOSS packages.

I'm saying that your company's CTO, especially one with a very large companies, could likely identify two or three engineers who they pull into a meeting and say "reach out to these guys and get them whatever they need. Here's my cell, call me the moment you need the plane or additional resources."

Seriously, if a CTO has a budget of a few hundred million dollars and thousands of dedicated employees, how hard is it to throw a few crumbs to the open source community to change this situation from being one of a burden on a volunteer effort to, instead, one where they feel like they're in the middle of an international event where their knowledge and services are vital to keeping the internet alive?

Again, I'm exaggerating, but you see where I'm going with this. It's a missed opportunity for some seriously great PR out of a seriously bad situation.

Some time ago, I saw this suggestion from a disaster relief specialist directed towards those who want to help with disaster relief: the best thing you can do after a disaster is stay away. Taking yourself to the disaster zone very often at best consumes scarce resources from those present to manage the disaster trying to bring you up to speed and at worst creates new problems that need solving.

It's not hard to extend this to the kind of software security flaw here. If I'm a developer on a package with a critical security vulnerability that needs to be fixed now, sending me extra developers who know absolutely nothing about the code I'm working on isn't going to be helpful--it's just going to waste my time trying to bring them up to speed (or more often, telling them to just go away). If I actually need help, I'll ask the people who I know can help me; trying to sift through unsolicited help to figure out who actually has the skills to do so would take too much time.

So think hard about what help Google et al could actually be providing to help log4j here. If you have to resort to clear exaggeration to find examples... maybe that's a sign that there actually isn't all that much that they could be doing that would actually be helpful.

> So think hard about what help Google et al could actually be providing to help log4j here.

I think you make a perfectly valid point and one that shouldn't be overlooked. How about this:

"Here's a $100K and an isolated penthouse suite down the road rented for the month where you can focus on fixing the problem and not be interrupted by screaming children. Here's a phone number if you need to delegate any specific tasks to additional teams."

Incentive to help. No added pressure. Just one practical example.

I don’t quite understand why you keep coming back to luxury apartments and private jets.

If children and family were viewed as too much of a distraction, I’m pretty sure the CTO (in this scenario) would simply choose a developer who lacks those distractions.

Let’s say the engineers chosen do have family. Why wouldn’t the company just comp a room in a local hotel?

I'm so confused. I thought we were talking about a single volunteer open source developer responsible for a vital tool, and it was too onerous to give them additional staff.

If you want to help someone, give them cash. A blank check. Not "here's what I think would be helpful and now you should arrange to use it". Not a week at a penthouse, not a butler, not a private jet. Enough cash to pay for those things if they want them.

Just ask them. "What do you need to get this done and pushed out?" Then give them what they ask for. Listen instead of talking.

It isn't quite simple; when negotiating it is better to give cash. When donating it is better to give goods. Particularly if there is more than one person involved on the receiving side.

In this instance either would be reasonable.

I am not aware of a single circumstance in which donating goods is better than cash. What makes you think that?

You didn’t explain why it’s a perfect valid point It doesn’t seem reasonable for Johnson & Johnston at a valuation of 1/2 trillion to free load You are kind of talking about greater good, perhaps those charitable donations should go to medical research or homeless shelters rather than reducing the burden on for profit companies

The valid point is that too many cooks can spoil the soup. Mythical man month, if you will. Adding people who don't have the institutional knowledge to a software project even if they are rock stars at their own companies could do more harm inadvertently when trying to fix something time critical. So the additional proposal made here acknowledges that, and instead tries to remove as many non-work distractions and discomforts as possible for the people who CAN reliably fix this fast.

For sure, but what could be done is eliminating any non-superflous task so they can focus on resolving that specific problem.

Have a team handle all github issues and media inquiries. Another team focus on initially evaluating all incoming pull requests to check for egregious errors or applicability.

Only after making it through the gauntlet would the original maintainers need to read and/or respond to them.

Especially when such overwhelming public attention and pressure overtakes a relatively small team like this one.

There is still a risk that the kind of time required for the maintainers to have to get those teams up to speed on the project and how it works and what needs to be done could be just as much of a distraction. Adding more triage teams might be good in the future, but for now, adding more outsiders without proper context might just add stress.

As with openssl, what needs to be done is that these volunteers need to be given cash so this is more than just a volunteer project. If a particular corporate entity doesn’t want to sponsor some of the maintainers to work on it full time, then the project needs full-time sponsorship by the Linux Foundation, ASF or under the OpenJDK.

This explains how such a “solution” Benefits log4j and log4j users. The part I question from the start is why “google should” Vs “google should” pump the equivalent money into medical research vs “google should” make what is does better

When there's a disaster, there will be emergency responders from other jurisdictions lined up on the state lines as the commanders call to ask if assistance is needed.

This situation is fairly urgent, but I think you might not realize just how many people a CTO at one of these companies manages. There are going to be "OSS fires" more or less constantly so "some major OSS project has a bad vuln" is not the sort of thing that gets a CTO at a company like Google or Facebook out of bed. I've only seen this happen a very few times and they were for problems that were way more serious and complex.

But that is not to say that nothing is being done. At Google, at least, there are organized efforts staffed with plenty of people that are trying to solve the much much much bigger problem of "secure all of our open source dependencies and all future dependencies" rather than the individual problem of "secure this one dependency."

And PR? Google has been running projects like OSSFuzz for years and I haven't really seen it materialize as a large amount of positive PR, even in the tech community.

> And PR? Google has been running projects like OSSFuzz for years and I haven't really seen it materialize as a large amount of positive PR, even in the tech community.

Google's Project Zero is both very helpful and gets them A LOT of PR, both tech and mainstream.

GPZ isn't oncall for urgent bugfixes and, while a truly excellent project filled with great people, isn't the core team responsible for safe imported code.

> There are going to be "OSS fires" more or less constantly so "some major OSS project has a bad vuln" is not the sort of thing that gets a CTO at a company like Google or Facebook out of bed.

If all your IT projects have an RCE vulnerability that’s relatively easy to exploit, that should keep you up at night.

The RCE existed prior to this disclosure. If I can't sleep today, why should I have been able to sleep a week ago? The dirty secret is that an absolutely enormous amount of code is vulnerable and that the solution to software security is not "fix RCEs as they are discovered as fast as possible." If having RCEs keeps you up at night, then I don't believe that there is a single engineer at almost any company in the world that interfaces with the internet that should be able to sleep.

The actual solutions here are at a more abstract layer than individual vulns.

> The RCE existed prior to this disclosure. If I can't sleep today, why should I have been able to sleep a week ago?

A week ago, this vulnerability might have been known at most to a few three-letter agencies. Today, every two-bit script kiddie will be trying to exploit it. It's not hard to see how the situation has changed.

No, the fact of having these vulnerabilities is not a problem (I mean, obviously, but at the level you describe). The problem is having them be known to the world. Especially with a level of publicity like this.

As with earlier comments this seems to oversimplify the problem of throwing people at a problem. Adding people to a project puts more pressure on the current maintainers, to authenticate, validate, train and support newcomers.

Apache has processes for this, and project maintainers pointed people in that direction repeatedly (e.g. https://github.com/apache/logging-log4j2/pull/608#issuecomme..., https://github.com/apache/logging-log4j2/pull/608#issuecomme...).

The Apache foundation receives funding from a large number of organizations already: https://www.apache.org/foundation/thanks.html

Perhaps the right question to ask here is: what did Apache do to help their members in this event?

You can ask this question of the Apache foundation independently, without adding pressure on the project maintainers at this time.

I am guessing FANG engineers (and most engineers in general) would unanimously suggest "delete this ridiculous , ill-conceived JNDI integration ASAP. If people want JNDI integration, use a custom opt-in log4j appender. Don't let this shit be enabled by default". Yet that may not sit well with the log4j folks.

Log4j taking user input to run and execute code is crazy. You don’t think it “sits well”?

This is almost laughably naive. The irony is that you are criticizing poor IT leadership, by offering a suggestion that is beyond poor.

"Let's get 3 of your best guys in touch with the guys over at log4j...."

As if you can just wire up some engineers to reach out to each other on unrelated projects and magic will happen.

Throwing new people who never seen your project at the mainteners during critical situation will just slow them down.

Adding random new people generally does not immediately speeds up the process, but it is actively harmful in the middle of crisis.

> I mean, are you willing to spend your weekend working around the clock to fix my problems?

Surely the difference is you are getting paid, and if your boss says, help these guys out, you can do it? As opposed to some guys with jobs who have a project on the side. The big guys could even do something like offer to pay the maintainers and maybe they can take leave or something.

I agree with both sentiments. The big guys are under no obligation to fix an issue in some library they happen to use. But the log4j guys are under even less obligation when they do it in their spare time.

Everyone should enjoy their weekends.

> Surely the difference is you are getting paid, and if your boss says, help these guys out, you can do it?

No, I'm not getting paid. What leads you to believe in that? My targets are defined yearly and are very well defined, and patching random FLOSS projects is not one of them. And what leads you to believe that others, such as my boss, don't have their own milestones to meet, and instead take random FLOSS requests from random people on the internet?

A FANG is not a magical entity where any engineer can drop everything they're doing at the drop of a hat to work on external projects, let alone one whose only possible outcome is at best total indifference and at worse we get the company to own a problem affecting everyone for no reason whatsoever.

I'm not under the impression that GP means to suggest you personally have any obligation to donate time to OSS by virtue of being an employee at a large company.

Something I believe we agree on is that it is in the interest of large tech companies to spend time fixing critical security bugs in their own programs, regardless of who originally wrote the malfunctioning code and for whom said code was written.

One way to fix those bugs would be to create a patch for the external OSS library in instances where such a library is the origin of the vulnerability. This is especially practical when that library is used heavily as a basic piece of the company's common software development framework.

GP appears to be arguing that these patches should be upstreamed instead of simply being maintained internally until the bug is patched by someone else in the OSS community.

I think that what throwaway is saying, perhaps without trying to do so, is that you can't expect people in a FANG to care about the best interest of their employer, not if there are metrics set up that don't reflect the interest in question. You can't pay six figures salaries and expect to find people without razor sharp focus on personal gain.

I really don't understand why you are defining this as a random, external project. Your software is dependent on this project! It's right in the term "dependency"!

> A FANG is not a magical entity where any engineer can drop everything they're doing at the drop of a hat to work on external projects

If all our projects suffer from a known RCE vulnerability, I’m fairly certain my boss will be happy for me to drop everything to get it resolved.

That’s not nearly at FAANG, but it’s certainly not an undue burden on the schedule.

I’m not suggesting you donate time. I’m suggesting that if a large company depends on open source projects, it may be in their best interests to either use some engineering resources to help out those projects, i.e their engineers would do it as part of their job, or to spend their money on the maintainers of those projects.

If the big guys don’t want to do that, fair enough. But the open source maintainers are not under any obligation to work to anyones time lines either.

> You'd think, in the spirit of open source, these multi-billion dollar companies--like Apple and Google and Amazon--would (...) mitigate the problems.

Your "(...)" elides the word "help," which completely changes the meaning of the quote, and your reply is constructed uncharitably as if that word wasn't in the original statement.

Somehow, I find what you are saying here to be totally unplausible.

> Log4j's maintainers have a far more difficult and challenging job than FANGs

You are saying that the companies that built advanced ML-based Chess/Go engines like Alpha Zero/Go can't solve a simple logging bug involving string substitution?

If your company ends up using the product in all your teams/project and products wouldn't it be in the company's interest to keep the product safe?

How do we know you're not a CTO/C--/manager in your 'faang' just taking this opportunity to bitch about how bad and unreliable open source is? You do have a track record when it comes to this.

> I mean, are you willing to spend your weekend working around the clock to fix my problems?

Wow, that's cynical even for a 'faang' dude.

same. my evenings and weekend are totally gone to put out this fire. which I wouldn't do if i wasn't obligated to

Speaking as an individual, of course you want to sit by the pool this weekend.

But as a professional representative of your org. surely you'll recognized the unsustainability of the situation and that it's far from ideal even in the pure self-interest of the company in question.