Hacker News Reader

Comment by lorenzohess | Hacker News Reader

Why did one developer have access, even if read-only, to more than 3,800 internal repos?

Read-only access to all non-sensitive code is how things should be. Huge engineering culture and productivity booster. It’s also very useful to keep each other honest (I’ve found so many “interesting” things hidden away in organizations with tight read access restrictions).

It’s called “inner source”, I’m also a fan of such a culture.

Devs not having read access to all code seems like a massive org smell. What’s worse, in many cases not having access doesn’t just prevent you from seeing it it also prevents you from knowing it exists. Now you don’t know what to ask for, who to ask, or what to not implement again.

There is no security risk that you could use to convince me that ”devs should only have access to code they need to modify”.

Yeah I worked in a company that blocked access to their main (terrible) product from some devs. They are not doing too well...

in my org, devs don’t have access to customer data directly, and sysadmins don’t have access to modify code.

It’s a simple rule from a simpler time, to limit the risk of total compromise.

Repos should not contain customer data.

Private Repos, in githubs case, might be customer data.

I think this might be more aimed at ensuring that if an attacker gains access to cloud login credentials via a compromised dev machine, those credentials can't then be used to access customer data.

Not saying it’s good but I think it’s quite common for devs to have read only access to everything. I suspect that with all the recent news, including this, the needle might start to shift a bit.

I think it’s actually non-trivial to determine how many repos you should have read-only access to. I frequently hop through multiple repos that I don’t contribute to, just to understand how the system is architected and what it does at different stages. We even have an internal Claude skill for finding relevant repo for a given problem which relies on personal gh access (via CLI). It _can_ be done more securely but those defaults built over many years will take time to change.

I think it is pretty common that devs have read only access to all source code.

The real question is why github has 3800 internal repos.

Shoot dude, the engineering organization I mentor/teach at a high school has ~75 internal repos.

Robot source code; satellite ground station hardware; satellite ground station software; visualization; satellite hardware; satellite software; nuttx + its submodules for 2 different projects; linux kernel fork; circuitpython fork; raspberry pico tools fork; embedded programming/debugging tools; my lecture notes; my automated grading tooling; etc etc etc. That's just me + ~35 students in classes.

Pretty easy to see how when you have scale you can get to a few thousand.

3800 repos without any orgs/groups must be fun..

*assuming github dogfoods github

each employee with personal fork of some company microservice

It's normal that a dev has *access* to all the code.

But did he clone all the repos into his machine? I doubt it. So, the hacker extracted all the 3800 repos using the employee's machine as a gateway? I doubt it as well, I'm sure they would have detected this huge amount of data much earlier than transferring all of it?

> The real question is why github has 3800 internal repos.

I guess they mean customer's private repos?

> I guess they mean customer's private repos?

I don't think so. It is even worse if a random developer has access to customers' private repos.

Good point. Then why in the world would a company have 3,500 repos? Do they create a repo for each employee?

They’ve been developing git and GitHub for over a decade. It really isn’t surprising they have made thousands of internally available repos. They probably have hundreds just for running automated tests alone.

I am sure many of their employees create repos. Is that strange?

It doesn’t mean they are all masterpieces of elaborate production code.

That is not unheard of at a large software company.

I'm personally up to 400 or so

All the attackers need to do is steal an SSH key and they'd be able to clone everything, no?

Nah GitHub/MS doesn't allow SSH keys for their internal stuff. You have to use git-credential-manager, which enforces MFA

Depends how it's set up. Many companies add an IP address check so if you don't come via their VPN (or are not in the office) the connection will be rejected before any auth is asked.

So you'd need to authenticate for the VPN, which often has 2nd factor.

But I have no idea of how they are set up.

Security is often overlooked internally and seen as source of friction. I worked at a popular US social media firm and it wasn't hard to get a permission that allows me to delete the entire company's dataset. Often arguments around "I'm working on org-level initiative and I need to get permission to get it done" would easily get me the permission.

It _is_ a source of friction.

I can think of _one_ product that allows you to set up low-friction access management, and AFAIK most users of that product don't set it up that way.

Software engineers _should_ be able to request access to dev resources JIT during their day-to-day work, have that access auto-approve in >99% of cases, have it auto-expire if they don't actually use the resources, and have all of that be subject to anomaly detection/approval escalations and other auditing.

Instead in most orgs it's like fill out a form, get your manager (who's always in meetings) to approve and then wait some number of days for a human to click-ops your request. At best you can open a PR and have the changes applied in an hour or two.

You _should_ be able to get access to things pretty much immediately if you need them and they're not sensitive. Then we could deny by default without cratering productivity.

Please name the product (that seems a good idea)

Security is often an excuse to block other teams to do legitimate work and so often it's fairly braindead. Security IMO needs to get it's act together, passkeys is a great example of security gone wrong from a UX design perspective because you can't hold them to the same standards as product or infra teams, they have the special privilege of breaking things and it increasing their metrics.

Tell them to make a better UX and they lose their minds in a huffy puff of fake crisis mode or get avoidant with stonewalling 'secret security stuff' that you can't hold them to account for. Or eat 50% of developer machine performance for "endpoint security" and the carnival of sadness goes on and on.

Signal is an example of security as a product that was actually designed for user UX in mind to give one example.

It’s the big advantage that small companies have over big ones.

I’ve ridden startups through the phase where they transition to “responsible adults”, and start putting in policies and locking things down and generally behaving like the giant corporations they expect to be one day (and that the locker downers came from and are used to).

You can feel the deceleration, like taking your foot off the gas on the freeway. I’ve sat through all hands meetings where the ceo asked why we don’t ship as fast anymore, and since by that time most of the fast moving folk have moved on, nobody has an explanation.

Why not? If you don't rely on security by obscurity, having access to code is not a security issue.

If you want to move fast, you need access. Unfortunately and obviously this allows threat actors to move fast, too. The tradeoff had a different risk profile a year ago, heck a couple weeks ago.

Because every developer asking for permission 3,800 times is exhausting for everyone.

Sounds like a great way to have outages because you can’t tell what legacy features are still in use or not. Or even worse, not being able to ever refactor or clean up because you have no means to discover your dependencies.

Reply on news.ycombinator.com