Hacker News Reader

Comment by leftcenterright | Hacker News Reader

Red flags for me:

- Github alerted Okta about the access, they were not able to detect this themselves (https://docs.github.com/en/organizations/keeping-your-organi...)

- It only says "access to code repositories" (it does not say anything about the level of that access, it might as well mean write access, capability to trigger actions etc.)

- Not relying on the confidentiality of source code is great, but malicious CD workflow actions would still be a risk if attackers had that level of access.

- No information about the entry point for compromise.

I doubt their 'commitment to transparency'.

View on HN

IAM on GitHub needs so much <3. So broad, much ow.

For example, I trialed major security vendor's enterprise product. They required their app be granted Admin on the GitHub org. All they needed to do was create issues, PRs, and read source code for analysis. There are scopes for that.

I was eventually on a call with a principle engineer in this company, who kept saying they needed this permissions, and I kept showing him the API docs that showed that wasn't so. Eventually he said, "well, we won't _use_ all those permissions, so just give them to us anyway, because it's easier this way." Sure, I'll give you the ability to change all my code, add/remove users, drop repos... etc, and trust that some day, when you're hacked, someone will not use those over granted permissions maliciously?

Security is hard. Be careful what permissions you give your 3rd party GitHub integrations.

Ugh. Doesn't raise the trust in their competence of protecting admin access credentials to GitHub. The same mindset leads to "We use just one shared ssh cert, because it is easier. And our VPN solution is a 2nd factor in any case".

>>"well, we won't _use_ all those permissions, so just give them to us anyway, because it's easier this way."

Devs have been doing that since the dawn on computing. Ohh your App needs to be able to write to a protected folder on windows. Dont document what folder just force the app the run as Admin.

Early Android Apps asked for all the permissions, all the time because of lazy devs

security is hard, and gets in the way of what the devs wnat to do so they just find ways to bypass it

"because of lazy devs" :thinking:

Definitely not because of the pesky product managers and sales teams who want a new feature to sell yesterday to boost their EOY bonus...

Early Android had pretty coarse permissions IIRC. It wasn't quite "root or nothing" but somewhat closer to that than it is today.

It was only recently that PATs got the ability to be scoped per repo, and even that's still in beta.

But frustratingly fine grained scoping doesn't work for repos part of an Organisation! Like, what!

You need to choose from a drop-down that the token owner is the organization, not you. Then you can create a token for a repo of the org. Your org must opt-in to beta.

Ask them to sign a document accepting all liability in those situations. I think the conversation will quickly change..

"Sorry, we don't accept redlines or riders for accounts that are less than $750k ACV."

I had exactly this same experience with Vercel, and we backed out of using them for our major open source repo as a result.

> I trialed major security vendor's enterprise product

> just give them to us anyway, because it's easier this way

Wow. The state of security is still sad in our profession, if even major security vendor(s) don't adhere to basic principles like "principle of least privilege".

It's not just security vendors, it's everyone.

You can't even set up popular software like Tailscale with a github login without it requiring access to your organization's private repositories.

It's like mobile phone permissions in the old days where your calculator needs access to your contacts and location.

I thought technology companies learned this lesson a decade ago, apparently not.

Heh. Reminds of one of Symantec's "Enterprise" products.

Turned out, if you're logged into the central (on prem) server it has the ability to run commands as root/superuser on any of the connected clients (generally servers themselves).

The commands run this way are _not logged_ and don't show up in any system audit logging.

After we pointed this out as a security problem in itself, they released a new version that _apparently_ had this functionality removed (was in the release notes).

But digging into the new release, they'd just moved the functionality into different binaries and hoped no-one would notice. :(

The mind boggles at what some of these places will try.

"Required functionality..." They're just not telling you who the requirements come from.

My experience with security vendors is that there's a lot security vendors who check checklist compliance solutions that on paper helps to be compliant but in reality are extremely insecure malwares.

Definitely - also its pretty easy to lose track of all things. Started a tool to audit github apps and misc permissions for an org which is currently basic atm but hopefully in the future more checks will be added - 3rd party integrations and apps are up there: https://github.com/crashappsec/github-analyzer. Any issues or feature requests are welcome and hopefully will expand it soon!

Name and shame.

> - It only says "access to code repositories" (it does not say anything about the level of that access, it might as well mean write access, capability to trigger actions etc.)

Given the exceptionally broad granularity of github permissions with oauth at least I'd be concerned. I've repeatedly had to avoid using something because while I wanted to grant permission to, say, read one particular repo I'd have to allow write access to all private repos. Githubs solution is to give broad access to a user that has limited access, although then they shout at you for creating more users if you're not a paying member yourself (but may be in an organisation).

https://docs.github.com/en/developers/apps/building-oauth-ap...

GitHub recently introduced fine-grained tokens that can be scoped to a single repository that might work for your use-case https://github.blog/2022-10-18-introducing-fine-grained-pers...

Unfortunately this only seems to be available for repos you own yourself and not if an org owns the repo, making it useless in a company context until that is expanded. Great for personal stuff tho.

Fine-grained access tokens are available on org-owned repos too, but the org has to opt in (for some reason).

They work, but the organization needs to approve the token (and its scope). As an org admin, I prefer it this way because I can audit what access developers give out to what repositories.

The new tokens are still in Beta, so there are some other limitations: for example, GitHub Packages do not support them yet, so you cannot use them in NPM/yarn to get your private packages hosted on GitHub.

Doesn't this rather defeat the point? If using a PAT is simpler than using the org token then I'll just use a PAT that has access to every single repository in the org.

GitHub Org admins have the option to block all regular PATs, and only allow fine-grained tokens that they pre-approve. This block is “off” by default, but I expect the best practice soon will be to enforce this rule.

Oh, aside from the pre-approval, that would be really nice. I need stuff scoped to the org that is not necessarily connected to my personal account (and all it’s repositories).

GitLab has had this feature for years.

I'm excited for FGPATs, but they're still in beta and still have a lot of shortcomings at this time.

overbroad roles + permissions is such a problem on every platform. github surprisingly bad given their technical sophistication overall

on my long-term wishlist: a standard way to aspirationally reduce your permissions footprint

like if I'm a twitter plugin, and I need one operation for which there is only a broad permission, I request the broad permission ('read and write everything'), but include an optional manifest making it clear that I only need the one action ('add followers')

this 1) enables a linter ecosystem to verify my footprint is narrower than it seems, and 2) over time lets the platform tailor offer narrower perms based on observed use-cases

permissions matter bc users can audit them more easily than code

Since GitHub alerted Okta, I'm assuming they use the regular, hosted github.com. I'm kinda shocked a security company doesn't have a private GitHub Enterprise server behind a firewall.

if they had, it'd be worse - they'd probably never know they were hacked...

Hosting something yourself does not make it magically more secure. Even if you hire a small team of really smart people, they'd have to work pretty hard to do as good of a job as the many 100s working on security at GitHub...

One would assume that their code is never accessible over the public Internet. It is pretty much over for Okta now.

So you expect "critical" companies to self-host everything?

Add to it that they reviewed "all recent commits to Okta software repositories." Due diligence or indicative of the threat actor having write access?

Many unanswered questions.

A good reason to give engineers PGP keys and turn on the "required code signing" feature on your org. Alas, security and productivity are perpetual odds.

Just wondering - would this meaningfully impact productivity beyond causing engineers to have to learn how to sign a commit (which would presumable take less than an hour, once)?

Actually generating a key and signing commits is pretty easy. I think the harder part would be ensuring all devs safely store the keys, rotate them regularly, etc.

Just a friendly reminder that both GH and GL now support using SSH keys for signing commits, and 1Password (and KeePassXC, FWIW) will safely store those SSH creds off-disk:

https://docs.github.com/en/authentication/managing-commit-si...

https://docs.gitlab.com/ee/user/project/repository/ssh_signe...

https://developer.1password.com/docs/ssh/agent/

https://keepassxc.org/docs/KeePassXC_UserGuide.html#_ssh_age...

Although in full transparency, I still use GPG for my needs, since I better understand its workflow

Note that your GPG key is discarded, and GitHub signs your commit itself with GitHub.com's own GPG key when anyone uses the GitHub UI to merge your PR.

All those "verified" buttons you see on a typical repo history tend to actually be for the GitHub.com signing key, which is shared by everyone. Your GPG signature is only used to convince GitHub to sign the final commit with its key.

It is possible to put your GPG signature on the merged commits, so that people can trust the commits came from you. That may be especially appropriate for security software. But you have to do the merges (or rebases as you prefer) outside GitHub for that, and push those merges directly to the main branch. That's what I do when I can, but it's not common practice. Many orgs require all merges to be done via GitHub, so end up with GitHub.com's shared signature on everything instead of their own.

So genuine question, how do I detect access to my github repository before github alerts me about something nefarious ?

> You can stream audit and Git events data from GitHub to an external data management system.

For a company of Okta's scale and importance, this should be part of SIEM.

- https://docs.github.com/en/enterprise-cloud@latest/admin/mon...

GitHub's Audit logging is quite lacking, it doesn't include API requests.

Another red flag is that they should mention what they will do with clear-text credentials that are in the source code that was stolen.

There are always creds in private/company repos! Lots.

> There are always creds in private/company repos! Lots.

I disagree with the "always" in this statement. Sloppy, lazy private repos sure. It is possible to have them completely absent in any and all repos though I've only seen and been part of such an effort once, it takes a lot of work to make sure it happens and I have little faith in most companies following through with that.

In my experience it's a pretty low bar to keep private credentials outside of source code. If a "security" company like Okta has secrets in their source code, that's embarrassing and unexpected. Any competent team of 2+ developers should be able to avoid this.

However, what's more common is secrets in CI variables. If their GitHub was breached, they should be more concerned with whether the attackers had access to GitHub Actions logs or secrets.

In this day and age, I really do not understand why one of the first steps when spinning up a new repo (for this type of app) is not leveraging a tool like dotenv, and then a config system that uses environment variables for things like db credentials/etc. Yes it takes another hour of time to get that all going but, in the long run, you'll thank yourself!

I've worked in places where all the code was open to all teams, with the exception of DevOps because they had too many hardcoded secrets and never had time to clean up their mess. I get it, they really were spread thin... but it should have never happened in the first place.

Okta employee here. I can assure you that there are no clear-text credentials in our source code.

Reply on news.ycombinator.com