Macaroons Escalated Quickly | Hacker News Reader

Macaroons Escalated Quickly

fly.io

263 pointsmkeeter172 comments

Back in 2021, when I wrote the API token survey post, I had some, uh, caveats about Macaroons, and linked to a talk from a developer at Chain who had gone through the experience of deploying them. Don't get me wrong, it's still an excellent talk†, but after working on implementing them for a year or two I find it much less damning than I did at the time.

One problem Chain seemed to run into was coupling between their services. They had a fast Golang "ledger" and a slow Rails "dashboard", and found that Macaroons forced all their requests to loop through the "dashboard". This was because they'd opted to do short-lived Macaroon tokens that needed to be reissued every 5 minutes (eminently sane), and only their dashboard could generate the proper token?

Another problem they had with Macaroons: their Rails dashboard has an RBAC permissons interface. But once they've issued a Macaroon, users can't change permissions with the interface anymore. But that's a problem with all stateless tokens, not just Macaroons; in fact, Macaroons probably help here, because if you're presenting a Macaroon to the dashboard interface in the first place, the RBAC interface can just attenuate it for you and hand it back to you.

I still think this is a design that mostly makes sense only if you have a problem domain for which both attenuation and delegation make sense. Most basic CRUD apps don't have these problems, and using Macaroons would be a mistake for them.

† https://www.youtube.com/watch?v=MZFv62qz8RU

I'm curious what your thoughts are on implementing a sane authorization system in 2024. You mentioned writing roles onto macroons but what does your policy look like? Is your surface area simple enough that it's not a concern?

I've been on various security teams with disjoint product-facing authz, internal authz and service authz policy engines / mechanisms etc. Additionally, authz gets baked into service code and product interfaces, so it's hard to change later.

The idea is that Macaroons are a low-level IAM language, designed close to the components that they pertain to, encoded directly into the tokens, and that higher-level IAM policies "compile down" into those tokens.

If you need more content, I would be keen to read a blog post on how you designed authz at Fly.io. I think it'd be a unique point of view, given your security background and ability to write well.

In the third_party_caveat code sample, I think you mean to be passing in the tag rather than the url, and the arguments at the call site are in the wrong order.

Edit: Or I guess along with the url.

You're right; I extracted that function from a gnarlier function I didn't want to use in the post (it was the same logic, but the function was so long it broke up the post too much) and wasn't careful enough in moving the code. Fixed, sort of! Thanks!

Go easy on me; I literally wrote this in IDLE.

> I still think this is a design that mostly makes sense only if you have a problem domain for which both attenuation and delegation make sense.

I would argue that supporting attenuation and delegation (almost?) always makes sense. It's very common that I find myself wishing I could grant some piece of software some but not all permissions to some resources; this can be relatively straightforward when the authorization scheme supports attenuation and delegation, but is painful when it doesn't.

As you mention in the post, users often want to do things that the developer hadn't thought of - I'd argue that's still the case even for what you might say is a basic CRUD app.

(Yes, this is related to the old ACL-vs-capability-model debate.)

Now that said, I'm not saying Macaroons make sense for every use case - just that I think attenuation and delegation do.

Limited-access tokens that you can give to others are often useful, but there are other ways to do it.

If your application is a monolith with an API, you're going to have to implement all the permissions yourself anyway, right? That includes the UI for setting up the permissions and generating the cookie, the code for checking permissions, and the code for actually doing the work. Might as well keep the cookies very simple and put the permissions in the database.

Once you have plugins with their own permissions, possibly running on their own servers, it starts to make more sense.

I feel that the issue there is that their architecture was unable to fit the macaroons correctly.

I remember when I saw a presentation by the macaroon authors a few years back, there were pending patents that Google filed around them. While the authors claimed Google wouldn't sue anyone, I'm always a bit skeptical about such claims. I thought macaroons would be helpful for some of my use-cases, but since I now knew there were patents that'd be wilful infringement so I didn't bother.

I can't find the patents now, so perhaps they were rejected or withdrawn. I had assumed that was why macaroons hadn't caught on more widely.

Edit: Found the patent: https://patents.google.com/patent/US9397990B1/

The Pythonish pseudo-code, rendered to an image (figure 7) -- is that common nowadays? Though the patent I see is dated 2013.

maybe this is similar to google's patenting of dropout for neural networks? you can never know but so far there haven't been many adverse effects and they claim that they patent it so others can't maliciously patent and enforce it.

dropout patent link: https://patents.google.com/patent/US9406017B2/en

That was what the authors claimed when I asked them about the macaroon patent. It'd be nice if Google had a legal document associated with patents they never plan to enforce, or the constraints around when they might enforce them (e.g. only against patent trolls) that a company could rely on.

I don't disagree but maintaining an arsenal of defensive parents is Enterprise IP legal 101. All the big companies do this. The goal is to avoid litigation by mutually assured destruction. At least that's what they tell you. Many projects grant you patents as part of, for example, an OSS project's license.

It's not unusual to include a clause that voids any such terms of you litigate over your own parents, for example -- hence the mutually assured destruction. Can you imagine what would happen if Google and Facebook tried to duke out some dumb software patent in court? A waste for all parties.

Lastly Google has a wealth of resources here: https://google.github.io/opencasebook/patents/#patents-in-op...

There are so many stupid patents out there about everything we could possibly work on, it is actually reassuring to see that Google is assigned to some of them, rather than to some storefront in Marshall, Texas.

Google has an open pledge to not litigate open source uses of its patents: https://www.google.com/patents/opnpledge/pledge

What does a "open pledge" like that realistically mean, in case they someday broke that pledge? Would the court-case 100% surely get thrown out? Am I legally protected because of this pledge?

Just look at their litigation record. They have been around for long enough to have a meaningful track record.

That doesn't mean things won't change in the future.

Their track record in other places has shown a significant decline (perhaps related to DoubleClick completing their reverse-acquisition of them).

That's my take on the death of don't be evil. Doubleclick was the worst kind of company and the merger with google seems to have diluted enough evil into google that google's immune system failed to kill it.

> Just look at their litigation record.

I guess if one looks at those, it'll look like they won't sue you?

But my hypothetical case is what about if they do sue you, does this actually protect you, or not? Does a "pledge" have enough meaning that it could change the outcome in a court case?

Yes.

The patents page (https://www.google.com/patents/opnpledge/patents/) does not include the macaroon patent. Maybe they forgot to update it?

Meanwhile, fly.io is not open source. Some of their software is, but not all of it. Same likely holds for the SaaS you're thinking of starting.

Macaroons are a cookie[1], Macarons are a French dessert[2].

I have a pet peeve about this mix-up in particular, and they seem to have mixed them up, calling the French thing macaroons.

1: https://www.savoryexperiments.com/almond-coconut-macaroons/

2: https://sallysbakingaddiction.com/french-macarons/

Ultimately, of course, it doesn't really matter as long as the person selling them knows what you're asking for. Which they will.

https://www.youtube.com/watch?v=nzcHeO43kgE

Wow, thank you. That was a real broccoli = kale moment for me.

Macaroons and macarons are both cookies. They're both desserts as well.

It's basically the same word though(they have the same etymology).

In Norwegian they're called coconut and French "makron", respectively.

Macaroons are the coconut cookie. Macarons are the fluffy egg white ones. Very different things.

But the word macaroon is derived from macaron. Obviously they are not the same thing, I never said that, did I?

Small detail, there's a picture of macarons and not macaroons.

A macaron is a sandwich-like cookie that's filled with jam, ganache, or buttercream. A macaroon is a drop cookie made using shredded coconut.

Macaroon is just the English word for the French word macaron.

ducks for cover

Macaroon is a completely different confectionery: https://en.m.wikipedia.org/wiki/Macaroon

Sure, but Macarons (the burger-looking things) are also sometimes known as "macaroons". Yeah.

Examples:

* https://missmacaroon.co.uk/

* https://www.floristgrays.co.uk/design-202300010/valentines-m...

* https://www.parisiennesouthwell.com/product-page/macaroons-v...

The fact that people say it doesn't make it any less wrong. :)

If enough people do it for long enough, it does. After all, that's how we got the rest of the words.

The joys and woes of ever-evolving language / culture

To the persistent victor…

The people in your 3 examples are wrong.

And Macaron is the French word for Macaroni. Think how that makes the Italians feel.

Don't let anyone try to convince you that up is down. Word was spelled wrong. Period.

So, are Macaroons are like a "blockchain", where each block is an attenuation (ie reducing permission and scopes) of the permission of the original token? So a bearer can get a token, reduce its scope (without talking to the server) and pass it down to other less privileged entities. That's cool!

I kind of had to look into the paper linked to really understand what they are. I feel the article could have given a bit more context on it :)

Somewhat. It is symmetric key based, so only the initial issuer can do verification. It is not part of a distributed data store, so there can be multiple copies (and derivatives) of a macaroon floating around.

Basically, a trusted service creates an initial atom and shares it with the initial party.

That trusted atom can be copied and shared. Anyone with a copy can amend additional information to it, and share those copies. Based on the copy you get, you can't remove or alter any information received.

So it works best for representing restrictions - maybe my user agent gets a token that gives full access for a project. It shares a version with another service endpoint which is restricted to read-only access. That service endpoint sees it needs to kick off a background process, so it creates a version which also adds on a 4 hour time limit.

None of these changes are necessarily authenticated - I don't know that it specifically was the user agent that shared the token with the service or added the read-only restriction, at least with the core tech.

You have to create a domain language to specify all of these things. If you want restrictions to be authenticated, you need to describe how to sign them individually with PKI.

While HMAC is fast, you still have the whole system rooted in a single trusted service that can mint any messages it wants or lie about the validity of a given message.

So while it is indeed a chain of cryptographic elements, it is missing the multi-party validation and operation transparency which are the common selling points of distributed ledgers.

Biscuits have single roots of trust too. The (superficial) distinction is that Biscuits can be verified (completely) on systems that don't hold the root keys. You can't do that with a Macaroon, but, as we explain in this post, you don't have to. I don't think we'd meaningfully gain security from Biscuits --- though we would potentially get flexibility from Datalog, and our deployment story might have been a little bit simpler.

(We already need hardware-isolated secret stores, so the deloyment win is probably marginal for us, but it's a bigger deal for other shops).

Purely symmetric cryptography is one of the features of Macaroons. It's a reason cryptography hipsters like it, not a reason to dunk on it.

Yep, comparing SHA2-based macaroons and ED25519-based biscuits, you'll have size and computational performance advantages for macaroons as well as post-quantum resistance.

But that doesn't help you if you want to solve the network performance and failure issues with verifying macaroons.

There may be strategies to solve some of the biscuit disadvantages using different signature mechanisms like BLS, but that might not help with performance and will not currently get you post-quantum resistance.

The back half of this piece is largely about the network performance and deployment issues of Macaroons and how we addressed them. This is a real system, running in production; it isn't a proposal.

> None of these changes are necessarily authenticated - I don't know that it specifically was the user agent that shared the token with the service or added the read-only restriction, at least with the core tech.

Macaroons are very much grounded in capability thinking, where it is a “feature” that principals such as the user agent in your case don’t need to have representable identities. So there’s nobody to authenticate, except the fact that it was a holder of the previous macaroons tag, which means they had the authority it represented.

In other words, not needing something like a PKI is somewhat the point of it all.

If you have the need to authenticate the intermediate principals, eg because they have pre-existing identities already, then the capability model in general may just be a distraction. They could just sign their desired attentuations with their own identity key.

One use of macaroons is to have caveats restricting the request itself used to achieve higher security than bearer tokens. This might be "only good for this service" audience restrictions or "only good for this hashed request message" as proof-of-possession.

This creates a role division, where the user agent caveats before the proof-of-possession are potentially authorized differently than the ones after. It also creates the possibility that you will need to override caveats per policy, say when that message wasn't a terminal call but rather hitting an ingress to a complex internal services framework which would still like to leverage the macaroon.

Basically - if a role has a requirement that they always terminate with a particular caveat, you know a section of caveats was created by software in that role. This is more of a second level of authorization than it is authentication though, I would agree.

In one of our iterations, we did indeed have a mandatory rule that each set of caveats added by a holder was terminated by a “sign off” caveat. But in the end we concluded that if that mattered in the actual enforcement, then one wasn’t really doing things “the capability way” and then that principal probably needed some stronger authentication than merely having held the unattenuated macaroon.

But that’s theoretical thinking and the real world is indeed more shades of gray.

See you in Madrid!

Are macaroons similar to UCAN?

https://ucan.xyz/

I wouldn't use the word "blockchain" as that is primarily a tool for generating consensus. Macaroons are each independent and can be "forked" freely.

So while I agree that the data structure itself is very similar the actual algorithms that are applied on top are wildly different.

Macaroons are also implemented and used by pypi.org's implementation named Warehouse:

https://warehouse.pypa.io/development/token-scanning.html

Also see:

https://pypitoken.readthedocs.io/en/latest/

Yes, although PyPI doesn't currently do much attenuation or delegation with them (this is largely my fault, since I didn't fully understand their power when picking them for the implementation).

That's been slowly changing, however -- as of a few months ago, PyPI issues slightly more compact API tokens that make better use of discrete caveats. They're also used on the Trusted Publishing[1] side to make the API token self-expiring.

[1]: https://docs.pypi.org/trusted-publishers/

Compulsory plug when someone uses macaroons: using an HMAC is overkill.

See https://github.com/rustyrussell/runes for a simpler alternative and implementation (this has C and Python, but there's also a Rust implementation because why not?)

However, the "no db access" property has proven to be untenable in practice. Users end up wanting to see what runes are issued, blacklist them, know when they were last used, and have rate limits. The last two are a killer, requiring some state to be kept (unless your system allows you to return a modified rune to the user, which is a different workflow from normal bearer creds).

Oops, I read further: you did use unique ids, and blacklists! I did that too. Now you're not db-lookup-free on use, so you need to think hard about what all this actually gains :(

Huh, I never thought about using length extension attacks as a protocol feature. However, you still need the initial secret, analogous to the HMAC key, and it's not a complicated algorithm to justify much effort in removing.

And don't forget that HMAC is more resistant to collision than the underlying hash. For example, I think it's still unknown if HMAC-MD5 is weak, even though MD5 definitely is.

> The community that formed around building open source “standard” Macaroons decided to use untyped opaque blobs to represent candidates.

I assume "candidates" was supposed to be "caveats" - and as an author of a "standard" macaroon implementation, I completely agree that this is the biggest downfall of Macaroons. With no common caveat language (and no independent "dischargers") it really limits their use to within a single org. And at that point you're basically asking everyone to invent their own token format anyway.

Though I don't personally use them much anymore - I think the use-cases for Macaroons are much more limited if you have a Zanzibar! - I appreciate seeing Macaroon discussions pop up and this post and the related discussions it linked out to were a great read.

Zanzibar and macaroons are actually pretty complimentary.

These look pretty useful cool...

I'm curious a bit though, here, and in other posts in HN, I often see that HMAC, a symmetric key is preferred, going as far as suggesting, in JWT, that other algorithms should not be implemented. Why is that? What are the problems with say RSA? (Ignorning that I'm not sure Asymmetric keys work with Macaroons design at all)

From my perspective, Asymmetric keys have been a great boon, in that I can keep the private key, secured on my single auth server, but then freely distribute the public key to the edge, greatly increasing responsiveness, and reducing the bottleneck on the Auth server.

Is there some security concern I've been missing?

This post was written exceedingly well. Few posts execute humour and whimsy without coming off as insincere or just not very funny (I'm guilty myself).

Bravo!

The tone is entertaining, but some of the snarkiness around the code is bit frustrating at times. e.g.

> # do i really need to say I'm not serious about this?

Can you just come out and say what you mean here? Like, presumably it's bad, insecure code, but can't you just spell it out for the benefit of your audience?

I'm probably being really picky here - just find this kind of developer in-crowd signalling to be really irritating and counterproductive.

The Python code here is essentially just pseudocode. In reality, if you build a Python Macaroon implementation, you'd pull in pyca/cryptography and use an actual AEAD, rather than rolling your own authenticated cipher out of pure HMAC. But the point is that this isn't real code, just enough to make the concepts concrete.

I can always appreciate that these types of stripped-down examples are merely for illustrative, conceptual purposes...but the ignoramus in me would also appreciate links to fleshed-out examples that take into account the shortcomings of the simpler example.

Our actual Macaroon code is linked at the bottom of the article.

That's a great explanation, and pretty much exactly what I'd love to see instead of the original slightly mysterious comment :)

Yes, you articulated what I think often when I read a tpt post. Enjoy the casual writing but feel like I’m frequently left out of the joke.

I understand all these concepts in general but looking at a block of code… I’m supposed to 1) know if it’s serious/not and 2) he’s being sarcastic about it not.

I even know Python well but the whole post is so “inside baseball” am not sure what to think. Guessing the bulk of the audience is not grizzled security engineers.

(I’d recommend another summary/details tag to put the grandparent explanation directly into the fly post.)

Which parts seemed "inside baseball" to you?

hmac and attenuation for starters.

I read your related piece a few years ago but apparently didn’t retain much, due to not using it every day. Knowledge went out to backup tape.

So, come to yesterday’s piece. I remember the abbev. HMAC but have to look it up. Didn’t get the use case. No idea what attenuation meant in this context… thought of sound and plowed ahead. Gave up by the middle of the piece, lost.

But, then saw the link to the old piece, and read it top to bottom. Ok, now I get it! Reread and got the second piece. Understand finally but still not entirely clear why you were dissing your work.

Read another related piece on why json and xml are discredited for this kind of work.

Often upfront in a piece the author will say read this first, define some jargon, and “why’s”. (The why is often what I care about most.) These go well with the details/summary tags. Well, looks like you used a button for that but same idea.

Experienced dev here but haven’t just spent two years building an iam system. Slightly more acknowledgment of that upfront would work wonders.

OK! That's helpful.

The only unserious code in this post, for what it's worth, is a couple functions that make an authenticated stream cipher out of HMAC, because the Python standard library doesn't have an encryption function that I could find.

Ok, don’t believe there’s full encryption because it changes often and better to let the community experts handle. (Besides the ancient crypt module.)

But the recent secrets module might have some building blocks you could use.

This whole thread reminds me of a Hollywood sequel—need to spend ten minutes regurgitating the backstory to an audience who maybe saw the movie two years ago.

Hence the unseriousness of the code. Like I said, in real code you'd use pyca/cryptography.

I studied that code and the comment for a good 10 minutes. As far as I can tell it just obfuscates, and is not actually implementing authenticated encryption. It would help to just come out and say that part out loud.

True fact: Salsa20 is itself a hash function, keyed, and running in a counter mode.

>I'm probably being really picky here - just find this kind of developer in-crowd signalling to be really irritating and counterproductive.

Why is it "developer in-crowd signalling" rather than just a joke? IOW I read this as saying the jokes were "as bad as" virtue-signalling.

It's a statement that won't make sense to someone who's new to the topic.

I don't think it's related to virtue signalling at all, the two just share the word "signalling".

It's probably more relevant to your previous blog post that did a survey of technologies, but I'm curious if y'all ever considered a fully centralized approach. Is there something about Fly that benefits from a decentralized approach?

Disclosure: I work on SpiceDB alongside maintainers like the pymacaroons author

Can you define "centralized" and "decentralized" for me? What would a "fully centralized" approach to this problem look like?

Sure thing! These terms may not be the best way to communicate these concepts. Would love your take on that, too.

I used the term decentralized to describe claim-based token-passing authorization strategies because the data defining their access is being source from various services, attenuated, and passed around with the request.

I use the term centralized to describe something where all the data defining access and the policies are sourced from one service. I personally work on a Google-inspired system, but there are plenty of policy engines using a similar approach.

Couple thoughts.

* With regards to the "core Macaroons" we implement with first-party caveats, we conceptually are centralized, in that there's a single source of truth about which apps belong to which orgs &c that caveats are evaluated against.

* The basic challenge we have is providing a comparable experience in every data center we're operating in around the world, so a "purely" centralized model would be problematic for us; people in Sydney would get a much worse experience than people in Ashburn. We get around this with LiteFS, to an extent. This post undersells how important LiteFS is to the deployment model here.

* With respect to having, like, a Zanzibar-like access matrix dealy: part of the thing here is not wanting to have to provide a single coherent permissions model to our users, because our problem domain is so broad and we will get it wrong. Providing orthogonal tools to let users express things seems like the better strategy to us? But there are upsides to both approaches.

* Of course, one of the big wins here, that we have a plugin interface for authorization, depends on some measure of decentralization.

I feel like this is kind of a scattershot response, but hopefully I hit some of what you were talking about, and you can let me know if I've missed the mark completely.

Off topic, but related: The art you see on Fly posts? It's made by the inimitable Annie Rugyt: https://annieruygtillustration.com/ She's wonderful, and has a knack for making brands come alive with illustration.

Remember the RethinkDB mascot? That was her too!

She's a full-time member of the team. This is the least of the art she's queued up for the prodsec team's posts this quarter. :)

Her big project right now is mascotizing Frankie the Balloon, for swag.

>and escape hatches like Mutation (for our GraphQL API).

Can you explain in a bit more detail what you mean by escape hatches (sounds dangerously fun)

The Organization -> Apps -> Volumes (whatever) structure models our problem domain and is a sort of coherent abstraction. If we get that abstraction wrong, it's a pain to retrofit changes. Caveats like "Mutation", on the other hand, are simply strings representing calls you can make in our API; they're not a model of anything. I don't think we use `Mutation` for anything right now, but it's there in case we (or you) ever need a token that authorizes a particular set of GQL calls that don't fit the model we came up with.

I like the "solve the now" perspective here, and having code examples is very helpful to understand some of the rational behind the approach. Having read your previous "tedious survey"[0] post on various token formats, I generally agree with a lot of your conclusions. Curious though about your thought process wrt macaroons vs biscuits.

To me the one major downside of macaroons has always been the single shared root symmetric key. Many use cases are addressed by third party attenuation, but then there are the problems like key rotation, having to do online verification, no built in encryption, no peer-to-peer support through an "untrusted" fly.io, and no third party token verification without decryption like in signcryption[1] schemes. Of course this is traded off by having to do PK issuance and management so I can see the simplicity of it.

Is fly.io scoping this pretty hard to just auth tokens with third party attenuation, or do you see further development and maybe moving to other token systems like biscuit when/if the need arises to address those known issues?

fwiw I've done a bit of research work myself on a token format using signcryption [2] where I explored addressing some of these ideas (but not the attenuation side of it yet, which I get is a big deal here).

[0] https://fly.io/blog/api-tokens-a-tedious-survey/

[1] https://github.com/jedisct1/libsodium-signcryption

[2] https://github.com/michelp/pgsodium/blob/feat/signcryption-t...

We're not going to use Biscuits. The "single root symmetric key" thing is a systems design challenge for us, not a cryptographic one: we simply isolate the key handling on a software-HSM-like verifier service, running on dedicated hardware. If you're coding authz logic for Fly.io's platform code, your experience will be that you can manipulate and check Macaroon caveats without ever having access to the keys themselves, because we split Macaroon-checking into "verification" and "caveat clearing".

The threats in this design that would force us to do abrupt key rotations and mass token invalidation are thus pretty much the same as the ones we'd face with Biscuits; the "trusted code base" of key-handling code is comparably small.

More detail here:

https://github.com/superfly/macaroon/blob/main/macaroon-thou...

> Until now! I dragged Fly.io into implementing them. Suckers!

Let's be real: they knew you would do this eventually. It was inevitable.

I'm curious why the connection from Rails is "mTLS-y" rather than actual mTLS. The macaroon repo mentions a Noise transport. Maybe to avoid dealing with X.509 certificates by distributing trusted public keys via LiteFS?

> We didn’t use the pre-existing public implementation because we were warned not to. [...] Macaroons decided to use untyped opaque blobs to represent caveats. We need things to be as rigidly unambiguous as they can be.

Another place where the implementations are limiting is in the nonce for discharge macaroons: it has to match the "challenge" exactly. I think the fly.io implementation makes the nonce a structured object that includes the challenge / key ID as a field, and extracts that field during third party caveat verification, which is nice. It makes it possible to include a random nonce (in the cryptographic sense) or revocation ID for instance.

I experimented recently with discharges containing service-specific info, e.g. a discharge from an auth service could contain the user's name (for logging), etc. It felt dangerous though, because there's potential for confusion on which service is allowed to provide what info - if we have a 3P caveat for an auth service and a revocation checking service, we want to be sure that the user profile info comes from the auth service, not the other one, and it gets complicated to encode that in the token.

Maybe this is what "proof" macaroons solve? They're not mentioned in the blog post or macaroon-thought.md, but seemed to be about making positive statements on something.

This has been a perpetual problem we’re trying to solve at our org and while I was on that team I was pushing for the idea of macaroons (we just use JWT and we suffer many headaches because of it).

My suggestion at the time was macaroons with some hierarchal PKI - servers themselves have a macaroon that delegates the permission to sign tokens that meet certain criteria, delegated by us the root organization. So in most situations a typical key hierarchy would look like “root key -> signing servers -> normal services -> bearer token”.

If a token is stateless then this just becomes another caveat and signature - show me that you have the authority to sign this key under this attenuation. Signing server? You should only be signing keys for key services. Data service? You should only be issuing tokens with X caveat. Etc.

If you want revocation though it’s not really stateless - at minimum you have to store a list of revoked tokens until the parent tokens that issued them have expired. In fact you would want to build this as a routine capability on every single level - signing services should rotate their own keys (this is that “clearing caveats” thing) and issue their own revocations, and then the assertion is that any signing from that key is invalid from that moment forward. Similarly, if a browser session wants to refresh its token, it needs to commit to never using the old one again, for the lifespan of the IAM token, and we need to track that for the lifespan of the IAM token. But there is a defined TTL after which you can clear the revocation - and it’s the lifespan of the token used to issue it.

Which is a very Redis-with-cross-region-replication sort of task, or perhaps Postgres for persistent backing. The good news is that there is no logical split-brain, the mere fact that another region issued a revocation is ipso facto all you need to know, when and why it happened is kinda irrelevant to the fact that the token is now revoked.

In other words - signing servers probably have an IAM representing them as an instance, with their own internal secret/private key, which is used to issue their own "bearer token" (public key) which rotates periodically. And when they do a clearing etc they are the authority that signs that the new macaroon is valid. And after some time their own bearer token is revoked (eg after 2pm any signings are invalid) and they rotate to signing with a new bearer token.

If you are going to rely on propagating revocation efficiently it needs to be a routine fact that’s occurring regularly, right? And if you don’t see a revocation from a signing server on some expected timeframe, it is probably in fact a sign of split-brain occurring… you’ve lost a server or a region and it’s in fact dubious to continue accepting that token anyway.

So this sort of leads to an inherent “valid -> historically valid but not for new tokens -> expired” lifecycle that mirrors the IAM/bearer model that clients also use.

This actually closes the "fail open" nature of the revocation - it is impossible to have a "split brain" in the usual sense of revocation not propagating to a region etc, because you know that a signing-server key isn't valid for more than 15 minutes anyway etc. And perhaps you could even add a "parentKey" field and use the bearer token itself as a carrier of these revocations (if the user has a new key and it's validly signed, then SOMEONE must have revoked the old key...) although I haven't rigorously thought that one through.

I am drawn to this revocation idea like a moth to a flame despite knowing how much complexity is going to live in making sure that service doesn’t ever go down or lose data or go split-brain. But if you are doing this “clearing” idea at least you don’t end up with a giant tree of historical caveats and sub-signatures.

On the other hand the reality is that really nobody wants true stateless especially for browser tokens… “log me out everywhere” is practically table stakes and that implies some kind of revocation. And revocation is implicitly state - it’s either valid or revoked, that’s state even if you aren’t mutating the token itself, you’re still mutating the validity of the token. Even if it’s a global “user revoked all tokens at 2pm” it’s still mutating state etc. The idea of revocations lets the “happy path” be stateless, and handle this smaller amount of replication of revocations etc, but otoh it’s still very load-bearing and revocations fail-open. But since the token is limited in time and scope anyway, that’s potentially a valid tradeoff…

But I haven't actually implemented any of this, so, take it for what it's worth.

I cut this back drastically from earlier drafts which talked more about how the verification service was deployed, but wanted to save at least one diagram with a Muppet in it, and spaced on blanking out the mTLS/TLS thing. Long story short: we originally deployed the verifier on NATS, not with direct HTTP, and so TLS wasn't an option; we wrote a Go implementation of Noise. It's just HTTP now (but still Noise for now).

The signing interface is mTLS-like (client-authenticated) because you can't just let any component sign. But really any component can verify, so all you need is server authentication.

Makes sense. The Muppet was definitely needed. Thank you!

I wonder why they went chose macaroons over biscuits. The later is only mentioned once in that article, but doesn't talk about why they were not selected.

I like Biscuits, wrote about them before, and recorded a podcast episode with Geoffroy Couprie about them†.

The short answers:

* When we started working on this design, there was still some uncertainty about the elliptic curve chaining construction (as in: they hadn't, or maybe had just started to, work out which one they were going to use).

* We do a lot of verifications, and the prospect of running lots of 25519 computations every time we have to check a token was daunting; by comparison, HMAC is essentially "free".

* I'm not especially scared of Ed25519, but asymmetric signature cryptography is much more complex than computing a MAC. Like I said in the post, one of the great charms of Macaroons is how much mileage it gets out of a construction that everyone understands, and that doesn't have sharp edges.

* Geoffroy sold me on the value of using a logic language to express caveats, but that was much more complicated than what we felt we needed. Another charm of Macaroons is that they're rigidly coherent; just an unordered collection of true/false predicates, loop over them and if any fail reject the request. I feel safe expressing things with Macaroon caveats --- not in the sense of "the token implementation doesn't have bugs", I'm sure Biscuit's Datalog is fine, but rather in the "I'm not going to mess this up as a user" sense.

* We needed implementations in a bunch of different languages. Biscuits have them, but owning the implementation ourselves derisked things for us. We could have owned an implementation of Biscuits, but I'm not smart enough to do that well.

† https://securitycryptographywhatever.com/2022/01/29/biscuits...

Oh my goodness -- I had this idea!!

When I first read a blog post titled "Macaroons are better than cookies" I started thinking about the operational side and I really didn't like the fact that you had to make a network round-trip to some auth-service to validate the macaroon--because that's what the state of the art already was, and it kinda stunk to pay that delay on every single query. (Of course I understood that you could horizontally scale the machines doing auth and load-balance and all that, but I worked at smaller companies where you had a small rack in a colo space or an air-conditioned closet, creating VMs to do all of that ate into a relatively limited resource and made sysadmins raise eyebrows.)

I tried for the longest time to put together a solution with hash functions and private-key crypto; I think the thing that bugged me the most was that I never was able to prove that you couldn't do it with those primitives. Eventually I had something similar with public-key crypto but at the time public-key usually meant RSA, so we were talking 1024-bit signatures for each step. Also I think the idea was that you'd furnish a zero-knowledge proof that you knew the key, which was hella complex.

Two things then happened: first, if you're talking PKI then you're talking about revocation too; I convinced myself that revocation was necessary for these sorts of credentials (and I now always feel vindicated when places are like "log out all other instances?") but I also convinced myself that revocation required the network hop in any case (which is not true, revocation is actually pretty infrequent and you can push revoked serials to services via event bus and in most cases the latency to ingest those is not a huge issue). So I convinced myself that what I was looking for was ultimately a purely academic exercise, was the first thing.

And the second was that I managed to simplify my PKI-plus-hashes-plus-zero-knowledge craziness to just what Biscuits are: "I'm signing a public key plus caveats," (but I didn't add a query language, I was just going to use a map of well-known-keys to values, "valid_before: 123...890"), "I'm going to transmit this to the recipient with its corresponding private key, which they will strip off and generate a new key to mint a new Biscuit."

And then RSA was big and slow and I forgot all about this until this HN thread. Ed25519 is the perfect way to fix that -- that and distributed revocation. I still don't see myself using it in practice but it's nice that a library exists if I do... thanks for the walk down memory lane, I will queue up the podcast episode, maybe for tonight or tomorrow.

Can't speak for them, but I've heard people balk at the Datalog part (IMO not an issue, but I'm not everyone)

Surprised to see JSON here. I recall discussing JWT with tptacek a few years ago, and one of the concerns was the {a: x, a: y} ambiguity in JSON parsers. Was that a concern with this design?

My other reaction was that this sounds scary:

> a token with no caveats restricts nothing. It’s a god-mode token. Don’t honor it.

I guess it's technically "fail closed" but seems kind of brittle

I think it becomes more obvious later in the post that the Python code here is for illustrative purposes only. We don't use JSON. Our actual tokens are strictly typed. But also: it doesn't matter that much, because all of the predicates in a Macaroon are evaluated independently. There's no opportunity to confuse a previous caveat with a subsequent one. That's one of the strengths of having a rigidly coherent design like Macaroons, rather than just a bag of keys and values like JWTs do.

Ah thanks for clarifying! I knew the code was illustrative but didn't realize the JSON part was too

The XML digital signature spec had that problem in spades. It took us about five times as long to button it down as it did to implement.

By the time I was done with that project, the document for partners and vendors (if anyone wanted to implement their own instead of using ours) was several times longer than the spec, what with all the extra MUST and MUST NOT situations. Which is not that hard when you're dealing with swiss cheese.

Document.findById and Element.findById being able to return different (non-null) results being the most egregious one I can remember.

XML is a disaster here all of its own, but is at least unambiguous at the parser level.

JSON says "if your document does this, parser behavior is undefined". So you need to declare that the parser used should behave in particular ways.

Both are best solved a robust specification and supplemental test vectors. If you define meta rules (e.g. JSON parsers must either fail or return the last object property), you still retain the value you hoped for by using something like JSON or XML in the first place.

Isn't it Turing Complete with namespaces? (that's another thing on the SHALL NOT list). Can't be unambiguous if you don't halt.

Turing complete? Not really, parsing still forms static data structures in linear time.

The fundamental screw-up with XML is that it was a textual markup format (e.g. HTML or word processor documents) that got usurped for cross platform data transfer. Among many other problems, that put pressure on its design to be unambiguous to generalized tools - which wasn't an initial design consideration.

So you had an awful lot of making people use document markup processing tools to serialize and parse data structures, schema and transform systems that tried to target both mindsets and did a bit of a poor job at each, and lots and lots of abstractions to try to make it so you only had to deal with the problem once.

Even JSON is a bit of a pain the farther away from late-bound or untyped scripting languages you get.

Storj is also leveraging macaroons. Great write-up on decentralized access control here: https://medium.com/@kleffew/what-is-capability-based-securit...

> we want Macaroon tokens to be safe to transmit between users

How does this work for auditing? If user A gives user B a token (perhaps after adding a caveat) how does this system audit log determine who did what? Does that require a third party system?

You can get as many independent Macaroons from us as you want, so while there are ways to handle this with third-party caveats (and never talking to our servers to get another token again), the simpler thing for token provenance with our system is the same as it would be with any other: just issue multiple tokens.

There is no centralized auditing of derivative macaroons, but there could be of verification of those macaroons - you still require a centralized service which knows the core HMAC secret to give a thumbs up or thumbs down.

To that end you can define auditing information even when there aren't really caveats - user A appending that they are giving a version of the macaroon to user B with no restrictions on use.

Since there isn't a domain language, you have to define all this. JSON can be useful to have an extensible notation for describing this, although you need to up-front declare the difference between information which can be ignored and caveats which much result in failure if not understood.

Note also that in our scheme, if Alice attenuates a token and gives it to Bob, Bob will still have to hit our login service to "activate" the token (that's the word I should have used in the post and I'm kicking myself for not thinking to express it that way) by clearing the third-party auth caveat we put on every token. So it's not as if we have no auditing of token delegation.

Still: if only for revocation purposes, I'd probably just mint a new token for e.g. a contractor (and certainly for every user). The nice thing about the system is that you can edit the tokens after we give them to you so you're not stuck with our role definitions, which I think is a significant win, but doing all of IAM without talking to our servers more than once, while a cute technical stunt, is probably not all that valuable to users.

> Bob will still have to hit our login service to "activate" the token.

I think thats what I was saying - on first verify your centralized system will know the contents of the token and can do appropriate auditing.

In practice I find that keeping processing simple is a challenge. It doesn't naturally say as simple as one would like, with orthagonal caveats that can be processed independently and independent of centralized knowledge.

For example, you might expect the user to indicate the system they are calling in a caveat, an audience value.

However, if you either make this true a restriction on further use it will prevent any sort of delegation, which may include delegation to internal systems that the user didn't know about.

You don't want to outright ignore it either.

So you have some sort of policy interpretation of that restriction, such as 'this audience means this set of services can use this macaroon' or 'we will audit internal use based on this audience but take no action', or even crazier you color caveats before and after this point as being ones set by the user vs ones set by the service.

This sounds a bit contrived, but it does loop back into one of the core usages I've seen for macaroons - gaining proof of possession or other sender verification rather than using bearer-style tokens.

I am not a customer, but I do like the fly.io "voice."

Typo/bug: pretty sure variable 'oldTag' should be 'oldTail'

Thanks! They're basically interchangeable --- HMAC produces a "MAC tag", and in the context of the Macaroon protocol the last visible tag is called a "tail". But I'm guessing I screwed up editing the code when I switched everything to "tail". :)

it seems to me that macaroons could get quite big in order to adequately describe allowed resources and permissions? Especially if the API is broad? Much bigger than JWTs.

The resource descriptions are pretty parsimonious; they're binary-encoded MsgPack integers, for the most part. But the cryptography eats up a lot of bytes. They're bigger than JWTs, but probably by a factor less than 2 (I don't know the median JWT size). Basic take: unless you can golf your tokens down to a single terminal line, it doesn't much matter how much bigger or smaller they are. These are all smaller than X.509 documents.

Ah i see

I'm (perhaps naively) imagining an exhaustive list of resources in every macaroon, but i guess the permissive evaluation of caveats would help here since it should mean only the most restricted users would have many caveats. Also I suppose if the caveat evaluation code can deal with wildcards then it can be reduced further. ok OK!

I'll have to play around and see how these look in practice. We do have a good use-case for this with both delegation and attenuation... sadly our identity provider is married to JWTs and moving off it would take a lot of work.

Anyways. Great Post! Certainly got me thinking, and now i'm tempted to just blend this with our existing system of JWTs :smirk:

The modal caveat is probably (in vibes-based notation):

    (Organization 4567 ops=read,write,create,control)

Followed by the second modal:

    (Organization 4567 ops=read,write,create,control)
    (Apps (App 345 ops=read,write,create,control))

You can make a _much_ more complicated token. And, for instance, our deploy tokens for CI/CD systems are pretty complicated, and they even have a disjunctive caveat, but most people are at most probably just going to lock their tokens down to a particular app, or turn them into read-only or start/stop-only tokens.

(We modeled everything just to be safe, and also because that stuff is super valuable for service tokens, when we ourselves want to take platform actions on behalf of the user and have it be traceable back to a user authorization. I feel like I did not say enough about how much I like where service tokens are pointing for us.)

TIL macarons [1] are also known as macaroons [2]. Where I'm from, macaroons are dense, sweet coconut treats and the French sweets are known as macarons. But from the Wikipedia article I see macaroons mean many different things to many different people.

[1] https://en.wikipedia.org/wiki/Macaron

[2] https://en.wikipedia.org/wiki/Macaroon

it messed with my head to hear that the english world interprets an "entrée" as a main course

American English, but not British. Dinner here is a starter, main course and dessert. "Entrée" messes with my head as well, since it very much sounds like a starter.

It's French for starter, even

> Dinner here is a starter, main course and dessert.

Yep. Check out the online menus in English restaurants and pubs. 'Mains' is the typical term used.

It's also the electrical supply. What a country.

And the water supply. And the central building on farms (in Scotland, at least). It's a silly country.

Don't even think about "a la mode" then.

I speak French, not Italian!

Switzerland has the similar Luxemburgerli, which my Swiss friends who reside there (as opposed to me, a Swiss person who lives somewhere else) swear is NOT the same as a macaron... I have never understood the purported differences.

I was just about to complain. Thanks for nipping it in the bud!

This feels relevant to the conversation: https://twitter.com/BigSpiderBack/status/864309383106240512/...

> TIL macarons [1] are also known as macaroons [2]

Not to forget macarrons! Where I'm from, macarrons [3] is a kind of pasta (penne).

[3] https://ca.wikipedia.org/wiki/Macarrons

In English "macaroni" is a kind of pasta. (I'm being deliberately vague because which pasta it is seems to vary.)

And the original name for that type of pasta in Italian is "maccheroni" (which are straight tubes, unlike the ones used abroad for "mac and cheese")

Or in English macaroni https://en.wikipedia.org/wiki/Macaroni

> Where I'm from, macaroons are dense, sweet coconut treats and the French sweets are known as macarons.

Same for me (New England, USA).

Both are delicious, but in my dialect they are not the same thing.

This is not a dialect thing, this is a factual thing. They are different, and anyone that uses macarOOn to mean a sweetly filled meringue sandwich is wrong.

Oh funny, I actually came to this comment thread to point out that they'd accidentally used a picture of macarons instead of macaroons in the blog. But I guess not necessarily!

The title and graphic suggest an article about the most delicious treat known to humans, and the post is about software. My disappointment is immeasurable and my day is ruined.

Yeah I clicked here thinking this would be an interesting story about why macarons are so popular these days or something. When I saw a code block I closed immediately. I'm a programmer but I don't care about every random, poorly-named library or framework that exists.

To be fair, it's not a library or framework, it's a format for certain types of security tokens. They're kind of like fancier cookies, hence the name "macaroon". Not so poorly-named, in my opinion (to the extent that "cookies" was ever a sensible name). Just the right level of whimsy while still calling back to what they are.

A macaroon is a mixed up ball of coconut and egg whites. A macaron is the delicious fancy sandwich cookie.

And for completeness, a Macron is the president.

Not to be confused with micron, which is the size of most politicians’ accomplishments after they get elected.

This is an obscure form of nominative determinism. Biscuits tend to be named after French leadership (eg Bourbon, Napoleon, etc), thus obviously someone whose name is nearly a biscuit should become leader of France.

(There are also Garibaldi biscuits, which would at first seem to dilute the theory... but he was born in France!)

Referer/referrer, macaron/macaroon, the web isn’t good with typos…

And macaroni is a kind of pasta. Make it make sense.

It is an oddly specific choice to use the word macaroon for this, given the widespread confusion over macaron vs. macaroon and how many other unambiguous variants of "cookie" are available for naming a piece of software that is a newer fancier cookie. I do not accept Wikipedia's milquetoast concession to call a macaron a "French macaroon".

Was it intentionally named as an allusion to this confusion? Software is full of hand-wringing over naming and the importance of naming. Is it supposed to be a kind of cookie that is often mistaken for a different cookie?

Or is this just another referer, a mistake that has been accepted as canon. Referer has the benefit of not being a real word at all though, so is less confusing.

As far as I remember (maybe Arnar or Úlfar or another of the authors would have a different memory, though) we just wanted a “cool cookie” name. We knew about the -oon/-on distinction and liked the sound of -oon.

I view it as a mild plus that the confusion around the name has educated so many about confectionery taxonomies.

If you type a name in your search engine of choice and the intended link is not on the first page of results, then that name might not be the best.

When I saw the title and graphic I assumed it would be an article about conflict over cultural/language imperialism.

Furthermore, I live where macaroon and macaron are spelled, pronounced, and taste different.

Not to be confused with macarons, which are also delicious but have no SW design named after them!

Or macron, which is a president.

OK, spurred by your comment I consulted my large Larousse dictionary and found these two: maçon (builder) and mâcon (a wine). I challenge others to find others in the ‘mc[r]*n’ form.

Well, there is Macron the sports apparel maker...

To make matters worse, the title and featured image make it very clear that they have confused "macaroons" and "macarons."

Macaroon: https://en.wikipedia.org/wiki/Macaroon Macaron: https://en.wikipedia.org/wiki/Macaron

THANK YOU

came here to say this

This hurts my braincell.

On "macaroon":

> The name "macaroon" is borrowed from French macaron

On "macaron":

> A macaron, or French macaroon, is a ...

found this helpful video posted on a different subthread: https://www.youtube.com/watch?v=nzcHeO43kgE&t=622s

To be fair, this is a mistake that started with the Google paper, and everyone else just copies the mistake.

The paper calls them Macaroons as a play on (browser) Cookies with layers (of caveats) - so clearly they meant macarons as well, since a macaroon doesn't have layers. Or at least, that's always been my interpretation of the name. It's possible it was just an arbitrary play on hMAC cookies and not the layers?

I had that thought, although according to another comment the definitions have crossover. Probably because people so frequently confuse the two, but here we are.

What I really want to know is how fly.io has recruited such talent. @tptacek (who wrote this article), @chrismccord (creator of elixir Phoenix/Liveview), and others.

Very impressive, but also slightly worrying since some parts of fly.io have such terrible DX/documentation. I've only recently started using them, but the documentation around billing and databases is shockingly bad for a team that is clearly so deeply stacked.

Out of curiosity, what database documentation was lacking?

IIRC they don’t provide a managed database, just like a template

Right. They provide a convenience function that spins up a machine running postgres and adds the right environment variables to any machines you already have running.

It's "just another machine," but that's not obvious when you're setting it up! Two bits of documentation that are lacking here are:

1. How does this affect billing, especially if you're on the hobby plan? (Answer, as far as I can tell: since it's just another machine, it can be free if and only if you resource it such that it fits in your free "allowances." The defaults are NOT these, however, so you have to be careful if you want to mess around for free).

2. Once the db machine is created...how can I get credentials for it? A connection string is generated upon machine creation but...what if you don't write it down?

Yeah i feel your pain here. Fly has a lot of things that feel like they're hidden features, or at least, undocumented. Like, its trivial to get all the env vars from a machine, but i couldn't find this in the documentation anywhere. Someone else i was working with told me how and now..i know?

fly ssh console --command env

To be fair, it is mentioned in their docs, i just couldnt find it searching things like "how to find my environment variables".

part of it is common sense: fly ssh console == ssh, and then you run env to print the environment variables in your current session (which may not be the exact set of envs your app sees).

There's a whole backstory to this, because we got a big influx of Heroku developers a couple years ago, and Heroku developers all expect to be able to dump all their environment variables, including secrets, from the command line or whatever. We don't do that: our API is like a diode for secrets, and once they're set, they're sealed into a cryptographic vault on separate hardware. It is and has to be the case that you can shell into a machine and get the secrets, which apps all expect to receive in environment variables, but we don't like that you can do that, and wouldn't want to encourage it. Environment variables are hazmat!

Right ok that makes sense. The reason i wanted this was for the reason mentioned earlier in this thread: you spin up a DB but don’t have the credentials. Fortunately i had stored them as an env var and was able to recover them, but otherwise i felt at a loss on how to gain access.

Right, that's exactly the kind of environment variable we don't want you to be able to easily retrieve.

Interestingly, there's a big intersection with Macaroons here. We're replacing Vault (for customer secrets, not for TLS keys or infra secrets) with an internal, Macaroon-integrated secret store called Pet Semetary, and one of the use cases for it is to come up with a high-assurance way to give customers straightforward access to secrets without having to reverse engineer them from their machines.

At any rate: this is all a consequence of our security model. A vulnerability in our Rails API server (which is not, like, beyond the frontiers of possibility) would give you enough access to jumble up everyone's secrets, but not to read them. We're happy to take our lumps for how annoying that makes our DX sometimes. :)

How do you want these things stored? And further, is there a way to regain access if someone looses their db credentials in the fly environment?

As you can see from the thread, and from, like, logical deduction, you can recover credentials from a shell on the Fly Machine your app runs on (because, after all, your app needs access to those secrets). But just dumping them from the API implies an API server that has access to the secrets, and that's a hard no from us (it won't be soon, though, because [see upthread]).

This is not everybody's optimal DX, and we get that. It's a whole public cloud platform, so there are lots of tradeoffs we have to make.

Part of it is probably that smart people want to work on harder and more ambitious problems, which YC advisors seem to say a lot. And also smart people want to work together

As I understand it, fly.io built a cloud from scratch, from the hardware up. That's HARD. It's typically something only well-funded companies like Amazon / Google / Microsoft do, with billions of dollars in products to justify it. Cloudflare is another company that pretty much did it, and it seems to have taken similarly ambitious engineering (although maybe theirs isn't fully general purpose)

https://fly.io/docs/about/security/ - We run on our own hardware deployed in secure data centers like Equinix

Most cloud companies in the 2010's built on top of other clouds like AWS. For example, Heroku and I'm pretty sure dotCloud, the company that became Docker.

I always thought that was a weird design, but it definitely saves you a lot of time and hassle. At the cost of a lot of software complexity and platform risk.

Even so, there were at least 20 or 30 of these companies, and most of them didn't make it.

Google App Engine (one of the first PaaS like Heroku, circa ~2007) also built on top of Google's data centers and Borg of course. They didn't manage their own hardware -- they used another cloud for that!

From the outside, "hosting" may all look kind of similar, but there are worlds of difference under the hood.

---

I'll also say that the subject of this blog post -- cloud security -- is extremely hard.

FWIW I joined the team that published the Macaroons paper ~10 years ago, and I tried to implement Macaroons as a side project then (for about a week, it wasn't very serious). I was looking for something simpler and more elegant, but there's no real magic bullet, and I think this post kinda concedes that toward the end.

But when you have really hard problems, it's not surprising if the solution is really complex!

The problem can actually be infinitely hard, because you have adversaries that are both more powerful, and that adapt.

Google learned this the hard way ~10 years ago when it learned that BOTH the United States government and the Chinese government had successfully attacked it :-/

People seem to forget about these incidents:

https://en.wikipedia.org/wiki/Operation_Aurora

https://en.wikipedia.org/wiki/MUSCULAR

https://www.washingtonpost.com/world/national-security/nsa-i...

So basically, if you are successful, then it earns you more problems :)

I think they've stopped talking about it but there used to be commentary from Google's "Better to turn down a good person than hire a bad one" logic (Or was it Facebook?) about how a C-employee can't evaluate an A-employee, and so they hire B-employees and it becomes a race to mediocrity.

My last hiring cycle I just missed working at a place with a couple guys who I dove pretty deep with on some esoteric topics over beers at a handful of tech meetups. The job req got weird and I had already accepted a position by the time it got sorted out. But lots of people have stories like that. It's one of the reasons people go to those meetups. I think for a lot of us it's being able to talk to someone who can keep up with your stream of technical talk, but the business contacts are a lovely bonus.

lol, the picture in the post is of a macaron, not a macaroon

https://www.foodnetwork.com/recipes/packages/baking-guide/ma...

Reply on news.ycombinator.com