Three things to never build yourself: auth, notifications, payments | Hacker News Reader

Three things to never build yourself: auth, notifications, payments

courier.com

213 pointstroygoode185 comments

Never outsource Auth. Maintain control over user accounts. That's the life blood of your business. If you have to ask everyone to reset their password because your auth provider increases their pricing or goes out of business, the churn will likely kill your company.

I would say the same for Stripe, but at least they'll help you migrate off their platform. Auth providers cant help you because the passwords are hashed... You need the same algo or you cant authenticate using the data they have.

And the only way off without a mass password reset is a silent migration in the background: migrate the user when they login.. but we all know that will take months and you will never get 100% to login during the migration period.

Pick an auth provider and you better believe in their business as much as your own. You will incur damage when you leave.

It is probably true that outsourcing your authentication to another party is probably bad, you probably should not write your own. So you should likely use a package for your auth code written by someone who focuses heavily on that.

In Elixir-land we're lucky. Jose Valim took his experience from writing Devise (the dominant Ruby gem for auth) and made auth generators for Phoenix apps!

You own the user table, but the auth logic is written by an expert.

> Never outsource Auth. Maintain control over user accounts.

This is some of the worst advice I've ever seen on HN. Don't take on risk you don't understand and cannot afford to mitigate. Don't be the next Equifax.

100%. I'm old enough that when I started my career we were still storing credit card numbers on our servers (and many companies were unfortunately doing so in the raw...). PCI compliance luckily knocked some sense into the industry. Crazy to me that some people don't realize that PII and secrets like passwords are basically in the same category now.

It is really not difficult to use BCrypt or Argon2 with your backend of choice.

Proper password and session management have been made accessible for many years if you use any "modern" backend.

> Its really not difficult

Good luck to you!

It's true the happy path and common error cases are easy to account for. But it just takes one overlooked bug or vulnerability either in your code or a library's code to send it all crashing down.

This is true across the board. One tiny crack in a bridge might mean it will collapse. That doesn't mean we should stop building bridges.

Sure, but it certainly means we should leave bridge building to those who are experts in that field.

> It is really not difficult to use BCrypt

Something I once saw in production:

An otherwise normal usage of bcrypt, that "prehashed" the password with sha512 before passing the resulting hash to bcrypt. This is conceptually reasonable (although not very useful) for reasons I don't think are worth going into, but had two critical problems.

1. It technically broke bcrypt's side-channel protection against timing attacks (bcrypt runs in constant time, while sha512 does not).

2. The much worse issue was that the binary hash was being passed to bcrypt. If bcrypt encounters a null byte in the input, it considers that to be the end of the input and ignores everything after, c-string style. Every byte of the binary hash had a 1/256 chance of being a null byte, which means 1/256 passwords result in a hash with the first byte being null, which means bcrypt will accept any of them as equivalent to the others. You could login to 0.4% of accounts by guessing ~256 completely random passwords (and a larger number of passwords were similarly but less extremely weakened)

You can say "well that's their fault for mucking with the input to bcrypt" but I bet most people would believe that kind of operation was safe before it's explained to them why it's not. It's very easy to accidentally destroy the security of a crypto scheme without knowing it, and thinking that what you're doing is perfectly run of the mill and safe.

Thank you for sharing that experience. It was a very interesting read.

However, lest anyone get the wrong opinion, password hashing is a very simple, straightforward, solved problem and has been for years.

One person doing something monumentally stupid doesn't make Auth some kind of cryptic minefield.

> Stop trying to make something sound difficult that isn't. Password hashing is a very simple, straightforward, solved problem and has been for years.

Try asking the following question on Twitter:

"While passing binary data, which example is safer for storing passwords?

[ ] $password | sha256 | bcrypt

[ ] $password | bcrypt"

What do you think the average programmer would say? Most would probably say they are either equal, or the first one, without knowing this specific thing, because most people don't implement their own password-hashing, they use library/framework provided ways that has been established as best practice already.

But, can't blame them really, the difference is marginal and innocent on the surface, but once you understand the implementation, you'll see the holes.

I think the question isn't particularly relevant because many programmers are just using bcrypt in a simple way that's very well documented. I think it's reasonable to expect people to know that "auth is dangerous, don't just do your own special thing and assume it'll be fine." If you want to do something that diverges from the simple case, then please don't try to roll your own auth.

My point is precisely that it wasn't "monumentally stupid," but rather a pretty reasonable decision given what the average person would expect bcrypt to provide functionally. The behavior I described is an unexpected landmine that people using bcrypt need to be aware of. There are likely others, though probably not enough that I would describe it as a minefield.

I would wager that most people reading this, if earnestly asked by a coworker "do you think it's fine for me to hash a password before passing it to bcrypt? I want to be able to support passwords over 72 characters and bcrypt truncates its input." would answer something along the lines of "I don't see how it could hurt" rather than "that's dangerous because a binary hash would result in a large portion of the passwords being hashed as an empty string"

The engineers that originally implemented and reviewed this were not idiots, they just weren't security experts.

My answer would definitely be: I don't know, but why bother. We know bcrypt by itself works. We don't know whether it works combined with another hash, and it's extra code we need to maintain.

It's monumentally stupid to think auth is easy. Yes there are standard cookbooks and checklists out there. The hubris to think that auth fails due to a crypto scheme choice is why average programmers consistently fail at it.

The yearly report of leaks in Fortune 500 companies should be proof enough of this.

EDIT: To elaborate. Crypto scheme is only one tiny facet of a successful authentication solution. Where do you store the hash? What language and stack are you using? What is the maturity of libraries available to you? What protocols? And many more seemingly tiny decisions. All it takes is a lazy developer that imports an insecure transient dependency or snoozes on a CVE.

> You can say "well that's their fault for mucking with the input to bcrypt" but I bet most people would believe that kind of operation was safe before it's explained to them why it's not.

What makes you think this foolish developer would not do the exact same thing before sending the data to an externally hosted auth service?

The external auth service's API would receive textual input (expecting a string read from a textbox), not binary. A hash that included null bytes would output those in hex as ASCII zeroes, which wouldn't cause the same problem when the external service received the value.

And that is, in fact, how you should perform a hash before bcrypt as well.

bcrypt accepts text input, not binary.

So if you use a pre-hash, you pass the hex or base64 output from the hash to bcrypt.

It's not that non-standard. You are describing the method Dropbox wrote a post about some years ago:

https://dropbox.tech/security/how-dropbox-securely-stores-yo...

> First, the plaintext password is transformed into a hash value using SHA512. This addresses two particular issues with bcrypt. Some implementations of bcrypt truncate the input to 72 bytes, which reduces the entropy of the passwords. Other implementations don’t truncate the input and are therefore vulnerable to DoS attacks because they allow the input of arbitrarily long passwords. By applying SHA, we can quickly convert really long passwords into a fixed length 512 bit value, solving both problems.

> Next, this SHA512 hash is hashed again using bcrypt with a cost of 10, and a unique, per-user salt. Unlike cryptographic hash functions like SHA, bcrypt is designed to be slow and hard to speed up via custom hardware and GPUs. A work factor of 10 translates into roughly 100ms for all these steps on our servers.

> Finally, the resulting bcrypt hash is encrypted with AES256 using a secret key (common to all hashes) that we refer to as a pepper. The pepper is a defense in depth measure. The pepper value is stored separately in a manner that makes it difficult to discover by an attacker (i.e. not in a database table). As a result, if only the password storage is compromised, the password hashes are encrypted and of no use to an attacker.

Your point 1 seems wrong. SHA-512 is constant time for any reasonable password length and any reasonable implementation. It hashes 1024-bit blocks in constant time, as it only uses 64-bit word operations such as add, shift, xor etc in a fixed order with fixed constants. There are no input data-dependent lookups, or other variable-time operations.

If you are considering the string copy that happens when the unhashed password is copied into the block to be variable time, probably you shouldn't count that against SHA-512 because there will be variable-length string operations before bcrypt starts its hashing too. Not to mention parsing it out of HTTP parameters or reading it from a GUI control. (If you really care about timing observations, perhaps you'd pre-hash the password on the client so what is transmitted is fixed-length).

Point 2 is of course a real problem. Though to be honest I've always regarded the NUL check and (sometimes, implementation-dependent) maximum length as unnecessary, arbitrary flaws in bcrypt - so bcrypt gets the blame. On grounds of purity and doing the right thing, I refuse to use raw bcrypt, and in fact I use: SHA-512 -> Base64 -> bcrypt when password hashing.

Does that make me wrong? Only if Dropbox is wrong (I hope they just forgot to mention Base64 in their old security article.)

> Your point 1 seems wrong. SHA-512 is constant time for any reasonable password length and any reasonable implementation.

I'm aware of this, and it's why there's a "technically" in my original point. Of course in practice it makes no difference, and I consider the better use of bcrypt's input space when prehashing to outweigh leaking a pretty useless amount of side-channel information for extraordinarily long passwords.

Though as a follow up anecdote, one of the engineers there had set a 300kB password to test the prehashing implementation, and just continued to use it (via password manager). He was potentially leaking a bit of timing information. I'm not even sure if a password that long made a timing difference that was detectable through network jitter, but if it did and some hypothetical attacker was able to learn that his password was hundreds of kilobytes long, well... good luck.

> in fact I use: SHA-512 -> Base64 -> bcrypt when password hashing. Does that make me wrong?

No, using a base64 representation of the sha hash makes your scheme perfectly reasonable. Also note that the base64 representation of sha512 is a bit longer than the typical bcrypt input length, so you're shrinking the collision space somewhat by truncating your sha hash. Certainly not enough that it would matter at all, just a point I find interesting. Hypothetically, a binary representation with a pass that removed or replaced NUL bytes would preserve more entropy, but we're well past the point where it matters and would be playing with fire to consider that complexity.

256 bits is enough collision resistance, so I don't mind truncating SHA-512, even down to 256 bits.

If you find collision space reduction interesting, then you might find it even more interesting that truncating SHA-512 makes it more secure not less.

This (and the higher rounds safety margin) is why I choose truncated SHA-512 over SHA-256 for this sort of application, even though the number of output bits is the same:

Check out the red boxes on https://en.wikipedia.org/wiki/Template:Comparison_of_SHA_fun...

What's been interesting to me is that during tech interviews, we eagerly have applicants roll their own sorting, or searching, or caching. But you never have questions about crypto or auth. Why one and not the other? Are both not critical to get right?

And then I'm reminded of the bug that was hidden in plain sight for 20 years in Programming Pearls[1] in their implementation of a binary search. This same algorithm was copied in Java and elsewhere, near verbatim. So, yeah, maybe don't roll your own searching either. The experts have enough difficulty.

Frankly, I don't trust most programmers to get anything right. People toss around "battle-tested" libraries and we all just take it for granted that that's what they are. But I recently found out one incredibly popular package for one incredibly popular language is doing something fairly dumb. Not insecure (that I can tell). Just simply unnecessary. They are signing session cookies. For reasons I could not fathom. Signing doesn't prevent cookie theft. It's just another layer that someone thought would add security over just sending a random string (hopefully your system/language's random generator is actually random). And that's how a lot of these mistakes happen. Let's add sha512 on top of our crypto. Because more = better, right?

Speaking of random, there was another popular package out there for generating UUIDs that has a github issue where someone discovered the package was generating collisions. Yikes.

[1] https://ai.googleblog.com/2006/06/extra-extra-read-all-about...

Isn't signing the session cookie to prevent cookie tampering? You can send a random string, but you would need to persist the random string somewhere in order to verify against it later

The cookie value has to be stored in some sort of map\dictionary that links to the session data. If the cookie value is a random string, you don’t need to store any more data than you would with a hashed cookie value. Or maybe I’m misunderstanding your point?

ah, okay. I think I misunderstood. I was thinking about the case where the data is stored in the cookie itself (i.e. an encrypted cookie), and you would need a way to verify that the contents weren't altered. If it's storing a session id, then there isn't a reason to also sign it. Sounds like they conflated the two approaches.

In what scenario do you need to have personal info associated with accounts and don't need to have full access to it? I see no user privacy benefits to outsourcing auth, unless all you need is to do is to verify someone's a real person.

But he's arguments are pretty sound to me. Do you have any counter-arguments than a generic statement?

I guess the question is what's more likely, Facebook, Google or Apple leaving the oauth game or a bug in an internal solution.

what's more likely - you being able to talk to Facebook, Google, or Apple to let your app/site use their authorization solution once their algorithm has decided to ban you, or you fixing your bug.

You're missing the original point. rgbrenner is talking about auth solutions like Auth0 where users can use username + password to sign up/login, and when you eventually are gonna have to leave (which, if your business is successful, you eventually will), you'll need to reset passwords one way or another.

Of course, if you only use third-party authentication for your service, it doesn't really matter what's in-between there, you can work around it.

That they will ban you for one reason or other has a very high possibility of occurring before you go out of business.

You and I have different definitions of very high possibility.

I had an app get banned for copyrighted content by Google because there was a movie poster in one of the backgrounds of the screenshots (camera type app in 2013). It will happen, probably for something ridiculous. In every job I’ve worked at, it had happened or they’d been threatened to be banned, at least once.

I don't think the main problem with Equifax was the Auth info they lost. It was their own business data, e.g. social security numbers etc, which wouldn't have been outsourced to their Auth provider anyway.

This is literally the Auth0 first contact sales pitch designed precisely to scare you out of even trying to assess the risk of building it yourself.

The reality is, though, it's never been easier than it is today to build secure auth - yes, it's still hard to do it right - but less hard than it ever has been before. So many security features are now a checkbox on a cloud provider's web interface that would have been a manual implementation of the underlying protocol a decade ago. So many languages have matured battle-tested OS libraries for handling the especially sensitive parts like crypto.

This notion that it is a "no-brainer" decision to happily trade your customer database and a % of revenue to remove the burden of stewardship seems crazy to me.

Not to mention there’s a whole lot of middle ground between writing your own auth logic and using a hosted service. Any reasonably popular web-oriented language has plenty of fully featured, high quality libraries you can often just drop in.

Hell, we use Laravel at work, which includes pretty robust auth features out of the box. It feels like such table stakes stuff for a web framework that I find the idea of paying for a third-party cloud service absolutely bananas.

It also includes pretty solid notification and payment solutions too, though these usually plug into services like SNS and Stripe and don’t really fit the “self hosted” bill.

I think that's where the confusion lies. Are people suggesting someone to write their own auth from the ground up, or write auth in coordination with their stack's auth library of choice?

It appears to me the assumption these people / articles make is that nobody is using existing hardened production ready libraries, but hand writing everything from the ground up for their todo app.

Because otherwise their message falls flat on its face.

> but less hard than it ever has been before

I have a very vested interest in this (founder Clerk.dev) - but the exact opposite experience is what led us to start Clerk.

Auth was easy when it was just email verification and choosing the right hashing algorithm. I used to pull Devise off the shelf and get it going in 30 minutes.

But these days, the simple solution is far from complete. I grew most frustrated trying to setup the rest...

- Single sign-on / Oauth with proper de-duping so users can always sign in regardless of how they signed up

- 2fa so users can better secure their accounts

- Integrations with a leaked password corpus (haveibeenpwned) to prevent credential stuffing attacks

- Active device tracking and remote session revocation

You can launch without these things, but it's definitely impacting your overall security and the conversion rates through your sign up and sign in flows.

I found these features quite simple to implement. What gave me a headache was advanced anomaly detection (both from the per-account and per-requester perspectives) and handling. For me, there is no bottom of the well on those things, and they are the killer features of auth as a service.

> So many languages have matured battle-tested OS libraries for handling the especially sensitive parts like crypto.

We use CSPRNG/SHA256/SHA512/AES/3DES/RSA primitives in .NET without any trouble. Microsoft made it pretty difficult to fuck things up as long as you don't sprinkle Math.Random in with your session token logic like a jackass.

Well, we implemented Auth using Auth0. The experience wasn't terrible, but not fantastic either. Probably best described as "ok".

However, now they've been bought by Okta, which has a pretty bad reputation.

So, I guess we'll get to find out eh?

Using the same authentication algorithm doesn't seem like a particularly big price to pay, assuming they started with a sensible one. And there's nothing stopping one from making it support the old style but add a more up-to-date algorithm for new password sets, either.

You can do both. Just use an open source project for auth instead of reinventing a commodity wheel.

You can also choose to self host. Keycloak and FusionAuth (disclosure, I am an employee) let you self host. You then have the user database in your systems.

> And the only way off without a mass password reset is a silent migration in the background: migrate the user when they login.. but we all know that will take months and you will never get 100% to login during the migration period.

Actually, not true. I can't speak for every auth provider, but FusionAuth and Auth0 both let you have the password hashes. If you know the algo (ask your provider!), you can load the hashes (and other ancillary password data like the salt) and your users will never be the wiser.

Here's a guide I wrote about how to migrate off of Auth0: https://fusionauth.io/docs/v1/tech/guides/auth0-migration/ The end goal of the guide is to move to FusionAuth, but the steps to get your password hashes out of Auth0 (the 'Exporting Users' section) will work no matter where you migrate to.

I'm currently implementing Keycloak for my startup. It's not perfectly documented, but I really like all the things it can do for us and how extensible it is.

I'd suggest checking out FusionAuth, but I'm a bit biased :) .

I have heard good and bad things about Keycloak, but I know one thing for sure: it's far better than rolling your own auth.

Good luck with your startup!

I understand that you want to promote your solution and you have a disclaimer in the previous comment - and that's great. But if you keep insisting then you should IMHO at least provide some reasons on why your solution may be better than e.g. Keycloak and not just posting the name a couple times and see if it sticks.

Fair point. Here's what I'd say if someone was choosing between Keycloak and FusionAuth:

I believe that FusionAuth has a better developer experience. Everything is an API, the docs are regularly updated (though we can always do better). There's an easy way to set up developer environments to a known state (Kickstart) that I don't believe Keycloak has an analog for.

FusionAuth supports limited memory environments. We have folks running in 384MB of memory. From what I've read, Keycloak wants more resources, though Keycloak X is apparently a good alternative. I don't know if Keycloak X has feature parity.

Theming is easier with FusionAuth, more typical auth flows can be themed, and themes can be entirely API managed: https://www.keycloak.org/docs/latest/server_development/#_th... vs https://fusionauth.io/docs/v1/tech/apis/themes/

If you want to run thousands of tenants,, FusionAuth is better. Compare https://keycloak.discourse.group/t/maximum-limit-of-realms/8... vs https://fusionauth.io/blog/2021/03/29/seegno-thousands-tenan...

Both options offer support (Keycloak via the Redhat SSO package), but you can view our pricing without talking to anyone. I wasn't able to find pricing for Redhat SSO (that's usually not a good sign, but maybe someone who has engaged with them can add more, I could be mistaken and it could be super affordable).

FusionAuth has a 100% free as in beer edition with unlimited enterprise or social connections, users, and tenants. (There are certain usage restrictions; you can't package FusionAuth and resell it without a paid license, for example.)

Keycloak has a large community and is 100% open source. Those are definitely strengths that I want to acknowledge.

It's always good to evaluate software as critical as auth (or payments or notifications, for that matter) with a POC simply because everyone's situation is a bit different and it does get embedded in your systems (source: I picked Stripe for a startup and evaluated moving off multiple times to save money but didn't end up doing so in part because of effort and opportunity cost).

You convinced me. I've been eyeballing Keycloak, will definitely give FusionAuth a shot first. :)

Awesome, would love to hear how it goes.

I've never checked out your product, but we use Keycloak extensively and I can say it is a great Auth solution and I've used several and even built some (poorly).

The only real downside of Keycloak IMO is the documentation. Because it covers so many bases, and is very extensible, the official doc is enormous. There are a lot of lightweight articles about how to use it, I'd use one of those to get started.

Some great things about it are:

- it's widely used, so you can google your way out of most problems

- it does everything. When someone asks if you have 2FA or support Yubikey - the answer is yes.

- it's very extensible. Great APIs, and you also have access to the underlying database should you really need it (we never have, I'm sure some do).

- it's open source. IMO this is a non-negotiable for an auth system

What other FOSS like Keycloak we have available in 2021? Looking to update my knowledge on options

There is ForgeRock and WSO2 which we have used in the past. Perhaps they have moved on but would not recommend vs Keycloak.

Ones that I've heard of in conversation with customers:

   * IdentityServer (.NET)
   * The ORY suite (hydra, etc)
   * Gluu

I'd say Keycloak is the most common one I hear.

There are about 15 FOSS packages listed here, but you'll need to weed through what is an auth server and what is not: https://en.wikipedia.org/wiki/List_of_single_sign-on_impleme...

Awesome, appreciate the perspective. A few qs for you:

My understanding is that you ended up writing java code to extend keycloak. Is that wrong?

Is keycloak resource hungry or have I heard wrong? Have you tried keycloak X? Is that better?

Is there a way to configure a dev environment? Is Terraform the preferred IaC solution?

How many realms do you use typically?

> My understanding is that you ended up writing java code to extend keycloak. Is that wrong?

Depends what you mean by "extend". For many operations, there are REST APIs, e.g. to create a realm (somewhat equivalent to a tenant in the SaaS world - though see below), it's simply POST /auth/admin/realms. There are also more elaborate extensions such as login listeners that if I remember correctly, you would build in Java, but we don't need those.

> Is keycloak resource hungry or have I heard wrong? Have you tried keycloak X? Is that better?

We haven't found it to be disproportionally resource hungry at all. I'm sure there are scenarios we haven't exercised but resource usage not a problem for us. Not fond of the way vendors throw around allegations like that without hard facts.

> Is there a way to configure a dev environment? Is Terraform the preferred IaC solution?

Depends on your own preferences, we just bring it up in a docker container.

> How many realms do you use typically?

Everyone will skin this cat differently, but for us (SaaS):

- one big realm that most tenants use.

- one admin realm for trusted internal access (i.e. our support team).

- one big realm that candidates use (we are an HR platform)

- one realm per tenant that needs SSO. e.g. if they want to authorise login using their own GSuite account, they get their own realm with their own idp.

As someone else pointed out, having large numbers of realms is a weak point of Keycloak. I look forward to us having enough SSO clients that that becomes a problem for us :)

Thanks for sharing your experience.

Always good to hear from someone who's actually used software rather than second or third hand perspectives.

> not perfectly documented

Understatement of the year!

> It's not perfectly documented

I want to chime in to this too.

If anyone from the Keycloak project is reading: keycloak is awesome, but the documentation is really the most painful part.

The main problem is that it assumes that people installing and/or configuring keycloak already are JBoss/J2EE experts.

Please assume knowledge of GNU/Linux systems, the OAuth/OIDC stuff, LDAP and SAML, but do not assume knowledge of JBoss.

They know about that quite well actually. But there's also Keycloak X in the works which is based on Quarkus. It's not mature enough yet but I hope they tackle a lot of the bad things.

+1.

I maintained and extended a self rolled auth system built on top of django and it was a constant headache. We lost weeks of engineering productivity fighting with keeping the various libraries up to date and running into the usual "forgot password" edgecases.

I spent a few days with FusionAuth for a demo project and once I grokked it I really wished we had used it or something similar instead. Amazing abstraction layer to have at your disposal.

Making people reset their password is not that big of a deal. I have run into this multiple times, the most recent being amtrak.com. It just told me "you have to reset your password" and I got a link in email like if I had forgotten it. I have also received many "we had a security incident you have to reset your password now" emails in recent years.

There is a whole line of thought that accounts should be passwordless and accessed via an email reset link as part of normal login procedure.

Which is great until you lose access to your email.

if you loose access to your email, you may just be out of luck. a good service will not allow you to change your registered email address without verifying that you own the old one

Not in my experience. Every account I can think of allows me to change my email address without verifying the old one.

And how would you even login to your email using that method :)

Yeah I hate that method. It's good in theory but terrible for the user. I go to a website I want to use. I click the magic link. The email takes 10 seconds to show up. In that 10 seconds I find something else to do while I'm waiting. I get distracted and find the magic link in my email 10 minutes later. It's now expired and I have to start over. Or just give up.

With a password, I click the site I want to use, my password manager fills in the password for me, and I click "login" and bam, I'm done.

> The email takes 10 seconds to show up.

Even longer if you have greylisting turned on, it gets temporarily rejected which means you then have to wait N minutes before it gets accepted (assuming they retry sensibly and don't just spam retries every 2 minutes and get fail2ban'd.)

By which time it might well have timed out anyway...

A faulty line that's miserable UX and barely better security, but sure, a line.

You sure it wasn't phishing. SCNR

Yes

Every phishing mail I've ever got talked about something something security click this link pl0x. OP is right in the sense that not everyone will follow (in a given time frame) and "You will incur damage".

...but the same is true if you would have a leak. So whatever I guess.

It's one thing to host your own/maintain your own auth database, but there is a definite benefit of using a tool like Okta SSO to link your authn to other services, if that's your business.

I've built auth systems from scratch and used third party auth services. Doing it from scratch always feels reasonable at first when it's just basic username and password, But I start to regret it once a client starts asking for more features. MFA, federation, auditing seem to inevitably become requirements at some point.

If going with a third-party service, my advice would be to make sure you understand how to export the data out of the system, including hashed passwords using a non-proprietary algorithm.

> Auth providers cant help you because the passwords are hashed... You need the same algo or you cant authenticate using the data they have.

You absolutely can migrate away from Auth providers. Auth0 (disclaimer: former employer) hashes passwords with Bcrypt, which is most likely the same exact algorithm you would choose. Bcrypt stores salts & rounds as part of the hash making those hashes fairly portable.

Even if you wanted to change algorithms, there are ways to do this so that it is transparent to the user, such as running the plaintext password through a new algorithm after a successful authentication and storing it in a new column.

He mentioned transparent migration and the problems encountered with it.

> Auth providers cant help you because the passwords are hashed... You need the same algo or you cant authenticate using the data they have.

Seeing as you have that it becomes a non-issue

Perhaps I'm incorrectly assuming auth providers don't expose an API for exporting all that data. Such an API would seem to be a fairly attractive attack vector.

Auth0 and fusionAuth don’t have APIs but will process requests manually.

I would expect that the way to do this sort of migration is to run it until the costs associated with maintaining the migration infrastructure exceed the projected losses due to churn. You don't need to worry much about auth provider pricing because these are typically billed on the basis of monthly-active-users, so even if you have a small number of users who aren't active, they'll hardly show up on your bill. When this pool of unmigrated users is small enough that you can tolerate churning a percentage of them, you can turn off your migration infrastructure and enforce a hard password reset.

That said, I don't know what kind of business has a significant percentage of users who churn because of a one-time password reset. I can't imagine any b2b contracts being canceled because the service provider enforced a password reset. Maybe retail sites are different?

I am on the side of not using one of these Auth companies, but if you are learning or just want to make a proof of concept fast, it definitely helps to have that taken care of for you.

Why not build in abstraction?

If you have a middleware that lets you swap providers, you can start off with a ready made solution and switch if necessary.

Obviously, this still means you need to maintain control over your data; but that's surely something a startup will do in any scenario, right?

I agree with Payments (PCI-DSS hell) & Notifications (especially email & SMS), but Authentication? Fuck no. Unless there is a business requirement for sharing your authN scope with multiple external systems, you should not worry about plugging your app's auth into someone else's idea of a good time. Consuming external authentication providers like Active Directory is something we do today, but it barely registers as "more complex" than integrated user management because its still the same server+process handling everything and all of the authentication state still lives on the same box (AD writes audit logs but these are out of scope from the actual auth flow).

We started trying to plug our app into an external SAML authentication flow (Azure) and it is making it much harder to prove certain semantics around user session lifetime and app security. Maybe it would be "easier" if we had just started with cloud native shiny BS, but I doubt it. Auth state would still be scattered across multiple systems, even if they all lived in Azure (or wherever).

Any time you take a problem that exists on one computer and spread it across many computers, it gets exponentially more difficult and prone to error.

> Any time you take a problem that exists on one computer and spread it across many computers, it gets exponentially more difficult and prone to error.

Hard to argue with that.

Unfortunately sometimes you can't say "we have the single source of truth" about a user; federation exists for organizational reasons.

Crap, I’ve done all three! Well, maybe not all at once though.

I can see why not to do notifications, but it’s hard to avoid doing some form of it if you’re making an app with focus on low latency realtime updates.

The bigger problem is that out of all three categories, I only really feel content with trusting Stripe, and only so much. Auth0 has its issues, and decent authentication systems you can roll on your own are plentiful. Maybe building it yourself no longer makes sense, but I do think owning your user database is a good idea, even if you do primarily lean on OAuth.

I'm curious how people handled payments before Stripe? Did you integrate with banks directly?

How different is that in terms of what Stripe offers?

There were, and still are, many gateway providers with APIs for payments. Authorize.net, NMI, USAePay, BrainTree, Payeezy, even Paypal was common enough once upon a time. And those are just a handful that I can list off the top of my head. (I know that some of those I listed have acquired each other recently as well.)

I guess it depends on what counts as handling payments yourself. Stripe can handle just about everything to managing subscriptions and emailing invoices and more. At least nowadays. It’s a lot more advanced than systems that roughly just provide an API to manage payment methods and charge them, like Authorize.net for example.

Also, handling the abstraction between multiple kinds of payment methods (cryptos, different card networks, PayPal, etc.) may also be a considerable amount of complexity that a payment vendor can help with.

I have never directly interacted with the financial system at its lowest levels. I’ve heard it is quite a trip.

We used CyberCash (I think) and Authorize.net, which were to Stripe as Altavista was to Google search.

That is to say that even 20 years ago there were middlemen in the payments space, they just were pretty horrendous to work with.

> Authorize.net

Arrreghhjh!!

Brings back horrible memories from the .com bubble for me.

F authorize.net.

You would typically integrate with a gateway who integrates with one or more banks who integrate with the schemes to get global coverage.

There are generally a number of domestic gateways in each region (I’ve worked with gateways in Australia, Malaysia and a couple of other places) but there are also larger providers.

There are name brands like Braintree and lesser known but still quite large, like Adyen. The API quality is extremely variable.

Checking out multiple gateways is a good idea. Stripe is supposed to be great but IIRC is relatively expensive.

I'm in the same camp. Years ago I have done all three myself too. Now I'm in new place where everything is done by AWS, and I do admit it is convenient but also... quite slow.

I don't think deciding not to do something out of fear of doing it wrong is the right approach... If it's business decision then at least it's backed with some reasoning - don't do it because it's cheaper to buy it.

It's been said a few times here, but it's important so I'm calling it out on the top level:

Don't roll your own auth. But do feel free to use a battle tested auth library / or framework hosted in your own app.

Then you get both benefits: 1. You don't have to put your destiny in someone else's hands, and 2. You're not getting the risks associated with building your own auth.

Regarding auth, I absolutely bought to JWT cool-aid but honestly if you're still on the monolith phase just use the most popular auth framework for your language. JWT adds a lot of complexity and room for misconfiguration, you do get something in return of course - it is stateless (hence scalable), works great with microservices, and improves your security model somewhat by separating issuing from verification. But you do have to pay for it, expiry gets tricky, the client code gets trickier, permissions get trickier. For most people it just isn't worth it

We are going through this currently. Have a large new system going in which relies on OAuth and JWTs and our IAM team is now spending a lot of time & energy with the developers on all of the use/edge cases with tokens, expiry, security, and whether the code should be in the client or the server. In the end it'll work out, but I completely agree that grabbing the most popular auth framework for your language will save a lot of headaches in the vast majority of cases.

To make things easy I usually use "alg":"none". It makes using jwts a breeze.

https://datatracker.ietf.org/doc/html/rfc7518#section-3.6

I hope this is a joke.

It was, but judging by the downvotes my delivery was off :)

Like a sibling comment said, hopefully this was tongue in cheek.

If you use "none", anyone can forge a JWT that says anything. I always say:

* You should have some other way of verifying that the JWT was unchanged by the client, like say being on a private network or using client TLS certs

and

* You should benchmark and know that the signing overhead is a significant source of performance degradation in your system.

Otherwise, sign your JWTs! :)

I recently discovered the TLS_NULL_WITH_NULL_NULL SSL ciphersuite (https://datatracker.ietf.org/doc/rfc5246/) - it makes analyzing network traffic so much simpler, and you don't even have to deal with certificates any more!

I think that a standalone auth solution is a good idea when you start to have multiple applications (microservices or not) that want to have the same user database. These could be multiple custom apps or COTS or OSS.

You don't have to use JWT to get a standalone solution (see for example this guide which I wrote: https://fusionauth.io/docs/v1/tech/guides/single-sign-on/ JWTs are used briefly, but most of the heavy lifting is done with application sessions).

If you have a single application, definitely use devise, passportjs, spring security or whatever is in your framework. 100% agree. But pretty quickly you often are adding in a forum, helpdesk, GSuite for employees, etc etc and having a single source of truth for a user is good.

More general: Things to never build yourself: Things outside your core business. Are you selling a notification service? Build it yourself, otherwise find either self-hosted or hosted solutions, depending on situation.

I've always thought this statement from Martin Fowler was right on target.

> Boiled down this means that if the business process you are supporting is part of your competitive advantage you should build custom software, if not you should buy a package and adjust your business process to fit the way the package works.

Source: https://martinfowler.com/bliki/PackageCustomization.html

This is a great read.

https://www.joelonsoftware.com/2001/10/14/in-defense-of-not-...

Predating Fowler by about 10 years.

There’s nothing I dread more than customizing software, and then having the supplier release a critical update that requires you to re-do the work and search for bugs. It’s so expensive and error prone, it really is just better sometimes to treat your version as a hard fork or else make your own system

Sure, this is understandable wisdom. However, outsourcing has its risks too, as you need to trust who you are outsourcing to, often with your critical data, and your user’s data. Not only that, but you hard-depend on someone who does auth. Will these companies all be around in 5 years? Do they meet your security and compliance needs? etc.

I know why people are against building custom solutions. I’m just gonna say that I personally felt safer running third party Django apps that handled authentication but on a server I controlled, vs for example, Auth0. Not to mention the pricing model. The disadvantage is that you may have to put some effort into helping maintain it if it’s part of your critical path, but that seems like a good tradeoff to me. The point is to get an MVP out quickly, right? If things work out, having the resources to maintain an auth system seems reasonable.

This is not to pick on Auth0 so much; it seemed cool when I tried it out. It’s just scary to me to have my user database out of my control.

I've had to have this discussion so many times.

Every one of these external dependencies is... a dependency. Each one brings with it the risk of the external company going under, prices rising to unacceptable levels, intermittent outages, etc. These are all independent events. For the sake of argument, if the risk of a company going under any given day is 1 in 100, then using five separate companies means our risk is 5 in 100 (1 in 20).

We need to weigh the risks plus impact against the advantages.

High risk + low impact or low risk + high impact are generally acceptable.

Auth is a space where generally it's moderate risk and high impact while the advantages are... quite low versus just throwing Keycloak onto your cluster and getting on with your day.

> Are you selling a notification service?

Except when there are none, you have to build it yourself.

And sometimes you just can't change your processes enough for no package customization.

I'm currently building what's essentially an internal specialized platform that is used for data science/engineering as well as actually providing the models.

The data is strictly private to the level that you might actually end in jail if you have a breach. While we use AWS, said use of AWS is more than half the reason we have to build custom solution around Keycloak, because a) AWS SSO is purposefully limited in a way that prevents us from using it b) AWS really doesn't work well with SSO for getting credentials for AWS SDK applications.

We also need to self-host because we can't outsource auth, including possibly requiring a legal shield against known hostile actors like USA. (shit's complex on that)

Then we have essentially "normal business requirements", meaning that we actually want SSO that results in user logging in exactly once - when they login to their VDI in closed-off VPC. And that turns out to be really complex, especially in case of accessing AWS services later :/

I like the conclusion here:

> If it’s a core business function — do it yourself, no matter what.

https://www.joelonsoftware.com/2001/10/14/in-defense-of-not-...

Of course, then the issue becomes "what is core?"

There’s a huge range between doing everything yourself and handing your users to another company and make all of your business depend on an external SaaS provider‘s whims.

As a dictum this is sound but its much better to frame this as a business case, eg. cost benefit analysis and risk analysis. This is what it is rooted in. When you have a limited resource (time/money) make sure they are being optimized for greatest return if you are a startup.

No. Auth is not that hard, it's not hard to scale. It's the bloodline of every interaction between you and your customers. Nearly every "web" language has a open source Auth solution with modern jwts or jwts on cookies. Bcrypt and salting solves "storing passwords". Further more once you do it it's highly repeatable across products.

exactly. Don't be clever and try to do things like re-invent bcrypt or hmac or do things through obscurity. Use the libs. I've written many auth systems, it is not rocket surgery. It is one of those things where you need to do some reading and understand what is going on, but I'd argue you should do that for just about everything you are working on.

This article should have been titled "getting started with Courier notifications" or something similar, it's a tutorial with a bit of "don't roll your own" advice at the beginning.

So this post is just advertisement for Courier's own service?

As a frontend developer, I've been wanting to expand my skills as a server-side developer, and have been a little lost on where to get started in building an auth system, and how I'd approach deeply understanding it enough to do it right (regardless of whether it's the right business choice or not, it seems likely worthwhile to be able to do it). Does anyone have any good resource that would provide a robust exploration into building this kind of system, either in a specific language or just a general one? I'm talking sane abstractions that you'd see Fowler talking about, and something that I could actually follow while writing code to do it.

In the early 2000s, a significant part of my income came from payment integrations. Everybody and their grandmother was jumping on ecommerce, and payment processors made it a bloody pain. To this day, I will never understand why, but it seems like they deliberately made it hard. And expensive. One of the companies I worked for had contracts with dozens of different payment processors, a lot of thought went into how to direct payments to the cheapest processor for whatever type of transaction (country, amount, card types, risk of chargebacks etc).

I'm glad Stripe mostly killed all that.

Methinks "Three things I can carefully build myself and gain a competitive advantage over everyone else who thinks they can't..."

This is a biased article. I would not let another company handle authentication or notifications for my apps. Payments, yes.

Sure, it is obviously biased – I started Courier because I think developers shouldn't be building their notifications infra themselves.

Why would you offload payments instead of e.g. connecting directly to payment gateways yourself, but wouldn't do the same for auth & notifications?

Payment has lots of legal pitfalls if done wrong not to mention the financial damage.

FWIW given GDPR, CCPA, et al so do auth & communication. I'd honestly have a big problem with our team wanting to store passwords ourselves...

You don't store passwords. And regarding GDPR, I have problems giving sensible data to third parties. Another point of failure and no control for me what happens to the data.

Both auth and notifications get extraordinary complex at even minimal scale. Obviously each business needs to decide which areas make sense to invest from tech perspective but LinkedIn and AirBnB each spend around $20m/year just on their notification systems.

https://medium.com/airbnb-engineering/airbnbs-promotions-and...

https://engineering.linkedin.com/blog/2018/03/air-traffic-co...

> notifications for my apps

What kind of notifications are we talking about where you can get away with self-hosting? AFAIK, Push Notifications on smartphones have to go through Google and Apple's servers. Maybe if you're doing email notifications?

I think there are a lot of 3rd party wrapper around Google & Apples Push APIs, which are not really needed. You can send Push Notification yourself just fine.

For email though, not sure, that's a completely different topic. It comes with many headaches, using a service like SendGrid or Mailchimp seems like a good idea, and if you really want to roll our own mail servers down the line you can still do that.

I'm no mobile developer, so I don't really know what I'm talking about. Please enlighten me if you can. Doesn't push notifications on iOS require using Apple Push Notification service? That was my last impression I got when I talked with someone about it.

Yes, you send a request to Apple's APIs and they will deliver a push notification. It's very simple really. Now there are a lot of wrappers around that (like courier.com) and instead of using Apple's API you know use the wrappers API. That also often requires some kind of 3rd party SDK to be integrated in your app.

These services are supposed to make things easier and have additional features like SMS notifications and so on, but, well not sure, never really used them myself.

Right, that was my initial point. You can't self-host notifications as they always go through either Apples or Androids servers, with neither company provide a way to self host that part.

What are good wrappers (not services!!!) for the apple and Google (and browser) notification APIs that handle the entire process? I only used Firebase for this purpose as when I started to look for self hosted solutions, they were all scattered libs with 'only 20000 easy steps to set up notifications for x!'. Maybe this has changed and I missed it: I would like to move away from Firebase.

For email we run bsd (not that that matters, it is just a factoid) servers, correctly configured which seem to work well (mail does not get to spam unless it is).

It's kind of funny that you'd allow someone else to manage money, but you wouldn't let someone else manage something that has the potential to cost you an equal or greater amount of money in GDPR violation fees and lawsuits.

Exactly!

How does using an external service for auth shield you from having to be responsible with your users‘ data?

GDPR compliance is not something you can just buy.

I think this was in reference to someone managing compliance of your notifications.

Never is a really big qualifier. It is possible for practiced coders to build secure systems. This is how we get trusted systems.

Trusting auth systems is always a gamble. Remember when Google 2FA was only actually checking the first factor? I sure do.

In my business the simple account and password ID model is insufficient. Most of our accounts are for families or organizations with complex contact protocols. Because on site service is the product we have the luxury of extensive physical in person verification. Assuming that every auth issue is for a typical Internet SaaS is a huge mistake.

I'm really surprised a Notifications as a Service company would suggest you shouldn't build notifications yourself. Shawking.

I'm a hardware designer, apologies if this is a dumb question about "auth".

If you're creating a product that is primarily or exclusively for mobile devices, what's the downside of relying on "log in with Google" and/or "sign in with Apple"?

Some people would not login then, and a vocal minority will complain loudly.

For your business, this may be a positive overall.

I often wonder why startup devs earn such high salaries if apparently they outsource everything, and are very opinionated about it, and have the quotes from business gurus to support those opinions.

Connecting things to the correct parts of other things can be just as messy and difficult as writing code :)

This article is from a business perspective.

I sometimes write auth/SAML solutions. They're not super complicated once you understand the basics of the system.

Not to be jumping on bandwagons but it seems to me like decentralized block chain based accounts abstract both payments and auth out of your app and into a unified audited protocol. And since its a ledger, couldn't you include notification information aswell? I feel like this is the killer feature of all these dapps, 2 of these features come built in.

Do you have any examples of blockchain based account management?

A lot of people are talking about auth/jwt.

I'm wondering for those who build it:

- how did you implement impersonation

- how did you implement tenants ( and switching in between). Most have a follow-up dialog after logging in, which you can select.

- did you implement something in your gateway that checks the token, instead of every service separately ( too lighten the load to the STS)

Missing the obvious - rolling your own crypto

You probably don't need any crypto code at all if you don't roll out your own auth or payment.

As a serial entrePAYneur and the core foundational member of multiple payments companies responsible for all hardware and software architected, assembled and constructed with my very own hands I would sternly disagree with this article. I have done exactly what this article states not to do and have done so multiple times with each better than the last, it only takes an interest and an insatiable desire to learn. The world will come to learn the importance of "owning all the code" and "data control" as the cyber issues continue to cost countless billions and in time more and more bodies. Security will become paramount to everything and outsourcing for "cheap" will have a much greater cost than one realizes in the moment. Those that take greater risks have much greater reward and those risks shape the future for where we are today. My current efforts are the really fun stuff given the world's current temperature on so many fronts.

Always remember that which you read is just someone else's flavor of an experience in which they try to convey their ways on you. I am not bashing what they wrote just recognize how things would be if risks are not taken.

> As a serial entrePAYneur

I don’t follow

> core foundational member of multiple payments companies

If you’re building a payments company, your product is literally doing the payments yourself!

> with each better than the last,

So, the first ones weren’t very good huh? Perhaps you could have outsourced them?

> Security will become paramount to everything and outsourcing for "cheap" will have a much greater cost than one realizes in the moment.

You’re conflating the use of a 3rd party for cost reduction with using a 3rd party due to efficacy and efficiency due to specialization. There’s zero chance you’re as good at Auth as a whole team of security engineers at an auth provider, period.

I’m biased but I would add: monitoring

Yes! I think that the benefits of an outsourced monitoring service are huge compared to the amount of money you pay.

There are also simple ones that you can get started with for free/cheap.

I did all three with a team of 12 engineers.

But this was in the context of a cloud/infrastructure provider, for which IAM, billing and payments are the 3 pillars of the ecosystem. You just can't outsource these business foundations to a third party.

This list should start with Encryption and have a long break before it continues.

Should be titled "Three things you can't build yourself" as Apple mandates that you us theirs.

And crypto!

I think it's about services not libraries.

Right on. I was thinking from a capability perspective of an application.

From the title only, crypto was my first thought too.

All 3 carry great examples of APIs managing the many APIs within.

What great examples do you know?

Challenge accepted

What’s wrong with basic auth over https...

Don’t tell me what to do.

Letting other people handle Auth and Notifications is a great way to get the rug pulled out from under you.

Exactly, how are all the Auth0 customers feeling about that acquisition?

unclear if we will all have to switch off Auth0 with the acquisition but I've built a company with Auth0 as the auth provider and definitely would not take it back, saved us so much time and allowed us to focus more time on product market fit

In the end this is the whole goal of something like stripe, or even something like Courier. Let your engineers spend more time on PMF and building cool new features.

On the flip side the absolute mess of a headache my day job had during their recent what 3+ hour downtime, stopping our paying customers from accessing our perfectly working application that they pay for, because early days devs decided to use Auth0 instead of a battle tested rails auth lib has made us prioritize refactoring auth0 out of our system.

Feeling fine.

As far as I can tell, nothing has actually changed from the customer perspective. It's not like Auth0 is going away or becoming unusable. Okta didn't spend $6.5B to burn all of Auth0's customers.

Nervous as we’re on an /extremely/ cheap (almost free) legacy plan.

Why can't you switch vendors?

One aspect that's difficult to navigate is migrating the passwords and MFA secrets.

Don't know why you are getting downvoted. Asking a vendor how you can get this sensitive data out before you commit to them is a great idea.

That's not true at all. Most vendors, if you make them specify it in a contract, will provide you with a way to export the hashes and some vendors support importing users with existing hashes. Okta in particular will import existing hashes.

My company moved to Okta from a home grown solution in 6 months and I suspect after the work we put in place to facilitate that would allow us to move somewhere else in even less time as long as they supported importing hashes.

> if you make them specify it in a contract

Does this mean that anyone who didn’t know to do this is screwed?

Not necessarily. You just depend on their generosity.

There's a big gap between "never build yourself" and "letting other people handle it". Yes, you should have full control over your auth, run it yourself and keep the data - but you still should not build it yourself, you should use well-established solutions/libraries built by others instead of trying to figure out e.g. what's the proper way to salt passwords.

Reply on news.ycombinator.com