What Is Tokenization? | Hacker News Reader

What Is Tokenization?

basistheory.com

71 pointsmooreds31 comments

Bit of an annoying title, assumes tokenisation only means one thing. The toy compiler course I did explained that tokenisation was breaking a string into parts "tokens". Probably other meanings too I'm guessing

Yup, thought it was about text tokenization, i.e. for nlp/machine learning purposes.

Yeah, synonyms exist. If you get annoyed by technical terms having different meanings in different fields, you are going to have a frustrating ride through life.

For some time I though it was all about the security implications of strtok().

Yep, overloaded term. Anyone got a better idea for what to call it in the context of sensitive data within applications?

We call it PII (Personal Identifiable Information) at my workplace

Hashing, perhaps. It's probably what they're doing in background.

Blogspam. Vendor selling tokenisation says tokenisation is good.

How on earth would this stop security being expensive and difficult, as the intro seems to promise?

It's probably useful mostly to companies which want to avoid the security levels needed to comply with the Payment Card Industry Data Security Standard (PCI DSS). Consider a seller with a shopping cart program. One that outsources the collection of the payment data, and gives back a token. The token can be used for billing the customer, but the payor, payee, and maybe the shipping address are encapsulated in the token. So possession of the token only lets you submit new transactions that match on those items. The seller doesn't have to have PCI DSS compliance.

This is useful for systems which rebill the customer. They don't have to store the CVV, for which there are high security requirements. If an unauthorized party gets the token, they can cause rebilling, but not buy from others or, hopefully, change the shipping address. Also, in case of trouble, it's easy to invalidate tokens. People have to re-auth, but not get new credit cards.

It's a similar concept to OpenID, in some ways.

Can't say such things here. Apparently questioning OP's notivation and accusing of bias/spam/astroturfing is against HN rules. ¯\_(ツ)_/¯

Hiya. Doesn't bother me at all to have you question my motivation.

I'm not connected with this company at all, just ran across it and thought it was an interesting concept.

Sure, it's content marketing (they have a product to sell). But I found it educational.

It seems to me like this scheme is only a half-measure for protecting something, because if an attacker compromises your app and steals a token they can still use it like your app does, right?

Say you have wrapped a credit card number in a token. When a user buys something, you call the API to charge the CC token instead of a CC number. But if an attacker steals a token, they can make that same API call and charge a card to pay themselves.

I guess the mitigations tokens give you over bare CC numbers in that scenario are:

* The attacker probably needs your API key or similar to use along with the token, instead of having a bare CC they can easily use anywhere

* Better auditing of token use

* Tokens are revocable in case you discover a breach

Still, due to that risk I don't see how this "eliminates" compliance risk.

It should be coupled to strong mutual authentication between the client and server.

It’s generally not considered safe to expose the raw tokens to your users to use in your api. In fact if your using the system to try to avoid the headaches of PCI DSS, if you expose tokens as means of payment they’ll consider the token to be a card equivalent.

Instead the best practice would be to tokenize the data, accept it through one channel (an AttachCard api or whatever) and then return to your api consumer another opaque token to use for actions like authorization or clearing. From here there you can ensure that tokens are never reused etc.

This way if tokens leak they should be “inert” and no actions should be able to be taken with them.

> It’s generally not considered safe to expose the raw tokens

I think the article muddies the waters on this point because the the sentence defining tokens in the intro says they "are safe to expose".

Yeah, I can’t really speak for the article. I ran a large production tokenization system for a payments company. Most of my experience is within the narrow context of protecting credit card numbers.

In other contexts maybe you’re more accepting of public exposure. For example tokenization can be an effective GDPR mitigation, but i don’t think folks are particularly worried about reuse attacks when the information contains a name or email (I’m sure there are instances where they are concerned as well)

But at least when talking about tokenizing actionable information (credit card, bank account, whatever) implementors should be careful to keep them out of their public apis.

> But if an attacker steals a token, they can make that same API call and charge a card to pay themselves.

I can't speak for every payments product but in the one I work with tokenized cards are tied to one merchant. A compromised token can only be used to process payments to that merchant.

Same here for the many payment processors I’ve worked with in the past. Except for the one time the processor swore up and down that tokens were tied to a merchant when they weren’t. It took several weeks of pestering support with example curl commands to convince them to escalate to someone who knew how to run a curl command. Then the bug was fixed within an hour.

Credit card tokenization is a thing, and commonly used in retail. It leaves the last four digits of the number intact to allow searching for transactions from the POS; but other numbers are scrambled. Credit card data is otherwise stored in the POS systems tokenized. It takes all teeth out of credit card data for PCI requirements.

In this case, the tokenization is the measure itself.

> This company doesn’t want to go through the trouble of securing their employees’ data within their own system, so they use a tokenization platform for storing sensitive employee data

This looks like something useful.

I think the so called "Web3" movement would be more Interesting if it was focused of building such infrastructure.

Like NFT’s perhaps?

So, like, give unique IDs to things and then move them around?

The most useful kinds of tokenization will "anchor" the data to a specific platform, and allow you to perform common operations without slinging plaintext around.

So it's a token and an interface, where the minimum interface is "give me the plaintext back", but richer interfaces are possible and more useful.

So e.g. once you tokenize a card, you can perform billing operations on the token. The original CHD is "too liquid" and can be used anywhere in the financial system. If the token platform ensures only you can get paid with the token that can the limit the scope of badness.

You can also apply auditing and enforce extra access control / policy around operations or exchange of tokens for plaintext. In a complex system this can be better than letting sensitive information propagate to different data stores or business units. It also means you can build out the auditing & controls just once instead of needing controls in N systems.

It's also a decent tactic for "compliance engineering" in terms of being able to draw a box around which parts of the system you expect your auditor(s) to want to look at really closely. Things can rapidly get expensive when too much stuff comes into audit scope.

I love this description and the concept of “compliance engineering”!

an analogy i like is that if encryption is a locked box you're passing around, using a token is like passing around a safe deposit box key instead- the real secret is still secure at the bank, probably encrypted as if by a teller's key (not with your token-"key", which is not an encryption key, just a reference to the box, and you can make as many tokens to the same box as you want.. so that's a rough edge to the analogy)

but anyway, it's just passing references around, so i think the main reason it gets confusing is because it's discussed as an alternative to encryption, despite being kind of the opposite of encryption (passing around a token and protecting the vault, vs passing around ciphertext and protecting the key)

I thought they were talking about something more like canonicalizing and hashing the data.

generally tokens are arbitrary codes without any mathematical relationship to the data, so that a token being leaked doesn't reveal anything about the secret (you might want to invalidate leaked tokens tho)

Bingo! The underlying value is stored safely away from the systems passing around the reference - this makes it simpler and safer for the underlying systems to use the plaintext vs needing to share the private key with each system needing to touch the data!

We have been using a token vendor for PCI level 1 compliance and it's been most excellent.

I can't tell you how many times over the years I've seen tens of millions of rows of sensitive data(e.g. ssn) sitting in databases unencrypted. Software devs as an industry really needs to take this more seriously, but often businesses simply will not allocate the funds to do this right because of minimal risks to the corporation. At least with PCI the companies are forced to take card data security seriously.. but for instance nacha data, not nearly as much.

Absolutely the way to handle sensitive data in every app, so much so that I'm working on an open source version.

. Hey ledgerdev, I’m the VP of Product at Basis Theory and I’d love to hear more of your experiences with tokenization!

How are you ultimately using them internally? What’s the ultimate business value you’re seeing?

You’re open source project sounds really awesome as well, I’d love to jam on it with you sometime!

We do something like this with our customers' log files.

Before we send them from their system to ours, we pass all PII facts through a salted SHA256 wherein the salt is unique per trace session. This allows us to correlate sensitive PII across areas of our product without actually unmasking the private knowledge. John Doe will hash the same within the same error tracing session, but across tracing sessions they might as well be 2 entirely different people.

Reply on news.ycombinator.com