I failed to make LinkedIn fix their broken international domain URL parser | Hacker News Reader

I failed to make LinkedIn fix their broken international domain URL parser

helmstedt.dk

222 pointshelstegt89 comments

I'm not surprised the author can't convince LinkedIn to change anything. Had a similar experience with Tinder where Paying in their web application using a danish credit card failed because their cc service returned "visadankort", which had to be changed to "visa" through the developer console for the payment to go through. Got tired of that and filed a bug. Oh the battle it took to actually convince them that an entire country is unable to pay in the web app using their de facto credit card.

I'd have to assume "supporting Danish credit cards" is lower on their list of revenue-impacting issues than you expect.

If Denmark were a US state, it wouldn't even be in the top 20 in terms of population. If you assume that Denmark has lower usage of Tinder than US, CA, or UK, and then you assume that not all Danish people are paying with Danish credit cards (many of them would be using Play Store subscriptions, which seems like it would bypass the issue you're describing), the affected population is probably really small.

Then you also have to consider that most of Tinder's paying users are heterosexual men, and they know that they can screw those men over without losing their business because Tinder is an irreplaceable marketplace to find women.

With all those assumptions, Tinder isn't being as stupid as they sound, but rather coldly rational.

The question is what the cost of that fix would be. Once the ticket reaches the developer it's probably quickly done and pays for itself with a handful registrations. However getting from support to a developer is the complex part. (Support has to understand the issue and then it has to be routed to the correct team ...)

I completely understand why you see this as a 30-min fix, and therefore they should just knock it out.

That's not how they would likely see it. If their devs are 100% utilized (which I'm sure they are), then fixing any issue -- even a 30-min one -- is going to mean that a different issue is delayed.

The question becomes this: at what point is fixing this issue more valuable to the company than any other issue the same dev could work on?

And the answer is possibly "never", depending on what high-impact bugs, features, or technical debt they have on their priority list.

You rationalize dysfunction into a strategy.

It's not like they are doing Moon landing there 24/7. It's a simple bug with direct impact on the revenue. They should take a half hour break and fix it.

I think you vastly oversimplify how easy this fix is likely to be. For all we know this may mean touching code in multiple repos, each with hard-coded lists of which cards are supported for some case or another. At least that is what it would mean to change something like this where I am now.

Oh wow, editing a hard coded list. Does one need a chisel for that?

That is not how things work in the real world. Sure, the fix itself might take 30 minutes. But then you have to validate it. And run a canary. And crosscheck your revenue numbers to make sure it didn't have an adverse affect elsewhere.

That's a lot of work for something that will probably mean very little extra revenue.

I assume they already have the quality assurance pipeline in place at Tinder and don't need to reinvent it from scratch for every bug report.

Of course they do. But it still takes time, effort, and resources to run through the QA pipeline.

https://xkcd.com/277/

Depending on the amount of technical debt, it could very well be a number of hard coded lists in a number of projects that may be under multiple teams and which may or may not include first party code. It’s not that fixes like this should be hard, it’s that sometimes they are hard despite themselves, for organizational reasons.

Edit: just for clarity, I’d love to fix this bug myself. I also have a million other easy fixes that I’d like to do and I have a manager to help me prioritize them.

It's not just about time, but focus as well.

Other tasks might also be more critical to revenue generation.

At the end of the day, there are always going to be some issues left by the wayside so I think you really have to pick your battles.

It's a 100% reproducible, mentally unchallenging, revenue impacting bug, which is frankly a rare combination. There is no excuse in not fixing it, unless you imply the thousands of Tinder devs are in core meltdown situation all the time.

Maybe they ran out those property list updating chisels.

How is it dysfunction to keep your devs focused on the tasks you've already identified as the most important?

I can't imagine how producing a PR for this with test cases, getting it reviewed, then shepherding through a typical QA/Staging/Prod pipeline would take less then a day for any code base of reasonable complexity.

Perhaps it wouldn't literally take 8 hours, but the focus shift of working on this would likely keep a dev from making much progress on other issues that day.

> How is it dysfunction to keep your devs focused on the tasks you've already identified as the most important?

Because software is never that cut and dried. There's what supporters think is important, what PMs think is important, what front-line engineers think is important and what leadership thinks is important.

These priorities are frequently different and their actual impact is quite often disjoint from their position in the organization.

Minimizing turn upkeep (over head) is worthwhile. Ignoring it is ludicrous.

They didn't claim it wouldn't impact revenue. Just that there are probably bigger issues that impact more revenue that should be worked on first from a purely ROI point of view.

If and when this issue has a higher ROI than anything else, it should be worked on. The flow of new issues with higher ROI may prevent that from ever happening.

I think every developer can easily name 5+ minor bugs in a product they're working on that they know exist and know roughly how to fix, but may never get around to fixing. Or maybe I'm the weird or unlucky one, but that's been true of every production software I've ever worked on.

What can be argued strongly, I think, is that ROI calculation should take into account secondary effects such as the app's reputation in the country or the appearance of discriminating against a country.

Hopefully there is some good logic and systems in place to actually calculate ROI, and it's not just a PM chasing trends and pushing out new features based on their gut.

I wouldn't say a user-facing bug that prevents collection of revenue is minor.

It's also a little alarming this isn't caught via QA or automated tests. Did this ever work? What other stuff is slipping through? Are we sure the report even escaped support and got triaged?

Financial systems are an absolute nightmare, so it doesn't surprise me that some oddball detail in a small country eludes tests. Pretty much every processor, network, and country has a million of those oddball details, and they change regularly.

At work, the platform I have to deal with has dozens of fields on every request and response that are specific to Brazil!

There is a secondary effect here besides ROI: image. In my mind I now link LinkedIn to "not even able to handle a creditcard payment". Get enough of those and ROI gets impacted.

This one isn't LinkedIn.

> Had a similar experience with Tinder

But I agree with your overall point.

Fixing an issue around being unable to pay with a credit card is likely anything but a 30 min issue, unless there is some triviality happening, like a config issue. But even then you need to add tests for this. A day or two is the absolute minimum and that would be if this is just a config issue. If it's not, then think of weeks.

It should be fairly trivial.

From the issue as described, the value "visa" worked, but "visadankort" didn't. Denmark's earlier, national debit card system is called DanKort. Nowadays, almost all DanKort cards are issued through Visa, so they are "Visa DanKort" -- within Denmark, there's the option to bypass some sort of fee for a lower cost of processing.

Presumably the value "visadankort" is there to distinguish these cards. However, they can be processed exactly as normal Visa cards.

This is not unique to Denmark, there are a list of card prefixes like this (see 4571 for DanKort[1]), but perhaps the different processing fees in unusual.

[1] https://www.stevemorse.org/ssn/List_of_Bank_Identification_N...

> That's not how they would likely see it. If their devs are 100% utilized (which I'm sure they are), then fixing any issue -- even a 30-min one -- is going to mean that a different issue is delayed.

This has not been my experience.

If you work at a company where it's possible, I strongly suggest shadowing a supporter for a day. You'll leave with a 5 page long rage-list of issues that weren't adequately captured (and would take you a few minutes to knock out) and a new relationship established that will help unblock that bottleneck of information flow.

IME, this is in part because supporters aren't as familiar with the technical aspects and don't know how to frame the issues they encounter in a way that would get traction with either PMs or engineers. As such are not well prioritized relative to issues that are fleshed out technically. I've worked at a company where supporters were spending huge amounts of time resolving issues that I knocked out in 5 minutes after shadowing and materially improved their operational efficiency.

In a word, what's missing is a culture of ownership.

I think you mean "customer support" when you say "supporter". If not, please disregard.

Tinder doesn't offer much customer support. They're definitely not spending a substantial amount of time on issues that affect the subset of users that use Danish cards, don't pay through the app, and have one of the cards that has this issue. If you told me the number of affected people was under 5,000, I would believe you.

Yep, the place I used to work referred to "customer support" as "supporters." I assumed the term was used more broadly than it appears to be, my bad.

I'm curious who handles those emails. I'm putting myself in the position of a software engineer at Tinder, I personally would have gotten care-mad enough to fix it. I go out of my way to get to know customer support folks where I work so I can get better insight into how people are actually using the product and where the pain points lie.

"features" is the wildcard there.

People who have already demonstrated they're trying to give you money is a relatively known payoff when that bug is fixed. "New features" that may never see the light of day, or may actually have a negative revenue impact... those are somehow sexier and often manage to take priority over mundane issues like... hundreds of peoples' payments failing over a month.

This is absurd. The return on investment for a bug like this must be extraordinarily high. The OP can fix it themselves by changing a string in the developer console.

A permanent new revenue stream from an entire country for the price of 1/16 of one employee's workday.

What kind of fires are burning at Tinder for that to not be a no brainer?

Side note: if there are any devs here from Tinder, I recommend you go fix this on your own initiative and get yourself some quantifiable money made for your employer for your next performance review.

1. It's not an entire country. It's people who use a Danish card and can't/won't subscribe through the Play Store. That's probably a tiny number of people.

2. Fires are almost always burning when you're supporting an app with millions of users expecting instant functionality, including instant messaging. They're also constantly working against bots and spammers.

It's not "coldly rational" if your fix is simple enough that can be fixed on the browser console and charging CCs directly avoid the 30% fee of the store (though Tinder does seem to tack that to the app store price)

lower on their list of revenue-impacting issues

Some companies do what's right. Some only prioritize based on revenue maximization.

Locking out an entire country is clearly wrong. Only prioritizing for revenue maximization is only ambiguously right.

It's not like fixing a bug is going to bankrupt Tinder.

1. This isn't locking out an entire country.

2. Tinder is constantly at war with bots and scammers. You have no way to know that fixing a CC issue for a tiny fraction of users is more ethical than their other bugs.

Companies spend thousands on things that have dubious impact on the bottom line. This is something that has a direct and measurable impact. If the solution for the parent commenter was as simple as changing something in developer console, then the change for tinder shouldn't take more than a few days or a week.

That's like using poor ROI to argue against curb cuts.

There are intangible, non-quantifiable benefits to doing things better. Our inability to concoct a price is not a defensible reason to let entropy and chaos win. Sometimes Freedom Markets™ just need some TLC.

This kind of hyper-optimization misses the forest for the trees. You need to have some love for your product and if you do, you do not let issues like this fester. Your bean counters can’t capture the full complexity of the world. They can’t capture the general sense of “shittiness” that users feel when your product fails in stupid ways and you don’t fix it. They can’t appreciate how that feeling snowballs into the success of a competitor five or ten years down the line.

I dunno it’s more about the principle of not fixing the product. I think software teams would benefit greatly by setting principles that they follow instead of justifying everything by revenue.

Not saying you support the revenue-only model btw :)

If it helps, every foreigner visiting Denmark has a similar experience in reverse when we try to pay with our Visa cards :P

As a funny side note, I cannot load https://xn--coronaprver-ngb.dk/ in Brave, which also seems to be confused by Internationalised domain names.

This is what I see:

> This server could not prove that it is xn--coronaprver-ngb.dk; its security certificate is from *.coronaprover.dk. This may be caused by a misconfiguration or an attacker intercepting your connection.

Edit: Hah! And HN is also guilty of being confused by this. My comment now says "xn-coronoprver-ngb" when viewed from the outside, but my comment field correctly says "coronaprøver" when editing the comment...

What’s the problem? That certificate is only valid for coronaprover.dk, not coronaprøver.dk or xn--coronaprver-ngb.dk.

The problem is that Brave fails when you navigate to "coronaprøver.dk" while both Chrome and Firefox works when you go there. Although Firefox and Chrome redirects to two different sites, they don't present a certificate error, while Brave does.

That's not my experience. Going to https://xn--coronaprver-ngb.dk/ in Firefox presents the invalid certificate, but going to http://xn--coronaprver-ngb.dk/ redirects me to https://www.coronaprover.dk/. It seems Brave always tries with https, and thus it fails. Not that it is confused by IDN.

The link provided was https:// so it would be a serious security issue if it downgraded to http.

It is the website that is broken (or at least does not support that link).

I used coronaprøver.dk to illustrate the point. The more authoritative URL is coronaprover.dk and coronaprøver.dk is registered to prevent domain squatting/phishing attempts. Although they probably should have made a solution without certificate errors.

I just successfully navigated to that site in Brave. The only problem was that I had to accept the invalid certificate.

My Firefox complains about the certificate too.

Yes, because the certificate is not valid for the site they linked to - it's only valid for the ASCII-named site. Everything seems to be working as intended here on the browser side.

Same in Chromium.

This is actually one of the lazy ways to deal with https://en.wikipedia.org/wiki/IDN_homograph_attack

The better way would probably be to use chromes list of when to allow IDN and when to convert to punycode: https://chromium.googlesource.com/chromium/src/+/master/docs...

Working with internationalized URLs (IRIs) is a Cursed Problem.

Here is the IRI for the Wikipedia page on semiconductors. When you click on it, it should show you a page with a picture of a silicon wafer on the left.

https://ar.wikipedia.org/wiki/شبه_موصل

Now try copy/pasting that link text into different web apps and see how they respond. HN seems to handle it correctly. Twitter doesn’t, it considers شبه_موصل to be just some extra text after the URL.

That's really a problem of picking up a url/uri/iri in text without delimeters. You see the same thing when there's 'uncommon' ascii characters in urls, like a trailing period, or commas or exclamation points. The proper thing to do is to enclose links in <>, as was done in well-formed email, but that wasn't super consistent either, and it hasn't transitioned to the world conciousness.

I don’t want to rely on users figuring out how to use delimiters. If the URIs with trailing periods break, so be it, people can use <>, but don’t make people use <> just because the URL isn’t English.

So I just tried pasting this into the editor for the issue tracker I work on [0]. When I traverse the URL from left to right with the keyboard cursor it goes as "normal" up to the the last / then jumps to the end, then traverses "backwards" then jumps back to the end. Is this right or do we have a bug? Mixing LTR and RTL is super confusing.

Does recognize it as a link just fine though.

0: https://kitemaker.co

That sounds correct to me, but there's no canonical answer. The question is, do you want to change keyboard cursor context from "next character" mid-sentence from LTR to RTL? If you do that, the cursor will get "stuck" at the word boundary where next and previous mean the same thing (left is both "prev" for LTR and "next" for RTL - how should the computer interpret that?)

At some point Linkedin changed their site from something that at least worked to some javascript that i can actually watch loading and assembling the page because it's so slow.

And you expect ... support for internationalized domain names?

When this happens is it literally just companies trying to save money so they only need to hire JS developers?

Or are static sites not webscale anymore? The web framework I use for my personal stuff just takes templates in and fills it with whatever a request requires, and it's definitely easier than dealing with JS wat-isms.

I think it’s more related to “tribes” than anything else. A great designer that can do passable js is going to be more valuable to a company on the front end, and a great dev that does quality work that will last for 5 years without refactoring is more valuable to a company on the backend, while a great “full stack” engineer is more valuable to the company helping multiple teams in a DevOps role. Not to mention that those folks will naturally gravitate towards those distinct roles.

Over time, the frontend design types will want more control over what they can and can’t do userland, and nobody else gives a damn because it doesn’t affect their day-to-day and in many cases makes their codebase cleaner / more “pure” because UI concerns get really messy for a variety of reasons.

It also generally feels safest to a company to put junior programmers on the frontend where even if they mess something up, at least the whole underlying system won’t be hosed. That allows juniors and mids to build chops more rapidly.

Anyway that’s all just my anecdata / speculation.

Pretty sure it's just to combat scraping

... or combat people, you know, actually checking their linkedin accounts once in a while?

I'm not actively looking for a job. I used to check linkedin randomly like twice per week to see what happens. After their new slow motion interface, that changed to once every 3 months. Sorry, recruiters with incredible opportunities.

I suspect each component belongs to a separate team (each with its own backend infrastructure & load times).

Some people here are claiming that it's just not worth for big companies to invest in such "niche" bugs for "small markets".

But I think it's more than that.

I think many US-based companies seriously underestimate difficult issues such as localisation or fail to understand phenomena such as multilingualism. I think this creates a lot of problems for international markets, but somehow these problems are probably not very visible.

For example, why is it that I can't see reviews in multiple languages on the Play Store? This would be valuable information both for customers as well as for developers who speak multiple languages.

+1 for leading me to https://en.wikipedia.org/wiki/Punycode, I had no idea this monstrosity exists :)

I have a few such domains registered for (valid!) vanity domains and emails.

Amount of people using them? Slim. Amount of email-clients confused? Depressing.

> tried e-mailing their security e-mail (no reply) No wonder, it's not a security issue. At least not yet, have you tried injecting new-lines?

Totally agree. It was just the only e-mail I could find and I was pessimistic about having the issue escalated through customer support. I think Vegard tried his best to help me, though. Once I got the problem escalated, I naively thought they were actually going to fix it.

"punycode" does seem to break a few things in userland, but obviously should exist for the web to be more useable in more languages.

I've delved into the subject a bit and as someone else has mentioned, "homograph attacks" are a thing.

Looking deeper, it seems each TLD has a set of dictionaries and rules for what is an acceptable combination of characters, and what isn't (to avoid homograph attacks)- and as I understand it, the solution is on a per TLD basis, as in different dictionaries and rules per TLD.

For 3rd parties like LinkedIn, I could understand why they may be averse to embracing IDN domains but for a company of their size and reach, they really should be looking after their wider user base.

For working with IDN domains, I've felt the easiest way to store/process them is to store them as their ASCII value and convert for human consumption.

Tried hunting for a C library that deals with the dictionary issues and was left not entirely sure whether libidn supports them.

Why worry about the dictionaries? No legitimate user will accidentally enter a Greek Alpha rather than an A, for example. If a malicious user does this it won't resolve anyway because the domain is banned by the rules and can't be registered.

Yeah, perhaps less important for LinkedIn but more pertinent for other parties like domain registrars and TLS cert issuers- apparently the 'rules' ala the registry's DNS is an evolving thing.

https://www.theregister.com/2020/03/04/homograph_attacks_sti...

Ignoring the specific characters without a local frame of reference would require at least a DNS lookup.

This kind of tricks is used by sending mail to users with legitimate looking links, which can be used for phishing.

Am I misunderstanding or are you looking for Unicode normalization?

http://site.icu-project.org/design/normalization/custom

I maintained a domain database and normalised to ASCII with libidn, sometimes the input data was not from zone files and preferably would've been able to double check the characters used in a potential domain to ensure it's something that's registerable, without any network required. That was my motivation for looking into the topic originally.

I had a similar problem on LinkedIn when I tried to share an SDK I was working on. It's called Compound.js. The post editor thought that "Compound.js" is a URL, which 404ed, breaking the post preview, which was supposed to link to the actual URL which was later in the post body.

I too would try hard to not having to rewrite url parser

I bet things can get batshit crazy and introduce some vulns

Well it would be nice it it were fixed, but aren't people in Denmark wary of using the three special characters æ, ø and å for this reason?

IIRC it's normal to have borsen.dk instead of børsen.dk, Denmark's equivalent of the FT. And probably a few others as well, I'd expect any website connected to Århus to use some other similar spelling.

Also LinkedIn doesn't seem very easy to deal with in general. I've been trying to find a way to categorize my contacts since forever (personal/business/school), but there doesn't seem to be a way. All the boards where people write have some rep pasting in a standard non-answer.

They shouldn't have to be though; internationalized domain names have been a standard for a decade now. Browsers, DNS servers and web servers should be on top of that.

I mean sure, 10+ years ago they would have used plain ascii domain names, but nowadays they should be free to user their own language and script. At least two thirds of the world population and internet users does not use latin script.

Everyone uses Latin script because too much depends on it. Even "https" and ".dk".

Localized scripts are exclusionary to everyone else who has no hope of typing them or even identifying them in a unicode table to copy and paste.

A funny situation happens in China where even the most computer illiterate old person can type latin letters on a PC with enough patience just by matching appearance to their keyboard, but many old people are utterly blocked from typing Chinese characters because that requires learning a special input method which is almost a whole nother language itself. Before smartphones, they had to write them with a stylus on a writing tablet to get around this problem.

Half the English alphabet is excluded from domain names too. When are we getting capital letters? Let's just tolerate never having them and not worry, same as those other languages' odd letters. ExpertSexChange.com and PenisLand.net could use them.

> Localized scripts are exclusionary to everyone else

It's the other way around, not supporting other letters is excluding everyone else because essentially only English doesn't have diacritics.

"Año" and "ano" are not the same word, degrading to ascii only is not an acceptable solution. Imagine for example that the letters 'd', 'q' and 'w', weren't availabe in domain names for some reason and you had to use alternatives. That's how it feels.

By exclusionary, I mean it excludes people, not words.

But if ñ is so important, then so are capital letters because "Olive" is a person and "olive" is a fruit. Imagine typing English without capital letters.

Yep, it is very uncommon to see IDNs used in Denmark, even though æøå are very common in words. As you say, they’re usually transposed to avoid dealing with IDN pain.

It’s bad enough in web browsers, if you try to use IDNs in e-mail, it’s a complete mess.

This is the problem I had in mind when building https://chota.link. I have seen too many broken previews. This little app lets you create short link and specify metatags. Try this short link on LinkedIn: https://chota.link/jVZ2Hd. I tested it and seems to be working.

Any reason on the name? Chota is a despective way to say "Penis" in South America.

Interesting, in (southern?) Brazil it refers to the female genital.

In Hindi, Chota means short...

In Hindi, Chota means short. Apologies to South Americans.

I remember when I tried to buy a DNA test at 23andme, and they failed to parse my email that was in a form xxx@yyy.zzz.org . They couldn't, and their support suggested using a hotmail address. That's where I got really suspicious of their overall competence, not only IT, and found another DNA test provider.

There are also issues with their preview image cache, it's been broken for at least three years already but they still haven't fixed it... Once your url has the wrong image, good luck updating it, you might as well redirect everybody to a new url.

I found this kind of intervention positive - this seems an effective way to prevent multiple phishing attempts & misinformation.

The problem is, LinkedIn thinks they don’t have customer support or bug reporting, when really they just delegated it to social media.

Reply on news.ycombinator.com