Show HN: Parse no-protocol URL, no-domain URI and email addresses etc from text | Hacker News Reader

Show HN: Parse no-protocol URL, no-domain URI and email addresses etc from text

github.com

74 pointsandrew-kang-g30 comments

To the OP: I would suggest normalising the protocol. Your examples show that you retain the case from the raw data:

    "protocol": "HTTP"
    "protocol": "https"

As it stands, users will end up normalising by themselves and you can save them the effort. FWIW this part is typically called the scheme, not protocol (e.g. Perl has `URI::Find::Schemeless` which performs a similar task)

    URI = scheme:[//authority]path[?query][#fragment]

See https://tools.ietf.org/html/rfc3986#section-3

To that end, you might also want to consider adding support for fragments. :)

Well in JS, it's called protocol https://developer.mozilla.org/en-US/docs/Web/API/URL/protoco...

Node calls it a protocol scheme. https://nodejs.org/dist/latest-v12.x/docs/api/url.html#url_u...

Congratulations! You're all right!

Interesting, thanks. I have virtually zero JS experience so didn't know that, and were I to try and pick it up this would've been a surprise to me - I naively assumed it would use the same terminology as the RFC and the libraries in languages with which I am familiar (rust, golang, python, perl). A quick look for the URI data type in a handful of other languages shows they also use "scheme". JS appears to be the outlier here.

In particular, the scheme doesn't necessarily refer to a protocol (e.g. application-specific schemes like `chrome:` that don't involve network access at all).

I can't figure out what this tool does, even after spending several minutes on the linked github page, and following the npmjs link on that page.

But clearly HN likes it (it's number two...); can anyone explain it to me?

Looks like it's able to detect URL in "natural language" texts. Like for example "this is alinkhttps://wikipedia.org". Check a live demo at https://jsfiddle.net/AndrewKang/xtfjn8g3/

I think the "no-protocol" in the post message suggests the http(s) part is unnecessary. Makes sense - it's a pretty easy problem when the scheme is there

I’m just wondering if it can parse things like « pls-no-spam at google’s email service », which a human can easily parse to pls-no-spam@gmail.com. I suppose POS tagging with some heuristics can get it working for a few cases.

And if it can, I feel the most users for this would be email harvesting spam bots. Which would not be super nice. Although in ideal and less common scenarios it could actually be useful for IR purposes.

Not sure why people bother to use the << spam at google's email >> approach when they can use an email forwarding service that lets you turn email addresses off, like https://idbloc.co (disclaimer, I built this)

I think these are a great idea. The reason that people use that method is because it is free, and the suggested alternative costs money.

I typically use throw away email addresses instead since they're free. If I actually want to use the service/account some day I'll sign up properly.

I expect someday the primary email providers will generate aliases on demand.

Fastmail lets you convert myusername@fastmail.com to whateverwebsiteorservice@myusername.fastmail.com

It's not a literal alias, but it works out fine.

One of these days I'm going to propose the Thunderbird + K-9 mail addons that say 'set the sent from email address to be equal to the sent to email address', that way if service@myusername.fastmail.com is the recipient, it will be the sender to any replies, rather than the MUA default replying with myusername@fastmail.com (but that's a separate discussion)

I didn't know that was possible, really neat!

> I typically use throw away email addresses instead since they're free. If I actually want to use the service/account some day I'll sign up properly.

The example above is about giving a direct line to be contacted through email on a personal site or blog, heck even a professional one.

> I expect someday the primary email providers will generate aliases on demand.

I know at least ProtonMail does it. But also theres the email#tag@gmail.com trick that works nicely enough.

That example is quite common within academic context: most of the researchers and professors I know, who have a university homepage, use this.

I guess its because they do want legitimate people to reach them easily (students, collaborators etc) and if spams do go through — well they'd probably ignore it, just like they ignore other emails.

Because I don't want to shutdown the address later: people may try to use it to reach me.

I use anti spam addresses to register to new services.

Not sure why people bother using idbloc.co instead of burnermail.io

Sanitising user input seems a good use-case too - I'd like it built in to my email client, for example.

(More likely to be useful than Outlook's typically wildly incorrect guesses at when someone's suggesting a meeting/calendar event!)

It fails with ipv6 URLs like http://[::1]:3001/

https://github.com/jakeogh/hostname-validate/blob/master/hos...

eventually it will get integrated with: https://github.com/jakeogh/kcl/blob/master/kcl/htmlops.py

Hacker News also fails to parse that. Give it another decade or two; we'll figure out how URLs work.

Not the first time newly built stuff can't handle IPv6... quite sad.

Why "pattern-dreamer"?

It looks interesting, but the readme nor the title doesn't make clear what is the project about. Would someone care to explain?

This project attempts to parse url's and email addresses from text, even when the protocol is missing or they're mangled in some other fashion (i.e. google.com/test-url?id=2 gets parsed and identified as valid URL, but invalid.nontld/test not. Or "test@example.euSample Text" get's parsed as test@example.eu)

So it needs to be updated every time new tld is added? That seems... suboptimal. Also it will not work for intranet addresses (valid.insidemylan/test).

Suggestion: change parse to extract

That would be even more clear.

FWIW, Apple’s implementation of this is called “NSDataDetector”.

Thanks for all the comments. Now ip-v6 and intranet cases are available to be extracted from v 1.6.

Reply on news.ycombinator.com