Microsoft.Recognizers.Text: numbers, units, and date/time in multiple languages | Hacker News Reader

Microsoft.Recognizers.Text: numbers, units, and date/time in multiple languages

github.com

111 pointsnailer31 comments

Open articleView on HN

It's way bigger that that: covers not just JS but Java, c# and more, and many languages

https://github.com/microsoft/Recognizers-Text

Ok, we've changed the URL to that from https://github.com/microsoft/Recognizers-Text/tree/master/Ja.... Thanks!

The link should be updated to this.

It could do with a link early on to some docs about what a recogniser is.

Used this for a quick proof of concept I was putting together where I needed to parse the output of an OCRed date and number (price) string (with known country). Seems to work very nicely on the sample size of 3 I tried with - so I can't speak to how well it works on a more diverse sample set - but it's great to have something like this available for free.

In my second job we maintained our own, also Regex based, logic for this and it was a nightmare so having a library to do it is quite the timesaver!

Can someone be awesome and ELI5 this? Not sure if I'm dumb or the people behind this have a very focused target audience in mind.

From what I understood, is a library that you can sends strings with "twenty thousand ninety eight" on it, and it tells you there is the number 20098 there.

More practically: 'last October' gives you a date range.

https://github.com/microsoft/Recognizers-Text/tree/master/.N...

I think these examples explain the functionality very well.

+1 for this. I have some sense reading the page but would love an ELI5.

It seems like its meant for standardizing/parsing freeform input, which I guess could be useful for things like chatbots?

For example you send the user localized text, "Did you receive your package?" then parse the yes/no and continue your decision tree with something like https://www.nuget.org/packages/Microsoft.Recognizers.Text.Ch...

It's a part of the underpinnings of LUIS, Microsoft's Azure service for language understanding, and indeed part of building things like chatbots.

An interesting comparison is that Microsoft.Recognizers.Text is a cross-platform cousin to JS library compromise/one: https://compromise.cool/

Are these done via NER or old school heuristics?

It's all Regex.

> The Patterns folder contains all the regular expressions that fulfill the recognizers logic. It's divided by supported language.

jesus! https://github.com/microsoft/Recognizers-Text/blob/master/Ja...

Worth noting that these are machine-generated from a more verbose and clear codebase.

It looks like most PR's are for RE definitions in yaml files. Is there something that generates the yaml?

Yep. Regular expressions are great, but they rapidly fall over into write-only code for nontrivial applications.

https://github.com/microsoft/Recognizers-Text/blob/master/Pa...

I'm one of the maintainers of this project. Comments, suggestions, or contributions are very welcome! Specially regarding expansion to support new languages/cultures.

We're looking at multiple ways to improve the project.

Hey thanks! I'm the submitter and am using the the project in a real app - I appreciate the comprehensiveness of the project!

One piece of feedback is: the documentation https://github.com/Microsoft/Recognizers-Text/tree/master/Ja... doesn't seem to cover the the various terms used in the results. Eg from my code:

      const recognizedDateTimes = recognizeDateTime(filterValue, "English");

      // TODO: clarify what is distinct between each 'recognizedDateTime'
      // Seems like these are the 'recognisable items' that were found but I can't find docs.
      const firstDateFound = recognizedDateTimes[0];

      // TODO: clarify what is a 'resolution'
      // Seems like this is each possible value for each recognisable item but I can't find docs.  
      // eg if the user enters 'october'
      // values[0] is in the past
      // values[1] is in the future
      // we always want the past, hence picking 0.
      const firstResolution = firstDateFound?.resolution?.values[0] || null;

A simple explainer in the README would be great.

Thanks for the feedback. Could you please open an issue on GitHub? Thanks.

Sure and done. https://github.com/microsoft/Recognizers-Text/issues/3064

This is awesome! A nice replacement for the Duckling service if you need fast regex based NER in your code.

Finally! Maintaining Duckling as a native Python library isn’t trivial unfortunately

Now if only Microsoft would put some of their recognizer smarts into Excel so that ZIP codes like "02201" don't get converted into "2,201".

You already can do that. Albeit it doesn’t detect it as a zip code specifically, but you can format the cells so that values that look like numbers aren’t automatically converted into numbers.

Excel does have its quirks (CSV handling is one of my pet peeves) but it does also have a surprising amount of sophisticated adjustable logic that is often hidden in plain sight.

Definitely. I wish there was a global/permanent setting to not strip leading zeros. I imagine the design decision might have been made decades ago for Lotus 1-2-3 compatibility or something, but it's almost certainly cost much more time and mistakes than it has saved.

This might be the setting you're looking for? https://www.howtogeek.com/816620/microsoft-is-finally-fixing...

That would break the business logic of millions of small businesses globally who rely on write-only excel spreadsheets to encode their business logic.

There's no Ruby yet.

Reply on news.ycombinator.com