LanguageTool – FOSS Style and Grammar Checker for 25 Languages | Hacker News Reader

LanguageTool – FOSS Style and Grammar Checker for 25 Languages

languagetool.org

84 pointsjka24 comments

Open articleView on HN

I have recently rolled this out for our users.

We have found it to be very good, cheaper than Grammerly.

Works with most Google Workspace apps, unlike Grammerly.

I didn't realise it was open source until now

And support many more languages than Grammerly.

Love your ironic sense of humor -- intentionally misspelling Grammarly in a comment on a style and grammar checker. Too rich!

It is interesting that the fundamental parts of grammar checking are still driven by rules engines and GOFAI expert systems. Even the gramamarly core is written in common lisp as a GOFAI app. Most of the modern attention/transformer NLPs are used to supplement the front end tasks like dependency parsing.

I'm not sure I believe this represents state of the art in this area. I think they just have put enough hours into writing rules it does a passable job for the biggest languages it supports. If you get enough rules it can look like progress.

I looked at the engine for the Japanese parsing and it looks like they are parsing Japanese in fundamentally the same way they are English and it's has been known for around 20 years that is insufficient. Japanese requires tagging be done in order to tokenize it not the reverse as is done with English.

Take a look at spaCy if you're interested in what's possible nowadays, but it's not an end user product.

Edit: it's not quite that bad they're using GoSen which stopped being used only about 10 years ago.

Yes, I put in Japanese that was nonsense, and it passed with flying colors.

Yeah, I figured.

That's why there are almost no NLP tools on the market that support Asian languages, in particular Japanese, Korean, and somewhat Chinese (Chinese is its own thing because of the large number of engineers it has to throw at problems).

I'm part of the team that makes Zoho Writer's grammar checker[1] and we did go with a modern neural-net based NLP technique to detect grammatical errors. It had a huge win in terms of the number of errors it caught, but we could never tell the user what exactly is wrong with that sentence, because.. well, deep learning.

So we've taken a hybrid approach where a modern AI flags errors and then run it through a rule engine to try and figure out what exactly is wrong. This way we get the best of both worlds. That lack of explanation is also why Grammarly is sticking with GOFAI systems perhaps. But the future of this problem area is certainly AI augmented with rule engines.

[1] - https://www.zoho.com/writer/free-grammar-checker.html

But if the NLP is able to flag more errors than the rules engine, how could the rules engine explain the errors?

I'm not sure I understand what you're asking, but neural net-based NLP can't explain itself while the rules engines just use simple pattern matching techniques, so it can.

OP is saying that the rules engine can't catch a bunch of errors that the ML based NLP can. In other words the ML system has a bigger input domain than the rules engine. I am confused about how they plan to use the rules engine to explain errors that it won't be able to catch at all.

Oh, I expect that you'd use NN NLP to do accurate parsing/tagging/NER then write rules based on the output of that parsing.

If you look at https://explosion.ai/demos/displacy, you can see the high-level information you can extract from sentences nowadays, which better matches our understanding of how grammar rules work.

And that's an approach that does generalize to languages like Japanese.

Yes, that's currently the problem. I'll be that Grammarly is doing something similar itself.

Do you support Japanese or other East Asian languages?

Boy, I would love to see an API for the analytical tools.

It's great, I've been running the self-hosted version for a few months now. I run it using a container[0] on a $10/mo Digital Ocean VPS using Dokku[1], along with other services. The main downside is that it's a bit of a memory hog, possible because it's written in Java? Otherwise, I haven't had to mess with it after the initial set up. I like that I'm not sending everything I write to another 3rd party service I don't control.

As for the plugin, it definitely catches more issues than the stock browser spellchecks. The main issue I have with it (maybe someone can point me in the right direction) is that it always tries to autodetect the language. This is fine for longer texts, but often fails on shorter strings like headlines. This leaves my words highlighted red because it thinks I'm writing bad German, or Swedish (which is fair). I haven't been able to figure out how to force it to only use US English.

[0]: https://hub.docker.com/r/erikvl87/languagetool [1]: https://dokku.com/

I don't understand, why are you running it on a separate server ? You can just install it on your computer and it will likely be faster and work also when you don't have internet (and free). IDK about other OSes but on archlinux it's just a package to install.

github: https://github.com/languagetool-org/languagetool

languages: https://dev.languagetool.org/languages

Improving LanguageTool: https://languagetool.org/dev

I find it more interesting the lack of comments than anything else - I'm not sure this software is a good idea, but regardless, nobody uses long form messaging anymore? I.e. if you write anything, you likely already have a solution you are content with, or you use twitter or YouTube comments and largely couldn't care less...

I wonder how many varieties of various languages it will recognize, and how many it will flag as erroneous. For example, is Catalan a variety of Spanish, a language, or erroneous Castilian? How about Gascon vis-à-vis French? I also wonder whether AAVE will be recognized on a level with Indian English, for example.

I installed LanguageTool as a browser add-on in Firefox and took a look at its settings. It does have a "Language Variants" setting, where one of the options is "Interpret Catalan as ...", and the options are "Catalan" or "Catalan (Valencian)"

Can anybody tell me what these tools are used for? Where’s the need? Who are the people using them? Thanks.

I use a similar tool (Grammarly) for all business related communication, as a non-native English speaker, to avoid simple grammar mistakes. For example, I often make the were/where, it's/its typos because, due to my native language, I think of words phonetically while I type them out, even if English isn't. It also highlights very long sentences, which reminds me to rephrase and break them up for legibility.

I wish it would have been native tool, not Java. It would be much lighter and easier to use then.

I'm using LT every day, can't complain.

One of my “must have” tools.

Reply on news.ycombinator.com