Hacker News Reader

Comment by tryauuum | Hacker News Reader

Unrelated story about Russian language.

The first letter of russian alphabet is А, the last one is Я. So it's natural to try to match russian words with '[А-Яа-я]+'. But this is a recipe for disaster, this regexp doesn't match words with 'Ё' in them like "Артём".

This is due to the fact that regexp ranges work on byte values. All letters of russian language have neatly ordered byte values, except for the Ё.

View on HN

English is probably the only commonly spoken language where naïve char range matching kind of sort of works. I say ”kind of sort of” because [a-zA-Z] trivially fails to match all words in many English texts that haven’t been lossily compressed to ASCII, including this comment.

It is practically always wrong to match on [a-z] unless you’re parsing a computer language whose spec guarantees that it works.

I always wanted to know, how easy is it to type naïve on a common western keyboard?

Do you have to press some obscure keyboard shortcut?

"i or similar works. ^i, `i, 'i, etc. for the others.

how easy is it to type naïve on a common western keyboard?

In macOS, you can either use Command-u (for "umlat") followed by i, or hold down the i key for a second and press 2 to select the ï from the pop-up menu.

> Command-u

option-u (aka alt-u).

Generally speaking, command is for application-level or os-level commands, control is for text edition, and alt is for alternate characters (all can be shifted and command "overrides" the rest).

You're right, it's Option-u. Most of the key labels on my MacBook have long since been scratched away.

This has happened with every single Apple keyboard I've ever used. I suspect it's my fault, since I'm a key pounder, having learned to type on an IBM Selectric typewriter.

On Ubuntu, I use xmodmap to turn Print Screen into a Compose key. Then it's: <compose>"i

https://en.m.wikipedia.org/wiki/Compose_key

I use the Macintosh keyboard map in Linux. So I do <right alt>+e to ‘, <right alt>+n to ~.

On a Swedish keyboard, there's a dead key for ¨, so you press that followed by i to get ï.

It's not very clear why the Swedish keyboard has that key, since ä and ö each have their own keys. The layout has other quirks as well, such as keys for §, ½ and the useless "currency sign", ¤.

Yes! I think the "mine" character should be switched for the dollar sign.

BTW, the dead key could be from german, for writing their U:s.

Mac os quietly removed the paragraph sign for me when switching keyboards to non-apple ones if you used a non-standard layout that had moved the key they use to detect what kind of keyboard you are using.

It wouldn't have been much of a bother, had I not used the key as my Emacs leader...

I'm Windows-based and wanted a keyboard layout that will allow me typing easily Polish and French at the same time, without switching keyboard layouts (PL == US+AltGr for accents; while FR layout is insane, because apart from being AZERTY, all special chars are in different places, and you need a Shift to type numbers; and the way to type accents is also special).

I found "Polish international" [1] layout which honestly can be perfect for many people. It's optimized to be compatible with regular Polish keyboard (hence with US keyboard too), and maybe not the fastest if you type a lot special chars, but it's extremely intuitive:

ï = AltGr+:, i

ü = AltGr+:, u

é = AltGr+/, e

è = AltGr+\, e (since it's extremely common, also aliased as AltGr+w)

If you're Windows based and want US-compatible keyboard layout that allows easily typing any special chars, I highly recommend it.

[1] https://translate.google.com/translate?sl=pl&tl=en&u=https%3...

I type English and French in Windows on the same QWERTY keyboard. I once learned to type on Azerty, but I mainly type English now on a standard US keyboard layout. For the French, I find the windows alt-numbers works the easiest for accented characters. Alt-130=é, Alt-133=à, Alt-135=ç, Alt-137=ê, Alt-138=è which covers 95% of the accented character usage. I have a little chart next to my desk with all the others (ï,ô,ù) they’re nearly all Alt-14x and Alt-15x. And then I’ll put é in the paste buffer because it is the most used and a bit quicker that way (for words like “préféré”).

The Alt-13x codes are not as quick as the Azerty keys, but good enough and once memorized are fairly easy with a keyboard that has a keypad (most PCs do, even my laptop). This is especially true because they are done with both hands simultaneously, as opposed to something like Cmd-e+e on a Mac. Actually, they are faster than finding the accented characters on my QWERTY virtual keyboard as I type this comment on iOS.

Those AltGr- combos seem complicated to me, I would much prefer a system such as AltGr-e =é, then AltGr-ee=è, AltGr-eee=ê, etc. To me that would be more intuitive than remembering the composing character (slash for aigüe, etc).

Reminds me of when the 'D' key broke in my physical keyboard long time ago. I liked that keyboard a lot and couldn't find a good replacement so I learnt to type Alt-100 do get 'd'.

You seem to be quite used to your Alt combination but as you said they really are not straightforward. I found another very simple solution, on Linux you can set a compose key (typically Alt gr or the contextual menu key). You type one after another, the compose key and then any two keys that make sense like ' followed by e (et vice versa), it will give you a é. It is both fast and easy to work with.

It is acceptable to write English without diacritics. "Naive" is accepted.

It is certainly acceptable, although some publications have diacritics as part of their house style, and if the author doesn't use them, the copy editor is supposed to insert them.

The New Yorker writing things like coöperate and reëlect is probably the most infamous, although they are not the only ones.

The Guardian style guide says to use spellings exposé, lamé, résumé, and roué – but not café. Although, when giving the name of an organisation/institution (restaurants and cafés included), the article should use whatever spelling is preferred by the management, including their choice about how to spell cafe/café (when that word is part of their name).

Personally, I'd always write café in formal English, because cafe just looks wrong to me. However, in something informal like a text message I probably wouldn't bother.

Sure, some publications might like that. And some publications still treat data as a plural of datum rather than a mass noun like water. But I don't think I know a native speaker who would look at "naive" and feel the way they do when thy see "could of".

FWIW, Economist explicitly names Naive as one to use without diacritic: https://cdn.static-economist.com/sites/default/files/store/S...

By default on a Mac you just hold down the key to get different options, similar to on an iPhone (and a presume touch-Android).

https://i.imgur.com/yuG063t.png

On any Unix, just enable the "compose" key, then <Compose>+"+i.

It's always something easy to remember, like " for umlauts, o for circles (©, ®), obviously ' for accents (ń) and so on.

You can see and modify the .XCompose file, and you can even put your own strings there like an email address or change the sequence. There are some community XCompose files on GitHub too.

You may or may not need to set GTK_IM_MODULE=xim to get GTK applications to use ~/.XCompose.

Also, Qt5 broke support for multi-character results when using compose sequences (everything after the first character is ignored).

You can set up your keyboard as US International. And then type [“] + [i]. It’s a very useful keyboard layout because the punctuation characters match the keyboard ana allows you to type English, Spanish, French, Dutch, German, Swedish, Norwiegan, Portuguese

Having punctuation keys as dead keys is too annoying so I use my own keyboard that has dead keys only when used with AltGr as well as some direct AltGr umlauts (ie AltGr+a=ä).

On both Windows and Linux, I've liked the "US international" keyboard layout as an alternate when I need to type letters with diacritical marks. In that layout, " ' ~ ` ^ are all dead keys, which modify the next letter typed. In addition, the right Alt key (usually Alt Gr on non-US keyboards) can be used to quickly type some commonly-used letters.

On Linux there is also a "US alternate international" layout with some extra dead keys to make it easier to do things like š or ž (Ctrl-RightAlt-< followed by s or z, respectively).

Forget ascii conversion, that also fails on contractions like "don't".

The easiest solution to this problem would be to just rename it to "naive".

ï isn't an English character though! For an English document, "naïve" is a mispelling, at best. That being said, you don't always have the luxury of doing things the "correct" way, especially if users are trying to cram god-knows what into a text field.

Nope. ”Naïve” is an accepted variant of ”naive” in every major English dictionary. [a-zA-Z] is never a ”correct” way to match natural language text.

Would it blow your mind that coöperate is technically correct?

That's the whole point of the diaeresis.

not that unusual - for German for instance üöäÜÖÄß need to be added so all words can be matched.

now, there is even a capital ß ;)

Out of curiosity I tried on my phone: ß

So my phone doesn't have that yet!

mine has it - ẞ the small one is ß

Is the capital supposed to be shorter?

It's wider. Depends on your font and it's support though.

Oddly enough, yes (depending on font of course); compare "Sf": "ẞß" should have the same capital and lowercase-ascender heights, the latter of which is often higher.

Less relatedly, I really hate when people use the eszett instead of a Greek beta. I just needed to get that out of my chest.

This would be an argument for just using [:alpha:] everywhere; presumably it does the correct thing based on locale?

No, alpha doesn't work, at least in "grep -P" with "ru_RU.UTF-8" locale:

  $ echo Test | grep -oP '[[:alpha:]]+'
  Test
  $ echo Артём | grep -oP '[[:alpha:]]+'
  $ echo Артём | grep -oP '[А-Яа-я]+'
  Арт
  м

This thing works, though I've never seen one in the wild:

  $ echo Артём | grep -oP '[\p{Cyrillic}]+'
  Артём

Standard classes only work for 8-bit locales, afaik, and also in some languages (e.g. perl) when a string encoding corresponds to an internal representation of what its engine thinks it is a "current" unicode format for the specific version of language and a checkout location. The fact that different tools stick to different engines, modes and normalization rules (bre/ere/posix/nfd/nfc/perl/pcre/php/ruby/icu/whatever) doesn't help either. Full cross-platform unicode matching is a can of worms that you usually don't want to open. It is basically CSV of the encoding world. /\S+?/ to the rescue.

https://regular-expressions.mobi/unicode.html

https://regular-expressions.mobi/refunicode.html

https://regular-expressions.mobi/posixbrackets.html

Well, that is horrifying. TIL.

  $ echo Артём | grep -oE '[[:alpha:]]+'
  Артём

Yep, -E vs. -P

The problem here is simply a bad regex pattern. You need to use something that supports Unicode an enable it. (Usually a flag or option you have to set because Unicode matching is slower and often not needed.) Then last but not least you need to use character classes and not define ranges yourself unless you actually need a range. Like you cant use [:digit:] or \d if you really only want 1-9 without the 0.

Here are some examples to match your string all of them use PCRE flavor

(UCP)[[:alpha:]]+

(UCP)[[:alnum:]]+ This would includes digits

(UCP)[[:word:]]+ This includes "word" chars

(UCP)\w+ same as above

Reply on news.ycombinator.com