Hacker News Reader

here's another "fun" grep locale oddity:

   $ echo HI | LANG=en_US.utf8 grep '^[a-z]'
   HI
   $ echo HI | LANG=C grep '^[a-z]'
   $

apparently en_{GB,US}.utf8 orders a-z like aAbBcC..zZ.

   $ echo ZI | LANG=en_US.utf8 grep '^[a-z]'
   $

This is what I get:

    $ echo HI | LANG=C grep '^[a-z]'
    $ echo HI | LANG=en_US.utf8 grep '^[a-z]'
    $

How come?

I was able to reproduce the bug. It could be a version thing.

  ; grep --version
  GNU grep 2.6.3
  ; echo A | LANG=en_US.utf8 grep '[a-z]'
  A

No, I have the same version but not a similar result. I also have the en_US.utf8 locale installed.

I had the same problem with sort:

  $ sort <<EOF
  > Aa
  > aa
  > Ab
  > ab
  > EOF
  aa
  Aa
  ab
  Ab

I was going crazy because I was getting different results in OSX and Ubuntu. Setting the LANG to POSIX fixed it.

    $ sort --version
    sort (GNU coreutils) 8.14

For what it's worth, this gets me the same results under LANG=C and LANG=en_US.utf8

FWIW, more recent GNU grep releases (2.6 and later) run fine in UTF-8 mode.

Actually, why is SmartOS, "the Completely Modern Operating System", shipping with a five year old grep?

Minor correction: the "bug" in grep was fixed in GNU grep 2.7, not 2.6. I recall that because I wrote an article about it on my blog. See the release notes for GNU grep 2.7 here: http://savannah.gnu.org/forum/forum.php?forum_id=6521

When the author shows the output of "which grep" it shows the path "/opt/local/bin/grep"; "/opt" is typically for additional installations of software. I haven't used SmartOS myself but my best guess is someone installed GNU grep manually and the PATH just included "/opt/local/bin" before "/usr/bin".

I guess SmartOS reserves /usr for the Solaris base install and puts additions in /opt/local. If you look at http://wiki.joyent.com/display/jpc2/SmartOS+Smartmachine+Dat... and click "to see list of installed software", it includes grep 2.5.3.

> This is not really a 2000x performance "win", this is GNU grep(1) being so bad at dealing with UTF-8 to the point of being basically unusable. > > The PoSix locale system is a nightmare, but the implementation in GNU coreutils is even worse, when it is not insanely slow it is plain broken.

Actually, GNU grep as of last year was much faster than BSD grep, as long as the C locale is used:

http://lists.freebsd.org/pipermail/freebsd-current/2010-Augu...

The reason why using UTF8 makes it much slower is that dealing with Unicode correctly is a very expensive business. But pretending that languages other than english do not exist is not an answer.

The Plan9 grep deals with Unicode correctly and is very, very fast. Claiming that

> dealing with Unicode correctly is a very expensive business

is plain FUD.

UTF-8 has zero overhead in this case (not near zero, but actual zero overhead). Grep doesn't even need to be aware of UTF-8. It doesn't need to be aware where a rune starts and ends in the byte stream. UTF-8 is designed in such a way that matching a regexp only requires byte comparisons, exactly as for ASCII.

Unfortunately, it's not as simple as that. There is more than one way of representing many text sequences using Unicode; for example, combining diacritics vs accented characters. To do the Right Thing (tm), you need to normalize, and that adds at least another pass over the whole string.

If you just deal with it on a byte by byte basis, you're not actually Unicode compatible. It might be fast, but it's not Correct.

In addition, conforming Unicode implementations have to ensure that their input is well-formed Unicode, and reject non-conforming characters (for some security-related reasons). So the original goal of UTF8, that it could be usable for straight byte-level search (the UTF8 RFC mentions a goal of allowing Boyer-Moore to be applied directly) is no longer really possible.

edit: a discussion on this subject from a few months ago, where I was corrected on that point: http://news.ycombinator.com/item?id=2860932

It's not necessarily grep's responsibility to normalize strings. They could very well stipulate that it is the responsibility of the user to make sure the both the input and the search expression are already normalized. That way, the normalization could be done by a separate program, resulting in the reusability and separation of concerns that is The Unix Way.

I fully agree, if you redefine grep as a byte search program, rather than for text search. Rule it out for most languages apart from English.

To have any usable grep with applicability beyond programming languages, English-language log-files, etc. (i.e. massive Anglocentrism), the normalizing would need to be built-in. Sketched:

    #!/usr/bin/bash
    regex="$1"; shift
    unicode-normalize-to-utf8 "$@" | byte-grep "$(utf8-byteify-regex "$regex")"

> The Plan9 grep deals with Unicode correctly and is very, very fast. Claiming that

>> dealing with Unicode correctly is a very expensive business

> is plain FUD.

If you think this is true you don't know unicode enough. I don't know how plan9 grep is implemented but if it does it with near zero overhead then it's not actually supporting Unicode, but only with a small subset of Unicode that they like. I also probably agree that that's the best part of Unicode and the rest of Unicode could burn in a fire but it's still not Unicode.

> If you think this is true you don't know unicode enough.

Yes, well, that is sort of a given for any statement about unicode (including this one).

You are conflating UTF with Unicode.

OP needs to upgrade grep to the latest version. He is using 2.5.3, and latest version is 2.10.

Release notes for 2.6:

"This release fixes an unexpectedly large number of flaws, from outright bugs (surprisingly many, considering this is "grep") to some occasionally debilitating performance problems."

http://savannah.gnu.org/forum/forum.php?forum_id=6249

Adding to what you said, I also find Plan9 regular expressions to be significantly easier to grok in your head compared to POSIX or PCRE regular expressions that push a lot of complexity on the user's side (PCREs are not even true regexps as PCREs are not regular).

Plan9 regexps are very easy to reason about, more explicit and don't look as line noise as much.

RE with backreferences is strictly more powerful then RE without them. Implying that PCRE is somehow inferior due to backreferences support is quite fuddy.

I thought POSIX regexes allowed back references (as in "(*.)\1"), so they aren't regular either.

Yup, the author an older version of grep 2.5.3. I have grep 2.6.3 on RHEL6 and only got a 2x boost doing a similar test.