Comment by IanCal | Hacker News Reader

Comment by IanCal | Hacker News Reader

I like seeing more things come out helping people with data analysis in various languages, but this put me off:

> isWhitespace x = elem x " \t\r\n"

This is the kind of thing that makes me concerned about using this resource for real-world data. In real-world data you're going to get all kinds of crazy things coming in, and if you're assuming nobody will ever have something like a zero width non-breaking space, or a form feed, you're going to have a problem.

It's the kind of thing I see with people starting out when dealing with data, similarly the punctuation detection here: https://github.com/BinRoot/Haskell-Data-Analysis-Cookbook/bl...

If you rely on these things, you will have problems. Text is hard and weird and terribly more complicated than people usually expect.

Does haskell have good libraries for dealing with the more awkward parts? Can I easily remove all characters marked as whitespace in unicode, for example? Detecting and managing mangled encodings?

That line of code was demonstrative. The actual book uses 'Data.Char.IsSpace' which properly handled it.

"Returns True for any Unicode space character, and the control characters \t, \n, \r, \f, \v."

Before you chastise them for not handling something, verify it. I'm not affiliated with the book but you probably deterred people from buying it.

https://github.com/BinRoot/Haskell-Data-Analysis-Cookbook/bl...

I agree that line of code was a turn-off because of its oversimplification.

> That line of code was demonstrative.

Why demonstrate with something completely unrepresentative of the actual contents, then?

Well for one the line shouldn't really be one of the first bits of advertising for the book if the author also knows it's wrong. The second example is taken from the GitHub repo for the book though, and is exactly the same type of error.

> you probably deterred people from buying it.

Quite possibly, but I think with good reason. I don't know what's in the book but I'm concerned it won't contain things like a discussion of what whitespace is and is not, how to decide what you should do for your data and when isSpace might not do what you really need. I can't review it properly but at least one bit of code in the repo looks dangerous and one bit on the website looks dangerous.

For more realistic stuff, I generally use https://hackage.haskell.org/package/text-icu. For example, in an old project we had a type called `Texty`, which was roughly non-empty normalized text:

  mkTexty :: Text -> Maybe Texty
  mkTexty = \a ->
      let a' = T.dropAround Char.isSpace (ICU.normalize ICU.NFC a)
      in if T.null a' then Nothing
                      else Just (Texty a')
  {-# INLINE mkTexty #-}

Great, thanks for the link.

Yeah, that doesn't give high hopes for this book teaching best practices :/

If you don't want to depend on text-icu, there is support for more Unicode space characters in https://hackage.haskell.org/package/base-4.8.2.0/docs/Data-C...

    import           Data.Char             (isAlpha, isLower, isSpace)
    import qualified Data.Text             as T

giving you e.g.

    textIsAllSpace t = T.all isSpace t

Important note for non-Haskellers: the library OP is talking about is part of the standard base libraries usually distributed with GHC. `text-icu`, while excellent, is not.

Reply on news.ycombinator.com