Hacker News Reader

Comment by blitzd | Hacker News Reader

My beef with Julia when I last tried it out was the lack of flexibility within the 'data munging' sphere. The assumption appeared to be that all of your data would come in to as UTF-8, whereas in my org we need strong support for Latin1/cp1252, and UTF-16LE BOM. Is this something that has changed recently, or are these traders are economists just working with more sane data?

View on HN

AbstractStrings (the generic string abstraction) can have any encoding – only the standard built-in String type is UTF-8. If you've got data in another encoding, use the StringEncodings [1] package to read it.

[1] https://github.com/nalimilan/StringEncodings.jl

Pipe the data through a converter before feeding it into Julia.

Well - that's my point. I do use a converter, as it's built into python at every intersection that I need it, reading CSV, XLSX, database calls, etc.

Having to stick python scripts in front of all my Julia scripts everywhere that it would perform data access doesn't really seem viable. Thus, my beef with Julia.

Not a language issue per say then. It shouldn't a problem to add support for this. Julia is even designed from the start to work with Strings of different types. Something not common in most other languages I've worked with.

Most string functions operate on AbstractString types. You can create whatever subtype you like implementing alternative string encodings.

Building on this comment, it's actually surprisingly straightforward to add your own encodings into the core language. Just add the appropriate types and extend the necessary base methods.

In this case, it appears Julia 0.5 has a transcode method supporting UTF8/16/32. [1]

1: http://docs.julialang.org/en/release-0.5/stdlib/strings/

I'd venture a guess that implementing any string encoding support in a language I'm not that familiar with would likely be quite an undertaking.

If not, then it also begs the question - if it's just that simple, why's it not already there in the core alongside UTF-8?

Anyone new to the language working through data problems is going to pick it up and start at the data collection/wrangling stage, and immediately run into the 'you can implement that yourself/run it through a converter' mentality, and likely go back to whatever they were using before. I know we did.

Reply on news.ycombinator.com