Hacker News Reader

Comment by staticassertion | Hacker News Reader

How do data scientists feel about Nim? It feels very pythony, but way faster - could be a really cool language for them, similar to what Julia is trying to be.

Curious if anyone in a DS role has tried it out. I'm certainly tempted to try it out, I'm sure I'm losing lots of performance in parts of my analytics pipeline due to Python being garbage slow.

View on HN

I am trying to improve Nim data science ecosystem with https://github.com/mratsim/Arraymancer.

I think today the best way to try Nim is for slow Python utilities vi nimpy: https://github.com/yglukhov/nimpy

You can also use Nim in Jupyter the following way: https://github.com/apahl/nim_magic

My next step would be to improve interop between Arraymancer tensors and Numpy ndarrays.

Very cool work! Thanks for the links.

Re: speed, I'm seeing the Rust ecosystem get better and better; I'm also learning the language on the weekends with small projects.

But also: many many moons ago I had Matlab code that was really slow (and I wanted to run 10^LARGENUM simulations for Monte Carlo purposes) and decided to learn Fortran. It was surprisingly easy to learn, and it did cut my running time from afternoons to minutes. But in the process I found out that my Matlab performance problems were mostly due to arrays being dynamically resized all the time -- while static memory preallocation could be hacked easily in straight Matlab.

As for Python, numpy is a fast embedded minilanguage written in C; the difference between "pythonic" and "numpythonic" is not emphasized enough in bootcamps, I think.

Matlab usually flags cases like that in the editor suggesting you reserve space outside a loop.

My experience with numpy was that it was 11x slower than visual basic access in excel. I was deali g with a lot of small matricies instead of a big one and I think that contributed.

I want to try julia in production. About to try out rust in production.

Not really a data scientist myself, but using Nim for my PhD in physics to analyze the data I took for an axion search experiment, code at [1].

I just love it. Super productive to work with and in my case faster than the C++ framework I'm replacing.

The ecosystem of course is pretty small. But for me wrapping C code, porting something or reimplementing some libraries can be a lot of fun, too. In my experience every hole I've found was easily plugged so far.

[1] https://github.com/Vindaar/TimepixAnalysis (recently due to ..reasons.. all development happened on the open PR)

I work as a Data Scientist/Machine Learning Engineer and I think that biggest issue with Nim are lacking libraries. Working as a Data Scientist is very iterative process, you spend most of the time preprocessing data. It's often try and see approach so interactive repl is a must. While I know Nim has some library to be used with Jupyter, I found it too immature. Other reason to favor Python is it's libraries - they are often written in cython which yields comparable (but of course a bit slower) performance to C. And to speed up custom code you can almost always use numba (llvm-based jit compiler that works with CPython).

Nim is used by leading data scientists. Eg.: https://github.com/mratsim/Arraymancer

The lack of mature libraries in the style of pandas, sklearn or the tidyverse, and the lack of a stable REPL is a major drawback for "production" data science. On the other hand though I've had fun playing with it in a personal capacity precisely because of the rough edges.

If I want to fit a particular kind of model I'll probably have to roll my own, meaning I have to be familiar with what the algorithm is doing and why, rather than just importing something straight from sklearn.

I also need to pay more attention to the code as I'm writing it instead of lazily debugging by trial and error in the REPL like I can with Python.

Basically, I wouldn't recommend it at work, but it's been fun and useful at home.

Since you mention tidyverse, I might shamelessly mention that I'm currently working on a sort of port of ggplot2 and some dplyr features in Nim [1]. It's super WIP, but for most plots and simple data frame operations (complicated stuff I do beforehand) I need it has mostly replaced plotly [2] (or in some cases matplotlib).

However, don't expect the included data frame to be fast. That in combination with a dynamic nature has been out of scope for me alone.

[1] https://github.com/Vindaar/ggplotnim/tree/addDocs

[2] https://github.com/brentp/nim-plotly

The options in python to speed up your code are vast without really having to do much, on a fundamental level. My outdated experience with Nimrod told me that it was a tad too opinionated E.g. Everything is a string in dbs is mentioned in the article.

It was an interesting language, for me it felt like a stepping stone to something better. But the docs were really bad,

"Nimrod" experience is quite outdated. There have been great improvements in documentation in the last year.

> The options in python to speed up your code are vast without really having to do much

This does not match my experience. The #1 way I see to optimize Python is to not write Python at all, and instead to FFI out.

there is this great discussion about the Nim Language on Julia Discourse: https://discourse.julialang.org/t/version-1-0-released-of-ni...

Reply on news.ycombinator.com