Hacker News Reader

Comment by umvi | Hacker News Reader

I've mostly seen the opposite - pandas and jupyter notebooks shipped directly to production because the data scientists and AI guys didn't know how to do anything but python. As a result, the solutions were not performant and often had lots of runtime crashes due to python's more loose typing

View on HN

If they're shipping notebooks to production and having so many crashes, I'd question that they even know how to do Python.

When do you ship notebooks to production? Jupiter was never meant for external clients.

While terrifying, it is not uncommon to see python notebooks make it to production.

Oh god why. I thought (and hoped) that GP didn't actually mean this.

I see how a team or an organization can eventually get to this point. It just saddens me that they got there.

why not, many data related tasks are rather ad-hoc, it's a waste of time to make a long lasting software out of every ad-hoc request

Seen the same with exposed matlab web apps. Not just the Python guys, kind of shocking how much group think exists on any platform.

Quite a number of AI edtech sites use notebooks in production for assignments, as an example of when it happens.

Usually what happens is:

1. "We got it all working in the notebook"

2. "Great, ship it"

3. (data scientist takes the notebook code almost verbatim, wraps it in a basic CLI or HTTP API and it gets shipped off in a docker container for other services to consume)

In practice that’s not too crazy. It’s fast easy to debug and works with most CI/CD tools.

If you ever want to work in teams that kind of setup works extremely well.

I'm not sure how you're going to fix this given the data science tools are in Python. Are you gonna implement a half-broken 20% of numpy/scipy for a one-off program and then try to port?

These libraries hide a lot of complexity and implementing even a few operators is a project.

i dont think they are complaining about the libraries, but rather the scratchpads and notebooks that people use for ideation and evaluation being moved directly into a production environment because the authors don't have the experience or time to build more structured, efficient and maintainable code.

Rust + Polars comes to mind.

Reply on news.ycombinator.com