Ask HN: Is Python integration necessary for a commercially-viable data product? | Hacker News Reader

Ask HN: Is Python integration necessary for a commercially-viable data product?

1 pointmattewong6 comments

(re "data product", say for example, Cockroach DB which I have no affiliation with and already has a python integration)

With platforms like DataBricks and Airflow that are heavily inclined towards python, it feels somewhat unavoidable to offer a python "import myproduct" option. But, is it truly necessary to be part of that python ecosystem to survive as a data product?

I know: python has a vibrant community, it's a virtuous cycle because every data lib is in python and therefore every data platform / analyst uses it etc, why would you ever not want to just add your product to that ecosystem etc. I get that, but that's also why I'm asking the question

Expose a "REST" API and webhooks and you can have practically as many clients or "integrations" as you want.

Here are design principles:

- Anything you can do with point and click, you must be able to do with an API call.

- It should theoretically be possible to clone your product using its API.

- Users can be anything (people, systems, your own product). For example, our product is built using a plugin architecture, our subsystems hit our API.

- Someone interested in an event should get that event and be able to subscribe to it. (Again, users can be people or systems, and that event could be the basis of some downstream actions).

In what sense? A lot of people like to do data analysis with jupyter, pandas, scikit-learn, etc. They don't need to be treated like children with half-baked "integrations", instead they know how to read industry standard data files

https://pandas.pydata.org/docs/user_guide/io.html

This is a sincere question and I have no idea what you mean by 'treated like children with half-baked "integrations"'?

The intended context of the question, which admittedly could be more clear-- is for a data product that could be used for any data. Take Cockroach DB for example, which already has python integration ( https://www.cockroachlabs.com/blog/sample-app-python-and-coc... )

With platforms like DataBricks and Airflow that are heavily inclined towards python, it feels somewhat unavoidable to need to offer a way to access the product via "import" in python.

So the question is: in that scenario, could a commercial product still survive-- or even thrive-- without doing so?

For me the use of the word "integration" is a bad smell. All too often I've seen it used in cases where people have tried to substitute 15 hours of talking to sales people for 15 minutes of Python coding for 15. That is, connecting to an API is one of the easiest things in computing but when people starting talking about "integration" they are making it one of the hardest.

If your product is a database (I make a query, you return results) then I think many users are going to want Python bindings to the database.

If this product is going to "thrive" people are going to need some way to connect with it. What alternative to using Python to do you imagine somebody wants?

OK thanks, I get that now. I suppose there could be at least two alternatives:

1. use "un-integrated" python code e.g. run a subprocess independently that just runs an external binary / shared lib / etc

2. avoid solutions that do not require a python environment entirely, or, use them in a less-commonly-employed manner that would not run the product through python (e.g. Airflow bash operator)

Both of the above seem perfectly reasonable to me, but I'm not sure whether others would agree.

Reply on news.ycombinator.com