Hacker News Reader

Comment by itg | Hacker News Reader

In my experience, the title 'Data Scientist' has become to mean data analyst at most companies, meaning working with Excel, Tableau, and SQL. Maybe R if you're lucky.

Companies doing ML/AI will usually have a small team of Research Scientists who mostly hold PhDs, and a team of supporting ML Engineers.

View on HN

>Companies doing ML/AI will usually have a small team of Research Scientists who mostly hold PhDs

How many companies are doing genuine AI research, as opposed to applying the research and tooling to their unique business problems?

I’d be more inclined to think that at a startup, because of the lack of infrastructure, a data scientist will do a lot more than research.

It’s even worse at established companies because data scientists still only do the data plumbing and simplistic analytics tasks, but not due to anything reasonable, like the “many hats” needs of a startup, but instead because of IT dysfunctiom and bureaucracy.

Think it depends but it does happen. We’ve been able to carve ourselves out of IT with our own infrastructure and our data scientists specifically focus on research and analysis. The only issue is when we try to get anything to prod and we hit IT.

> “The only issue is when we try to get anything to prod and we hit IT.”

Which inevitably means the one person on the data science team who is good with linux and docker suddenly becomes the IT wizard, and their time gets sucked up by having to find ways to go around utterly stupid barriers and tactics used by IT to avoid doing work to help you.

Source: I am currently this person for my team.

I feel your pain. I am in it too.

Would love to get your thoughts some time - I'm building a product to make life a little bit easier (3 step deploy model as API) but with a vision towards more broaded deployment usecases. Your advice will be valuable and much appreciated!

As someone who is about to be tasked with building this sort of infrastructure for a hospital system what would be your dream architecture? I hate IT hurdles and I am hoping I can avoid building them into our infrastructure.

The number one thing is to make containers a first class deployment and provisioning artifact. As long as dev teams can control their own containers, they will be able to do what they need no matter how arbitrary or assumption-breaking.

Do not ever require dev teams to go through IT to get their chosen tools deployed to the right places or with the right resources provisioned. Never.

This is the root of all evil with infra teams: if they see themselves or their mandate as being gatekeepers of provisioned resources, then dev teams have lost and you as a data scientist / ML engineer, you’ll never get your work done.

As an ML engineer, I want to define the entire runtime and development environments of any analytics artifacts or web services that I create, and to change these environments as needed, as indicated by what’s required to get the job done.

Let me define containers, hook them up in whatever CI tools are used, push and pull them from some internal container repository, and describe the configuration for the resources they need. Offer that as the contract to dev teams and then infra’s job is to maintain the underlying data center that physically supports running the containers and occasional hand holding for special exceptions, networking, secrets management, and cost tracking.

Good, that's my current plan! I appreciate your input.

Very well said. I can’t stress the container environment enough not to mention the ability to control the toolset.

Used to be this person at a previous company. I didn't run molecular dynamics simulations on supercomputers so I could explain to the CISO why a machine learning library isn't a virus.

Don't know why you're getting downvoted for this - we have the same pain. We're not allowed to have a sysadmin or administer our own servers but IT is busy supporting thousands of non-computery-users. To them we're a fly in the ointment.

I find it quite interesting as technically speaking both IT and data science are technical. Pardon the pun.

The real value of data science teams comes from great infrastructure in addition to a solid team so it is a bit of a pity sometimes.

Reply on news.ycombinator.com