We've been through that and are building our platform to address some parts of this problem so projects have a higher likelihood of success.
In our experience building machine learning products and executing projects for large organizations over many years, the problems are in different parts of the journey.
There were problems with us, problems within the client, and problems in the interactions.
The problems with us were related to the way we did machine learning projects. We did turn-key solutions, from data acquisition to training models, to building software for the client's domain experts not only to use these models, but train new ones with fresh data after we are gone.
Some were pure academics who taught at universities, but relied on our colleagues to set up their environments, upgrade their systems, deploy their models, get them the data, etc.
Others were good at developing software but had to be dragged into the ML realm to use models.
We had different profiles. Some could do it all: from signal processing, to writing software for embedded devices for data acquisition, to setting up environments and infrastructure, to writing web and mobile applications, to deploying models and applications.
There were a lot of frictions given that these profiles lived in different realms and were optimizing for different things and there had to be overlap to get things done, given that we were a tiny team that could not afford to hire specialists full time given that we were a consultancy company and our revenue was based on projects.
But even when we had many projects, they were in parallel but with the same people. This is not easy as the same people working on six or seven different projects with different code bases can drive people nuts. The context switch, the email you send asking for data gets answered when you are working on something else. Scheduling meetings. Jumping from one to the other. The PhD tapping who wants to show results to the client and who taps on your shoulder to make that happen while you're busy deploying their earlier work to the client, which you can't figure out where it is because tracking experiments, if it was done, was ad-hoc. Excel sheets, logs, pen and paper, in their head, over email.
The problems in the client organizations were especially when not all stakeholders are aligned. You are on a project where only executives are involved, and you ask for the involvement of their domain experts but get denied. So you build things based on what you "think" they would want, or you don't get data in a timely manner because you have to set a meeting with legal, security, data warehouse, sales, and marketing who get onboarded at a later stage, or a new privacy law gets passed and it changes everything.
The people in the client organization having different agendas, or not committing to the projects because they resent not having been consulted, or people being afraid of this "AI thing replacing them" and starting to pull weird things, like not answering your emails, not giving you data, etc. Sometimes the right people you want to talk with who have deep domain expertise are not the ones you are talking with. Sometimes when you have the people with domain expertise you don't have the support of decision makers.
We have changed the way we do business and insist right from the start that they be at the table or we won't do it, and we've had much better results that way. There's an effort to explain things that is extremely important: what is machine learning, what can be done, what we can't do, defining the problem and success criterias and keeping that definition alive through the project and not let other things creep in, agreeing on the deliverables. We've been having better results like that.
The problem in the interactions is in the latency of communication sometimes. Some clients, even large ones, are super responsive and you have the full support of everyone. They know what they want, they know how bad they want it, and they'll make everything at your disposal to get things done. Top management is involved. People are helpful. You get to talk with everyone, and they have people who can work their magic to get you what you need and onboard their own people. Regular updates. A great cadence. You really see progress. We were successful in these contexts and they were happy. Repeat business in that case where they come with more problems, and you leverage the relations you have built during the previous projects. Everybody knows everybody. You basically are like colleagues. They have the stamina to see things through.
So, with the interactions and the problems we faced with clients, we have fixed things upstream by explaining how things will happen, explaining that it is important that the problem be specified, and to get the metrics right, etc.
With the problems we had ourselves, we're building our platform[0] to dramatically accelerate our execution. Collaborative notebooks, scheduled long-running notebooks, automatic experiment tracking for metrics, params, and models. One click deployments, live dashboard for monitoring.
All these things required a tap on the shoulder, or were blocking steps. There was a lot of time wasted on environments breaking when a "data scientist" required the help of another colleague to fix their environment, or to deploy their model. The automatic tracking takes off the burden to remember to track experiment because we were tired of trying to remember which model was best, or which data it trained on, or which parameters were used, and which metrics it had. Or when a developer wanted to build a feature that relied on a model from another project but had to be dragged into the "machine learning realm" even to use that model.
The short-term objective for the platform was to take the burden of what were doing manually, so we don't have to do it ourselves. This alone has dramatically increased our quality of life and reduced our stress level. We've also witnessed this effect in one of our current projects: what would have involved our people could already be done by the platform, so it prevented taps on their should with "new priority, folks...". This makes me happier.
The mid-term objective is to improve our readiness and step-up our ML game and include best practices right into the platform, without becoming rigid, as one of the things we didn't like in other solutions is the rigidity. The rigidity of what a "pipeline is", the rigidity of the environment where you drag and drop things, etc.
We're optimizing for getting things done, not to make the most appealing product visually, as stylesheets never held our machine learning projects back.
- [0]: https://iko.ai