What's the reason you need to abandon scikit-learn? You can run scikit-learn on Databricks, and many of our customers do.

Disclaimer: Databricks cofounder.