K-Means Clustering is one of those tools that once you understand it becomes more and more useful over time. You keep finding new ways to apply it, it's a bit like Quicksort, Bresenham or Divide and Conquer in that sense.
In general clustering algorithms tend not to be very useful because they are ultimately just very poorly defined latent variable models. Most people starting out with clustering don't even understand that they're making the assumption that K latent variables are responsible for generating the observed data.
K-means in particular has loads of problems for practical applications: the final location of the means is heavily dependent on the initial locations, the final location can also be heavily dependent on the particular dataset used (try bootstrap resampling to get variance of your means), assumes the data rests on a Euclidean surface, making geometric assumptions about high dimensional spaces is almost always a bad a bad idea (things being close by in Euclidean terms in high dimensional spaces often misses things that are nearly identical except on feature that is quite different).
With over a decade of modeling experience and building data science products I have never seen a product ship that relies on k-means and never seen analysis that isn't coming form someone junior in the field that relies on it.
K-means is a great demo the expectation maximization algorithm, which is a useful tool to understand, but generally I would avoid k-means, and clustering in general unless you have an explicit understanding of why you are doing it. Even then, there is probably a more stable and appropriate latent variable model you could use.
I used it exactly once, and it worked quite well :) But that was because I was clustering 1-dimensional data, which has a globally-optimal solution and there was a fast implementation for it available (see my post https://news.ycombinator.com/item?id=30675594). So it was okay in that instance to brute-force check a lot of different cluster numbers and then post-process the results with some heuristics.
I always liken the latter to 'automatic feature detection'.
That said, its worth noting that there are algorithms out there for determining optimal number of clusters for k-means (though personally I found them to be costly and subject to overfitting) like using the silhouette coefficient or elbow method [2].
[1]: https://www.geeksforgeeks.org/ml-mean-shift-clustering/ [2]: https://towardsdatascience.com/k-means-clustering-how-it-wor...
This implementation is also surprisingly fast, so you can use it to brute-force check many different numbers of clusters and check using silhouette distance. The advantage over traditional k-means is that you don't need to check multiple initializations for any given number of clusters, because the algorithm is deterministic and guaranteed to find the global optimum.
1. O(NK^2) dynamic programming from 1965 by James Bruce (https://dspace.mit.edu/bitstream/handle/1721.1/4396/RLE-TR-4...)
2. O(KNlogN) dynamic + divide&conquer algorighm from 1989 from Xiaolin Wu and John Rokne (http://doi.acm.org/10.1145/75427.75472)
3. O(NK) algorithm from 1991 by Xiaolin Wu (http://dx.doi.org/10.1016/0196-6774(91)90039-2)
There are some examples of this at https://stats.stackexchange.com/questions/133656/how-to-unde...
For example, word2vec uses k-means clustering using cosine similarity measure [1]. It works very, very well. The caveat is not many optimization variations of k-means will work with that "distance".
[1] https://github.com/tmikolov/word2vec/blob/master/word2vec.c#...
So I assume at least part of the usefulness is that the principle applies to unrelated domains?
It's not obvious how to do this with HDBSCAN or hierarchical clustering. Maybe you'd have to draw some kind of boundary around each cluster after fixing the parameters.
Eg. Wouldn't KNN be easier if you want to classify new groups later on? I'm probably missing something, but defining the groups upfront seems more labor intensive ( note: novice in this area )