[ENH] Tree (hierarchical clustering, dendrogram) of clusters

[mstrazar]

It is unfeasible to compute hierarchical clustering with datasets of more than a few thousand data points. Instead, a tree between the clusters is useful. Use cluster centroid (mean point) or other linkage options from Scipy to compute linkages.

Tree between 17 clusters:
tree.pdf

[lanzagar]

I agree with everything written: HC is unfeasible for large data, tree of clusters can be useful, cluster centroids can be used for this (provided by kmeans).
What exactly is the issue here?

One issue I have with this approach is that when the original data contains categorical variables, the centroids have a different domain (kmeans continuizes the data). But for numerical data (e.g. gene expression) this is not a problem.

[mstrazar]

It's more an enhancement than an issue. Certain clustering algorithms do not output cluster centroids, but only categorical labels. Hierarchical clustering would not even need to operate on centroids, but use the appropriate linkage method to aggregate distances between provided clusters. Categorical variables can be continuized too.

[lanzagar]

I see - so instead of using k-means you would like to use predefined clusters (categorical var in data), which can be a result of (any) clustering or otherwise given.
And in this case we currently don't have a good option to get the centroids, which can be used in hierarchical clustering.

In that case, the solution would be to have a Pivot table, which can compute averages (or other aggregates) to obtain the centroids of selected groups.
This is becoming a recurring issue :) We ended up with a similar conclusion many times in the past few months. +1 for Pivot tables, we just need to come up with a good widget proposal.

[ajdapretnar]

Ahem, #3823. 😁 Would this help? @mstrazar If so, this would be a nice thing to add to the docs.

[janezd]

Closed due to inactivity. Probably partially solved by Pivot widget.