Hierarchical clustering#Agglomerative clustering example

{{Short description|Statistical method of analysis which seeks to build a hierarchy of clusters}}

In data mining and statistics, hierarchical clustering{{cite book |first=Frank |last=Nielsen | title=Introduction to HPC with MPI for Data Science | year=2016 | publisher=Springer |isbn=978-3-319-21903-5 |pages=195–211

|chapter=8. Hierarchical Clustering | url=https://www.springer.com/gp/book/9783319219028 |chapter-url=https://www.researchgate.net/publication/314700681 }} (also called hierarchical cluster analysis or HCA) is a method of cluster analysis that seeks to build a hierarchy of clusters. Strategies for hierarchical clustering generally fall into two categories:

Agglomerative: Agglomerative: Agglomerative clustering, often referred to as a "bottom-up" approach, begins with each data point as an individual cluster. At each step, the algorithm merges the two most similar clusters based on a chosen distance metric (e.g., Euclidean distance) and linkage criterion (e.g., single-linkage, complete-linkage){{Cite journal |last=Murtagh |first=Fionn |last2=Contreras |first2=Pedro |date=2012 |title=Algorithms for hierarchical clustering: an overview |url=https://wires.onlinelibrary.wiley.com/doi/10.1002/widm.53 |journal=WIREs Data Mining and Knowledge Discovery |language=en |volume=2 |issue=1 |pages=86–97 |doi=10.1002/widm.53 |issn=1942-4795}}. This process continues until all data points are combined into a single cluster or a stopping criterion is met. Agglomerative methods are more commonly used due to their simplicity and computational efficiency for small to medium-sized datasets {{Cite journal |last=Mojena |first=R. |date=1977-04-01 |title=Hierarchical grouping methods and stopping rules: an evaluation |url=https://academic.oup.com/comjnl/article-lookup/doi/10.1093/comjnl/20.4.359 |journal=The Computer Journal |language=en |volume=20 |issue=4 |pages=359–363 |doi=10.1093/comjnl/20.4.359 |issn=0010-4620}}.
Divisive: Divisive clustering, known as a "top-down" approach, starts with all data points in a single cluster and recursively splits the cluster into smaller ones. At each step, the algorithm selects a cluster and divides it into two or more subsets, often using a criterion such as maximizing the distance between resulting clusters. Divisive methods are less common but can be useful when the goal is to identify large, distinct clusters first.

In general, the merges and splits are determined in a greedy manner. The results of hierarchical clustering are usually presented in a dendrogram.

Hierarchical clustering has the distinct advantage that any valid measure of distance can be used. In fact, the observations themselves are not required: all that is used is a matrix of distances. On the other hand, except for the special case of single-linkage distance, none of the algorithms (except exhaustive search in $\mathcal{O}(2^n)$ ) can be guaranteed to find the optimum solution.{{cn|date=October 2024}}

Complexity

The standard algorithm for hierarchical agglomerative clustering (HAC) has a time complexity of $\mathcal{O}(n^3)$ and requires $\Omega(n^2)$ memory, which makes it too slow for even medium data sets. However, for some special cases, optimal efficient agglomerative methods (of complexity $\mathcal{O}(n^2)$ ) are known: SLINK{{cite journal | author=R. Sibson | title=SLINK: an optimally efficient algorithm for the single-link cluster method | journal=The Computer Journal | volume=16 | issue=1 | pages=30–34 | year=1973 | publisher=British Computer Society | url=http://www.cs.gsu.edu/~wkim/index_files/papers/sibson.pdf | doi=10.1093/comjnl/16.1.30| doi-access=free }} for single-linkage and CLINK{{cite journal | author=D. Defays | title=An efficient algorithm for a complete-link method | journal=The Computer Journal | volume=20 | issue=4 | pages=364–6 | year=1977 | publisher=British Computer Society | doi=10.1093/comjnl/20.4.364| doi-access= }} for complete-linkage clustering. With a heap, the runtime of the general case can be reduced to $\mathcal{O}(n^2 \log n)$ , an improvement on the aforementioned bound of $\mathcal{O}(n^3)$ , at the cost of further increasing the memory requirements. In many cases, the memory overheads of this approach are too large to make it practically usable. Methods exist which use quadtrees that demonstrate $\mathcal{O}(n^2)$ total running time with $\mathcal{O}(n)$ space.{{Cite journal |last=Eppstein |first=David |date=2001-12-31 |title=Fast hierarchical clustering and other applications of dynamic closest pairs |url=https://dl.acm.org/doi/10.1145/351827.351829 |journal=ACM Journal of Experimental Algorithmics |volume=5 |pages=1–es |doi=10.1145/351827.351829 |issn=1084-6654|arxiv=cs/9912014 }}

Divisive clustering with an exhaustive search is $\mathcal{O}(2^n)$ , but it is common to use faster heuristics to choose splits, such as k-means.

Cluster Linkage

In order to decide which clusters should be combined (for agglomerative), or where a cluster should be split (for divisive), a measure of dissimilarity between sets of observations is required. In most methods of hierarchical clustering, this is achieved by use of an appropriate distance d, such as the Euclidean distance, between single observations of the data set, and a linkage criterion, which specifies the dissimilarity of sets as a function of the pairwise distances of observations in the sets. The choice of metric as well as linkage can have a major impact on the result of the clustering, where the lower level metric determines which objects are most similar, whereas the linkage criterion influences the shape of the clusters . For example, complete-linkage tends to produce more spherical clusters than single-linkage.

The linkage criterion determines the distance between sets of observations as a function of the pairwise distances between observations.

Some commonly used linkage criteria between two sets of observations A and B and a distance d are:{{cite web | title=The CLUSTER Procedure: Clustering Methods | url=https://support.sas.com/documentation/cdl/en/statug/63033/HTML/default/statug_cluster_sect012.htm | work=SAS/STAT 9.2 Users Guide | publisher= SAS Institute | access-date=2009-04-26}}{{cite journal |last1=Székely |first1=G. J. |last2=Rizzo |first2=M. L. |year=2005 |title=Hierarchical clustering via Joint Between-Within Distances: Extending Ward's Minimum Variance Method |journal=Journal of Classification |volume=22 |issue=2 |pages=151–183 |doi=10.1007/s00357-005-0012-9 |s2cid=206960007 }}

class="wikitable"

! Names

! Formula

Maximum or complete-linkage clustering

| $\max_{a\in A,\, b\in B} d(a,b)$

Minimum or single-linkage clustering

| $\min_{a\in A,\, b\in B} d(a,b)$

Unweighted average linkage clustering (or UPGMA)

| $\frac{1}{|A|\cdot|B$

\sum_{a \in A }\sum_{ b \in B} d(a,b).

| Weighted average linkage clustering (or WPGMA)

| $d(i \cup j, k) = \frac{d(i, k) + d(j, k)}{2}.$

| Centroid linkage clustering, or UPGMC

| $\lVert \mu_A-\mu_B\rVert^2$ where $\mu_A$ and $\mu_B$ are the centroids of A resp. B.

|Median linkage clustering, or WPGMC

| $d(i\cup j, k) = d(m_{i\cup j}, m_k)$ where $m_{i\cup j} = \tfrac{1}{2}\left(m_i + m_j\right)$

|Versatile linkage clustering{{cite journal | doi=10.1007/s00357-019-09339-z | last1=Fernández | first1=Alberto | last2=Gómez | first2=Sergio | title=Versatile linkage: a family of space-conserving strategies for agglomerative hierarchical clustering | journal=Journal of Classification | volume=37 | year=2020 | issue=3 | pages=584–597| arxiv=1906.09222 | s2cid=195317052 }}

| $\sqrt[p]{\frac{1}$

A|\cdot|B

\sum_{a \in A }\sum_{ b \in B} d(a,b)^p}, p\neq 0

|Ward linkage,{{cite journal |last=Ward |first=Joe H. |year=1963 |title=Hierarchical Grouping to Optimize an Objective Function |journal=Journal of the American Statistical Association |volume=58 |issue=301 |pages=236–244 |doi=10.2307/2282967 |jstor=2282967 |mr=0148188}} Minimum Increase of Sum of Squares (MISSQ){{Citation |last=Podani |first=János |title=New combinatorial clustering methods |date=1989 |url=https://doi.org/10.1007/978-94-009-2432-1_5 |work=Numerical syntaxonomy |pages=61–77 |editor-last=Mucina |editor-first=L. |place=Dordrecht |publisher=Springer Netherlands |language=en |doi=10.1007/978-94-009-2432-1_5 |isbn=978-94-009-2432-1 |access-date=2022-11-04 |editor2-last=Dale |editor2-first=M. B.}}

| $\frac$

A|\cdot|B

A\cup B

\lVert \mu_A - \mu_B \rVert ^2 = \sum_{x\in A\cup B} \lVert x - \mu_{A\cup B} \rVert^2 - \sum_{x\in A} \lVert x - \mu_{A} \rVert^2 - \sum_{x\in B} \lVert x - \mu_{B} \rVert^2

|Minimum Error Sum of Squares (MNSSQ)

| $\sum_{x\in A\cup B} \lVert x - \mu_{A\cup B} \rVert^2$

|Minimum Increase in Variance (MIVAR)

| $\frac{1}$

A\cup B

\sum_{x\in A\cup B} \lVert x - \mu_{A\cup B} \rVert^2 - \frac{1}

\sum_{x\in A} \lVert x - \mu_{A} \rVert^2 - \frac{1}

\sum_{x\in B} \lVert x - \mu_{B} \rVert^2

= \text{Var}(A\cup B) - \text{Var}(A) - \text{Var}(B)

|Minimum Variance (MNVAR)

| $\frac{1}$

A\cup B

\sum_{x\in A\cup B} \lVert x - \mu_{A\cup B} \rVert^2 = \text{Var}(A\cup B)

|Hausdorff linkage{{Cite journal |last1=Basalto |first1=Nicolas |last2=Bellotti |first2=Roberto |last3=De Carlo |first3=Francesco |last4=Facchi |first4=Paolo |last5=Pantaleo |first5=Ester |last6=Pascazio |first6=Saverio |date=2007-06-15 |title=Hausdorff clustering of financial time series |url=https://www.sciencedirect.com/science/article/pii/S0378437107001124 |journal=Physica A: Statistical Mechanics and Its Applications |language=en |volume=379 |issue=2 |pages=635–644 |doi=10.1016/j.physa.2007.01.011 |arxiv=physics/0504014 |bibcode=2007PhyA..379..635B |s2cid=27093582 |issn=0378-4371}}

| $\max_{x\in A\cup B} \min_{y\in A\cup B} d(x,y)$

|Minimum Sum Medoid linkage{{Cite conference |last=Schubert |first=Erich |date=2021 |title=HACAM: Hierarchical Agglomerative Clustering Around Medoids – and its Limitations |url=http://star.informatik.rwth-aachen.de/Publications/CEUR-WS/Vol-2993/paper-19.pdf |conference=LWDA’21: Lernen, Wissen, Daten, Analysen September 01–03, 2021, Munich, Germany |pages=191–204 |via=CEUR-WS}}

| $\min_{m\in A\cup B} \sum_{y\in A\cup B} d(m, y)$ such that m is the medoid of the resulting cluster

|Minimum Sum Increase Medoid linkage

| $\min_{m\in A\cup B} \sum_{y\in A\cup B} d(m,y)
- \min_{m\in A} \sum_{y\in A} d(m,y)
- \min_{m\in B} \sum_{y\in B} d(m,y)$

|Medoid linkage{{Cite conference |last1=Miyamoto |first1=Sadaaki |last2=Kaizu |first2=Yousuke |last3=Endo |first3=Yasunori |date=2016 |title=Hierarchical and Non-Hierarchical Medoid Clustering Using Asymmetric Similarity Measures |url=https://ieeexplore.ieee.org/document/7801678 |conference=2016 Joint 8th International Conference on Soft Computing and Intelligent Systems (SCIS) and 17th International Symposium on Advanced Intelligent Systems (ISIS) |pages=400–403 |doi=10.1109/SCIS-ISIS.2016.0091}}{{Cite conference |date=2016 |title=Visual Clutter Reduction through Hierarchy-based Projection of High-dimensional Labeled Data| conference=Graphics Interface |url=https://graphicsinterface.org/wp-content/uploads/gi2016-14.pdf | first1=Dominik|last1=Herr|first2=Qi|last2=Han|first3=Steffen|last3=Lohmann| first4=Thomas |last4=Ertl |access-date=2022-11-04 |website=Graphics Interface |language=en-CA |doi=10.20380/gi2016.14}}

| $d(m_A, m_B)$ where $m_A$ , $m_B$ are the medoids of the previous clusters

| Minimum energy clustering

| $\frac {2}{nm}\sum_{i,j=1}^{n,m} \|a_i- b_j\|_2 - \frac {1}{n^2}\sum_{i,j=1}^{n} \|a_i-a_j\|_2 - \frac{1}{m^2}\sum_{i,j=1}^{m} \|b_i-b_j\|_2$

Some of these can only be recomputed recursively (WPGMA, WPGMC), for many a recursive computation with Lance-Williams-equations is more efficient, while for other (Hausdorff, Medoid) the distances have to be computed with the slower full formula. Other linkage criteria include:

The probability that candidate clusters spawn from the same distribution function (V-linkage).
The product of in-degree and out-degree on a k-nearest-neighbour graph (graph degree linkage).{{Cite book|last1=Zhang|first1=Wei|last2=Wang|first2=Xiaogang|last3=Zhao|first3=Deli|last4=Tang|first4=Xiaoou|title=Computer Vision – ECCV 2012 |chapter=Graph Degree Linkage: Agglomerative Clustering on a Directed Graph |date=2012|editor-last=Fitzgibbon|editor-first=Andrew|editor2-last=Lazebnik|editor2-first=Svetlana|editor2-link= Svetlana Lazebnik |editor3-last=Perona|editor3-first=Pietro|editor4-last=Sato|editor4-first=Yoichi|editor5-last=Schmid|editor5-first=Cordelia|series=Lecture Notes in Computer Science|language=en|publisher=Springer Berlin Heidelberg|volume=7572|pages=428–441|doi=10.1007/978-3-642-33718-5_31|isbn=9783642337185|arxiv=1208.5092|bibcode=2012arXiv1208.5092Z|s2cid=14751}} See also: https://github.com/waynezhanghk/gacluster
The increment of some cluster descriptor (i.e., a quantity defined for measuring the quality of a cluster) after merging two clusters.{{cite journal |first1=W. |last1=Zhang |first2=D. |last2=Zhao |first3=X. |last3=Wang |title=Agglomerative clustering via maximum incremental path integral |journal=Pattern Recognition |volume=46 |issue=11 |pages=3056–65 |date=2013 |doi=10.1016/j.patcog.2013.04.013 |citeseerx=10.1.1.719.5355 |bibcode=2013PatRe..46.3056Z}}{{cite book |last1=Zhao |first1=D. |last2=Tang |first2=X. |chapter=Cyclizing clusters via zeta function of a graph |chapter-url= |title=NIPS'08: Proceedings of the 21st International Conference on Neural Information Processing Systems |date=2008 |isbn=9781605609492 |pages=1953–60 |publisher=Curran |citeseerx=10.1.1.945.1649}}{{cite journal |first1=Y. |last1=Ma |first2=H. |last2=Derksen |first3=W. |last3=Hong |first4=J. |last4=Wright |title=Segmentation of Multivariate Mixed Data via Lossy Data Coding and Compression |journal=IEEE Transactions on Pattern Analysis and Machine Intelligence |volume=29 |issue=9 |pages=1546–62 |date=2007 |doi=10.1109/TPAMI.2007.1085 |pmid=17627043 |hdl=2142/99597 |s2cid=4591894 |hdl-access=free }}

Agglomerative clustering example

Image:Clusters.svg

For example, suppose this data is to be clustered, and the Euclidean distance is the distance metric.

The hierarchical clustering dendrogram would be:

Image:Hierarchical clustering simple diagram.svg

Cutting the tree at a given height will give a partitioning clustering at a selected precision. In this example, cutting after the second row (from the top) of the dendrogram will yield clusters {a} {b c} {d e} {f}. Cutting after the third row will yield clusters {a} {b c} {d e f}, which is a coarser clustering, with a smaller number but larger clusters.

This method builds the hierarchy from the individual elements by progressively merging clusters. In our example, we have six elements {a} {b} {c} {d} {e} and {f}. The first step is to determine which elements to merge in a cluster. Usually, we want to take the two closest elements, according to the chosen distance.

Optionally, one can also construct a distance matrix at this stage, where the number in the i-th row j-th column is the distance between the i-th and j-th elements. Then, as clustering progresses, rows and columns are merged as the clusters are merged and the distances updated. This is a common way to implement this type of clustering, and has the benefit of caching distances between clusters. A simple agglomerative clustering algorithm is described in the single-linkage clustering page; it can easily be adapted to different types of linkage (see below).

Suppose we have merged the two closest elements b and c, we now have the following clusters {a}, {b, c}, {d}, {e} and {f}, and want to merge them further. To do that, we need to take the distance between {a} and {b c}, and therefore define the distance between two clusters.

Usually the distance between two clusters $\mathcal{A}$ and $\mathcal{B}$ is one of the following:

The maximum distance between elements of each cluster (also called complete-linkage clustering):

:: $\max \{\, d(x,y) : x \in \mathcal{A},\, y \in \mathcal{B}\,\}.$

The minimum distance between elements of each cluster (also called single-linkage clustering):

:: $\min \{\, d(x,y) : x \in \mathcal{A},\, y \in \mathcal{B} \,\}.$

The mean distance between elements of each cluster (also called average linkage clustering, used e.g. in UPGMA):

:: ${1 \over$

\mathcal{A}|\cdot|\mathcal{B}

}\sum_{x \in \mathcal{A}}\sum_{ y \in \mathcal{B}} d(x,y).

The sum of all intra-cluster variance.
The increase in variance for the cluster being merged (Ward's method)
The probability that candidate clusters spawn from the same distribution function (V-linkage).

In case of tied minimum distances, a pair is randomly chosen, thus being able to generate several structurally different dendrograms. Alternatively, all tied pairs may be joined at the same time, generating a unique dendrogram.{{cite journal | doi=10.1007/s00357-008-9004-x | last1=Fernández | first1=Alberto | last2=Gómez | first2=Sergio | title=Solving Non-uniqueness in Agglomerative Hierarchical Clustering Using Multidendrograms | journal=Journal of Classification | volume=25 | year=2008 | issue=1 | pages=43–65| arxiv=cs/0608049 | s2cid=434036 }}

One can always decide to stop clustering when there is a sufficiently small number of clusters (number criterion). Some linkages may also guarantee that agglomeration occurs at a greater distance between clusters than the previous agglomeration, and then one can stop clustering when the clusters are too far apart to be merged (distance criterion). However, this is not the case of, e.g., the centroid linkage where the so-called reversals{{cite book |first1=P. |last1=Legendre |first2=L.F.J. |last2=Legendre |chapter=Cluster Analysis §8.6 Reversals |title=Numerical Ecology |chapter-url=https://books.google.com/books?id=DKlUIQcHhOsC&pg=PA376 |date=2012 |publisher=Elsevier |isbn=978-0-444-53868-0 |pages=376–7 |edition=3rd |series=Developments in Environmental Modelling |volume=24}} (inversions, departures from ultrametricity) may occur.

Divisive clustering

The basic principle of divisive clustering was published as the DIANA (DIvisive ANAlysis clustering) algorithm.{{cite book |first1=L. |last1=Kaufman |first2=P.J. |last2=Rousseeuw |chapter=6. Divisive Analysis (Program DIANA) |title=Finding Groups in Data: An Introduction to Cluster Analysis |chapter-url=https://books.google.com/books?id=YeFQHiikNo0C&pg=PA253 |orig-year=1990 |date=2009 |publisher=Wiley |isbn=978-0-470-31748-8 |pages=253–279}} Initially, all data is in the same cluster, and the largest cluster is split until every object is separate.

Because there exist $O(2^n)$ ways of splitting each cluster, heuristics are needed. DIANA chooses the object with the maximum average dissimilarity and then moves all objects to this cluster that are more similar to the new cluster than to the remainder.

Informally, DIANA is not so much a process of "dividing" as it is of "hollowing out": each iteration, an existing cluster (e.g. the initial cluster of the entire dataset) is chosen to form a new cluster inside of it. Objects progressively move to this nested cluster, and hollow out the existing cluster. Eventually, all that's left inside a cluster is nested clusters that grew there, without it owning any loose objects by itself.

Formally, DIANA operates in the following steps:

Let $C_0 = \{1\dots n\}$ be the set of all $n$ object indices and $\mathcal{C} = \{C_0\}$ the set of all formed clusters so far.
Iterate the following until $|\mathcal{C}| = n$ :
Find the current cluster with 2 or more objects that has the largest diameter: $C_* = \arg\max_{C\in \mathcal{C}} \max_{i_1,i_2\in C} \delta(i_1,i_2)$

Find the object in this cluster with the most dissimilarity to the rest of the cluster:

i^* = \arg\max_{i\in C_*} \frac{1}

C_*

1}\sum_{j\in C_*\setminus\{i\}} \delta(i,j)

Pop

i^*

from its old cluster

C_*

and put it into a new splinter group

C_\textrm{new} = \{i^*\}

As long as

C_*

isn't empty, keep migrating objects from

C_*

to add them to

C_\textrm{new}

. To choose which objects to migrate, don't just consider dissimilarity to

C_*

, but also adjust for dissimilarity to the splinter group: let

i^* = \arg\max_{i\in C} D(i)

where we define

D(i) = \frac{1}{|C_*

1}\sum_{j\in C_*\setminus\{i\}} \delta(i,j) - \frac{1}{|C_\textrm{new}

\sum_{j\in C_\textrm{new}} \delta(i,j), then either stop iterating when

D(i^*) < 0

, or migrate

i^*

Add $C_\textrm{new}$ to $\mathcal{C}$ .

Intuitively, $D(i)$ above measures how strongly an object wants to leave its current cluster, but it is attenuated when the object wouldn't fit in the splinter group either. Such objects will likely start their own splinter group eventually.

The dendrogram of DIANA can be constructed by letting the splinter group $C_\textrm{new}$ be a child of the hollowed-out cluster $C_*$ each time. This constructs a tree with $C_0$ as its root and $n$ unique single-object clusters as its leaves.

Greedy Nature of the Algorithm

Hierarchical clustering is often described as a greedy algorithm because it makes a series of locally optimal choices without reconsidering previous steps. At each iteration, it merges the two clusters that are closest together based on a selected distance metric, always choosing the best immediate option available. This approach is "greedy" because it seeks to optimize the current decision rather than planning for the best possible overall clustering . Once two clusters are merged, the decision is final and irreversible, without the possibility of backtracking, which can lead to suboptimal results if earlier choices were not ideal. Despite this, the greedy nature of hierarchical clustering makes it computationally efficient and simple to implement, though it may not always capture the true underlying structure of complex datasets .

Limitations

Hierarchical clustering, particularly in its standard agglomerative form, presents several notable limitations: (a) Time Complexity: Hierarchical clustering, especially in its basic agglomerative form, has a high time complexity of O(n³). This becomes a significant bottleneck for large datasets, limiting its scalability {{cite journal |author=D. Defays |year=1977 |title=An efficient algorithm for a complete-link method |journal=The Computer Journal |publisher=British Computer Society |volume=20 |issue=4 |pages=364–6 |doi=10.1093/comjnl/20.4.364 |doi-access=}}. (b) Scalability: Due to the time and space complexity, hierarchical clustering algorithms struggle to handle very large datasets efficiently (c) Sensitivity to Noise and Outliers: Hierarchical clustering methods can be sensitive to noise and outliers in the data, which can lead to the formation of inaccurate or misleading cluster hierarchies {{cite journal |author=R. Sibson |year=1973 |title=SLINK: an optimally efficient algorithm for the single-link cluster method |url=http://www.cs.gsu.edu/~wkim/index_files/papers/sibson.pdf |journal=The Computer Journal |publisher=British Computer Society |volume=16 |issue=1 |pages=30–34 |doi=10.1093/comjnl/16.1.30 |doi-access=free}}. (d) Difficulty with High-Dimensional Data: In high-dimensional spaces, hierarchical clustering can face challenges due to the curse of dimensionality, where data points become sparse, and distance measures become less meaningful. This can result in poorly defined clusters{{Cite conference |last1=Herr |first1=Dominik |last2=Han |first2=Qi |last3=Lohmann |first3=Steffen |last4=Ertl |first4=Thomas |date=2016 |title=Visual Clutter Reduction through Hierarchy-based Projection of High-dimensional Labeled Data |url=https://graphicsinterface.org/wp-content/uploads/gi2016-14.pdf |conference=Graphics Interface |language=en-CA |doi=10.20380/gi2016.14 |access-date=2022-11-04 |website=Graphics Interface}}. (e) Inability to Handle Non-Convex Shapes and Varying Densities: Traditional hierarchical clustering methods, like many other clustering algorithms, often assume that clusters are convex and have similar densities. They may struggle to accurately identify clusters with non-convex shapes or varying densities {{Cite journal |last=Wani |first=Aasim Ayaz |date=2024-08-29 |title=Comprehensive analysis of clustering algorithms: exploring limitations and innovative solutions |journal=PeerJ Computer Science |language=en |volume=10 |pages=e2286 |doi=10.7717/peerj-cs.2286 |issn=2376-5992 |pmc=11419652 |pmid=39314716 |doi-access=free}}.

Software

= Open source implementations =

File:Iris dendrogram.png of the Iris dataset (using R). [https://cran.r-project.org/web/packages/dendextend/vignettes/Cluster_Analysis.html Source] ]]

File:Orange-data-mining-hierarchical-clustering.png.]]

ALGLIB implements several hierarchical clustering algorithms (single-link, complete-link, Ward) in C++ and C# with O(n²) memory and O(n³) run time.
ELKI includes multiple hierarchical clustering algorithms, various linkage strategies and also includes the efficient SLINK, CLINK and Anderberg algorithms, flexible cluster extraction from dendrograms and various other cluster analysis algorithms.
Julia has an implementation inside the Clustering.jl package.{{Cite web |title=Hierarchical Clustering · Clustering.jl |url=https://juliastats.org/Clustering.jl/stable/hclust.html |access-date=2022-02-28 |website=juliastats.org |language=en}}
Octave, the GNU analog to MATLAB implements hierarchical clustering in function "linkage".
Orange, a data mining software suite, includes hierarchical clustering with interactive dendrogram visualisation.
R has built-in functions{{Cite web |title=hclust function - RDocumentation |url=https://www.rdocumentation.org/packages/stats/versions/3.6.2/topics/hclust |access-date=2022-06-07 |website=www.rdocumentation.org |language=en}} and packages that provide functions for hierarchical clustering.{{Citation |last1=Galili |first1=Tal |title=dendextend: Extending 'dendrogram' Functionality in R|date=2021-10-28 |url=https://cran.r-project.org/package=dendextend |access-date=2022-06-07 |last2=Benjamini |first2=Yoav |last3=Simpson |first3=Gavin |last4=Jefferis |first4=Gregory}}{{Cite web|last1=Paradis|first1=Emmanuel|display-authors=etal |title=ape: Analyses of Phylogenetics and Evolution|url=https://cran.r-project.org/web/packages/ape/index.html|access-date=2022-12-28}}{{Cite web|last1=Fernández|first1=Alberto|last2=Gómez|first2=Sergio|date=2021-09-12|title=mdendro: Extended Agglomerative Hierarchical Clustering|url=https://sergio-gomez.github.io/mdendro/|access-date=2022-12-28}}
SciPy implements hierarchical clustering in Python, including the efficient SLINK algorithm.
scikit-learn also implements hierarchical clustering in Python.
Weka includes hierarchical cluster analysis.

= Commercial implementations =

MATLAB includes hierarchical cluster analysis.
SAS includes hierarchical cluster analysis in PROC CLUSTER.
Mathematica includes a Hierarchical Clustering Package.
NCSS includes hierarchical cluster analysis.
SPSS includes hierarchical cluster analysis.
Qlucore Omics Explorer includes hierarchical cluster analysis.
Stata includes hierarchical cluster analysis.
CrimeStat includes a nearest neighbor hierarchical cluster algorithm with a graphical output for a Geographic Information System.

Hierarchical clustering#Agglomerative clustering example

Complexity

Cluster Linkage

Agglomerative clustering example

Divisive clustering

Greedy Nature of the Algorithm

Limitations

Software

= Open source implementations =

= Commercial implementations =

See also

References

Further reading