Multispecies coalescent process

File:Multispecies coalescent.jpg

If we consider a rooted three-taxon tree, the simplest non-trivial phylogenetic tree, there are three different tree topologies but four possible gene trees.{{cite journal | vauthors = Hobolth A, Christensen OF, Mailund T, Schierup MH | title = Genomic relationships and speciation times of human, chimpanzee, and gorilla inferred from a coalescent hidden Markov model | journal = PLOS Genetics | volume = 3 | issue = 2 | pages = e7 | date = February 2007 | pmid = 17319744 | pmc = 1802818 | doi = 10.1371/journal.pgen.0030007 | doi-access = free }} The existence of four distinct gene trees despite the smaller number of topologies reflects the fact that there are topologically identical gene tree that differ in their coalescent times. In the type 1 tree the alleles in species A and B coalesce after the speciation event that separated the A-B lineage from the C lineage. In the type 2 tree the alleles in species A and B coalesce before the speciation event that separated the A-B lineage from the C lineage (in other words, the type 2 tree is a deep coalescence tree). The type 1 and type 2 gene trees are both congruent with the species tree. The other two gene trees differ from the species tree; the two discordant gene trees are also deep coalescence trees.

The distribution of times to coalescence is actually continuous for all of these trees. In other words, the exact coalescent time for any two loci with the same gene tree may differ. However, it is convenient to break up the trees based on whether the coalescence occurred before or after the earliest speciation event.

Given the internal branch length in coalescent units it is straightforward to calculate the probability of each gene tree.{{cite journal | vauthors = Pamilo P, Nei M | title = Relationships between gene trees and species trees | journal = Molecular Biology and Evolution | volume = 5 | issue = 5 | pages = 568–83 | date = September 1988 | pmid = 3193878 | doi = 10.1093/oxfordjournals.molbev.a040517 | doi-access = free }} For diploid organisms the branch length in coalescent units is the number of generations between the speciation events divided by twice the effective population size. Since all three of the deep coalescence tree are equiprobable and two of those deep coalescence tree are discordant it is easy to see that the probability that a rooted three-taxon gene tree will be congruent with the species tree is:

\begin{align}

P(congruence) & = 1 - \frac{2}{3} \exp(-T) = 1 - \frac{2}{3} \exp(-\frac{t}{2N_e})

\end{align}

File:Hemiplasy.jpg

Where the branch length in coalescent units (T) is also written in an alternative form: the number of generations (t) divided by twice the effective population size (N_e). Pamilo and Nei also derived the probability of congruence for rooted trees of four and five taxa as well as a general upper bound on the probability of congruence for larger trees. Rosenberg{{cite journal | vauthors = Rosenberg NA | title = The probability of topological concordance of gene trees and species trees | journal = Theoretical Population Biology | volume = 61 | issue = 2 | pages = 225–47 | date = March 2002 | pmid = 11969392 | doi = 10.1006/tpbi.2001.1568 | bibcode = 2002TPBio..61..225R }} followed up with equations used for the complete set of topologies (although the large number of distinct phylogenetic trees that becomes possible as the number of taxa increases makes these equations impractical unless the number of taxa is very limited).

The phenomenon of hemiplasy is a natural extension of the basic idea underlying gene tree-species tree discordance. If we consider the distribution of some character that disagrees with the species tree it might reflect homoplasy (multiple independent origins of the character or a single origin followed by multiple losses) or it could reflect hemiplasy (a single origin of the trait that is associated with a gene tree that disagrees with the species tree).

The phenomenon called incomplete lineage sorting (often abbreviated ILS in the scientific literatures{{cite journal | vauthors = Jarvis ED, Mirarab S, Aberer AJ, Li B, Houde P, Li C, Ho SY, Faircloth BC, Nabholz B, Howard JT, Suh A, Weber CC, da Fonseca RR, Li J, Zhang F, Li H, Zhou L, Narula N, Liu L, Ganapathy G, Boussau B, Bayzid MS, Zavidovych V, Subramanian S, Gabaldón T, Capella-Gutiérrez S, Huerta-Cepas J, Rekepalli B, Munch K, Schierup M, Lindow B, Warren WC, Ray D, Green RE, Bruford MW, Zhan X, Dixon A, Li S, Li N, Huang Y, Derryberry EP, Bertelsen MF, Sheldon FH, Brumfield RT, Mello CV, Lovell PV, Wirthlin M, Schneider MP, Prosdocimi F, Samaniego JA, Vargas Velazquez AM, Alfaro-Núñez A, Campos PF, Petersen B, Sicheritz-Ponten T, Pas A, Bailey T, Scofield P, Bunce M, Lambert DM, Zhou Q, Perelman P, Driskell AC, Shapiro B, Xiong Z, Zeng Y, Liu S, Li Z, Liu B, Wu K, Xiao J, Yinqi X, Zheng Q, Zhang Y, Yang H, Wang J, Smeds L, Rheindt FE, Braun M, Fjeldsa J, Orlando L, Barker FK, Jønsson KA, Johnson W, Koepfli KP, O'Brien S, Haussler D, Ryder OA, Rahbek C, Willerslev E, Graves GR, Glenn TC, McCormack J, Burt D, Ellegren H, Alström P, Edwards SV, Stamatakis A, Mindell DP, Cracraft J, Braun EL, Warnow T, Jun W, Gilbert MT, Zhang G | display-authors = 6 | title = Whole-genome analyses resolve early branches in the tree of life of modern birds | journal = Science | volume = 346 | issue = 6215 | pages = 1320–31 | date = December 2014 | pmid = 25504713 | pmc = 4405904 | doi = 10.1126/science.1253451 | bibcode = 2014Sci...346.1320J }}) is linked to the phenomenon. If we examine the illustration of hemiplasy with using a rooted four-taxon tree (see image to the right) the lineage between the common ancestor of taxa A, B, and C and the common ancestor of taxa A and B must be polymorphic for the allele with the derived trait (e.g., a transposable element insertion{{cite journal | vauthors = Suh A, Smeds L, Ellegren H | title = The Dynamics of Incomplete Lineage Sorting across the Ancient Adaptive Radiation of Neoavian Birds | journal = PLOS Biology | volume = 13 | issue = 8 | pages = e1002224 | date = August 2015 | pmid = 26284513 | pmc = 4540587 | doi = 10.1371/journal.pbio.1002224 | veditors = Penny D | doi-access = free }}) and the allele with the ancestral trait. The concept of incomplete lineage sorting ultimately reflects on persistence of polymorphisms across one or more speciation events.

The probability density of the gene trees under the multispecies coalescent model is discussed along with its use for parameter estimation using multi-locus sequence data.

In the basic multispecies coalescent model, the species phylogeny is assumed to be known. Complete isolation after species divergence, with no migration, hybridization, or introgression is also assumed. We assume no recombination so that all the sites within the locus share the same gene tree (topology and coalescent times). However, the basic model can be extended in different ways to accommodate migration or introgression, population size changes, recombination.{{Cite web|url=https://academic.oup.com/sysbio/article/67/5/786/5017269|title = Modeling Hybridization Under the Network Multispecies Coalescent}}{{Cite book|chapter-url=https://hal.archives-ouvertes.fr/hal-02535622|title=Phylogenetics in the Genomic Era|chapter=The Multi-species Coalescent Model and Species Tree Inference|year=2020|publisher=No commercial publisher {{!}} Authors open access book }} [https://hal.inria.fr/PGE Authors open access book.]

The model and implementation of this method can be applied to any species tree. As an example, the species tree of the great apes: humans (H), chimpanzees (C), gorillas (G) and orangutans (O) is considered. The topology of the species tree, {{not a typo|(((HC)G)O))}}, is assumed known and fixed in the analysis (Figure 1). Let $D=\backslash \{D\_i\backslash \}$ be the entire data set, where $\{D\_i\}$ represent the sequence alignment at locus $i$, with $i=1,2,\backslash ldots\; ,L$ for a total of $L$ loci.

The population size of a current species is considered only if more than one individual is sampled from that species at some loci.

The parameters in the model for the example of Figure 1 include the three divergence times $\backslash tau\_\{HC\}$, $\backslash tau\_\{HCG\}$ and $\backslash tau\_\{HCGO\}$ and population size parameters $\backslash theta\_\{H\}$ for humans; $\backslash theta\_\{C\}$ for chimpanzees; and $\backslash theta\_\{HC\}$, $\backslash theta\_\{HCG\}$ and $\backslash theta\_\{HCGO\}$ for the three ancestral species.

The divergence times ($\backslash tau$'s) are measured by the expected number of mutations per site from the ancestral node in the species tree to the present time (Figure 1 of Rannala and Yang, 2003).

Therefore, the parameters are $\backslash Theta=\backslash \{\backslash theta\_H,\backslash theta\_C,\backslash theta\_\{HC\},\backslash theta\_\{HCG\},\backslash theta\_\{HCGO\},\backslash tau\_\{HC\},\backslash tau\_\{HCG\},\backslash tau\_\{HCGO\}\backslash \}$.

The joint distribution of $f(T\_i,t\_i\backslash mid\backslash Theta)$ is derived directly in this section. Two sequences from different species can coalesce only in one populations that are ancestral to the two species. For example, sequences H and G can coalesce in populations HCG or HCGO, but not in populations H or HC. The coalescent processes in different populations are different.

For each population, the genealogy is traced backward in time, until the end of the population at time $\backslash tau$, and the number of lineages $(m)$ entering the population and the number of lineages leaving it $(n)$ are recorded. For example, $m=3,n=2,$ and $\backslash tau=\backslash tau\_\{HC\}$, for population H (Table 1). This process is called a censored coalescent process because the coalescent process for one population may be terminated before all lineages that entered the population have coalesced. If $n\; \backslash geq\; 1$ the population consists of $n$ disconnected subtrees or lineages.

With one time unit defined as the time taken to accumulate one mutation per site, any two lineages coalesce at the rate $\backslash frac\{2\}\{\backslash theta\}$. The waiting time $t\_j$ until the next coalescent event, which reduces the number of lineages from $j$ to $j-1$ has exponential density

:$f(t\_j)=\backslash frac\{j(j-1)\}\{2\}\; \backslash frac\{2\}\{\backslash theta\}\; \backslash exp\backslash \{-\backslash frac\{j(j-1)\}\{2\}\; \backslash frac\{2\}\{\backslash theta\}t\_j\backslash \}\; ,\backslash quad\; j=m,m-1,\backslash ldots,n+1$

If $n\backslash geq\; 1$, the probability that no coalescent event occurs between the last one and the end of the population at time $\backslash tau$; i.e. during the time interval $\backslash tau-(t\_m+t\_\{m-1\}+\backslash ldots+t\_\{n+1\})$. This probability is $\backslash exp\backslash \{-\backslash frac\{n(n-1)\}\{\backslash theta\}[\backslash tau-(t\_m+t\_\{m-1\}+\backslash ldots+t\_\{n+1\})]$ and is 1 if $n=1$.

(Note: One should recall that the probability of no events over time interval $t$ for a Poisson process with rate $\backslash lambda$ is $e^\{-\backslash lambda\; t\}$. Here the coalescent rate when there are $n$ lineages is $\backslash lambda\; =\; \backslash frac\{n(n-1)\}\{\backslash theta\}$.)

In addition, to derive the probability of a particular gene tree topology in the population, if a coalescent event occurs in a sample of $j$ lineages, the probability that a particular pair of lineages coalesce is $1/\backslash binom\{j\}\{2\}=2/j(j-1),\backslash quad\; j=m,m-1,\backslash ldots,n+1$.

Multiplying these probabilities together, the joint probability distribution of the gene tree topology in the population and its coalescent times $t\_m,t\_\{m+1\},\backslash ldots,t\_\{n+1\}$ as

:$\backslash prod\_\{j=n+1\}^\{m\}\; \backslash Big[\backslash frac\{2\}\{\backslash theta\}\; \backslash exp\backslash Big\backslash \{-\backslash frac\{j(j-1)\}\{\backslash theta\}t\_j\backslash Big\backslash \}\backslash Big]\; \backslash exp\backslash Big\backslash \{-\backslash frac\{n(n-1)\}\{\backslash theta\}(\backslash tau-(t\_m+t\_\{m+1\}+\backslash ldots+t\_\{n+1\}))\backslash Big\backslash \}$.

The probability of the gene tree and coalescent times for the locus is the product of such probabilities across all the populations. Therefore, the gene genealogy of Figure 1,{{Cite book|title=Molecular evolution : a statistical approach| first = Ziheng | last = Yang | name-list-style = vanc |publisher= Oxford University Press |year= 2014 |isbn=9780199602605|edition=First|location=Oxford|pages=Chapter 9|oclc=869346345}} we have

\begin{align}

f(G_i\mid \Theta) & =[2/\theta_H\exp\{-6t^{(H)}_3/\theta_H\}\exp\{-2(\tau_{HC}-t^{(H)}_3)/\theta_{H}\}] \\

& {} \times [2/\theta_{C}\exp\{-2t^{(C)}_2/\theta_{C}\}]\\

& {} \times [2/\theta_{HC}\exp\{-6t^{HC}_3/\theta_{HC}\}] \times [2/\theta_{HC}\exp\{-2t^{HC}_2/\theta_{HC}\}]\\

& {} \times [\exp\{-2(\tau_{HCG}-\tau_{HG}-(t^{HC}_3+t^{HC}_2))/\theta_{HCG}\}] \\

& {} \times [2/\theta_{HCGO}\exp\{-6t^{HCGO}_3/\theta_{HCGO}\}] \times [2/\theta_{HCGO}\exp\{-2t^{HCGO}_2/\theta_{HCGO}\}]

\end{align}

The gene genealogy $G\_i$ at each locus $i$ is represented by the tree topology $T\_i$ and the coalescent times $t\_i$. Given the species tree and the parameters $\backslash Theta$ on it, the probability distribution of $G\_i=\backslash \{T\_i,t\_i\backslash \}$ is specified by the coalescent process as

:$f(G\backslash mid\backslash Theta)=\backslash prod\_i\; f(G\_i\backslash mid\backslash Theta)=\backslash prod\_i\; f(T\_i,t\_i\backslash mid\backslash Theta)$,

where $f(G\_i\backslash mid\backslash Theta)=f(T\_i,t\_i\backslash mid\backslash Theta)$ is the probability density for the gene tree at locus locus $i$, and the product is because we assume that the gene trees are independent given the parameters.

The probability of data $D\_i$ given the gene tree and coalescent times (and thus branch lengths) at the locus, $f(D\_i\backslash mid\; G\_i)$, is Felsenstein's phylogenetic likelihood.{{cite journal|vauthors=Felsenstein J|year=1981|title=Evolutionary trees from DNA sequences: a maximum likelihood approach|journal=Journal of Molecular Evolution|volume=17|issue=6|pages=368–76|doi=10.1007/BF01734359|pmid=7288891|bibcode=1981JMolE..17..368F|s2cid=8024924}} Due to the assumption of independent evolution across the loci,

:$f(D\backslash mid\; G)=\backslash prod\_i\; f(D\_i\backslash mid\; G\_i)$

The likelihood function or the probability of the sequence data given the parameters $\backslash Theta$ is then an average over the unobserved gene trees

:$f(D\; \backslash mid\; \backslash Theta)\; =\; \backslash int\; f(D\backslash mid\; G)\; f(G\backslash mid\; \backslash Theta)\; dG,$

where the integration represents summation over all possible gene tree topologies ($T\_i$) and, for each possible topology at each locus, integration over the coalescent times $t\_i$.{{cite journal|vauthors=Xu B, Yang Z|date=December 2016|title=Challenges in Species Tree Estimation Under the Multispecies Coalescent Model|journal=Genetics|volume=204|issue=4|pages=1353–1368|doi=10.1534/genetics.116.190173|pmc=5161269|pmid=27927902}} This is in general intractable except for very small species trees.

In Bayesian inference, we assign a prior on the parameters, $f(\backslash Theta)$, and then the posterior is given as

:$f(\backslash Theta\backslash mid\; D)=\; \backslash int\; f(\backslash Theta,\; G\backslash mid\; D)\; dG,$

where again the integration represents summation over all possible gene tree topologies ($T\_i$) and integration over the coalescent times $t\_i$. In practice this integration over the gene trees is achieved through a Markov chain Monte Carlo algorithm, which samples from the joint conditional distribution of the parameters and the gene trees

:$f(\backslash Theta,\; G\backslash mid\; D)\; \backslash propto\; f(D\backslash mid\; G)\; f(G\backslash mid\; \backslash Theta)\; f(\backslash Theta).$

The above assumes that the species tree is fixed. In species-tree estimation, the species tree ($S$) changes as well, so that the joint conditional distribution (from which the MCMC samples) is

:$f(S,\; \backslash Theta,\; G\backslash mid\; D)\; \backslash propto\; f(D\backslash mid\; G)\; f(G\backslash mid\; S,\backslash Theta)\; f(\backslash Theta)f(S),$

where $f(S)$ is the prior on species trees.

As a major departure from two-step summary methods, full-likelihood methods average over the gene trees. This means that they make use of information in the branch lengths (coalescent times) on the gene trees and accommodate their uncertainties (due to limited sequence length in the alignments) at the same time. It also explains why full-likelihood methods are computationally much more demanding than two-step summary methods.

The integration or summation over the gene trees in the definition of the likelihood function above is virtually impossible to compute except for very small species trees with only two or three species.{{Cite journal|last=Yang|first=Ziheng|date=2002-12-01|title=Likelihood and Bayes Estimation of Ancestral Population Sizes in Hominoids Using Data From Multiple Loci|url=https://www.genetics.org/content/162/4/1811|journal=Genetics|language=en|volume=162|issue=4|pages=1811–1823|doi=10.1093/genetics/162.4.1811|issn=0016-6731|pmid=12524351|pmc=1462394}} Full-likelihood or full-data methods, based on calculation of the likelihood function on sequence alignments, have thus mostly relied on Markov chain Monte Carlo algorithms. MCMC algorithms under the multispecies coalescent model are similar to those used in Bayesian phylogenetics but are distinctly more complex, mainly due to the fact that the gene trees at multiple loci and the species tree have to be compatible: sequence divergence has to be older than species divergence. As a result, changing the species tree while the gene trees are fixed (or changing a gene tree while the species tree is fixed) leads to inefficient algorithms with poor mixing properties. Considerable efforts have been taken to design smart algorithms that change the species tree and gene trees in a coordinated manner, as in the rubber-band algorithm for changing species divergence times, the coordinated NNI, SPR and NodeSlider moves.{{Cite journal|last1=Yang|first1=Z.|last2=Rannala|first2=B.|date=2014-12-01|title=Unguided Species Delimitation Using DNA Sequence Data from Multiple Loci|url= |journal=Molecular Biology and Evolution|language=en|volume=31|issue=12|pages=3125–3135|doi=10.1093/molbev/msu279|issn=0737-4038|pmc=4245825|pmid=25274273}}{{Cite journal|last1=Rannala|first1=Bruce|last2=Yang|first2=Ziheng|date=2017-01-04|title=Efficient Bayesian species tree inference under the multispecies coalescent|journal=Systematic Biology|volume=66|issue=5|language=en|pages=823–842|doi=10.1093/sysbio/syw119|pmid=28053140|pmc=8562347|issn=1063-5157|doi-access=free}}

Consider for example the case of two species (A and B) and two sequences at each locus, with a sequence divergence time $t\_i$ at locus $i$. We have for all $i$. When we want to change the species divergence time $\backslash tau$ within the constraint of the current $t\_i$, we may have very little room for change, as $\backslash tau$ may be virtually identical to the smallest of the $t\_i$. The rubber-band algorithm changes $\backslash tau$ without consideration of the $t\_i$, and then modifies the $t\_i$ deterministically in the same way that marks on a rubber band move when the rubber band is held from a fixed point pulled towards one end. In general, the rubber-band move guarantees that the ages of nodes in the gene trees are modified so that they remain compatible with the modified species divergence time.

Full likelihood methods tend to reach their limit when the data consist of a few hundred loci, even though more than 10,000 loci have been analyzed in a few published studies.{{Cite journal|last1=Shi|first1=Cheng-Min|last2=Yang|first2=Ziheng|date=2018-01-01|title=Coalescent-Based Analyses of Genomic Sequence Data Provide a Robust Resolution of Phylogenetic Relationships among Major Groups of Gibbons|url= |journal=Molecular Biology and Evolution|language=en|volume=35|issue=1|pages=159–179|doi=10.1093/molbev/msx277|issn=0737-4038|pmc=5850733|pmid=29087487}}{{Cite journal|last1=Thawornwattana|first1=Yuttapong|last2=Dalquen|first2=Daniel|last3=Yang|first3=Ziheng|date=2018-10-01|editor-last=Tamura|editor-first=Koichiro|title=Coalescent Analysis of Phylogenomic Data Confidently Resolves the Species Relationships in the Anopheles gambiae Species Complex|url= |journal=Molecular Biology and Evolution|language=en|volume=35|issue=10|pages=2512–2527|doi=10.1093/molbev/msy158|issn=0737-4038|pmc=6188554|pmid=30102363}}

The basic multispecies coalescent model can be extended in a number of ways to accommodate major factors of the biological process of reproduction and drift.

For example, incorporating continuous-time migration leads to the MSC+M (for MSC with migration) model, also known as the isolation-with-migration or IM models.{{Cite journal|last=Hey|first=Jody|date=April 2010|title=Isolation with Migration Models for More Than Two Populations|url= |journal=Molecular Biology and Evolution|language=en|volume=27|issue=4|pages=905–920|doi=10.1093/molbev/msp296|issn=1537-1719|pmc=2877539|pmid=19955477}}{{Cite journal|last1=Zhu|first1=T.|last2=Yang|first2=Z.|date=2012-10-01|title=Maximum Likelihood Implementation of an Isolation-with-Migration Model with Three Species for Testing Speciation with Gene Flow|url=https://academic.oup.com/mbe/article-lookup/doi/10.1093/molbev/mss118|journal=Molecular Biology and Evolution|language=en|volume=29|issue=10|pages=3131–3142|doi=10.1093/molbev/mss118|pmid=22504520|issn=0737-4038|doi-access=free}} Incorporating episodic hybridization/introgression leads to the MSC with introgression (MSci){{Cite journal|last1=Flouri|first1=Tomáš|last2=Jiao|first2=Xiyun|last3=Rannala|first3=Bruce|last4=Yang|first4=Ziheng|date=2020-04-01|editor-last=Rosenberg|editor-first=Michael|title=A Bayesian Implementation of the Multispecies Coalescent Model with Introgression for Phylogenomic Analysis|url= |journal=Molecular Biology and Evolution|language=en|volume=37|issue=4|pages=1211–1223|doi=10.1093/molbev/msz296|issn=0737-4038|pmc=7086182|pmid=31825513}} or multispecies-network-coalescent (MSNC) model.{{Cite journal|last1=Wen|first1=Dingqiao|last2=Nakhleh|first2=Luay|date=2018-05-01|editor-last=Kubatko|editor-first=Laura|title=Coestimating Reticulate Phylogenies and Gene Trees from Multilocus Sequence Data|url=https://academic.oup.com/sysbio/article/67/3/439/4568576|journal=Systematic Biology|language=en|volume=67|issue=3|pages=439–457|doi=10.1093/sysbio/syx085|pmid=29088409|issn=1063-5157|doi-access=free}}{{Cite journal|last1=Zhang|first1=Chi|last2=Ogilvie|first2=Huw A|last3=Drummond|first3=Alexei J|last4=Stadler|first4=Tanja|date=2018-02-01|title=Bayesian Inference of Species Networks from Multilocus Sequence Data|url= |journal=Molecular Biology and Evolution|language=en|volume=35|issue=2|pages=504–517|doi=10.1093/molbev/msx307|issn=0737-4038|pmc=5850812|pmid=29220490}}

The multispecies coalescent has profound implications for the theory and practice of molecular phylogenetics.{{Cite journal|last=Maddison|first=Wayne P. | name-list-style = vanc |date=1997-09-01|title=Gene Trees in Species Trees |journal=Systematic Biology|language=en|volume=46|issue=3|pages=523–536|doi=10.1093/sysbio/46.3.523|issn=1063-5157|doi-access=free}}{{cite journal | vauthors = Edwards SV | title = Is a new and general theory of molecular systematics emerging? | journal = Evolution; International Journal of Organic Evolution | volume = 63 | issue = 1 | pages = 1–19 | date = January 2009 | pmid = 19146594 | doi = 10.1111/j.1558-5646.2008.00549.x | doi-access = free }} Since individual gene trees can differ from the species tree one cannot estimate the tree for a single locus and assume that the gene tree correspond the species tree. In fact, one can be virtually certain that any individual gene tree will differ from the species tree for at least some relationships when any reasonable number of taxa are considered. However, gene tree-species tree discordance has an impact on the theory and practice of species tree estimation that goes beyond the simple observation that one cannot use a single gene tree to estimate the species tree because there is a part of parameter space where the most frequent gene tree is incongruent with the species tree. This part of parameter space is called the anomaly zone{{cite journal | vauthors = Degnan JH, Rosenberg NA | title = Discordance of species trees with their most likely gene trees | journal = PLOS Genetics | volume = 2 | issue = 5 | pages = e68 | date = May 2006 | pmid = 16733550 | pmc = 1464820 | doi = 10.1371/journal.pgen.0020068 | veditors = Wakeley J | doi-access = free }} and any discordant gene trees that are more expected to arise more often than the gene tree. that matches the species tree are called anomalous gene trees.

The existence of the anomaly zone implies that one cannot simply estimate a large number of gene trees and assume the gene tree recovered the largest number of times is the species tree. Of course, estimating the species tree by a "democratic vote" of gene trees would only work for a limited number of taxa outside of the anomaly zone given the extremely large number of phylogenetic trees that are possible.{{Cite journal|last=Felsenstein|first=Joseph | name-list-style = vanc |date=March 1978|title=The Number of Evolutionary Trees |journal=Systematic Zoology|volume=27|issue=1|pages=27–33|doi=10.2307/2412810|jstor=2412810 }} However, the existence of the anomalous gene trees also means that simple methods for combining gene trees, like the majority rule extended ("greedy") consensus method or the matrix representation with parsimony (MRP) supertree{{Cite journal|last=Baum|first=Bernard R. | name-list-style = vanc |date=February 1992|title=Combining trees as a way of combining data sets for phylogenetic inference, and the desirability of combining gene trees |journal=Taxon|language=en|volume=41|issue=1|pages=3–10|doi=10.2307/1222480|jstor=1222480 |issn=0040-0262}}{{Cite journal|last=Ragan|first=Mark A. | name-list-style = vanc |date=March 1992|title=Phylogenetic inference based on matrix representation of trees |journal=Molecular Phylogenetics and Evolution|language=en|volume=1|issue=1|pages=53–58|doi=10.1016/1055-7903(92)90035-F|pmid=1342924 |bibcode=1992MolPE...1...53R }} approach, will not be consistent estimators of the species tree{{cite journal | vauthors = Degnan JH, DeGiorgio M, Bryant D, Rosenberg NA | title = Properties of consensus methods for inferring species trees from gene trees | journal = Systematic Biology | volume = 58 | issue = 1 | pages = 35–54 | date = February 2009 | pmid = 20525567 | pmc = 2909780 | doi = 10.1093/sysbio/syp008 }}{{Cite journal|last1=Wang|first1=Yuancheng|last2=Degnan|first2=James H. | name-list-style = vanc |date=2011-05-02|title=Performance of Matrix Representation with Parsimony for Inferring Species from Gene Trees |journal=Statistical Applications in Genetics and Molecular Biology |volume=10 |issue=1 |doi=10.2202/1544-6115.1611 |s2cid=199663909}} (i.e., they will be misleading). Simply generating the majority-rule consensus tree for the gene trees, where groups that are present in at least 50% of gene trees are retained, will not be misleading as long as a sufficient number of gene trees are used. However, this ability of the majority-rule consensus tree for a set of gene trees to avoid incorrect clades comes at the cost of having unresolved groups.

Simulations have shown that there are parts of species tree parameter space where maximum likelihood estimates of phylogeny are incorrect trees with increasing probability as the amount of data analyzed increases.{{cite journal | vauthors = Kubatko LS, Degnan JH | title = Inconsistency of phylogenetic estimates from concatenated data under coalescence | journal = Systematic Biology | volume = 56 | issue = 1 | pages = 17–24 | date = February 2007 | pmid = 17366134 | doi = 10.1080/10635150601146041 | veditors = Collins T | doi-access = free }} This is important because the "concatenation approach," where multiple sequence alignments from different loci are concatenated to form a single large supermatrix alignment that is then used for maximum likelihood (or Bayesian MCMC) analysis, is both easy to implement and commonly used in empirical studies. This represents a case of model misspecification because the concatenation approach implicitly assumes that all gene trees have the same topology.{{cite journal | vauthors = Warnow T | title = Concatenation Analyses in the Presence of Incomplete Lineage Sorting | journal = PLOS Currents | volume = 7 | date = May 2015 | pmid = 26064786 | pmc = 4450984 | doi = 10.1371/currents.tol.8d41ac0f13d1abedf4c4a59f5d17b1f7 | doi-access = free }} Indeed, it has now been proven that analyses of data generated under the multispecies coalescent using maximum likelihood analysis of a concatenated data are not guaranteed to converge on the true species tree as the number of loci used for the analysis increases{{cite journal | vauthors = Roch S, Steel M | title = Likelihood-based tree reconstruction on a concatenation of aligned sequence data sets can be statistically inconsistent | journal = Theoretical Population Biology | volume = 100C | pages = 56–62 | date = March 2015 | pmid = 25545843 | doi = 10.1016/j.tpb.2014.12.005 | arxiv = 1409.2051 | bibcode = 2015TPBio.100...56R }}{{cite journal | vauthors = Mendes FK, Hahn MW | title = Why Concatenation Fails Near the Anomaly Zone | journal = Systematic Biology | volume = 67 | issue = 1 | pages = 158–169 | date = January 2018 | pmid = 28973673 | doi = 10.1093/sysbio/syx063 | doi-access = free }}{{cite journal | vauthors = Roch S, Nute M, Warnow T | title = Long-Branch Attraction in Species Tree Estimation: Inconsistency of Partitioned Likelihood and Topology-Based Summary Methods | journal = Systematic Biology | volume = 68 | issue = 2 | pages = 281–297 | date = March 2019 | pmid = 30247732 | doi = 10.1093/sysbio/syy061 | veditors = Kubatko L | arxiv = 1803.02800 }} (i.e., maximum likelihood concatenation is statistically inconsistent).

There are two basic approaches for phylogenetic estimation in the multispecies coalescent framework: 1) full-likelihood or full-data methods which operate on multilocus sequence alignments directly, including both maximum likelihood and Bayesian methods, and 2) summary methods, which use a summary of the original sequence data, including the two-step methods that use estimated gene trees as summary input and SVDQuartets, which use site pattern counts pooled over loci as summary input.

{{Reflist}}

Category:Statistical genetics

Category:Statistical inference

Category:Population genetics

Category:Phylogenetics