Kaplan–Meier estimator
{{Short description|Non-parametric statistic used to estimate the survival function}}
{{Use mdy dates|date=August 2022}}
The Kaplan–Meier estimator,{{cite journal |last1=Kaplan |first1=E. L. |last2=Meier |first2=P. |title=Nonparametric estimation from incomplete observations |journal=J. Amer. Statist. Assoc. |volume=53 |issue=282 |pages=457–481 |year=1958 |jstor=2281868 |doi=10.2307/2281868}}Kaplan, E.L. in a retrospective on the seminal paper in "This week's citation classic". Current Contents 24, 14 (1983). [http://www.garfield.library.upenn.edu/classics1983/A1983QS51100001.pdf Available from UPenn as PDF.] {{Webarchive|url=https://web.archive.org/web/20160412025025/http://garfield.library.upenn.edu/classics1983/A1983QS51100001.pdf |date=April 12, 2016 }} also known as the product limit estimator, is a non-parametric statistic used to estimate the survival function from lifetime data. In medical research, it is often used to measure the fraction of patients living for a certain amount of time after treatment. In other fields, Kaplan–Meier estimators may be used to measure the length of time people remain unemployed after a job loss,{{Cite journal | last1 = Meyer | first1 = Bruce D. | year = 1990 | title = Unemployment Insurance and Unemployment Spells | journal = Econometrica | volume = 58 | issue = 4 | pages = 757–782 | doi = 10.2307/2938349| jstor = 2938349 | s2cid = 154632727 | url = http://www.nber.org/papers/w2546.pdf }} the time-to-failure of machine parts, or how long fleshy fruits remain on plants before they are removed by frugivores. The estimator is named after Edward L. Kaplan and Paul Meier, who each submitted similar manuscripts to the Journal of the American Statistical Association.{{cite journal |last1=Stalpers |first1=Lukas J A |last2=Kaplan |first2=Edward L |title=Edward L. Kaplan and the Kaplan-Meier Survival Curve |journal=BSHM Bulletin: Journal of the British Society for the History of Mathematics |date=4 May 2018 |volume=33 |issue=2 |pages=109–135 |doi=10.1080/17498430.2018.1450055 |s2cid=125941631 |doi-access=free }} The journal editor, John Tukey, convinced them to combine their work into one paper, which has been cited more than 34,000 times since its publication in 1958.{{cite journal |title=Nonparametric Estimation from Incomplete Observations |journal=Journal of the American Statistical Association |year=1958 |volume=53 |issue=282 |pages=457–481 |doi=10.1080/01621459.1958.10501452 |url=https://www.tandfonline.com/doi/abs/10.1080/01621459.1958.10501452 |access-date=27 February 2023|last1=Kaplan |first1=E. L. |last2=Meier |first2=Paul }}{{cite news |url=http://articles.chicagotribune.com/2011-08-18/news/ct-met-meier-obit-20110818_1_clinical-trials-research-experimental-treatment |archive-url=https://web.archive.org/web/20170913190451/http://articles.chicagotribune.com/2011-08-18/news/ct-met-meier-obit-20110818_1_clinical-trials-research-experimental-treatment |url-status=dead |archive-date=2017-09-13 |title=Paul Meier, 1924–2011 |newspaper=Chicago Tribune |date=August 18, 2011 }}
The estimator of the survival function (the probability that life is longer than ) is given by:
:
with a time when at least one event happened, di the number of events (e.g., deaths) that happened at time , and the individuals known to have survived (have not yet had an event or been censored) up to time .
Basic concepts
A plot of the Kaplan–Meier estimator is a series of declining horizontal steps which, with a large enough sample size, approaches the true survival function for that population. The value of the survival function between successive distinct sampled observations ("clicks") is assumed to be constant.
An important advantage of the Kaplan–Meier curve is that the method can take into account some types of censored data, particularly right-censoring, which occurs if a patient withdraws from a study, is lost to follow-up, or is alive without event occurrence at last follow-up. On the plot, small vertical tick-marks state individual patients whose survival times have been right-censored. When no truncation or censoring occurs, the Kaplan–Meier curve is the complement of the empirical distribution function.
In medical statistics, a typical application might involve grouping patients into categories, for instance, those with Gene A profile and those with Gene B profile. In the graph, patients with Gene B die much quicker than those with Gene A. After two years, about 80% of the Gene A patients survive, but less than half of patients with Gene B.
To generate a Kaplan–Meier estimator, at least two pieces of data are required for each patient (or each subject): the status at last observation (event occurrence or right-censored), and the time to event (or time to censoring). If the survival functions between two or more groups are to be compared, then a third piece of data is required: the group assignment of each subject.{{cite journal |last1=Rich |first1=Jason T. |last2=Neely |first2=J. Gail |last3=Paniello |first3=Randal C. |last4=Voelker |first4=Courtney C. J. |last5=Nussenbaum |first5=Brian |last6=Wang |first6=Eric W. |title=A practical guide to understanding Kaplan-Meier curves |journal=Otolaryngology–Head and Neck Surgery |date=September 2010 |volume=143 |issue=3 |pages=331–336 |doi=10.1016/j.otohns.2010.05.007 |pmid=20723767 |pmc=3932959 }}
Problem definition
Let be a random variable as the time that passes between the start of the possible exposure period, , and the time that the event of interest takes place, . As indicated above, the goal is to estimate the survival function underlying . Recall that this function is defined as
:, where is the time.
Let be independent, identically distributed random variables, whose common distribution is that of : is the random time when some event happened. The data available for estimating is not , but the list of pairs where for , is a fixed, deterministic integer, the censoring time of event and . In particular, the information available about the timing of event is whether the event happened before the fixed time and if so, then the actual time of the event is also available. The challenge is to estimate given this data.
Derivation of the Kaplan–Meier estimator
Two derivations of the Kaplan–Meier estimator are shown. Both are based on rewriting the survival function in terms of what is sometimes called hazard, or mortality rates. However, before doing this it is worthwhile to consider a naive estimator.
= A naive estimator =
To understand the power of the Kaplan–Meier estimator, it is worthwhile to first describe a naive estimator of the survival function.
Fix and let . A basic argument shows that the following proposition holds:
:Proposition 1: If the censoring time of event exceeds (), then if and only if .
Let be such that . It follows from the above proposition that
:
Let
X_k = \mathbb{I}(\tilde \tau_k\ge t)
and consider only those
k\in C(t) := \{ k \, :\, c_k \ge t\}
, i.e. the events for which the outcome was not censored before time . Let
m(t)=|C(t)|
be the number of elements in
C(t)
. Note that the set
C(t)
is not random and so neither is
m(t)
. Furthermore,
(X_k)_{k\in C(t)}
is a sequence of independent, identically distributed Bernoulli random variables with common parameter
S(t)=\operatorname{Prob}(\tau\ge t)
. Assuming that
m(t)>0
, this suggests to estimate using
:
\hat S_\text{naive}(t)
= \frac{1}{m(t)} \sum_{k:c_k\ge t} X_k
= \frac
\{1\le k \le n\,:\, \tilde \tau_k\ge t\} |
\{1\le k \le n\,:\, c_k\ge t\} |
= \frac
\{1\le k \le n\,:\, \tilde \tau_k\ge t\} |
where the second equality follows because implies , while the last equality is simply a change of notation.
The quality of this estimate is governed by the size of . This can be problematic when is small, which happens, by definition, when a lot of the events are censored. A particularly unpleasant property of this estimator, that suggests that perhaps it is not the "best" estimator, is that it ignores all the observations whose censoring time precedes . Intuitively, these observations still contain information about : For example, when for many events with ,
= The plug-in approach =
By elementary calculations,
:
S(t) & = \operatorname{Prob}(\tau > t\mid\tau > t-1)\operatorname{Prob}(\tau > t-1) \\[4pt]
& = (1-\operatorname{Prob}(\tau\le t\mid\tau > t-1)) \operatorname{Prob}(\tau > t-1)\\[4pt]
& = (1-\operatorname{Prob}(\tau=t\mid\tau \ge t)) \operatorname{Prob}(\tau > t-1) \\[4pt]
& = q(t) S(t-1)\,,
\end{align}
where the second to last equality used that
:
By a recursive expansion of the equality
:
Note that here
The Kaplan–Meier estimator can be seen as a "plug-in estimator" where each
is estimated based on the data and the estimator of
is obtained as a product of these estimates.
It remains to specify how
k\in [n] such that
:
By a similar reasoning that lead to the construction of the naive estimator above, we arrive at the estimator
:
= 1 - \frac
\{1\le k\le n\,:\, c_k\ge s, \tilde \tau_k=s\} |
\{1\le k \le n\,:\, c_k\ge s, \tilde \tau_k\ge s\} |
= 1 - \frac
\{1\le k\le n\,:\,\tilde \tau_k=s\} |
\{1\le k \le n\,:\, \tilde \tau_k\ge s\} |
(think of estimating the numerator and denominator separately in the definition of the "hazard rate"
:
The form of the estimator stated at the beginning of the article can be obtained by some further algebra. For this, write
Note that if
:
As opposed to the naive estimator, this estimator can be seen to use the available information more effectively: In the special case mentioned beforehand, when there are many early events recorded, the estimator will multiply many terms with a value below one and will thus take into account that the survival probability cannot be large.
= Derivation as a maximum likelihood estimator =
Kaplan–Meier estimator can be derived from maximum likelihood estimation of the discrete hazard function.{{Cite web|url=https://web.stanford.edu/~lutian/coursepdf/STAT331unit3.pdf|title=STAT331 Unit 3, Kaplan-Meier (KM) Estimator|date=January 13, 2014|first=Lu|last=Tian|publisher=Stanford University|accessdate=12 May 2023}} More specifically given
:
and the likelihood function for the hazard function up to time
:
therefore the log likelihood will be:
:
finding the maximum of log likelihood with respect to
:
where hat is used to denote maximum likelihood estimation. Given this result, we can write:
:
More generally (for continuous as well as discrete survival distributions), the Kaplan-Meier estimator may be interpreted as a nonparametric maximum likelihood estimator.{{cite book |last1=Andersen |first1=Per Kragh |last2=Borgan |first2=Ornulf |last3=Gill |first3=Richard D. |last4=Keiding |first4=Niels |title=Statistical models based on counting processes |date=1993 |publisher=Springer-Verlag |location=New York |isbn=0-387-97872-0}}
Benefits and limitations
The Kaplan–Meier estimator is one of the most frequently used methods of survival analysis. The estimate may be useful to examine recovery rates, the probability of death, and the effectiveness of treatment. It is limited in its ability to estimate survival adjusted for covariates; parametric survival models and the Cox proportional hazards model may be useful to estimate covariate-adjusted survival.
The Kaplan-Meier estimator is directly related to the Nelson-Aalen estimator and both maximize the empirical likelihood.Zhou, M. (2015). Empirical Likelihood Method in Survival Analysis (1st ed.). Chapman and Hall/CRC. https://doi.org/10.1201/b18598, https://books.google.com/books?id=9-b5CQAAQBAJ&dq=Does+the+Nelson%E2%80%93Aalen+estimator+construct+an+empirical+likelihood%3F&pg=PA7
Statistical considerations
The Kaplan–Meier estimator is a statistic, and several estimators are used to approximate its variance. One of the most common estimators is Greenwood's formula:{{cite book |last1=Greenwood |first1=Major |date=1926 |title=A report on the natural duration of cancer |series=Issue 33 of Reports on public health and medical subjects |publisher=HMSO |oclc=14713088 }}
:
where
{{Hidden|titlestyle=background-color:#f2dfce; color:black;|contentstyle=border:1px #C4C3D0 solid;|header=For a 'sketch' of the mathematical derivation of the equation above, click on "show" to reveal|content=
Greenwood's formula is derived{{Cite web|url=https://www.math.wustl.edu/%7Esawyer/handouts/greenwood.pdf|title=The Greenwood and Exponential Greenwood Confidence Intervals in Survival Analysis|accessdate=12 May 2023}}{{self-published inline|date=November 2021}} by noting that probability of getting
and
:
\operatorname{Var}\left(\log \widehat{S}(t)\right) &\sim \frac{1}{{\widehat{S}(t)}^2} \operatorname{Var} \left(\widehat{S}(t)\right) \Rightarrow \\
\operatorname{Var}\left( \widehat{S}(t)\right) &\sim {{\widehat{S}(t)}^2}\operatorname{Var}\left(\log\widehat{S}(t) \right)
\end{align}
using martingale central limit theorem, it can be shown that the variance of the sum in the following equation is equal to the sum of variances:
:
as a result we can write:
:
\operatorname{Var}( \widehat{S}(t)) &\sim
{{\widehat{S}(t)}^2}\operatorname{Var}\left(\sum_{i:\ t_i\le t} \log\left(1 - \widehat{h}_i\right)\right) \\
&\sim
{{\widehat{S}(t)}^2}\sum\limits_{i:\ t_i\le t} \operatorname{Var}\left(\log\left(1 - \widehat{h}_i\right)\right)
\end{align}
using the delta method once more:
:
\operatorname{Var}( \widehat{S}(t)) &\sim
{{\widehat{S}(t)}^2}\sum_{i:\ t_i\le t}\left(\frac{\partial \log\left(1 - \widehat{h}_i\right)}{\partial \widehat{h}_i}\right)^2 \operatorname{Var}\left(\widehat{h}_i\right)\\
&={{\widehat{S}(t)}^2}\sum_{i:\ t_i\le t}\left(\frac{1}{1-\widehat{h}_i}\right)^2\frac{\widehat{h}_i \left( 1-\widehat{h}_i \right)}{n_i} \\
&= {{\widehat{S}(t)}^2}\sum_{i:\ t_i\le t} \frac{\widehat{h}_i}{n_i\left(1-\widehat{h}_i\right)} \\
&= {{\widehat{S}(t)}^2}\sum_{i:\ t_i\le t} \frac{d_i}{n_i(n_i-d_i)}
\end{align}
as desired.
----
}}
In some cases, one may wish to compare different Kaplan–Meier curves. This can be done by the log rank test, and the Cox proportional hazards test.
Other statistics that may be of use with this estimator are pointwise confidence intervals,{{cite journal |last1=Fay |first1=Michael P. |last2=Brittain |first2=Erica H.|author2-link=Erica Brittain |last3=Proschan |first3=Michael A. |title=Pointwise confidence intervals for a survival distribution with small samples or heavy censoring |journal=Biostatistics |date=1 September 2013 |volume=14 |issue=4 |pages=723–736 |doi=10.1093/biostatistics/kxt016 |pmid=23632624 |pmc=3769999 }} the Hall-Wellner band{{cite journal |last1=Hall |first1=W. J. |last2=Wellner |first2=Jon A. |title=Confidence bands for a survival curve from censored data |journal=Biometrika |date=1980 |volume=67 |issue=1 |pages=133–143 |doi=10.1093/biomet/67.1.133 }} and the equal-precision band.{{cite journal |last1=Nair |first1=Vijayan N. |title=Confidence Bands for Survival Functions With Censored Data: A Comparative Study |journal=Technometrics |date=August 1984 |volume=26 |issue=3 |pages=265–275 |doi=10.1080/00401706.1984.10487964 }}
Software
- Mathematica: the built-in function
SurvivalModelFit
creates survival models.{{Cite web|url=http://reference.wolfram.com/language/ref/SurvivalModelFit.html|title=Survival Analysis – Mathematica SurvivalModelFit|website=wolfram.com|access-date=2017-08-14}} - SAS: The Kaplan–Meier estimator is implemented in the
proc lifetest
procedure.{{Cite web|url=https://support.sas.com/documentation/cdl/en/statug/68162/HTML/default/viewer.htm#statug_lifetest_overview.htm|title=SAS/STAT(R) 14.1 User's Guide|website=support.sas.com|accessdate=12 May 2023}} - R: the Kaplan–Meier estimator is available as part of the
survival
package.{{cite web |last=Therneau |first=Terry M. |date=2022-08-09 |title=survival: Survival Analysis |url=https://cran.r-project.org/web/packages/survival/index.html |access-date=2022-11-30 |work=The Comprehensive R Archive Network}}{{cite book |first=Frans |last=Willekens |chapter=Statistical Packages for Multistate Life History Analysis |series=Use R! |title=Multistate Analysis of Life Histories with R |publisher=Springer |year=2014 |isbn=978-3-319-08383-4 |pages=135–153 |chapter-url=https://books.google.com/books?id=Cd2CBAAAQBAJ&pg=PA135 |doi=10.1007/978-3-319-08383-4_6 }}{{cite book |first1=Ding-Geng |last1=Chen |first2=Karl E. |last2=Peace |title=Clinical Trial Data Analysis Using R |publisher=CRC Press |year=2014 |isbn= 9781439840214|pages=99–108 |url=https://books.google.com/books?id=fGnRBQAAQBAJ&pg=PA99 }} - Stata: the command
sts
returns the Kaplan–Meier estimator.{{cite web |title=sts — Generate, graph, list, and test the survivor and cumulative hazard functions |work=Stata Manual |url=https://www.stata.com/manuals15/ststs.pdf }}{{cite book |first=Mario |last=Cleves |title=An Introduction to Survival Analysis Using Stata |location=College Station |publisher=Stata Press |edition=Second |year=2008 |isbn=978-1-59718-041-2 |pages=93–107 |url=https://books.google.com/books?id=xttbn0a-QR8C&pg=PA93 }} - Python: the
lifelines
andscikit-survival
packages each include the Kaplan–Meier estimator.{{Cite web|url=https://lifelines.readthedocs.io/en/latest/|title=lifelines — lifelines 0.27.7 documentation|website=lifelines.readthedocs.io|accessdate=12 May 2023}}{{Cite web|url=https://scikit-survival.readthedocs.io/en/stable/api/generated/sksurv.nonparametric.kaplan_meier_estimator.html|title=sksurv.nonparametric.kaplan_meier_estimator — scikit-survival 0.20.0|website=scikit-survival.readthedocs.io|accessdate=12 May 2023}} - MATLAB: the
ecdf
function with the'function','survivor'
arguments can calculate or plot the Kaplan–Meier estimator.{{Cite web|url=http://mathworks.com/help/stats/ecdf.html|title=Empirical cumulative distribution function – MATLAB ecdf|website=mathworks.com|access-date=2016-06-16}} - StatsDirect: The Kaplan–Meier estimator is implemented in the
Survival Analysis
menu.{{cite web|website=statsdirect.co.uk
|url=https://www.statsdirect.co.uk/help/Default.htm#survival_analysis/kaplan_meier.htm |title=Kaplan-Meier Survival Estimates|access-date=12 May 2023}}
- SPSS: The Kaplan–Meier estimator is implemented in the
Analyze > Survival > Kaplan-Meier...
menu.{{Cite web|url=https://statistics.laerd.com/spss-tutorials/kaplan-meier-using-spss-statistics.php|title = Kaplan-Meier method in SPSS Statistics {{pipe}} Laerd Statistics}} - Julia: the
Survival.jl
package includes the Kaplan–Meier estimator.{{Cite web|url=https://juliastats.org/Survival.jl/latest/km/|title = Kaplan-Meier · Survival.jl}} - Epi Info: Kaplan–Meier estimator survival curves and results for the log rank test are obtained with the
KMSURVIVAL
command.{{Cite web|url=https://www.cdc.gov/epiinfo/user-guide/command-reference/analysis-commands-kmsurvival.html|title =Epi Info™ User Guide - Command Reference - Analysis Commands: KMSURVIVAL|accessdate=30 Oct 2023}}
See also
References
{{Reflist|30em}}
Further reading
- {{cite book |last1=Aalen |first1=Odd |first2=Ornulf |last2=Borgan |first3=Hakon |last3=Gjessing |title=Survival and Event History Analysis: A Process Point of View |publisher=Springer |year=2008 |isbn=978-0-387-68560-1 |pages=90–104 }}
- {{cite book |last=Greene |first=William H. |author-link=William Greene (economist) |chapter=Nonparametric and Semiparametric Approaches |title=Econometric Analysis |publisher=Prentice-Hall |edition=Seventh |year=2012 |isbn=978-0-273-75356-8 |pages=909–912 |chapter-url=https://books.google.com/books?id=-WFPYgEACAAJ&pg=PA909 }}
- {{cite book |last1=Jones |first1=Andrew M. |first2=Nigel |last2=Rice |first3=Teresa Bago |last3=D'Uva |first4=Silvia |last4=Balia |chapter=Duration Data |title=Applied Health Economics |location=London |publisher=Routledge |year=2013 |isbn=978-0-415-67682-3 |pages=139–181 |chapter-url=https://books.google.com/books?id=7tdcCol9mNEC&pg=PA141 }}
- {{cite book |first1=Judith B. |last1=Singer |first2=John B. |last2=Willett |title=Applied Longitudinal Data Analysis: Modeling Change and Event Occurrence |location=New York |publisher=Oxford University Press |year=2003 |isbn=0-19-515296-4 |pages=483–487 |url=https://books.google.com/books?id=PpnA1M8VwR8C&pg=PA483 }}
External links
- {{cite web |url= http://www.cancerguide.org/scurve_km.html |title= Survival Curves: Accrual and The Kaplan–Meier Estimate |first= Steve |last= Dunn |website= Cancer Guide |series= Statistics |date= 2002 }}
- {{youTube|5C_zzD1pOAg|Three evolving Kaplan–Meier curves}}
{{Statistics|analysis}}
{{DEFAULTSORT:Kaplan-Meier estimator}}