Selection bias#Sampling bias

Sampling bias is systematic error due to a non-random sample of a population,[http://www.medilexicon.com/medicaldictionary.php?t=10087 Medical Dictionary - 'Sampling Bias'] Retrieved on September 23, 2009 causing some members of the population to be less likely to be included than others, resulting in a biased sample, defined as a statistical sample of a population (or non-human factors) in which all participants are not equally balanced or objectively represented.[http://medical-dictionary.thefreedictionary.com/Sample+bias TheFreeDictionary → biased sample]. Retrieved on 2009-09-23. Site in turn cites: Mosby's Medical Dictionary, 8th edition. It is mostly classified as a subtype of selection bias,[http://medical.webends.com/kw/Selection%20Bias Dictionary of Cancer Terms → Selection Bias]. Retrieved on September 23, 2009. sometimes specifically termed sample selection bias,{{cite journal|last1=Ards|first1=Sheila|last2=Chung|first2=Chanjin|last3=Myers|first3=Samuel L.|title=The effects of sample selection bias on racial differences in child abuse reporting|journal=Child Abuse & Neglect|volume=22|issue=2|year=1998|pages=103–115|pmid= 9504213|doi=10.1016/S0145-2134(97)00131-2|doi-access=free}}{{cite book|last1=Cortes|first1=Corinna|last2=Mohri|first2=Mehryar|last3=Riley|first3=Michael|last4=Rostamizadeh|first4=Afshin|title=Algorithmic Learning Theory |chapter=Sample Selection Bias Correction Theory |volume=5254|year=2008|pages=38–53|url=http://www.cs.nyu.edu/~mohri/postscript/bias.pdf|doi=10.1007/978-3-540-87987-9_8|series=Lecture Notes in Computer Science|isbn=978-3-540-87986-2|arxiv=0805.2775|citeseerx=10.1.1.144.4478|s2cid=842488}}{{cite journal|last1=Cortes|first1=Corinna|last2=Mohri|first2=Mehryar|title=Domain adaptation and sample bias correction theory and algorithm for regression|journal=Theoretical Computer Science|volume=519|year=2014|pages=103–126|url=http://www.cs.nyu.edu/~mohri/pub/nsmooth.pdf|doi=10.1016/j.tcs.2013.09.027|citeseerx=10.1.1.367.6899}} but some classify it as a separate type of bias.{{cite book|last=Fadem|first=Barbara|title=Behavioral Science|url=https://books.google.com/books?id=f0IDHvLiWqUC|year=2009|publisher=Lippincott Williams & Wilkins|isbn=978-0-7817-8257-9|page=262}}

A distinction of sampling bias (albeit not a universally accepted one) is that it undermines the external validity of a test (the ability of its results to be generalized to the rest of the population), while selection bias mainly addresses internal validity for differences or similarities found in the sample at hand. In this sense, errors occurring in the process of gathering the sample or cohort cause sampling bias, while errors in any process thereafter cause selection bias.

Examples of sampling bias include self-selection, pre-screening of trial participants, discounting trial subjects/tests that did not run to completion and migration bias by excluding subjects who have recently moved into or out of the study area, length-time bias, where slowly developing disease with better prognosis is detected, and lead time bias, where disease is diagnosed earlier for participants than in comparison populations, although the average course of disease is the same.

• Early termination of a trial at a time when its results support the desired conclusion.
• A trial may be terminated early at an extreme value (often for ethical reasons), but the extreme value is likely to be reached by the variable with the largest variance, even if all variables have a similar mean.

• Susceptibility bias
• Clinical susceptibility bias, when one disease predisposes for a second disease, and the treatment for the first disease erroneously appears to predispose to the second disease. For example, postmenopausal syndrome gives a higher likelihood of also developing endometrial cancer, so estrogens given for the postmenopausal syndrome may receive a higher than actual blame for causing endometrial cancer.{{cite journal |author1=Feinstein AR |author2=Horwitz RI |title=A critique of the statistical evidence associating estrogens with endometrial cancer |journal=Cancer Res. |volume=38 |issue=11 Pt 2 |pages=4001–5 |date=November 1978 |pmid=698947}}
• Protopathic bias, when a treatment for the first symptoms of a disease or other outcome appear to cause the outcome. It is a potential bias when there is a lag time from the first symptoms and start of treatment before actual diagnosis. It can be mitigated by lagging, that is, exclusion of exposures that occurred in a certain time period before diagnosis.{{cite journal |author1=Tamim H |author2=Monfared AA |author3=LeLorier J |title=Application of lag-time into exposure definitions to control for protopathic bias |journal=Pharmacoepidemiol Drug Saf |volume=16 |issue=3 |pages= 250–8 |date=March 2007 |pmid=17245804 |doi=10.1002/pds.1360|s2cid=25648490 }}
• Indication bias, a potential mixup between cause and effect when exposure is dependent on indication, e.g. a treatment is given to people in high risk of acquiring a disease, potentially causing a preponderance of treated people among those acquiring the disease. This may cause an erroneous appearance of the treatment being a cause of the disease.{{cite book |author=Matthew R. Weir |title= Hypertension (Key Diseases) (Acp Key Diseases Series) |publisher=American College of Physicians |location=Philadelphia, Pa |year=2005 |page=159 |isbn=978-1-930513-58-7}}

• Partitioning (dividing) data with knowledge of the contents of the partitions, and then analyzing them with tests designed for blindly chosen partitions.
• Post hoc alteration of data inclusion based on arbitrary or subjective reasons, including:
• Cherry picking, which actually is not selection bias, but confirmation bias, when specific subsets of data are chosen to support a conclusion (e.g. citing examples of plane crashes as evidence of airline flight being unsafe, while ignoring the far more common example of flights that complete safely. See: availability heuristic)
• Rejection of bad data on (1) arbitrary grounds, instead of according to previously stated or generally agreed criteria or (2) discarding "outliers" on statistical grounds that fail to take into account important information that could be derived from "wild" observations.{{cite journal|last1=Kruskal|first1=William H.|title=Some Remarks on Wild Observations|journal=Technometrics|volume=2|issue=1|year=1960|pages=1–3|doi=10.1080/00401706.1960.10489875|url=http://www.tufts.edu/~gdallal/out.htm}}

• Selection of which studies to include in a meta-analysis (see also combinatorial meta-analysis).
• Performing repeated experiments and reporting only the most favorable results, perhaps relabelling lab records of other experiments as "calibration tests", "instrumentation errors" or "preliminary surveys".
• Presenting the most significant result of a data dredge as if it were a single experiment (which is logically the same as the previous item, but is seen as much less dishonest).

Attrition bias is a kind of selection bias caused by attrition (loss of participants),{{Cite journal | doi = 10.1093/ije/dyh406| title = Empirical evidence of attrition bias in clinical trials| journal = International Journal of Epidemiology| volume = 34 |issue=1| pages = 87–88| year = 2005| last1 = Jüni | first1 = P. |last2=Egger |first2=Matthias | pmid = 15649954| doi-access = free}} discounting trial subjects/tests that did not run to completion. It is closely related to the survivorship bias, where only the subjects that "survived" a process are included in the analysis or the failure bias, where only the subjects that "failed" a process are included. It includes dropout, nonresponse (lower response rate), withdrawal and protocol deviators. It gives biased results where it is unequal in regard to exposure and/or outcome. For example, in a test of a dieting program, the researcher may simply reject everyone who drops out of the trial, but most of those who drop out are those for whom it was not working. Different loss of subjects in intervention and comparison group may change the characteristics of these groups and outcomes irrespective of the studied intervention.

Lost to follow-up, is another form of Attrition bias, mainly occurring in medicinal studies over a lengthy time period. Non-Response or Retention bias can be influenced by a number of both tangible and intangible factors, such as; wealth, education, altruism, initial understanding of the study and its requirements.{{Cite journal|last1=Jordan|first1=Sue|last2=Watkins|first2=Alan|last3=Storey|first3=Mel|last4=Allen|first4=Steven J.|last5=Brooks|first5=Caroline J.|last6=Garaiova|first6=Iveta|last7=Heaven|first7=Martin L.|last8=Jones|first8=Ruth|last9=Plummer|first9=Sue F.|last10=Russell|first10=Ian T.|last11=Thornton|first11=Catherine A.|date=2013-07-09|title=Volunteer Bias in Recruitment, Retention, and Blood Sample Donation in a Randomised Controlled Trial Involving Mothers and Their Children at Six Months and Two Years: A Longitudinal Analysis|journal=PLOS ONE|volume=8|issue=7|pages=e67912|doi=10.1371/journal.pone.0067912|issn=1932-6203|pmc=3706448|pmid=23874465|bibcode=2013PLoSO...867912J|doi-access=free}} Researchers may also be incapable of conducting follow-up contact resulting from inadequate identifying information and contact details collected during the initial recruitment and research phase.{{Cite journal|last=Small|first=W. P.|title=Lost to Follow-Up|date=1967-05-06|url=http://www.sciencedirect.com/science/article/pii/S014067366792377X|journal=The Lancet|series=Originally published as Volume 1, Issue 7497|language=en|volume=289|issue=7497|pages=997–999|doi=10.1016/S0140-6736(67)92377-X|pmid=4164620|s2cid=27683727 |issn=0140-6736}}

Philosopher Nick Bostrom has argued that data are filtered not only by study design and measurement, but by the necessary precondition that there has to be someone doing a study. In situations where the existence of the observer or the study is correlated with the data, observation selection effects occur, and anthropic reasoning is required.{{cite book |first=Nick |last=Bostrom |title=Anthropic Bias: Observation Selection Effects in Science and Philosophy |publisher=Routledge |location=New York |year=2002 |isbn=978-0-415-93858-7 }}

An example is the past impact event record of Earth: if large impacts cause mass extinctions and ecological disruptions precluding the evolution of intelligent observers for long periods, no one will observe any evidence of large impacts in the recent past (since they would have prevented intelligent observers from evolving). Hence there is a potential bias in the impact record of Earth.{{Cite journal | doi = 10.1111/j.1539-6924.2010.01460.x| pmid = 20626690| title = Anthropic Shadow: Observation Selection Effects and Human Extinction Risks| journal = Risk Analysis| volume = 30| issue = 10| pages = 1495–506| year = 2010| last1 = Ćirković | first1 = M. M. | last2 = Sandberg | first2 = A. | last3 = Bostrom | first3 = N. | bibcode = 2010RiskA..30.1495C| s2cid = 6485564}} Astronomical existential risks might similarly be underestimated due to selection bias, and an anthropic correction has to be introduced.{{Cite journal | doi = 10.1038/438754a| title = Astrophysics: Is a doomsday catastrophe likely?| journal = Nature| volume = 438| issue = 7069| pages = 754| year = 2005| last1 = Tegmark | first1 = M. | last2 = Bostrom | first2 = N. | pmid=16341005| bibcode = 2005Natur.438..754T| s2cid = 4390013| doi-access = free}}

{{main|Self-selection bias}}

Self-selection bias or a volunteer bias in studies offer further threats to the validity of a study as these participants may have intrinsically different characteristics from the target population of the study.{{Cite journal|last1=Tripepi|first1=Giovanni|last2=Jager|first2=Kitty J.|last3=Dekker|first3=Friedo W.|last4=Zoccali|first4=Carmine|date=2010|title=Selection Bias and Information Bias in Clinical Research|url=https://www.karger.com/Article/FullText/312871|journal=Nephron Clinical Practice|language=english|volume=115|issue=2|pages=c94–c99|doi=10.1159/000312871|issn=1660-2110|pmid=20407272|doi-access=free}} Studies have shown that volunteers tend to come from a higher social standing than from a lower socio-economic background.{{Cite web|date=2017-11-17|title=Volunteer bias|url=https://catalogofbias.org/biases/volunteer-bias/|access-date=2020-10-29|website=Catalog of Bias|language=en}} Furthermore, another study shows that women are more probable to volunteer for studies than males. Volunteer bias is evident throughout the study life-cycle, from recruitment to follow-ups. More generally speaking volunteer response can be put down to individual altruism, a desire for approval, personal relation to the study topic and other reasons. As with most instances mitigation in the case of volunteer bias is an increased sample size. {{Citation needed|date=January 2021|reason=A citation that mentions that increasing the sample size mitigates this would be beneficial}}

In the general case, selection biases cannot be overcome with statistical analysis of existing data alone, though Heckman correction may be used in special cases. An assessment of the degree of selection bias can be made by examining correlations between exogenous (background) variables and a treatment indicator. However, in regression models, it is correlation between unobserved determinants of the outcome and unobserved determinants of selection into the sample which bias estimates, and this correlation between unobservables cannot be directly assessed by the observed determinants of treatment.{{Cite journal | doi = 10.2307/1912352| jstor = 1912352| title = Sample Selection Bias as a Specification Error| journal = Econometrica| volume = 47| issue = 1| pages = 153–161| year = 1979| last1 = Heckman | first1 = J. J. }}

When data are selected for fitting or forecast purposes, a coalitional game can be set up so that a fitting or forecast accuracy function can be defined on all subsets of the data variables.

Selection bias is closely related to:

• publication bias or reporting bias, the distortion produced in community perception or meta-analyses by not publishing uninteresting (usually negative) results, or results which go against the experimenter's prejudices, a sponsor's interests, or community expectations.
• confirmation bias, the general tendency of humans to give more attention to whatever confirms our pre-existing perspective; or specifically in experimental science, the distortion produced by experiments that are designed to seek confirmatory evidence instead of trying to disprove the hypothesis.
• exclusion bias, results from applying different criteria to cases and controls in regards to participation eligibility for a study/different variables serving as basis for exclusion.

{{Div col|colwidth=30em}}

• {{annotated link|Berkson's paradox}}
• {{annotated link|Black swan theory}}
• {{annotated link|Cherry picking}}
• {{annotated link|Frequency illusion}}
• {{annotated link|Funding bias}}
• {{annotated link|List of cognitive biases}}
• {{annotated link|Participation bias}}
• {{annotated link|Publication bias}}
• {{annotated link|Reporting bias}}
• {{annotated link|Sampling bias}}
• {{annotated link|Sampling probability}}
• {{annotated link|Selective exposure theory}}
• {{annotated link|Self-fulfilling prophecy}}
• {{annotated link|Survivorship bias}}

{{div col end}}

{{reflist|30em}}

{{Biases}}

{{Medical research studies}}

Category:Sampling (statistics)

Category:Experimental bias

Category:Scientific method

Category:Causal inference