Expert review (method)
Expert review or expert evaluation is a method to evaluate survey questions from the perspective of one or more experts.
An expert review has two primary goals:{{Cite journal |last=Olson |first=Kristen |date=2010 |title=An Examination of Questionnaire Evaluation by Expert Reviewers |journal=Field Methods |volume=22 |issue=4 |pages=295–318 |doi=10.1177/1525822X10379795}}
- Identify potential problems related to data quality and data collection so they can be mitigated.
- Group survey items by how likely they are to result in measurement errors, such as to classify interviewer effects.
An individual expert or a panel of experts such as survey methodologists, subject matter experts (such as sociologists) or other people familiar with questionnaire design review the survey, and they may be asked to suggest revisions.{{Citation |last=DeMaio |first=Terry |title=Examining expert reviews as a pretest method |date=2004 |work=QUEST 2003: proceedings of the 4th Conference on Questionnaire Evaluation Standards, 21–23 October 2003 |volume=9 |pages=60–73 |editor-last=Prüfer |editor-first=Peter |url=https://www.ssoar.info/ssoar/handle/document/49195 |access-date=2024-02-21 |series=ZUMA-Nachrichten Spezial |place=Mannheim |publisher=Zentrum für Umfragen, Methoden und Analysen -ZUMA- |language=en |isbn=978-3-924220-27-3 |last2=Landreth |first2=Ashley |editor2-last=Rexroth |editor2-first=Margrit |editor3-last=Fowler |editor3-first=Floyd Jackson Jr.}} Because variation is found in the problems identified by each expert, a panel rather than one single expert is recommended.
The experts document their assessment and typically report the results in open-ended comments.{{Cite journal |last=Yan |first=T. |last2=Kreuter |first2=F. |last3=Tourangeau |first3=R |date=December 2012 |title=Evaluating Survey Questions: A Comparison of Methods |journal=Journal of Official Statistics |volume=28 |issue=4 |pages=503–529}} They can also conduct a systematic review using a coding tool such as the Questionnaire Appraisal System that is supported by an item taxonomy of the cognitive demands of a question and a detailed set of potential problem codes document the features that may lead to response error.{{Cite web |last=Willis |first=G |last2=Lessler |first2=J |date=1999 |title=Question Appraisal System (QAS-99) |url=https://www.cdc.gov/healthyyouth/evaluation/pdf/brief15.pdf |website=Centers for Disease Control and Prevention Website}}{{Citation |last=Maitland |first=Aaron |title=A Comparison of Five Question Evaluation Methods in Predicting the Validity of Respondent Answers to Factual Items |date=2020-01-02 |work=Advances in Questionnaire Design, Development, Evaluation and Testing |pages=75–90 |editor-last=Beatty |editor-first=Paul |url=https://onlinelibrary.wiley.com/doi/10.1002/9781119263685.ch4 |access-date=2024-02-21 |edition=1 |publisher=Wiley |language=en |doi=10.1002/9781119263685.ch4 |isbn=978-1-119-26362-3 |last2=Presser |first2=Stanley |editor2-last=Collins |editor2-first=Debbie |editor3-last=Kaye |editor3-first=Lyn |editor4-last=Padilla |editor4-first=Jose Luis}}
When applied to translated surveys, the expert panel also includes translators and they usually produce individual reports and then convene a resolution meeting that generates an improved version of the survey translation.{{Cite book |last1=Pan |first1=Yuling |url=https://www.taylorfrancis.com/books/mono/10.4324/9780429294914/sociolinguistics-survey-translation-yuling-pan-mandy-sha-hyunjoo-park |title=The Sociolinguistics of Survey Translation |last2=Sha |first2=Mandy |date=2019-07-09 |publisher=Routledge |isbn=978-0-429-29491-4 |location=London |pages=118–124 |doi=10.4324/9780429294914 |s2cid=198632812}}
Compared to empirical pretesting methods, an expert review is more time-efficient and less costly to complete.{{Cite journal |last=Presser |first=S |author-link=Stanley Presser |last2=Blair |first2=J |title=Survey pretesting: Do different methods produce different results? |journal=Sociological Methodology |volume=24 |pages=73–104}} An expert review can also be combined with other methods to enable multimode pretesting.{{Cite journal |last=Ridolfo |first=Heather |last2=Thompson |first2=Ashley |date=2023-10-05 |title=Conducting Small-Scale Multimethod Questionnaire Evaluation |url=https://www.surveypractice.org/article/88101-conducting-small-scale-multimethod-questionnaire-evaluation |journal=Survey Practice |language=en |volume=16 |issue=1 |doi=10.29115/SP-2023-0017|doi-access=free }}{{Cite web |last=Sha |first=Mandy |date=March 2012 |title=Translation Review and Cognitive Testing of ACS Language Assistance Guides in Multiple Languages |url=https://www.census.gov/library/working-papers/2012/acs/2012_Sha_01.html |access-date=2024-01-08 |website=United States Census Bureau}}{{Cite book |last=Aizpurua |first=Eva |title=The essential role of language in survey research |publisher=RTI Press |year=2020 |isbn=978-1-934831-23-6 |editor-last=Sha |editor-first=Mandy |pages=129–150 |chapter=Pretesting methods in cross-cultural research (Chapter 7) |doi=10.3768/rtipress.bk.0023.2004 |editor-last2=Gabel |editor-first2=Tim |doi-access=free}}