Statistically improbable phrase

{{Short description|Phrase that appears more often in a single work than in a large sample size}}

A statistically improbable phrase (SIP) is a phrase or set of words that occurs more frequently in a document (or collection of documents) than in some larger corpus.{{cite web|url=http://courses.cms.caltech.edu/cs145/2011/wikipedia.pdf |title=SIPping Wikipedia |website=Courses.cms.caltech.edu |access-date=2017-01-01}}{{cite web|url=https://www.plagiarismtoday.com/2012/07/03/how-long-should-a-statistically-improbably-phrase-be/|title=How Long Should a Statistically Improbably Phrase Be?|author=Jonathan Bailey|date=3 July 2012|work=Plagiarism Today}}{{cite journal|url= |title=Identifying duplicate content using statistically improbable phrases|first1=Mounir|last1=Errami|first2=Zhaohui|last2=Sun|first3=Angela C.|last3=George|first4=Tara C.|last4=Long|first5=Michael A.|last5=Skinner|first6=Jonathan D.|last6=Wren|first7=Harold R.|last7=Garner|date=1 June 2010|journal=Bioinformatics|volume=26|issue=11|pages=1453–1457|via=bioinformatics.oxfordjournals.org|doi=10.1093/bioinformatics/btq146|pmid=20472545|pmc=2872002}} Amazon.com uses this concept in determining keywords for a given book or chapter, since keywords of a book or chapter are likely to appear disproportionately within that section.{{cite web|url=https://www.amazon.com/gp/search-inside/sipshelp.html|title=What are Statistically Improbable Phrases?|access-date=2007-12-18|publisher=Amazon.com}}{{cite news|url=https://www.washingtonpost.com/wp-dyn/content/article/2005/08/29/AR2005082901873.html|title=Amazon's Vital Statistics Show How Books Stack Up|last=Weeks|first=Linton|work=The Washington Post|date=August 30, 2005|access-date=September 8, 2015}} Christian Rudder has also used this concept with data from online dating profiles and Twitter posts to determine the phrases most characteristic of a given race or gender in his book Dataclysm.{{cite book |last=Rudder |first=Christian |date=2014 |title=Dataclysm: Who We Are When We Think No One's Looking |location=New York |publisher=Crown Publishers |isbn=978-0-385-34737-2}} SIPs with a linguistic density of two or three words, adjective, adjective, noun or adverb, adverb, verb, will signal the author's attitude, premise or conclusions to the reader or express an important idea.

Another use of SIPs is as a detection tool for plagiarism. (Almost) unique combinations of words can be searched for online, and if they have appeared in a published text, the search will identify where. This method only checks those texts that have been published and that have been digitized online.

For example, a submission by, say, a student that contained the phrase "garden style, praising irregularity in design", might be searched for using Google.com and will yield the original Wikipedia article about Sir William Temple, English political figure and essayist.

==Example==

While common words like "the" appear frequently in most texts, a phrase like "explicit Boolean algorithm" might occur much more often in a document about computers than it does in general English. Therefore, "explicit Boolean algorithm" would be considered a statistically improbable phrase in that context.

Statistically improbable phrases of Darwin's On the Origin of Species could be: temperate productions, genera descended, transitional gradations, unknown progenitor, fossiliferous formations, our domestic breeds, modified offspring, doubtful forms, closely allied forms, profitable variations, enormously remote, transitional grades, very distinct species and mongrel offspring.[http://crookedtimber.org/2005/04/02/sociologically-improbable-phrases/ Sociologically Improbable Phrases] Crooked Timber April 2005