Bot prevention

Studies suggest that over half of the traffic on the internet is bot activity, of which over half is further classified as 'bad bots'.{{Cite book|last1=Amin Azad|first1=Babak|last2=Starov|first2=Oleksii|last3=Laperdrix|first3=Pierre|last4=Nikiforakis|first4=Nick|title=Detection of Intrusions and Malware, and Vulnerability Assessment |chapter=Web Runner 2049: Evaluating Third-Party Anti-bot Services |date=2020|editor-last=Maurice|editor-first=Clémentine|editor2-last=Bilge|editor2-first=Leyla|editor3-last=Stringhini|editor3-first=Gianluca|editor4-last=Neves|editor4-first=Nuno|series=Lecture Notes in Computer Science|volume=12223|language=en|location=Cham|publisher=Springer International Publishing|pages=135–159|doi=10.1007/978-3-030-52683-2_7|isbn=978-3-030-52683-2|pmc=7338186}}

Bots are used for various purposes online. Some bots are used passively for web scraping purposes, for example, to gather information from airlines about flight prices and destinations. Other bots, such as sneaker bots, help the bot operator acquire high-demand luxury goods; sometimes these are resold on the secondary market at higher prices, in what is commonly known as 'scalping'.{{cite book|last1=Chiapponi|first1=Elisa|last2=Dacier|first2=Marc|last3=Todisco|first3=Massimiliano|last4=Catakoglu|first4=Onur|last5=Thonnard|first5=Olivier|title=Service-Oriented Computing – ICSOC 2020 Workshops |chapter=Botnet Sizes: When Maths Meet Myths |date=2021|series=Lecture Notes in Computer Science|volume=12632|pages=596–611|doi=10.1007/978-3-030-76352-7_52|isbn=978-3-030-76351-0|s2cid=232203240}}{{cite web|last1=Marks|first1=Tod|title=Why Ticket Prices Are Going Through the Roof|url=https://www.consumerreports.org/money/why-ticket-prices-are-going-through-the-roof/|website=Consumer Reports}}{{cite web|title=Bad Bot Report 2021|url=https://www.exclusive-networks.com/se/wp-content/uploads/sites/25/2020/12/Imperva-Bad-Bot-Report-2021.pdf|access-date=23 August 2021|publisher=Imperva}}

Various fingerprinting and behavioural techniques are used to identify whether the client is a human user or a bot. In turn, bots use a range of techniques to avoid detection and appear like a human to the server.

Browser fingerprinting techniques are the most common component in anti-bot protection systems. Data is usually collected through client-side JavaScript which is then transmitted to the anti-bot service for analysis. The data collected includes results from JavaScript APIs (checking if a given API is implemented and returns the results expected from a normal browser), rendering complex WebGL scenes, and using the Canvas API.{{Cite book|last1=Jonker|first1=Hugo|last2=Krumnow|first2=Benjamin|last3=Vlot|first3=Gabry|title=Computer Security – ESORICS 2019 |chapter=Fingerprint Surface-Based Detection of Web Bot Detectors |date=2019|editor-last=Sako|editor-first=Kazue|editor2-last=Schneider|editor2-first=Steve|editor3-last=Ryan|editor3-first=Peter Y. A.|chapter-url=https://research.ou.nl/en/publications/7e71fdd1-36c3-4d36-a10f-8ff5729974b3|series=Lecture Notes in Computer Science|volume=11736|language=en|location=Cham|publisher=Springer International Publishing|pages=586–605|doi=10.1007/978-3-030-29962-0_28|isbn=978-3-030-29962-0|s2cid=202579603}}

TLS fingerprinting techniques categorise the client by analysing the supported cipher suites during the SSL handshake.{{cite web |title=Qualys SSL Labs - Projects / HTTP Client Fingerprinting Using SSL Handshake Analysis |url=https://www.ssllabs.com/projects/client-fingerprinting/ |website=www.ssllabs.com}} These fingerprints can be used to create whitelists/blacklists containing fingerprints of known browser stacks. In 2017, Salesforce open sourced its TLS fingerprinting library (JA3).{{cite web |last1=Althouse |first1=John |title=Open Sourcing JA3 |url=https://engineering.salesforce.com/open-sourcing-ja3-92c9e53c3c41 |website=Medium |language=en |date=5 February 2019}} Between August and September 2018, Akamai noticed a large increase in TLS tampering across its network to evade detection.{{cite web |title=Bots Tampering with TLS to Avoid Detection - Akamai Security Intelligence and Threat Research Blog |url=https://blogs.akamai.com/sitr/2019/05/bots-tampering-with-tls-to-avoid-detection.html |website=blogs.akamai.com}}{{cite web |title=Bots increasingly tampering with TLS to outfox filters |url=https://portswigger.net/daily-swig/bots-increasingly-tampering-with-tls-to-outfox-filters |website=The Daily Swig {{!}} Cybersecurity news and views |language=en |date=17 May 2019}}

Behaviour-based techniques are also utilised, although less commonly than fingerprinting techniques, and rely on the idea that bots behave differently to human visitors. A common behavioural approach is to analyse a client's mouse movements and determine if they are typical of a human.{{cite book|last1=Wei|first1=Ang|last2=Zhao|first2=Yuxuan|last3=Cai|first3=Zhongmin|title=Biometric Recognition |chapter=A Deep Learning Approach to Web Bot Detection Using Mouse Behavioral Biometrics |date=2019|series=Lecture Notes in Computer Science|volume=11818|pages=388–395|doi=10.1007/978-3-030-31456-9_43|isbn=978-3-030-31455-2|s2cid=203847308}}

More traditional techniques such as CAPTCHAs are also often employed, however they are generally considered ineffective while simultaneously obtrusive to human visitors.{{cite book|last1=Chu|first1=Zi|last2=Gianvecchio|first2=Steven|last3=Wang|first3=Haining|title=From Database to Cyber Security |chapter=Bot or Human? A Behavior-Based Online Bot Detection System |date=2018|series=Lecture Notes in Computer Science|volume=11170|pages=432–449|doi=10.1007/978-3-030-04834-1_21|isbn=978-3-030-04833-4}}

The use of JavaScript can prevent some bots that rely on basic requests (such as via cURL), as these will not load the detection script and hence will fail to progress. A common method to bypass many techniques is to use a headless browser to simulate a real web browser and execute the client-side JavaScript detection scripts. There are a variety of headless browsers that are used; some are custom (such as PhantomJS) but it is also possible to operate typical browsers such as Google Chrome in headless mode using a driver. Selenium is a common web automation framework that makes it easier to control the headless browser. Anti-bot detection systems attempt to identify the implementation of methods specific to these headless browsers, or the lack of proper implementation of APIs that would be implemented in regular web browsers.

The source code of these JavaScript files is typically obfuscated to make it harder to reverse engineer how the detection works. Common techniques include:{{cite web |title=JavaScript Obfuscator Tool |url=https://obfuscator.io/ |website=obfuscator.io}}

• Minification
• String arrays
• Control flow flattening
• Dead code injection
• debugger statements, to prevent use of debuggers like DevTools

Anti-bot protection services are offered by various internet companies, such as Cloudflare{{cite web|title=Cloudflare Bot Management|url=https://www.cloudflare.com/products/bot-management/|website=Cloudflare|language=en-us}} and Akamai.{{cite web|title=Bot Manager|url=https://www.akamai.com/us/en/products/security/bot-manager.jsp|access-date=23 August 2021|website=Akamai Technologies}}{{cite web|title=Akamai Bot Manager|url=https://developer.akamai.com/akamai-bot-manager|website=Akamai Technologies|language=en}}

In the United States, the Better Online Tickets Sales Act (commonly known as the BOTS Act) was passed in 2016 to prevent some uses of bots in commerce.{{cite news|last1=Sisario|first1=Ben|date=9 December 2016|title=Congress Moves to Curb Ticket Scalping, Banning Bots Used Online|work=The New York Times|url=https://www.nytimes.com/2016/12/08/business/media/ticket-scalping-bots-act.html}} A year later, the United Kingdom passed similar regulations in the Digital Economy Act 2017.{{cite web|last1=Keepfer|first1=DLA Piper-Francis|date=10 January 2018|title=UK Government criminalises the use of ticket tout bots|url=https://www.lexology.com/library/detail.aspx?g=d4c8584c-e088-46d2-9c39-07e190474d3c|website=Lexology|language=en}}{{cite web|date=23 April 2018|title=New law will ban use of bots to bulk buy tickets|url=https://www.which.co.uk/news/2018/04/new-law-will-ban-use-of-bots-to-bulk-buy-tickets/|website=Which? News}} The effectiveness of these measures is disputed.{{cite journal|last1=Elefant|first1=Sammi|date=2018|title=Beyond the Bots: Ticked-Off Over Ticket Prices or The Eternal Scamnation|url=https://escholarship.org/uc/item/056242s2|journal=UCLA Entertainment Law Review|language=en|volume=25|issue=1|doi=10.5070/LR8251039716|issn=1073-2896|doi-access=free}}

{{reflist}}

Category:Internet security

Category:Internet fraud