Adversarial machine learning

At the MIT Spam Conference in January 2004, John Graham-Cumming showed that a machine-learning spam filter could be used to defeat another machine-learning spam filter by automatically learning which words to add to a spam email to get the email classified as not spam.{{Cite web |title=How to beat an adaptive/Bayesian spam filter (2004) |url=http://blog.jgc.org/2023/07/how-to-beat-adaptivebayesian-spam.html |access-date=2023-07-05 |language=en}}

In 2004, Nilesh Dalvi and others noted that linear classifiers used in spam filters could be defeated by simple "evasion attacks" as spammers inserted "good words" into their spam emails. (Around 2007, some spammers added random noise to fuzz words within "image spam" in order to defeat OCR-based filters.) In 2006, Marco Barreno and others published "Can Machine Learning Be Secure?", outlining a broad taxonomy of attacks. As late as 2013 many researchers continued to hope that non-linear classifiers (such as support vector machines and neural networks) might be robust to adversaries, until Battista Biggio and others demonstrated the first gradient-based attacks on such machine-learning models (2012{{cite arXiv|eprint=1206.6389|class=cs.LG |first1=Battista|last1=Biggio|first2=Blaine|last2=Nelson|title=Poisoning Attacks against Support Vector Machines|date=2013-03-25|last3=Laskov|first3=Pavel}}–2013{{Cite book |last1=Biggio |first1=Battista|last2=Corona|first2=Igino|last3=Maiorca|first3=Davide|last4=Nelson |first4=Blaine|last5=Srndic|first5=Nedim|last6=Laskov|first6=Pavel|last7=Giacinto|first7=Giorgio |last8=Roli|first8=Fabio|title=Advanced Information Systems Engineering |chapter=Evasion Attacks against Machine Learning at Test Time |date=2013 |series=Lecture Notes in Computer Science|volume=7908|publisher=Springer |pages=387–402 |arxiv=1708.06131|doi=10.1007/978-3-642-40994-3_25 |isbn=978-3-642-38708-1|s2cid=18716873}}). In 2012, deep neural networks began to dominate computer vision problems; starting in 2014, Christian Szegedy and others demonstrated that deep neural networks could be fooled by adversaries, again using a gradient-based attack to craft adversarial perturbations.{{cite arXiv |eprint=1312.6199|class=cs.CV|first1=Christian|last1=Szegedy|first2=Wojciech|last2=Zaremba |title=Intriguing properties of neural networks|date=2014-02-19|last6=Goodfellow|first7=Rob|last7=Fergus |first6=Ian|first3=Ilya|last5=Erhan|first4=Joan|last4=Bruna|last3=Sutskever|first5=Dumitru}}{{cite journal|last1=Biggio|first1=Battista|last2=Roli|first2=Fabio|date=December 2018 |title=Wild patterns: Ten years after the rise of adversarial machine learning|journal=Pattern Recognition|volume=84|pages=317–331 |arxiv=1712.03141|bibcode=2018PatRe..84..317B |doi=10.1016/j.patcog.2018.07.023|s2cid=207324435}}

Recently, it was observed that adversarial attacks are harder to produce in the practical world due to the different environmental constraints that cancel out the effect of noise.{{cite arXiv |eprint=1607.02533 |class=cs.CV|first1=Alexey|last1=Kurakin|first2=Ian|last2=Goodfellow |title=Adversarial examples in the physical world|last3=Bengio|first3=Samy|year=2016}}Gupta, Kishor Datta, Dipankar Dasgupta, and Zahid Akhtar. "Applicability issues of Evasion-Based Adversarial Attacks and Mitigation Techniques." 2020 IEEE Symposium Series on Computational Intelligence (SSCI). 2020. For example, any small rotation or slight illumination on an adversarial image can destroy the adversariality. In addition, researchers such as Google Brain's Nicholas Frosst point out that it is much easier to make self-driving cars{{Cite journal |last1=Lim |first1=Hazel Si Min |last2=Taeihagh |first2=Araz |date=2019 |title=Algorithmic Decision-Making in AVs: Understanding Ethical and Technical Concerns for Smart Cities |journal=Sustainability |language=en |volume=11 |issue=20 |pages=5791 |arxiv=1910.13122 |bibcode=2019arXiv191013122L |doi=10.3390/su11205791 |doi-access=free |s2cid=204951009}} miss stop signs by physically removing the sign itself, rather than creating adversarial examples.{{Cite web|date=2019-11-21|title=Google Brain's Nicholas Frosst on Adversarial Examples and Emotional Responses|website=Synced |url=https://syncedreview.com/2019/11/21/google-brains-nicholas-frosst-on-adversarial-examples-and-emotional-responses/|access-date=2021-10-23}} Frosst also believes that the adversarial machine learning community incorrectly assumes models trained on a certain data distribution will also perform well on a completely different data distribution. He suggests that a new approach to machine learning should be explored, and is currently working on a unique neural network that has characteristics more similar to human perception than state-of-the-art approaches.

While adversarial machine learning continues to be heavily rooted in academia, large tech companies such as Google, Microsoft, and IBM have begun curating documentation and open source code bases to allow others to concretely assess the robustness of machine learning models and minimize the risk of adversarial attacks.{{Cite web|title=Responsible AI practices|url=https://ai.google/responsibilities/responsible-ai-practices/|access-date=2021-10-23|website=Google AI|language=en}}{{Citation |title=Adversarial Robustness Toolbox (ART) v1.8 |date=2021-10-23 |url=https://github.com/Trusted-AI/adversarial-robustness-toolbox |publisher=Trusted-AI |access-date=2021-10-23}}{{Cite web|last=amarshal|title=Failure Modes in Machine Learning - Security documentation|url=https://docs.microsoft.com/en-us/security/engineering/failure-modes-in-machine-learning|access-date=2021-10-23|website=docs.microsoft.com|language=en-us}}

Examples include attacks in spam filtering, where spam messages are obfuscated through the misspelling of "bad" words or the insertion of "good" words;{{cite journal|last1=Biggio|first1=Battista|last2=Fumera|first2=Giorgio|last3=Roli|first3=Fabio|year=2010|title=Multiple classifier systems for robust classifier design in adversarial environments|url=http://pralab.diee.unica.it/en/node/671|journal=International Journal of Machine Learning and Cybernetics|volume=1|issue=1–4|pages=27–41|doi=10.1007/s13042-010-0007-7|hdl=11567/1087824|issn=1868-8071|s2cid=8729381|access-date=2015-01-14|archive-date=2023-01-19|archive-url=https://web.archive.org/web/20230119094021/http://pralab.diee.unica.it/en/node/671|url-status=dead|url-access=subscription}}{{cite journal|last1=Brückner |first1=Michael|last2=Kanzow|first2=Christian|last3=Scheffer|first3=Tobias|date=2012|title=Static Prediction Games for Adversarial Learning Problems |url=http://www.jmlr.org/papers/volume13/brueckner12a/brueckner12a.pdf|journal=Journal of Machine Learning Research|volume=13|issue=Sep|pages=2617–2654|issn=1533-7928}} attacks in computer security, such as obfuscating malware code within network packets or modifying the characteristics of a network flow to mislead intrusion detection;{{Cite journal |last1=Apruzzese |first1=Giovanni |last2=Andreolini |first2=Mauro |last3=Ferretti |first3=Luca |last4=Marchetti |first4=Mirco |last5=Colajanni |first5=Michele |date=2021-06-03 |title=Modeling Realistic Adversarial Attacks against Network Intrusion Detection Systems |journal=Digital Threats: Research and Practice |volume=3 |issue=3 |pages=1–19 |doi=10.1145/3469659 |s2cid=235458519 |issn=2692-1626|arxiv=2106.09380 }}{{Cite journal |last1=Vitorino |first1=João |last2=Oliveira |first2=Nuno |last3=Praça |first3=Isabel |date=March 2022 |title=Adaptative Perturbation Patterns: Realistic Adversarial Learning for Robust Intrusion Detection |journal=Future Internet |language=en |volume=14 |issue=4 |pages=108 |doi=10.3390/fi14040108 |issn=1999-5903|doi-access=free |hdl=10400.22/21851 |hdl-access=free }} attacks in biometric recognition where fake biometric traits may be exploited to impersonate a legitimate user;{{cite journal|last1=Rodrigues|first1=Ricardo N.|last2=Ling|first2=Lee Luan|last3=Govindaraju|first3=Venu|date=1 June 2009|title=Robustness of multimodal biometric fusion methods against spoof attacks|url=http://cubs.cedar.buffalo.edu/images/pdf/pub/robustness-of-multimodal-biometric-fusion-methods-against-spoof-attacks.pdf|journal=Journal of Visual Languages & Computing|volume=20|issue=3|pages=169–179 |doi=10.1016/j.jvlc.2009.01.010|issn=1045-926X}} or to compromise users' template galleries that adapt to updated traits over time.

Researchers showed that by changing only one-pixel it was possible to fool deep learning algorithms.{{cite journal|last1=Su|first1=Jiawei|last2=Vargas|first2=Danilo Vasconcellos|last3=Sakurai |first3=Kouichi|date=October 2019|title=One Pixel Attack for Fooling Deep Neural Networks|journal=IEEE Transactions on Evolutionary Computation|volume=23|issue=5|pages=828–841|arxiv=1710.08864 |doi=10.1109/TEVC.2019.2890858|issn=1941-0026|s2cid=2698863}} Others 3-D printed a toy turtle with a texture engineered to make Google's object detection AI classify it as a rifle regardless of the angle from which the turtle was viewed.{{cite news|date=3 November 2017|title=Single pixel change fools AI programs|work=BBC News|url=https://www.bbc.com/news/technology-41845878|access-date=12 February 2018}} Creating the turtle required only low-cost commercially available 3-D printing technology.{{cite arXiv|eprint=1707.07397|class=cs.CV|first1=Anish |last1=Athalye|first2=Logan|last2=Engstrom|last3=Ilyas|first3=Andrew|last4=Kwok|first4=Kevin|year=2017 |title=Synthesizing Robust Adversarial Examples}}

A machine-tweaked image of a dog was shown to look like a cat to both computers and humans.{{cite magazine|date=2018|title=AI Has a Hallucination Problem That's Proving Tough to Fix|magazine=WIRED |url=https://www.wired.com/story/ai-has-a-hallucination-problem-thats-proving-tough-to-fix/|access-date=10 March 2018}} A 2019 study reported that humans can guess how machines will classify adversarial images.{{cite journal|last1=Zhou|first1=Zhenglong|last2=Firestone|first2=Chaz|year=2019 |title=Humans can decipher adversarial images|journal=Nature Communications|volume=10|issue=1|page=1334 |arxiv=1809.04120|pmc=6430776 |pmid=30902973|bibcode=2019NatCo..10.1334Z|doi=10.1038/s41467-019-08931-6 |doi-access=free}} Researchers discovered methods for perturbing the appearance of a stop sign such that an autonomous vehicle classified it as a merge or speed limit sign.{{Cite web|last=Ackerman|first=Evan|date=2017-08-04|title=Slight Street Sign Modifications Can Completely Fool Machine Learning Algorithms|url=https://spectrum.ieee.org/slight-street-sign-modifications-can-fool-machine-learning-algorithms |access-date=2019-07-15 |website=IEEE Spectrum: Technology, Engineering, and Science News}}

McAfee attacked Tesla's former Mobileye system, fooling it into driving 50 mph over the speed limit, simply by adding a two-inch strip of black tape to a speed limit sign.{{cite magazine|date=2020|title=A Tiny Piece of Tape Tricked Teslas Into Speeding Up 50 MPH|language=en|magazine=Wired|url=https://www.wired.com/story/tesla-speed-up-adversarial-example-mgm-breach-ransomware/|access-date=11 March 2020}}{{Cite web|date=2020-02-19|title=Model Hacking ADAS to Pave Safer Roads for Autonomous Vehicles|url=https://securingtomorrow.mcafee.com/blogs/other-blogs/mcafee-labs/model-hacking-adas-to-pave-safer-roads-for-autonomous-vehicles|access-date=2020-03-11|website=McAfee Blogs|language=en-US}}

Adversarial patterns on glasses or clothing designed to deceive facial-recognition systems or license-plate readers, have led to a niche industry of "stealth streetwear".{{cite magazine|last1=Seabrook|first1=John|date=2020|title=Dressing for the Surveillance Age|language=en|magazine=The New Yorker|url=https://www.newyorker.com/magazine/2020/03/16/dressing-for-the-surveillance-age|access-date=5 April 2020}}

An adversarial attack on a neural network can allow an attacker to inject algorithms into the target system.{{cite journal|last1=Heaven|first1=Douglas|date=October 2019|title=Why deep-learning AIs are so easy to fool|journal=Nature|language=en|volume=574|issue=7777|pages=163–166 |bibcode=2019Natur.574..163H|doi=10.1038/d41586-019-03013-5|pmid=31597977|s2cid=203928744 |doi-access=}} Researchers can also create adversarial audio inputs to disguise commands to intelligent assistants in benign-seeming audio;{{cite journal|last1=Hutson|first1=Matthew|date=10 May 2019|title=AI can now defend itself against malicious messages hidden in speech |journal=Nature|doi=10.1038/d41586-019-01510-1|pmid=32385365|s2cid=189666088}} a parallel literature explores human perception of such stimuli.{{cite arXiv|eprint=2003.12362|class=eess.AS|first1=Michael A|last1=Lepori|first2=Chaz |last2=Firestone|title=Can you hear me now? Sensitive comparisons of human and machine perception |date=2020-03-27}}{{cite arXiv|eprint=2001.08444|class=eess.AS|first1=Jon|last1=Vadillo |first2=Roberto|last2=Santana|title=On the human evaluation of audio adversarial examples|date=2020-01-23}}

Clustering algorithms are used in security applications. Malware and computer virus analysis aims to identify malware families, and to generate specific detection signatures.D. B. Skillicorn. "Adversarial knowledge discovery". IEEE Intelligent Systems, 24:54–61, 2009.B. Biggio, G. Fumera, and F. Roli. "[http://pralab.diee.unica.it/en/node/1103 Pattern recognition systems under attack: Design issues and research challenges] {{Webarchive|url=https://web.archive.org/web/20220520211435/https://pralab.diee.unica.it/en/node/1103 |date=2022-05-20 }}". Int'l J. Patt. Recogn. Artif. Intell., 28(7):1460002, 2014.

Attacks against (supervised) machine learning algorithms have been categorized along three primary axes:{{Cite journal|last1=Barreno|first1=Marco|last2=Nelson |first2=Blaine |last3=Joseph|first3=Anthony D.|last4=Tygar|first4=J. D.|year=2010|title=The security of machine learning|url=https://link.springer.com/content/pdf/10.1007/s10994-010-5188-5.pdf |journal=Machine Learning|volume=81|issue=2 |pages=121–148|doi=10.1007/s10994-010-5188-5|doi-access=free |s2cid=2304759}} influence on the classifier, the security violation and their specificity.

• Classifier influence: An attack can influence the classifier by disrupting the classification phase. This may be preceded by an exploration phase to identify vulnerabilities. The attacker's capabilities might be restricted by the presence of data manipulation constraints.{{cite book|last=Sikos|first=Leslie F.|title=AI in Cybersecurity|publisher=Springer|year=2019|isbn=978-3-319-98841-2|series=Intelligent Systems Reference Library|volume=151|location=Cham|page=50|doi=10.1007/978-3-319-98842-9|s2cid=259216663 }}
• Security violation: An attack can supply malicious data that gets classified as legitimate. Malicious data supplied during training can cause legitimate data to be rejected after training.
• Specificity: A targeted attack attempts to allow a specific intrusion/disruption. Alternatively, an indiscriminate attack creates general mayhem.

This taxonomy has been extended into a more comprehensive threat model that allows explicit assumptions about the adversary's goal, knowledge of the attacked system, capability of manipulating the input data/system components, and on attack strategy.B. Biggio, G. Fumera, and F. Roli. "[http://pralab.diee.unica.it/en/node/657 Security evaluation of pattern classifiers under attack] {{Webarchive|url=https://web.archive.org/web/20180518055115/http://pralab.diee.unica.it/en/node/657|date=2018-05-18}}". IEEE Transactions on Knowledge and Data Engineering, 26(4):984–996, 2014.{{cite book|last1=Biggio|first1=Battista|title=Support Vector Machines Applications|last2=Corona|first2=Igino|last3=Nelson|first3=Blaine|last4=Rubinstein |first4=Benjamin I. P.|last5=Maiorca |first5=Davide|last6=Fumera|first6=Giorgio|last7=Giacinto |first7=Giorgio|last8=Roli|first8=Fabio|date=2014|publisher=Springer International Publishing|isbn=978-3-319-02300-7|pages=105–153|chapter=Security Evaluation of Support Vector Machines in Adversarial Environments|doi=10.1007/978-3-319-02300-7_4|arxiv=1401.7727|s2cid=18666561}} This taxonomy has further been extended to include dimensions for defense strategies against adversarial attacks.{{Cite journal|last1=Heinrich|first1=Kai|last2=Graf|first2=Johannes|last3=Chen|first3=Ji|last4=Laurisch |first4=Jakob|last5=Zschech|first5=Patrick|date=2020-06-15|title=Fool Me Once, Shame On You, Fool Me Twice, Shame On Me: A Taxonomy of Attack and De-fense Patterns for AI Security|journal=ECIS 2020 Research Papers|url=https://aisel.aisnet.org/ecis2020_rp/166}}

Below are some of the most commonly encountered attack scenarios.

Poisoning consists of contaminating the training dataset with data designed to increase errors in the output. Given that learning algorithms are shaped by their training datasets, poisoning can effectively reprogram algorithms with potentially malicious intent. Concerns have been raised especially for user-generated training data, e.g. for content recommendation or natural language models. The ubiquity of fake accounts offers many opportunities for poisoning. Facebook reportedly removes around 7 billion fake accounts per year.{{Cite web |date=2021-09-27 |title=Facebook removes 15 Billion fake accounts in two years |url=https://www.techdigest.tv/2021/09/facebook-removes-15-billion-fake-accounts-in-two-years.html |access-date=2022-06-08 |website=Tech Digest |language=en-GB}}{{Cite web |agency=Associated Press |date=2019-05-23 |title=Facebook removed 3 billion fake accounts in just 6 months |url=https://nypost.com/2019/05/23/facebook-removed-3-billion-fake-accounts-in-just-6-months/ |access-date=2022-06-08 |website=New York Post |language=en-US}} Poisoning has been reported as the leading concern for industrial applications.

On social medias, disinformation campaigns attempt to bias recommendation and moderation algorithms, to push certain content over others.

A particular case of data poisoning is the backdoor attack,{{Cite journal |last1=Schwarzschild |first1=Avi |last2=Goldblum |first2=Micah |last3=Gupta |first3=Arjun |last4=Dickerson |first4=John P. |last5=Goldstein |first5=Tom |date=2021-07-01 |title=Just How Toxic is Data Poisoning? A Unified Benchmark for Backdoor and Data Poisoning Attacks |url=https://proceedings.mlr.press/v139/schwarzschild21a.html |journal=International Conference on Machine Learning |language=en |publisher=PMLR |pages=9389–9398}} which aims to teach a specific behavior for inputs with a given trigger, e.g. a small defect on images, sounds, videos or texts.

File:AI training data poisoning illustration (Nightshade).png

For instance, intrusion detection systems are often trained using collected data. An attacker may poison this data by injecting malicious samples during operation that subsequently disrupt retraining.B. Biggio, B. Nelson, and P. Laskov. "[http://pralab.diee.unica.it/en/node/751 Support vector machines under adversarial label noise] {{Webarchive|url=https://web.archive.org/web/20200803190135/http://pralab.diee.unica.it/en/node/751 |date=2020-08-03 }}". In Journal of Machine Learning Research – Proc. 3rd Asian Conf. Machine Learning, volume 20, pp. 97–112, 2011.M. Kloft and P. Laskov. "[http://www.jmlr.org/papers/volume13/kloft12b/kloft12b.pdf Security analysis of online centroid anomaly detection]". Journal of Machine Learning Research, 13:3647–3690, 2012.

Data poisoning techniques can also be applied to text-to-image models to alter their output, which is used by artists to defend their copyrighted works or their artistic style against imitation.{{Cite web |last=Edwards |first=Benj |date=2023-10-25 |title=University of Chicago researchers seek to "poison" AI art generators with Nightshade |url=https://arstechnica.com/information-technology/2023/10/university-of-chicago-researchers-seek-to-poison-ai-art-generators-with-nightshade/ |access-date=2023-10-27 |website=Ars Technica |language=en-us}}

Data poisoning can also happen unintentionally through model collapse, where models are trained on synthetic data.{{Cite web |last=Rao |first=Rahul |title=AI-Generated Data Can Poison Future AI Models |url=https://www.scientificamerican.com/article/ai-generated-data-can-poison-future-ai-models/ |access-date=2024-06-22 |website=Scientific American |language=en}}

As machine learning is scaled, it often relies on multiple computing machines. In federated learning, for instance, edge devices collaborate with a central server, typically by sending gradients or model parameters. However, some of these devices may deviate from their expected behavior, e.g. to harm the central server's model{{Cite journal |last1=Baruch |first1=Gilad |last2=Baruch |first2=Moran |last3=Goldberg |first3=Yoav |date=2019 |title=A Little Is Enough: Circumventing Defenses For Distributed Learning |journal=Advances in Neural Information Processing Systems |publisher=Curran Associates, Inc. |volume=32|arxiv=1902.06156 |url=https://proceedings.neurips.cc/paper/2019/hash/ec1c59141046cd1866bbbcdfb6ae31d4-Abstract.html}} or to bias algorithms towards certain behaviors (e.g., amplifying the recommendation of disinformation content). On the other hand, if the training is performed on a single machine, then the model is very vulnerable to a failure of the machine, or an attack on the machine; the machine is a single point of failure.{{Cite journal |last1=El-Mhamdi |first1=El-Mahdi |last2=Guerraoui |first2=Rachid |last3=Guirguis |first3=Arsany |last4=Hoang |first4=Lê-Nguyên |last5=Rouault |first5=Sébastien |date=2022-05-26 |title=Genuinely distributed Byzantine machine learning |journal=Distributed Computing |volume=35 |issue=4 |pages=305–331 |doi=10.1007/s00446-022-00427-9 |s2cid=249111966 |issn=1432-0452|doi-access=free |arxiv=1905.03853 }} In fact, the machine owner may themselves insert provably undetectable backdoors.{{cite arXiv |last1=Goldwasser |first1=S. |last2=Kim |first2=Michael P. |last3=Vaikuntanathan |first3=V. |last4=Zamir |first4=Or |date=2022 |title=Planting Undetectable Backdoors in Machine Learning Models |class=cs.LG |eprint=2204.06974}}

The current leading solutions to make (distributed) learning algorithms provably resilient to a minority of malicious (a.k.a. Byzantine) participants are based on robust gradient aggregation rules.{{Cite journal |last1=Blanchard |first1=Peva |last2=El Mhamdi |first2=El Mahdi |last3=Guerraoui |first3=Rachid |last4=Stainer |first4=Julien |date=2017 |title=Machine Learning with Adversaries: Byzantine Tolerant Gradient Descent |url=https://proceedings.neurips.cc/paper/2017/hash/f4b9ec30ad9f68f89b29639786cb62ef-Abstract.html |journal=Advances in Neural Information Processing Systems |publisher=Curran Associates, Inc. |volume=30}}{{Cite journal |last1=Chen |first1=Lingjiao |last2=Wang |first2=Hongyi |last3=Charles |first3=Zachary |last4=Papailiopoulos |first4=Dimitris |date=2018-07-03 |title=DRACO: Byzantine-resilient Distributed Training via Redundant Gradients |url=https://proceedings.mlr.press/v80/chen18l.html |journal=International Conference on Machine Learning |language=en |publisher=PMLR |pages=903–912|arxiv=1803.09877 }}{{Cite journal |last1=Mhamdi |first1=El Mahdi El |last2=Guerraoui |first2=Rachid |last3=Rouault |first3=Sébastien |date=2018-07-03 |title=The Hidden Vulnerability of Distributed Learning in Byzantium |journal=International Conference on Machine Learning |publisher=PMLR |pages=3521–3530 |arxiv=1802.07927 |url=https://proceedings.mlr.press/v80/mhamdi18a.html}}{{Cite arXiv |last1=Allen-Zhu |first1=Zeyuan |last2=Ebrahimianghazani |first2=Faeze |last3=Li |first3=Jerry |last4=Alistarh |first4=Dan |date=2020-09-28 |title=Byzantine-Resilient Non-Convex Stochastic Gradient Descent |class=cs.LG |eprint=2012.14368}} [https://openreview.net/forum?id=PbEHqvFtcS Review]{{Cite conference |conference=9th International Conference on Learning Representations (ICLR), May 4–8, 2021 (virtual conference) |last1=Mhamdi |first1=El Mahdi El |last2=Guerraoui |first2=Rachid |last3=Rouault |first3=Sébastien |date=2020-09-28 |title=Distributed Momentum for Byzantine-resilient Stochastic Gradient Descent |url=https://infoscience.epfl.ch/record/287261 |access-date=2022-10-20}} [https://openreview.net/forum?id=H8UHdhWG6A3 Review]{{Cite journal |last1=Data |first1=Deepesh |last2=Diggavi |first2=Suhas |date=2021-07-01 |title=Byzantine-Resilient High-Dimensional SGD with Local Iterations on Heterogeneous Data |journal=International Conference on Machine Learning |publisher=PMLR |pages=2478–2488 |url=https://proceedings.mlr.press/v139/data21a.html}} The robust aggregation rules do not always work especially when the data across participants has a non-iid distribution. Nevertheless, in the context of heterogeneous honest participants, such as users with different consumption habits for recommendation algorithms or writing styles for language models, there are provable impossibility theorems on what any robust learning algorithm can guarantee.{{Cite arXiv |last1=Karimireddy |first1=Sai Praneeth |last2=He |first2=Lie |last3=Jaggi |first3=Martin |date=2021-09-29 |title=Byzantine-Robust Learning on Heterogeneous Datasets via Bucketing |class=cs.LG |eprint=2006.09365}} [https://openreview.net/forum?id=jXKKDEi5vJt Review]

Evasion attacksB. Nelson, B. I. Rubinstein, L. Huang, A. D. Joseph, S. J. Lee, S. Rao, and J. D. Tygar. "[http://www.jmlr.org/papers/volume13/nelson12a/nelson12a.pdf Query strategies for evading convex-inducing classifiers]". J. Mach. Learn. Res., 13:1293–1332, 2012 consist of exploiting the imperfection of a trained model. For instance, spammers and hackers often attempt to evade detection by obfuscating the content of spam emails and malware. Samples are modified to evade detection; that is, to be classified as legitimate. This does not involve influence over the training data. A clear example of evasion is image-based spam in which the spam content is embedded within an attached image to evade textual analysis by anti-spam filters. Another example of evasion is given by spoofing attacks against biometric verification systems.

Evasion attacks can be generally split into two different categories: black box attacks and white box attacks.

Model extraction involves an adversary probing a black box machine learning system in order to extract the data it was trained on.{{Cite web|date=2020-04-06|title=How to steal modern NLP systems with gibberish?|url=http://cleverhans.io/2020/04/06/stealing-bert.html|access-date=2020-10-15|website=cleverhans-blog|language=en}}{{cite arXiv|eprint=2009.06112|class=cs.CR|first1=Xinran|last1=Wang|first2=Yu|last2=Xiang|title=Information Laundering for Model Privacy|date=2020-09-13|last3=Gao|first3=Jun|last4=Ding|first4=Jie}} This can cause issues when either the training data or the model itself is sensitive and confidential. For example, model extraction could be used to extract a proprietary stock trading model which the adversary could then use for their own financial benefit.

In the extreme case, model extraction can lead to model stealing, which corresponds to extracting a sufficient amount of data from the model to enable the complete reconstruction of the model.

On the other hand, membership inference is a targeted model extraction attack, which infers the owner of a data point, often by leveraging the overfitting resulting from poor machine learning practices.{{Cite web|last=Dickson|first=Ben|date=2021-04-23|title=Machine learning: What are membership inference attacks?|url=https://bdtechtalks.com/2021/04/23/machine-learning-membership-inference-attacks/|access-date=2021-11-07|website=TechTalks|language=en-US}} Concerningly, this is sometimes achievable even without knowledge or access to a target model's parameters, raising security concerns for models trained on sensitive data, including but not limited to medical records and/or personally identifiable information. With the emergence of transfer learning and public accessibility of many state of the art machine learning models, tech companies are increasingly drawn to create models based on public ones, giving attackers freely accessible information to the structure and type of model being used.

There is a growing literature about adversarial attacks in

linear models. Indeed, since the seminal work from Goodfellow at al.{{cite conference |last1=Goodfellow |first1=Ian J. |last2=Shlens |first2=Jonathon |last3=Szegedy |first3=Christian |date=2015 |title=Explaining and Harnessing Adversarial Examples |conference=International Conference on Learning Representations (ICLR)}} studying these models in linear models has been an important tool to understand how adversarial attacks affect machine learning models.

The analysis of these models is simplified because the computation of adversarial attacks can be simplified in linear regression and classification problems. Moreover, adversarial training is convex in this case.{{cite conference |last1=Ribeiro |first1=Antonio H. |last2=Zachariah |first2=Dave |last3=Bach |first3=Francis |last4=Schön |first4=Thomas B. |year=2023 |title=Regularization properties of adversarially-trained linear regression |url=https://openreview.net/forum?id=K8gLHZIgVW |conference=Thirty-seventh Conference on Neural Information Processing Systems}}

Linear models allow for analytical analysis while still reproducing phenomena observed in state-of-the-art models.

One prime example of that is how this model can be used to explain the trade-off between robustness and accuracy.{{cite conference |last1=Tsipras |first1=Dimitris |last2=Santurkar |first2=Shibani |last3=Engstrom |first3=Logan |last4=Turner |first4=Alexander |last5=Ma |first5=Aleksander |date=2019 |title=Robustness May Be At Odds with Accuracy |conference=International Conference for Learning Representations}}

Diverse work indeed provides analysis of adversarial attacks in linear models, including asymptotic analysis for classification {{cite conference |last1=Dan |first1=C. |last2=Wei |first2=Y. |last3=Ravikumar |first3=P. |date=2020 |title=Sharp statistical guarantees for adversarially robust Gaussian classification |url=https://proceedings.mlr.press/v119/dan20b.html |conference=International Conference on Machine Learning}} and for linear regression.{{cite conference |last1=Javanmard |first1=A. |last2=Soltanolkotabi |first2=M. |last3=Hassani |first3=H. |date=2020 |title=Precise tradeoffs in adversarial training for linear regression |url=http://proceedings.mlr.press/v125/javanmard20a.html |conference=Conference on Learning Theory}}{{cite journal |last1=Ribeiro |first1=A. H. |last2=Schön |first2=T. B. |date=2023 |title=Overparameterized Linear Regression under Adversarial Attacks |journal=IEEE Transactions on Signal Processing |volume=71 |pages=601–614 |arxiv=2204.06274 |bibcode=2023ITSP...71..601R |doi=10.1109/TSP.2023.3246228}} And, finite-sample analysis based on Rademacher complexity.{{cite conference |last1=Yin |first1=D. |last2=Kannan |first2=R. |last3=Bartlett |first3=P. |date=2019 |title=Rademacher Complexity for Adversarially Robust Generalization |url=https://proceedings.mlr.press/v97/yin19b.html |conference=International Conference on Machine Learning}}

A result from studying adversarial attacks in linear models is that it closely relates to regularization.{{Citation |last1=Ribeiro |first1=Antônio H. |title=Regularization properties of adversarially-trained linear regression |date=2023-10-16 |arxiv=2310.10807 |last2=Zachariah |first2=Dave |last3=Bach |first3=Francis |last4=Schön |first4=Thomas B.}} Under certain conditions, it has been shown that

• adversarial training of a linear regression model with input perturbations restricted by the infinity-norm closely resembles Lasso regression, and that
• adversarial training of a linear regression model with input perturbations restricted by the 2-norm closely resembles Ridge regression.

Adversarial deep reinforcement learning is an active area of research in reinforcement learning focusing on vulnerabilities of learned policies. In this research area, some studies initially showed that reinforcement learning policies are susceptible to imperceptible adversarial manipulations.{{cite journal |last1=Goodfellow |first1=Ian |last2=Shlens |first2=Jonathan |last3=Szegedy |first3=Christian |title=Explaining and Harnessing Adversarial Examples |journal=International Conference on Learning Representations |date=2015 |arxiv=1412.6572}}{{Cite book |last1=Pieter |first1=Huang |first2=Sandy |last2=Papernot |first3=Nicolas |last3=Goodfellow |first4=Ian |last4=Duan |first5=Yan |last5=Abbeel |title=Adversarial Attacks on Neural Network Policies |date=2017-02-07 |oclc=1106256905}} While some methods have been proposed to overcome these susceptibilities, in the most recent studies it has been shown that these proposed solutions are far from providing an accurate representation of current vulnerabilities of deep reinforcement learning policies.{{cite journal |last1=Korkmaz |first1=Ezgi |date=2022 |title=Deep Reinforcement Learning Policies Learn Shared Adversarial Features Across MDPs |journal=Thirty-Sixth AAAI Conference on Artificial Intelligence (AAAI-22) |volume=36 |issue=7 |pages=7229–7238 |doi=10.1609/aaai.v36i7.20684 |arxiv=2112.09025|s2cid=245219157 }}

Adversarial attacks on speech recognition have been introduced for speech-to-text applications, in particular for Mozilla's implementation of DeepSpeech.{{cite book |last1=Carlini |first1=Nicholas |last2=Wagner |first2=David |title=2018 IEEE Security and Privacy Workshops (SPW) |chapter=Audio Adversarial Examples: Targeted Attacks on Speech-to-Text |date=2018 |pages=1–7 |doi=10.1109/SPW.2018.00009 |arxiv=1801.01944 |isbn=978-1-5386-8276-0 |s2cid=4475201 }}

There are a large variety of different adversarial attacks that can be used against machine learning systems. Many of these work on both deep learning systems as well as traditional machine learning models such as SVMs and linear regression.{{Cite book|last1=Jagielski|first1=Matthew|last2=Oprea|first2=Alina |last3=Biggio|first3=Battista|last4=Liu |first4=Chang|last5=Nita-Rotaru|first5=Cristina|last6=Li |first6=Bo|title=2018 IEEE Symposium on Security and Privacy (SP) |chapter=Manipulating Machine Learning: Poisoning Attacks and Countermeasures for Regression Learning |date=May 2018|publisher=IEEE|pages=19–35 |arxiv=1804.00308|doi=10.1109/sp.2018.00057|isbn=978-1-5386-4353-2|s2cid=4551073}} A high level sample of these attack types include:

• Adversarial Examples{{Cite web|date=2017-02-24|title=Attacking Machine Learning with Adversarial Examples|url=https://openai.com/blog/adversarial-example-research/|access-date=2020-10-15 |website=OpenAI}}
• Trojan Attacks / Backdoor Attacks{{cite arXiv|eprint=1708.06733|class=cs.CR|first1=Tianyu |last1=Gu|first2=Brendan|last2=Dolan-Gavitt|title=BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain|date=2019-03-11|last3=Garg|first3=Siddharth}}
• Model Inversion{{Cite journal|last1=Veale|first1=Michael|last2=Binns|first2=Reuben|last3=Edwards |first3=Lilian|date=2018-11-28|title=Algorithms that remember: model inversion attacks and data protection law |journal=Philosophical Transactions. Series A, Mathematical, Physical, and Engineering Sciences|volume=376|issue=2133|arxiv=1807.04644|bibcode=2018RSPTA.37680083V|doi=10.1098/rsta.2018.0083 |issn=1364-503X |pmc=6191664|pmid=30322998}}
• Membership Inference{{cite arXiv|eprint=1610.05820|class=cs.CR|first1=Reza|last1=Shokri |first2=Marco|last2=Stronati|title=Membership Inference Attacks against Machine Learning Models |date=2017-03-31 |last3=Song|first3=Congzheng|last4=Shmatikov|first4=Vitaly}}

An adversarial example refers to specially crafted input that is designed to look "normal" to humans but causes misclassification to a machine learning model. Often, a form of specially designed "noise" is used to elicit the misclassifications. Below are some current techniques for generating adversarial examples in the literature (by no means an exhaustive list).

• Gradient-based evasion attack
• Fast Gradient Sign Method (FGSM){{cite arXiv|eprint=1412.6572|class=stat.ML|first1=Ian J.|last1=Goodfellow|first2=Jonathon|last2=Shlens|title=Explaining and Harnessing Adversarial Examples|date=2015-03-20|last3=Szegedy|first3=Christian}}
• Projected Gradient Descent (PGD){{cite arXiv|eprint=1706.06083|class=stat.ML|first1=Aleksander|last1=Madry|first2=Aleksandar|last2=Makelov|title=Towards Deep Learning Models Resistant to Adversarial Attacks|date=2019-09-04|last3=Schmidt|first3=Ludwig|last4=Tsipras|first4=Dimitris|last5=Vladu|first5=Adrian}}
• Carlini and Wagner (C&W) attack{{cite arXiv|eprint=1608.04644|class=cs.CR|first1=Nicholas|last1=Carlini|first2=David|last2=Wagner|title=Towards Evaluating the Robustness of Neural Networks|date=2017-03-22}}
• Adversarial patch attack{{cite arXiv|eprint=1712.09665|class=cs.CV|first1=Tom B.|last1=Brown|first2=Dandelion|last2=Mané|title=Adversarial Patch|date=2018-05-16|last3=Roy|first3=Aurko|last4=Abadi|first4=Martín|last5=Gilmer|first5=Justin}}

Black box attacks in adversarial machine learning assume that the adversary can only get outputs for provided inputs and has no knowledge of the model structure or parameters.{{Cite journal |last1=Guo|first1=Sensen|last2=Zhao|first2=Jinxiong|last3=Li|first3=Xiaoyu|last4=Duan |first4=Junhong|last5=Mu|first5=Dejun|last6=Jing|first6=Xiao|date=2021-04-24|title=A Black-Box Attack Method against Machine-Learning-Based Anomaly Network Flow Detection Models|journal=Security and Communication Networks|volume=2021|at=e5578335|doi=10.1155/2021/5578335|issn=1939-0114|doi-access=free}} In this case, the adversarial example is generated either using a model created from scratch, or without any model at all (excluding the ability to query the original model). In either case, the objective of these attacks is to create adversarial examples that are able to transfer to the black box model in question.{{Cite web|last=Gomes|first=Joao|date=2018-01-17|title=Adversarial Attacks and Defences for Convolutional Neural Networks|url=https://medium.com/onfido-tech/adversarial-attacks-and-defences-for-convolutional-neural-networks-66915ece52e7|access-date=2021-10-23 |website=Onfido Tech}}

Simple Black-box Adversarial Attacks is a query-efficient way to attack black-box image classifiers.{{Cite journal |last1=Guo |first1=Chuan |last2=Gardner |first2=Jacob |last3=You |first3=Yurong |last4=Wilson |first4=Andrew Gordon |last5=Weinberger |first5=Kilian |date=2019-05-24 |title=Simple Black-box Adversarial Attacks |url=https://proceedings.mlr.press/v97/guo19a.html |journal=Proceedings of the 36th International Conference on Machine Learning |language=en |publisher=PMLR |pages=2484–2493|arxiv=1905.07121 }}

Take a random orthonormal basis $v\_1,\; v\_2,\; \backslash dots,\; v\_d$ in $\backslash R^d$. The authors suggested the discrete cosine transform of the standard basis (the pixels).

For a correctly classified image $x$, try $x\; +\; \backslash epsilon\; v\_1,\; x\; -\; \backslash epsilon\; v\_1$, and compare the amount of error in the classifier upon $x\; +\; \backslash epsilon\; v\_1,\; x,\; x\; -\; \backslash epsilon\; v\_1$. Pick the one that causes the largest amount of error.

Repeat this for $v\_2,\; v\_3,\; \backslash dots$ until the desired level of error in the classifier is reached.

The Square Attack was introduced in 2020 as a black box evasion adversarial attack based on querying classification scores without the need of gradient information.{{Cite book |last1=Andriushchenko |first1=Maksym|last2=Croce|first2=Francesco|last3=Flammarion|first3=Nicolas |last4=Hein|first4=Matthias|title=Computer Vision – ECCV 2020 |chapter=Square Attack: A Query-Efficient Black-Box Adversarial Attack via Random Search |date=2020|editor-last=Vedaldi|editor-first=Andrea|editor2-last=Bischof |editor2-first=Horst|editor3-last=Brox|editor3-first=Thomas|editor4-last=Frahm|editor4-first=Jan-Michael |chapter-url=https://link.springer.com/chapter/10.1007/978-3-030-58592-1_29 |series=Lecture Notes in Computer Science|volume=12368|location=Cham|publisher=Springer International Publishing|pages=484–501|doi=10.1007/978-3-030-58592-1_29 |arxiv=1912.00049|isbn=978-3-030-58592-1|s2cid=208527215}} As a score based black box attack, this adversarial approach is able to query probability distributions across model output classes, but has no other access to the model itself. According to the paper's authors, the proposed Square Attack required fewer queries than when compared to state-of-the-art score-based black box attacks at the time.

To describe the function objective, the attack defines the classifier as $f:[0,\; 1]^d\; \backslash rightarrow\; \backslash reals^K$, with $d$ representing the dimensions of the input and $K$ as the total number of output classes. $f\_k(x)$ returns the score (or a probability between 0 and 1) that the input $x$ belongs to class $k$, which allows the classifier's class output for any input $x$ to be defined as $\backslash text\{argmax\}\_\{k=1,...,K\}f\_k(x)$. The goal of this attack is as follows:

$$\backslash text\{argmax\}\_\{k\; =\; 1,...,K\}f\_k(\backslash hat\{x\})\; \backslash neq\; y,\; ||\backslash hat\{x\}\; -\; x||\_p\; \backslash leq\; \backslash epsilon\; \backslash text\{\; and\; \}\; \backslash hat\{x\}\; \backslash in\; [0,\; 1]^d$$

In other words, finding some perturbed adversarial example $\backslash hat\{x\}$ such that the classifier incorrectly classifies it to some other class under the constraint that $\backslash hat\{x\}$ and $x$ are similar. The paper then defines loss $L$ as $L(f(\backslash hat\{x\}),\; y)\; =\; f\_y(\backslash hat\{x\})\; -\; \backslash max\_\{k\; \backslash neq\; y\}f\_k(\backslash hat\{x\})$ and proposes the solution to finding adversarial example $\backslash hat\{x\}$ as solving the below constrained optimization problem:

$$\backslash min\_\{\backslash hat\{x\}\; \backslash in\; [0,\; 1]^d\}L(f(\backslash hat\{x\}),\; y),\; \backslash text\{\; s.t.\; \}\; ||\backslash hat\{x\}\; -\; x||\_p\; \backslash leq\; \backslash epsilon$$

The result in theory is an adversarial example that is highly confident in the incorrect class but is also very similar to the original image. To find such example, Square Attack utilizes the iterative random search technique to randomly perturb the image in hopes of improving the objective function. In each step, the algorithm perturbs only a small square section of pixels, hence the name Square Attack, which terminates as soon as an adversarial example is found in order to improve query efficiency. Finally, since the attack algorithm uses scores and not gradient information, the authors of the paper indicate that this approach is not affected by gradient masking, a common technique formerly used to prevent evasion attacks.

This black box attack was also proposed as a query efficient attack, but one that relies solely on access to any input's predicted output class. In other words, the HopSkipJump attack does not require the ability to calculate gradients or access to score values like the Square Attack, and will require just the model's class prediction output (for any given input). The proposed attack is split into two different settings, targeted and untargeted, but both are built from the general idea of adding minimal perturbations that leads to a different model output. In the targeted setting, the goal is to cause the model to misclassify the perturbed image to a specific target label (that is not the original label). In the untargeted setting, the goal is to cause the model to misclassify the perturbed image to any label that is not the original label. The attack objectives for both are as follows where $x$ is the original image, $x^\backslash prime$ is the adversarial image, $d$ is a distance function between images, $c^*$ is the target label, and $C$ is the model's classification class label function:{{cite arXiv |title=HopSkipJumpAttack: A Query-Efficient Decision-Based Attack|language=en|last1 = Chen|first1 = Jianbo|last2 = Jordan|first2 = Michael I.|last3 = Wainwright|first3 = Martin J.|year = 2019|class=cs.LG |eprint = 1904.02144}} [https://www.youtube.com/watch?v=vkCifg2rp34 YouTube presentation]

$$\backslash textbf\{Targeted:\; \}\; \backslash min\_\{x^\backslash prime\}d(x^\backslash prime,\; x)\; \backslash text\{\; subject\; to\; \}\; C(x^\backslash prime)\; =\; c^*$$

$$\backslash textbf\{Untargeted:\; \}\; \backslash min\_\{x^\backslash prime\}d(x^\backslash prime,\; x)\; \backslash text\{\; subject\; to\; \}\; C(x^\backslash prime)\; \backslash neq\; C(x)$$

To solve this problem, the attack proposes the following boundary function $S$ for both the untargeted and targeted setting:

This can be further simplified to better visualize the boundary between different potential adversarial examples:

With this boundary function, the attack then follows an iterative algorithm to find adversarial examples $x^\backslash prime$ for a given image $x$ that satisfies the attack objectives.

1. Initialize $x$ to some point where $S(x)\; >\; 0$
2. Iterate below
3. Boundary search
4. Gradient update
5. * Compute the gradient
6. * Find the step size

Boundary search uses a modified binary search to find the point in which the boundary (as defined by $S$) intersects with the line between $x$ and $x^\backslash prime$. The next step involves calculating the gradient for $x$, and update the original $x$ using this gradient and a pre-chosen step size. HopSkipJump authors prove that this iterative algorithm will converge, leading $x$ to a point right along the boundary that is very close in distance to the original image.

However, since HopSkipJump is a proposed black box attack and the iterative algorithm above requires the calculation of a gradient in the second iterative step (which black box attacks do not have access to), the authors propose a solution to gradient calculation that requires only the model's output predictions alone. By generating many random vectors in all directions, denoted as $u\_b$, an approximation of the gradient can be calculated using the average of these random vectors weighted by the sign of the boundary function on the image $x^\backslash prime\; +\; \backslash delta\_\{u\_b\}$, where $\backslash delta\_\{u\_b\}$ is the size of the random vector perturbation:

$$\backslash nabla\; S(x^\backslash prime,\; \backslash delta)\; \backslash approx\; \backslash frac\{1\}\{B\}\backslash sum\_\{b=1\}^\{B\}\backslash phi(x^\backslash prime\; +\; \backslash delta\_\{u\_b\})\; u\_b$$

The result of the equation above gives a close approximation of the gradient required in step 2 of the iterative algorithm, completing HopSkipJump as a black box attack.{{cite arXiv|last1=Andriushchenko|first1=Maksym|last2=Croce|first2=Francesco|last3=Flammarion|first3=Nicolas|last4=Hein|first4=Matthias|date=2020-07-29|title=Square Attack: a query-efficient black-box adversarial attack via random search|class=cs.LG|eprint=1912.00049}}{{Cite web|date=2020-06-21|title=Black-box decision-based attacks on images|url=https://davideliu.com/2020/06/21/black-box-decision-based-attacks-on-images/|access-date=2021-10-25|website=KejiTech|language=en}}

White box attacks assumes that the adversary has access to model parameters on top of being able to get labels for provided inputs.

One of the first proposed attacks for generating adversarial examples was proposed by Google researchers Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy.{{cite arXiv|last1=Goodfellow|first1=Ian J.|last2=Shlens|first2=Jonathon|last3=Szegedy|first3=Christian|date=2015-03-20|title=Explaining and Harnessing Adversarial Examples|class=stat.ML|eprint=1412.6572}} The attack was called fast gradient sign method (FGSM), and it consists of adding a linear amount of in-perceivable noise to the image and causing a model to incorrectly classify it. This noise is calculated by multiplying the sign of the gradient with respect to the image we want to perturb by a small constant epsilon. As epsilon increases, the model is more likely to be fooled, but the perturbations become easier to identify as well. Shown below is the equation to generate an adversarial example where $x$ is the original image, $\backslash epsilon$ is a very small number, $\backslash Delta\_x$ is the gradient function, $J$ is the loss function, $\backslash theta$ is the model weights, and $y$ is the true label.{{Cite web|title=Adversarial example using FGSM {{!}} TensorFlow Core|url=https://www.tensorflow.org/tutorials/generative/adversarial_fgsm|access-date=2021-10-24|website=TensorFlow|language=en}}

$$adv\_x\; =\; x\; +\; \backslash epsilon\; \backslash cdot\; sign(\backslash Delta\_xJ(\backslash theta,\; x,\; y))$$

One important property of this equation is that the gradient is calculated with respect to the input image since the goal is to generate an image that maximizes the loss for the original image of true label $y$. In traditional gradient descent (for model training), the gradient is used to update the weights of the model since the goal is to minimize the loss for the model on a ground truth dataset. The Fast Gradient Sign Method was proposed as a fast way to generate adversarial examples to evade the model, based on the hypothesis that neural networks cannot resist even linear amounts of perturbation to the input.{{Cite web|last=Tsui|first=Ken|date=2018-08-22|title=Perhaps the Simplest Introduction of Adversarial Examples Ever|url=https://towardsdatascience.com/perhaps-the-simplest-introduction-of-adversarial-examples-ever-c0839a759b8d|access-date=2021-10-24|website=Medium|language=en}} FGSM has shown to be effective in adversarial attacks for image classification and skeletal action recognition.{{cite conference |last1=Corona-Figueroa |first1=Abril |last2=Bond-Taylor |first2=Sam |last3=Bhowmik |first3=Neelanjan |last4=Gaus |first4=Yona Falinie A. |last5=Breckon |first5=Toby P. |last6=Shum |first6=Hubert P. H. |last7=Willcocks |first7=Chris G. |title=Unaligned 2D to 3D Translation with Conditional Vector-Quantized Code Diffusion using Transformers |date=2023 |publisher=IEEE/CVF |arxiv=2308.14152 }}

In an effort to analyze existing adversarial attacks and defenses, researchers at the University of California, Berkeley, Nicholas Carlini and David Wagner in 2016 propose a faster and more robust method to generate adversarial examples.{{cite arXiv|last1=Carlini|first1=Nicholas|last2=Wagner|first2=David|date=2017-03-22|title=Towards Evaluating the Robustness of Neural Networks|class=cs.CR|eprint=1608.04644}}

The attack proposed by Carlini and Wagner begins with trying to solve a difficult non-linear optimization equation:

$$\backslash min(||\backslash delta||\_\{p\})\; \backslash text\{\; subject\; to\; \}\; C(x\; +\; \backslash delta)\; =\; t,\; x\; +\; \backslash delta\; \backslash in\; [0,\; 1]^n$$

Here the objective is to minimize the noise ($\backslash delta$), added to the original input $x$, such that the machine learning algorithm ($C$) predicts the original input with delta (or $x\; +\; \backslash delta$) as some other class $t$. However instead of directly the above equation, Carlini and Wagner propose using a new function $f$ such that:

$$C(x\; +\; \backslash delta)\; =\; t\; \backslash iff\; f(x\; +\; \backslash delta)\; \backslash leq\; 0$$

This condenses the first equation to the problem below:

$$\backslash min(||\backslash delta||\_\{p\})\; \backslash text\{\; subject\; to\; \}\; f(x\; +\; \backslash delta)\; \backslash leq\; 0,\; x\; +\; \backslash delta\; \backslash in\; [0,\; 1]^n$$

and even more to the equation below:

$$\backslash min(||\backslash delta||\_\{p\}\; +\; c\; \backslash cdot\; f(x\; +\; \backslash delta)),\; x\; +\; \backslash delta\; \backslash in\; [0,\; 1]^n$$

Carlini and Wagner then propose the use of the below function in place of $f$ using $Z$, a function that determines class probabilities for given input $x$. When substituted in, this equation can be thought of as finding a target class that is more confident than the next likeliest class by some constant amount:

$$f(x)\; =\; ([\backslash max\_\{i\; \backslash neq\; t\}Z(x)\_i]\; -\; Z(x)\_t)^\{+\}$$

When solved using gradient descent, this equation is able to produce stronger adversarial examples when compared to fast gradient sign method that is also able to bypass defensive distillation, a defense that was once proposed to be effective against adversarial examples.{{Cite web|title=carlini wagner attack|url=http://richardjordan.com/ucdfk/carlini-wagner-attack|access-date=2021-10-23|website=richardjordan.com}}{{Cite web|last=Plotz|first=Mike|date=2018-11-26|title=Paper Summary: Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods|url=https://medium.com/@hyponymous/paper-summary-adversarial-examples-are-not-easily-detected-bypassing-ten-detection-methods-faf040e54e93|access-date=2021-10-23|website=Medium|language=en}}

File:Proactive_arms_race.jpg

Researchers have proposed a multi-step approach to protecting machine learning.

• Threat modeling – Formalize the attackers goals and capabilities with respect to the target system.
• Attack simulation – Formalize the optimization problem the attacker tries to solve according to possible attack strategies.
• Attack impact evaluation
• Countermeasure design
• Noise detection (For evasion based attack){{Cite journal|arxiv=2007.00337|author1=Kishor Datta Gupta|first2=Zahid|last2=Akhtar|title=Determining Sequence of Image Processing Technique (IPT) to Detect Adversarial Attacks|last3=Dasgupta|first3=Dipankar|journal=SN Computer Science|year=2021|volume=2|issue=5|page=383|doi=10.1007/s42979-021-00773-8|issn=2662-995X|s2cid=220281087}}
• Information laundering – Alter the information received by adversaries (for model stealing attacks)

A number of defense mechanisms against evasion, poisoning, and privacy attacks have been proposed, including:

• Secure learning algorithmsO. Dekel, O. Shamir, and L. Xiao. "[https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/DekelShXi09.pdf Learning to classify with missing and corrupted features]". Machine Learning, 81:149–178, 2010.{{Cite journal|last1=Liu|first1=Wei |last2=Chawla|first2=Sanjay|year=2010|title=Mining adversarial patterns via regularized loss minimization|url=https://link.springer.com/content/pdf/10.1007/s10994-010-5199-2.pdf|journal=Machine Learning|volume=81|pages=69–83 |doi=10.1007/s10994-010-5199-2|doi-access=free|s2cid=17497168}}
• Byzantine-resilient algorithms
• Multiple classifier systemsB. Biggio, G. Fumera, and F. Roli. "[http://pralab.diee.unica.it/en/node/642 Evade hard multiple classifier systems] {{Webarchive|url=https://web.archive.org/web/20150115114531/http://pralab.diee.unica.it/en/node/642 |date=2015-01-15 }}". In O. Okun and G. Valentini, editors, Supervised and Unsupervised Ensemble Methods and Their Applications, volume 245 of Studies in Computational Intelligence, pages 15–38. Springer Berlin / Heidelberg, 2009.
• AI-written algorithms.
• AIs that explore the training environment; for example, in image recognition, actively navigating a 3D environment rather than passively scanning a fixed set of 2D images.
• Privacy-preserving learningB. I. P. Rubinstein, P. L. Bartlett, L. Huang, and N. Taft. "Learning in a large function space: Privacy- preserving mechanisms for svm learning". Journal of Privacy and Confidentiality, 4(1):65–100, 2012.
• Ladder algorithm for Kaggle-style competitions
• Game theoretic modelsM. Kantarcioglu, B. Xi, C. Clifton. [http://www.stat.purdue.edu/~xbw/research/BoweiXi.AdversarialClassification2010.pdf "Classifier Evaluation and Attribute Selection against Active Adversaries"]. Data Min. Knowl. Discov., 22:291–335, January 2011.{{Cite journal|last1=Chivukula|first1=Aneesh|last2=Yang|first2=Xinghao|last3=Liu |first3=Wei|last4=Zhu|first4=Tianqing |last5=Zhou |first5=Wanlei|date=2020|title=Game Theoretical Adversarial Deep Learning with Variational Adversaries|url=https://ieeexplore.ieee.org/document/8986751 |journal=IEEE Transactions on Knowledge and Data Engineering |volume=33|issue=11|pages=3568–3581|doi=10.1109/TKDE.2020.2972320|hdl=10453/145751 |s2cid=213845560|issn=1558-2191|hdl-access=free}}{{Cite journal|last1=Chivukula|first1=Aneesh Sreevallabh|last2=Liu|first2=Wei|date=2019|title=Adversarial Deep Learning Models with Multiple Adversaries|url=https://ieeexplore.ieee.org/document/8399545|journal=IEEE Transactions on Knowledge and Data Engineering|volume=31|issue=6|pages=1066–1079|doi=10.1109/TKDE.2018.2851247|issn=1558-2191|hdl-access=free|s2cid=67024195|hdl=10453/136227}}
• Sanitizing training data
• Adversarial training
• Backdoor detection algorithms{{Cite web|title=TrojAI|url=https://www.iarpa.gov/index.php/research-programs/trojai|access-date=2020-10-14|website=www.iarpa.gov}}
• Gradient masking/obfuscation techniques: to prevent the adversary exploiting the gradient in white-box attacks. This family of defenses is deemed unreliable as these models are still vulnerable to black-box attacks or can be circumvented in other ways.{{cite arXiv|eprint=1802.00420v1 |first1=Anish| last1=Athalye| first2=Nicholas|last2=Carlini| first3=David| last3=Wagner| title=Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Example| date=2018-02-01|class=cs.LG }}
• Ensembles of models have been proposed in literature, which have shown to be ineffective against evasion attacks{{cite arXiv |eprint=1706.04701 |class=cs.LG |first1=Warren |last1=He |first2=James |last2=Wei |title=Adversarial Example Defenses: Ensembles of Weak Defenses are not Strong |date=2017-06-15 |first3=Xinyun |last3=Chen |first4=Nicholas |last4=Carlini |first5=Dawn |last5=Song}} but effective against data poisoning attacks.{{Cite journal |title=Data poisoning attacks against machine learning algorithms |journal=Expert Systems with Applications |last1=Yerlikaya |first1=Fahri Anıl |date=2022-07-14 |url=https://doi.org/10.1016/j.eswa.2022.118101 |volume=208 |via=Elsevier Science Direct |last2=Bahtiyar |first2=Şerif|doi=10.1016/j.eswa.2022.118101 }}

• Pattern recognition
• Fawkes (image cloaking software)
• Generative adversarial network

{{reflist}}

• [https://atlas.mitre.org/ MITRE ATLAS: Adversarial Threat Landscape for Artificial-Intelligence Systems]
• [https://csrc.nist.gov/publications/detail/nistir/8269/draft NIST 8269 Draft: A Taxonomy and Terminology of Adversarial Machine Learning]
• NIPS 2007 Workshop on [https://web.archive.org/web/20120108072159/http://nips.cc/Conferences/2007/Program/event.php?ID=615 Machine Learning in Adversarial Environments for Computer Security]
• [http://pralab.diee.unica.it/en/ALFASVMLib AlfaSVMLib] {{Webarchive|url=https://web.archive.org/web/20200924221614/http://pralab.diee.unica.it/en/ALFASVMLib |date=2020-09-24 }} – Adversarial Label Flip Attacks against Support Vector Machines
• {{cite journal|last1=Laskov|first1=Pavel|last2=Lippmann|first2=Richard|year=2010|title=Machine learning in adversarial environments|journal=Machine Learning|volume=81|issue=2|pages=115–119|doi=10.1007/s10994-010-5207-6|s2cid=12567278}}
• Dagstuhl Perspectives Workshop on "[https://www.dagstuhl.de/en/seminars/seminar-calendar/seminar-details/12371 Machine Learning Methods for Computer Security]"
• Workshop on [http://aisec.cc/ Artificial Intelligence and Security], (AISec) Series

{{Artificial intelligence navbox}}

Category:Machine learning

Category:Computer security

Category:AI safety