A 16×16 Patch Attack on Vision Transformers with Differential Evolution
Abdulsalam, Abass Bamidele
Abdulsalam, Abass Bamidele
Author
Supervisor
Department
Computer Vision
Embargo End Date
Type
Thesis
Date
2022
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Vision transformers (ViTs) have recently started gaining popularity in the field of Ma- chine Learning (ML) and Computer Vision (CV). They have in reality exhibited excellent performance on many computer vision tasks including image classification. This makes ViT(s) a potential substitute for convolutional neural networks (CNNs). Previous tests on the limitation of machine learning models have shown that they are exposed to the possibility of danger through adversarial examples, which are visually-imperceptible perturbations carefully supplied to the inputs, making them deceptive. Extensive works have been dedicated to adversarial attacks on CNNs under black-box and white-box settings. However, despite this numerous works, similar studies are still underway in the case of ViTs especially in the black box setting. In essence, the logic behind ViTs is to split input images into sequence of small patches in which relationships are computed among pixels in these splitted patches while also encapsulating global inter- actions among them as its centre of attention. This is believed to strengthen such networks against local perturbations, such as those induced by adversarial attacks. In this thesis, we study the impact of an attack in a very limited setting in which one or a few patches can be subtly modified with the purpose of fooling ViT models. We build upon the famous one-pixel attack for fooling DNNs to generate a patch-wise adversarial perturbations based on differential evolution (DE). Compared to many of the current attack methods, our attack is a black-box attack which do not require any information about the model to fool it. When tested on imagenet 1k dataset, our result shows that more than 75% of the dataset can be misclassified with more coloured patches.
Citation
A.B. Abdulsalam, "A 16×16 Patch Attack on Vision Transformers with Differential Evolution", M.S. Thesis, Computer Vision, MBZUAI, Abu Dhabi, UAE, 2022.