Less is more? An ablation study on AutoAttack for adversarial robustness evaluation

AutoAttack is widely recognized as a standard adversarial robustness evaluation framework, yet the individual contributions of its components and mechanisms remain insufficiently explored. In this work, we present a comprehensive ablation study on the standard AutoAttack version, isolating the singular contribution of each component, focusing on the attack ensemble, random initialization, and Expectation over Transformation (EoT) optimization across four different state-of-the-art robust models. Our analysis reveals that simplified attack sequences often achieve results comparable to the complete AutoAttack sequence while requiring significantly fewer computational resources. Furthermore, our findings show that EoT generally provides modest improvements in attack success rate, while the benefits of random initialization may vary depending on the model architecture. By identifying which among the AutoAttack components has the most significant influence on the robustness evaluation, our work offers practical recommendations for designing efficient evaluation frameworks that balance thoroughness with computational cost considerations.