Implementing Vision Transformers (ViT) vs CNNs for industrial defect detection?

All answers to this question.

For high-resolution industrial tasks, a standard ViT is often inefficient because the attention mechanism scales quadratically with the number of patches. However, Swin Transformers (Shifted Window) are a perfect middle ground. They compute attention within local windows but shift those windows to provide hierarchical representation, similar to a CNN’s receptive field growth. In 2024, we switched from a heavy CNN ensemble to a Swin-B model for a similar quality control task and saw a 12% improvement in the mAP (mean Average Precision) for very small, non-local defects.

   Answered 2024-08-15 by Patricia Lewis

Are you planning to deploy this model on the edge or in the cloud? ViT architectures often require significantly more data and longer training times to outperform a well-tuned CNN. Do you have a labeled dataset in the range of 100k+ images, or are you relying on pre-trained models and fine-tuning with a smaller niche dataset?

   Answered 2024-08-17 by Steven Harris

• Steven, we are deploying on-prem with NVIDIA Jetson Orin modules. Our dataset is around 50k images, but we use heavy data augmentation. Your point about data hunger is well-taken; we noticed the ViT was overfitting early on. We’ve decided to start with a "ConvNeXt" architecture first—it’s a CNN that uses many of the ViT's design choices. It seems more robust for our current data scale while still providing that modern performance boost.

     Commented 2024-08-19 by Thomas Wright

If the "cracks" are very small, make sure you aren't downsampling too much. Using a "Sliding Window" approach for inference can help maintain the detail regardless of the model architecture.

   Answered 2024-08-20 by Barbara Scott

• I agree with Barbara. For semiconductor defects, the features are often just a few pixels wide. Downsampling is the enemy of accuracy in this specific domain.

     Commented 2024-08-21 by Patricia Lewis