Defect.AI
RUNNING TIME:
10/2023
—
03/2026
Total project duration:
30 Monate
The project
Detection of impairments and defects using computer vision methods has been increasingly adopted in recent years in different industry sectors, for example in media and manufacturing. This includes for example the detection of dust and scratches in film material, block and playback distortions due to data loss in digitised or digitally transmitted video, surface or structural impairments of manufactured products in RGB and hyperspectral images, and many more. Most deployed systems use highly sophisticated algorithms incorporating the specialised know-how of domain experts.
While deep-learning based approaches have brought significant progress in many computer vision applications, the specific nature of many defect detection tasks and the lack of training data (both due to the costs of creating datasets, but also due to the rare occurrence of many relevant types of defects) has hindered the adoption of these methods so far.
The aim of Defect.AI is to research basic technologies for data-efficient training of visual defect detection methods. They will make use of recent advances in machine learning such as foundation models and expert knowledge distillation.
The project is implemented together with our partner HS-ART-Digital.
Research groups
Downloads
Customers
Bundesministerium für Klimaschutz, Umwelt, Energie, Mobilität, Innovation und Technologie
vertreten durch die Österreichische Forschungsförderungsgesellschaft mbH (FFG)
Project partners
HS-ART-Digital
Project details
Defect.AI Defect.AI researched basic technologies for data-efficient training of visual defect detection methods. They use recent advances in machine learning such as foundation models, which are pre-trained in a self-supervised manner on large unannotated data sets and are then adjusted to a specific task using transfer learning. The work confirms the suitability of foundation models (like SWIN or other vision transformers) for defect detection tasks, especially in small dataset scenarios. An important aspect in media use cases is the capability of multi-image detection. Robustness of defect detection can be massively increased by using multiple consecutive images of a film or a video during the training and the inference process. By this approach temporal properties of defects can be taken directly into consideration by the model.
The following images show an example of single frame defects originating from dust and dirt on film (original image on the left), where defect detection (the detected defect region is shown in the middle) is the first and crucial step for subsequent digital film restoration (restored image is shown on the right).
Another important aspect is to capture the expert knowledge embedded in existing handcrafted detectors. Knowledge distillation is used to learn this specific knowledge, at scale and without requiring manual annotation. This legacy knowledge captured in a detection model can then be fine tuned with expert annotated, high-quality knowledge, where only a very low amount of expert annotated samples is required.
When there are only a few examples available, it is important to be able to supplement the data with high-quality examples. For this we researched methods to extract defect samples from an original image into an intermediate defect’s representation, and to generate from them a high number of defect samples on target images with various relevant content properties (like various motions, luminance…) and various other defect properties (e.g. different film grain noise), while at the same time preserving the original appearance of the defect. The following pictures show the original image with a defect contained the extracted defect sample area, the augmented high quality target defect area and the entire target image containing the augmented defect. Augmented defect samples are extremely helpful to train or fine tune a defect detection model for cases where only a small set of original defect samples or content properties is available.
The researched methods incorporating fine-tuning of foundation models with a multi-image approach and highest quality defect augmentation is a very efficient solution for defect detection, especially in small dataset scenarios.
