Detect and Classify the Defected Imprinted Characters on Metal Plates

The task is to detect imprinted alphanumeric characters “A”, “F”, “1”, “2“ and “4” on metal plates and classify them as “OK", “NOK-Scratch”, “NOK-Bubble” or “NOK-DeShape”. This is a Detection followed by a Classification task and can be realized as a pipeline composed of a detector followed by a classifier.

Dataset

The dataset contains 384 metal plates each with an imprinted sequence of alphanumeric characters. The characters “A”, “F”, “1”, “2“ and “4” are subject to inspection and are labeled as “OK", “NOK-Scratch” , “NOK-Bubble” or “NOK-DeShape”. A sample metal plate is shown below.

Data Loading & Annotation

You have 2 options:

• 1. Download the data annotated with LabelMe format. Load the imageset and the annotations that you downloaded, using ReliVision. To do that, import the imageset using “Import Data” and the annotation file per image folder using the “Import Annotated Dataset” (with the LabelMe annotation format option). Your annotated dataset will be available in the gallery.
• 2. Load the imageset and annotate the data yourself in any industry standard format you like using the ReliVision’s intuitive annotation functions. To do that, download the image folder only. Follow the ReliVision Knowledge Hub User Guide (ReliTrainer: Data) to define the target state/label (which is a label of the character as OK, NOK-Scratch , NOK-Bubble or NOK-DeShape), to annotate the characters (using rectangle annotation tool as the task is object detection) and to save them in an appropriate industry standard format (for which LabelMe is a good option).  Sample annotation of two metal plates, with OK (green), NOK-Scratch (red), NOK-Bubble (yellow) and “NOK-DeShape” (orange) characters are shown below:

Building A Character Detection and Classification Pipeline:

We will be building a pipeline that is composed of a series connected character detector and a character classifier. The character detector will be responsible for detecting the characters , if there is any, and outputting a bounding box. The classifier will be fed with character detection results (the cropped images of the characters detected) and will decide on the type of the character (OK, NOK-Scratch , NOK-Bubble or NOK-DeShape).

Training a Character Detector:

1. Prepare your data for character detector training: In order to train a character detection model, you need to have training images in which all characters, of all types, are annotated with a common label, such as “character”. Merge your labels into a single class called “Character”. Visit ReliVision Knowledge Hub User Guide to see how to do this (Edit ROI States).

2. Follow the ReliVision Knowledge Hub User Guide (ReliTrainer: Models) to train your AI model for metal nut defect detection. The main steps, detailed in the User Guide, include

• Model type selection: Detection
• Annotated dataset selection
• Automated or manual train/test split which essentially spares some data for training validation purposes.
• Hyper parameter setting: We have chosen the following in this use case:
• Epochs: 1000
• X-Y Resolution: 640 (default)
• Learning Rate: 0.01 (default)
• Momentum: 0.937 (default)
• Weight Decay: 0.005 (default)

Sample training and validation loss and accuracy curves as a function of epochs are as follows:

Training a Defect Classifier:

1. Prepare your data for character classifier training: In order to train a character classifier model, you need to have training images of individual (cropped) characters with labels . The dataset provided has rectangle annotations with 4 labels corresponding to different label types. The best option is to

Extract the rectangle annotations using the ReliVision data curation functions. See ReliVision Knowledge Hub User Guide to see how to do this.

2. Follow the ReliVision Knowledge Hub User Guide (ReliTrainer: Models) to train your AI model for whole image classification (OK, NOK-Scratch , NOK-Bubble or NOK-DeShape). The main steps, detailed in the User Guide, include

• Model type selection: Classification
• Annotated dataset selection
• Automated or manual train/test split which essentially spares some data for training validation purposes.
• Hyper parameter setting: We have chosen the following in this use case:
• Epochs: 100 (default)
• X-Y Resolution: 640 (default)
• Learning Rate: 0.01 (default)
• Momentum: 0.937 (default)
• Weight Decay: 0.005 (default)

Sample training and validation loss and accuracy curves as a function of epochs are as follows:

Building and Running the Pipeline:

Use the pipeline editor to build your pipeline by dragging and dropping the AI/Basic blocks. For this specific case, we do not need any Basic Block (Digital signal/image processing - DSP/DIP - functions). You will need to select an input data source (your raw image set), a detection AI Block (the defect detector you trained) and a defect classifier (the classifier you trained) connected in series as depicted below. Follow the ReliVision Knowledge Hub User Guide (ReliTrainer: Models / AI Pipeline).

Simply run your pipeline using the execute button at the top. Visit the ReliVision Knowledge Hub User Guide (ReliTrainer: Models / AI Pipeline) for more details. You can review your results using the ReliTrainer data annotation interface. Your pipeline’s outputs will be saved as a separate set of annotations for each image it is run on.  Visit the ReliVision Knowledge Hub User Guide (ReliTrainer: Data) for more details. Here is an output example:

Sample Training and Testing Performance:

Follow the ReliVision Knowledge Hub User Guide (ReliTrainer: Models) to test your AI model for X-Ray object detection. Testing involves running the trained model on a given dataset using the “Prediction” tab. Each image in your test dataset will be augmented/annotated with your AI model’s output mask. Sample output masks are as follows:

• Detector:
• 0.97 IoU @ training ; 0.96 IoU @ testing.
• False Positive Rate (FPR) = 0%
• Sensitivity = 100%
• Classifier:
• Accuracy = 99.1%