Validating generated model

In this tutorial, you will learn how to run a calibration session with mpVision to make sure your AI model is accurate and reliable.

In order to validate your newly generated AI model or to test an existing model with your data, you will need to run a calibration session. This is an important step in the process of implementing AI models in any production environment, and it is especially important in GxP and other regulated fields. The calibration session will help you understand how well your model performs on unseen data (data that was not used for training) and how to improve it, if needed.

To demonstrate this function for this tutorial, we will be using the same Kaggle dataset that we used in training tutorial. It contains images of three classes: two types of blood cells and platelets.

Creating a dataset

After logging in to the application, click on the "Datasets" under "Calibration" navigation item in the sidebar menu located on the left side of the application view. It will navigate you to the view containing the list of your calibration datasets (it will be empty if you haven't created any yet). Similar to the training datasets, you can create a new calibration dataset by clicking on the "Create new dataset" button located in the top left corner of the lists view.

Uploading images

Now that you have created a calibration dataset, you can start uploading images to it. First, click on the dataset name in the list of calibration datasets. This will navigate you to the view containing the gallery of images in the dataset. To upload images, either drag and drop them into the highlighted area in the center of the view, or click the highlighted area to open the file picker dialog. You can upload multiple images at once, but to keep it simple for this tutorial, we will upload only two images.

Labeling images

Just like how you labeled images in the training dataset, you need to label images in the calibration dataset. Except this time, the dataset will not be used for training the model, but rather for comparing the model outputs with assigned labels — the ground truth for model calibration.

First, click on the image you want to label in the gallery view. This will open the image in the labeling component where you can draw bounding boxes and polygons around the objects you want to label. Since our model is trained to detect blood cells types, specifically "RBC", "WBC" and "Platelets", we need to label them exactly the same way in the calibration dataset. That way system will be able to compare the model outputs and calculate the performance metrics.

And similarly to labeling training dataset images, you can use autodetection functions to speed up the process. For this tutorial, we have already labeled the images for you. Download them from this link, and use the import function in the dataset creation view to upload them. But this time, select "Datasets" under "Calibration" navigation item in the sidebar menu.

After completing the labeling process, we can start a calibration session.

Running a calibration session

To initiate a calibration session, navigate to the dataset gallery view by clicking on the dataset name in the list of calibration datasets. Or, if you are in image labeling tool, click on the calibration datasets breadcrumb located in the top left corner of the image view. In the gallery view, click on the "Calibrate" button located in the top right corner of the view. This will open a popover containing a select input with a list of your AI models — currently you will have one with the name you have provided in the training tutorial. Then select the "Start" button, and you should see a notification on the bottom right corner of the screen conveying that the calibration session has started.

In the right side from the "Calibrate" button, you will see a calibration sessions link that will show you the list of calibration sessions that have been initiated with this dataset. You can click on it to see the status of the run you just have started. When the calibration run is complete, you will be able to navigate to calibration report view by clicking on the "View" button located on the right side of the calibration session list item.

Calibration report

After the calibration session is complete, you can navigate to the calibration report view by clicking on the "View" button located on the right side of the calibration session list item. That view contains characteristics of the calibration session, such as the number of images processed, the number of objects detected, and the performance metrics of the model on the calibration dataset. The performance metrics includes average accuracies across all classes for all images as well as for each individual class.

Good model performance should have accuracy above 95% for all classes, but it is important to remember that the calibration dataset should be representative of the data that the model will be used on. If certain classes are particularly still challenging the AI model, it is recommended to add more images with those classes to the training data set and re-train the model. Model training is typically an iterative process.

When you achieve the desired accuracy, you can deploy the model to the production environment. You can also can go back to the training tutorial to improve the model according to your business metrics and run another calibration session.

For now, let's go ahead and run an inference session!