The Medical Image Segmentation Synthesis (MISS) Tool is software written in MATLAB that allows a user to produce synthetic segmentations and assess segmentation performance using a wide range of performance metrics implemented within the code. The MISS tool can be used through a graphical user interface (GUI) or as command-line functions inserted into a user’s own code. The GUI allows for visualization of the synthesis segmentation, interactive tuning of the synthesis parameters, and display of the segmentation evaluation results. The command-line mode allows for processing images in batches as well as providing flexible ways for users to integrate the MISS tool with their applications.

The MISS tool provides two basic components:

· Image segmentation synthesis function: This function allows a user to emulate a range of reader/algorithm segmentations through the adjustment of an input (truth) segmentation mask with various types of segmentation errors such as the addition of spiculations or other types of systematic contour changes. The user inputs an initial segmentation (truth) mask, and the image segmentation synthesis function outputs a new segmentation contour that includes the selected segmentation errors.

· Segmentation performance evaluation function: This function allows a user to compute 24 performance metrics, such as the Dice coefficient, Jaccard index (IoU), Medical Similarity Index (MSI), and Hausdorff Distance (HD), as part of a comprehensive segmentation evaluation applied to a single or set of imaging studies. The user provides a truth mask and a segmentation mask for each image, and the program outputs values for the 24-performance metrics chosen by the user.

Intended Purpose

The MISS tool supports multiple activities by end users and AI developers including:

· Investigating properties of segmentation performance metrics and informing segmentation metric selection.

· Investigating truthing methods and informing truthing method selection by allowing users to assess the impact of different augmentation methods for combining multiple segmentation (truth) masked provided by a set of truthers.

· Assessing the robustness of a segmentation algorithm by synthesizing controlled segmentation errors on a dataset that the algorithm is intended to be applied to and investigating the variability of the performance metrics.

· Evaluating the impact of segmentation errors on subsequent analyses through the synthesis of well controlled and customizable segmentation errors.

The MISS-tool allows users to customize segmentation errors by configurable parameters. For example, the magnitude of contour changes, the position and height of spiculations, and the area of overlapping between the truth mask and synthetic segmentation.

The intended users of this MISS tool include AI segmentation algorithm developers and assessors. The clinical use cases include AI-based segmentation applied to Digital Pathology and Radiology image datasets.

The Use of GUI

The GUI contains two options:

· Segmentation performance evaluation

· Image segmentation contour synthesis

To start the GUI, open the file “main_option.mlapp” and run it in MATLAB.

Main options

When running the MISS tool MATLAB-GUI, the Main Options window pops up.

If you have truth masks and binary segmentation results, to calculate segmentation performance metrics, select 1: Computing Segmentation Metrics.

If you have truth masks, to generate synthetic segmentation contours based on the truth masks, select 2: Synthesis of Segmentation Contours.

When either option is selected, the Main Options window is closed, and a new window will be shown. There is a “Return to main window” button on the new window to go back to the Main Options window.

Segmentation Performance Metrics

After clicking the “Computing Segmentation Metrics” on the main options window, you will see the window for Metrics Computation.

“Return to main window” button on the new window is for going back to the main window.

Evaluation of a single case

Load data

For evaluating segmentation performance of a single case, you click the “Segmentation contour” and “Reference contour” buttons in the “User Input (Single)” panel to input one segmentation contour and one (reference) truth mask.

The supported input file formats are: '*.jpg; *.tif; *.png; *.gif'. The image should have only one object (connected component) as the segmentation or truth mask (filled contour image).

Note: input images must have the same size.

After inputting an image, the file name of the image will show below the input button, the metrics values and images will display automatically.

To view result

The values of 24 metrics show in the “Segmentation Metrics” frame. On the Segmentation Viewer, the segmentation truth is shown as a blue mask, and the segmentation is shown as a green contour.

Note: by default, on Metrics Computation window the calculations of Hausdorff Distance (HD) and Average Hausdorff Distance (AVD) metrics are enabled. If you want to exclude the two metrics, uncheck the “Compute Hausdorff Distances (Slow)” box under the “Comparison” frame.

More details of these metrics are in the “Functions of segmentation evaluation metrics” section.

Results output

The values of 24 metrics auto-save in the “Workspace” of MATLAB, it is a MATLAB struct value named “metrics_value”.

By clicking the “Save metrics values” button, these values will be saved in a CSV file specified by the user.

Other options

· “Compute Hausdorff Distances (slower)” check box: on/off the calculations of Hausdorff Distance (HD) and Average Hausdorff Distance (AVD) metrics.

ßà

· “Display in filled contour” check box: on/off filling the area of segmentation. If checked the truth mask shows in filled red area, and the segmentation shows in filled blue area with green contour. Their overlapping area shows in purple.

ßà

Performance metrics computation on multiple cases

Load data

To load images and masks for multiple cases, you click the “Segmentation contours” and “Reference contours” buttons in the “User Input (Multiple cases)” panel to specify segmentation masks and truth (reference) masks.

The supported image formats are: '*.jpg; *.tif; *.png; *.gif'. The image should have only one object (connected component) as the segmentation or truth mask.

Note: 1) the numbers of segmentations and truth masks must be same and non-zero; 2) all input images must have the same size; 3) the program assumes one-to-one correspondence between the two lists, i.e., the first segmentation mask corresponds to the first truth mask, and so on.

After loading the image files, the number of image files will show below the buttons, and files’ names will print in MATLAB Command Window.

Results output

After loading the image files, you click the “Performance Metrics” button to start computing the 24 metrics values for each pair of segmentations and truth masks, and the results will be saved in a CSV file specified by the user.

Segmentation synthesis

Load data

First, you click the “Load Truth Mask” button (the blue arrow on above picture) to input a truth mask.

The supported image formats are: '*.jpg; *.tif; *.png; *.gif'. The image should have only one object (connected component) as the truth mask.

After loading a mask, the metrics values and images will display automatically, where the metrics reflect the segmentation error of the synthetic contour against the input truth contour. Without any modification, the synthetic segmentation is identical to the truth mask.

Note: by default, on Synthesis of Contour window the calculations of Hausdorff Distance (HD) and Average Hausdorff Distance (AVD) metrics are disabled due to a slow processing. If you want to include the two metrics, check the “Compute Hausdorff Distances (Slow)” box under the “Comparison” frame.

The “Recover” button is to undo all changes and recover the original mask image. For details of the “Compute Hausdorff Distances (slower)” check box and “Display in filled contour” check box, please see in the “Other options” section.

Synthesis Tools

Our tool provides three methods to modify the input truth mask and generate synthetic segmentation contours:

· Add spiculations – “Spiculation” button. An example:

· Contour change by Fourier Descriptor – “Fourier” button. An example:

· Affine transformation – “Affine” button. An example:

Each method has several parameters for the user to tune in order to generate different synthetic segmentations. For more details of these methods, please see in the “Functions of segmentation synthesis” section.

Add spiculations

Parameters

· Angle: angular location to put the spiculation, it is an angle value ranging from 0 to 359 degree.

· Height: the height of the spiculation, in terms of the number of pixels.

· Width: the width of the spiculation, in terms of the number of pixels.

For technical details of these parameters, please see in the “2.3 Spiculations” section of the paper:

Guan, S., Samala, R. K., Arab, A., & Chen, W. (2023, April). MISS-tool: medical image segmentation synthesis tool to emulate segmentation errors. In Medical Imaging 2023: Computer-Aided Diagnosis (Vol. 12465, pp. 273-281). SPIE. https://doi.org/10.1117/12.2653650

Operations (note these apply to the other

· Close: close the window.

· Apply to original: add the spiculation to the input mask. This operation will reset the mask that has been modified by other methods before adding a spiculation.

· Apply to current: add the spiculation to the current mask. This operation is to add a spiculation to a mask that has been modified by other methods (or to add multiple spiculations to a mask).

· Add params to Batch Processing: send and add all parameters of this modification to the Batch Processing window. For more details, please see in the “Batch Processing” section.

· Undo once: This operation will undo the last modification and recover the previous mask. When multiple changes have been applied, this operation can only undo the last change.

Note: changes by all the three Synthesis methods are counted, i.e., after the “undo once” in one Synthesis Tool window, the “undo once” function in other Synthesis Tool windows will not work anymore before the next modification is executed.

After clicking the “Apply to original” or “Apply to current”, the result will show on the Segmentation Viewer. Here we show an example of:

Angle: 270 (degree)

Height: 20 (pixels)

Width: 6 (pixels)

Apply to original.

Change contour by Fourier Descriptor

Parameters

· Detail: the number of lower-frequency Fourier Descriptors to be kept (FDs beyond this will be set to zero); the larger value for this parameter the more detail of the contour (high-frequency components) will be kept; ranging from 1 to the “Total #”, which is the total number of Fourier Descriptors for this contour.

· Range: the number of Fourier Descriptors to be modified; ranging from 0 to (Detail value – 1); the larger value for this parameter the greater change is applied to the contour.

· Magnitude: the magnitude of modification to the Fourier Descriptors to be made according to the Range; the larger value for this parameter the greater change is applied to the contour.

For technical details of these parameters, please see in the “2.4 Fourier Descriptor” section of the paper:

After clicking the “Apply to original” or “Apply to current”, the result will show on the Segmentation Viewer. Here is an example:

Detail: 20

Range: 12

Magnitude: 6

Apply to original.

Affine transformation

Parameters

· Resize: the ratio of size change on x-axis (vertical) and y-axis (horizontal). Here we show an example for x = 1.5, y=2 applying to original mask, without other changes:

· Rotation: the degree of counter-clockwise rotation around the center. Here is an example for rotation of90 degrees:

· translation: in terms of the number of pixels. Here is an example:

For technical details of these parameters, please see in the “2.1 Affine Transformation” section of the paper:

Results Output

After all changes have been applied to the mask, by clicking the “Save syn mask & metrics values” button, two files will be created: 1) the modified mask will be saved in a PNG file specified by the user, 2) and results of 24 segmentation metrics between the input truth mask and synthetic masks will be saved in a CSV file with a name PNG file name_Metrics.csv. The values of 24 metrics also auto-save in the “Workspace” of MATLAB, it is a MATLAB struct value named “metrics_value”.

Batch Processing

“Batch Processing” button allows you to apply several modifications, including Fourier Descriptor (FD), Spiculation (SP), and Affine transformation (AF), to one or many masks all at once.

Note: 1) To apply functions in the “Batch Processing” window, a truth mask must have been loaded from the main “Segmentation Viewer” window. 2) To click the “Return to main window” button in main “Segmentation Viewer” window, the “Batch Processing” window must be closed. 3) The modifications plan (MOD) will be deleted once the “Batch Processing” window is closed. To keep it, please click “Save to MOD file”.

Parameters

The parameters are grouped by the three types of modifications to masks, as shown in the “Synthesis Tools” before. To look at the details of the parameters for each type of modifications, please use their links: Fourier Descriptor (FD), Spiculation (SP), and Affine transformation (AF).

The parameters: Detail and Range for Fourier Descriptor (FD), will be presented and saved in percentage (%) forms. Where:

The details about these parameters can be found in Fourier Descriptor (FD).

Editing operations for modification plan

Add operations

1. Choose one operation to be added into the modification plan from the parameters field (right) by clicking its check box.

2. Edit the parameters.

3. Click “< Add” button. The operation will be added in queue with an order number.

Operations can also add/send into the modification plan from the Fourier Descriptor (FD), Spiculation (SP), and Affine transformation (AF) windows by clicking the “Add params to Batch Processing” button.

Remove operations

1. Select the operation to be removed on the Modifications list.

2. Click “Remove” button. This operation will be removed from the queue. The order numbers of operations after it will subtract one.

Change operations on the Modifications list

1. Select the operation to be changed on the Modifications list (left).

2. Choose one operation from the parameters field (right) by clicking its check box to substitute the selected operation (left).

3. Click “< Overwrite” button. The selected operation on the Modifications list (left) will be substituted by the new operation with checked box from the parameters field (right).

To click “Copy >” button, the parameters of selected operation will be copied to the according parameters field, and the title of operation on the parameters field (right) will flash for highlighting.

Applying modifications to mask(s)

Click the “Apply to Dashboard Sample (from original)” button, all operations on the Modifications list will be applied in their orders to the original mask loaded from the main window.

Note: this operation will reset the mask that has been modified by other methods before.

Click the “Apply to User Input (many)” button, all operations on the Modifications list will be applied in their orders to the selected masks by users. And the modified mask will be saved in PNG files under folder: “…/MISS-tool/Saved/Batch_saved/”, by adding “_syn” in their file names.

The supported image formats are: '*.jpg; *.tif; *.png; *.gif'. The image should have only one object (connected component) as the segmentation or truth mask. The image will be converted to binary image, of which “1” stands for the object (foreground) and “0” stands for the background.

Save & load modifications plan

Click the “Save to MOD file” button, the operations on the Modifications list will be saved in an XML file: “/MISS-tool/Saved/saved_batch_processing_params.xml”. To load saved operations into the Modifications list, click “Load from MOD file” and choose an XML file to open.

Note: “Load from MOD file” will reset the Modifications list. Unsaved operations will be lost.

The Use of Command (Code)

Similar to the GUI, the Command model has two parts:

· Segmentation performance evaluation for user’s inputs (truth and segmentation results)

· Image segmentation synthesis (truth masks provided by user)

Main functions

Function for segmentation evaluation metrics: seg_measures

This function is used for segmentation performance evaluation of user’s inputs (truth masks and segmentation results)

Usage: idx = seg_measures(truth_mask, seg_mask, compute_Hausdorff)

Input: {truth_mask, seg_mask}: two binary images {0, 1} for the truth (reference) mask and segmentation mask; same size

compute_Hausdorff: if compute Hausdorff Distances (slower); Boolean.

o true: compute HD and AVD

o false: do NOT compute HD and AVD

Output: A struct data includes 24 indexes of segmentation evaluation metrics ^[ref]

1. DICE Dice Coefficient

2. JAC Jaccard Coefficient

3. MSI Medical Similarity Index

4. FN False Negative (Confusion matrix)

5. FP False Positive (Confusion matrix)

6. TN True Negative (Confusion matrix)

7. TP True Positive (Confusion matrix)

8. TPR Sensitivity (Recall, true positive rate)

9. TNR Specificity (true negative rate)

10. FPR Fallout (false positive rate)

11. PPV Precision (also called positive predictive value)

12. ACC Accuracy

13. AUC Area under ROC Curve (one system state)

14. KAP Cohen Kappa

15. ARI Adjusted Rand Index

16. ICC Interclass Correlation

17. VS Volumetric Similarity Coefficient

18. MI Mutual Information

19. VOI Variation of Information

20. PBD Probabilistic Distance

21. GCE Global Consistency Error

22. HD Hausdorff Distance (optionally in voxel or millimeter)

23. AVD Average Hausdorff Distance (optionally in voxel or millimeter)

24. MHD Mahanabolis Distance

For example, idx.DICE is the Dice Coefficient of img1 and img2.

Note: if compute_Hausdorff = false, idx.HD = NaN, idx.AVD = NaN.

References and definitions for metrics: the 3.MSI is from Kim, Haksoo, et al. "Quantitative evaluation of image segmentation incorporating medical consideration functions." Medical physics 42.6Part1 (2015): 3013-3023. https://doi.org/10.1118/1.4921067 , and other metrics are from Taha, A.A., Hanbury, A. Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool. BMC Med Imaging 15, 29 (2015). https://doi.org/10.1186/s12880-015-0068-x

Functions for segmentation synthesis

Inputs of the functions for segmentation synthesis are the image containing a mask to be modified and several parameters. The image is the key input to these functions, and it should include only one area (connected component) as the segmentation/mask to be modified. And the output from a function (modified mask) is the synthetic segmentation.

Add spiculations: mod_spike

This function is used to add spiculations to the mask of user’s input.

Usage: syn = mod_spike(img, center, height, width , GUI_use)

Input:

· img: a binary image {0, 1}

· center: location to put the spiculation, it is an angle value ranging from 0 to 359 degree (360 = 0 degree).

· height: the height of the spiculation, the number of pixels.

· width: the width of the spiculation, the number of pixels.

· GUI_use: control the type of input and output of this function. If true, the input and output are boundaries of the mask; if false, the input and output are mask images; Boolean.

o true: mask boundary -> mask boundary

o false: mask image -> mask image

For technical details of these parameters, please see in the “2.3 Spiculations” section of the paper:

Output: the modified mask (synthetic segmentation) with an added spiculation, same size as the input image.

Note: Running this function is to add one spiculation to the input mask at a time. To add more spiculations to a mask needs to run the function multiple times.

Change contour by Fourier Descriptor: mod_fourier

This function is used to change the mask contour of user’s input using Fourier Descriptors.

Usage: syn = mod_fourier(img, detail, range, magnitude, GUI_use)

Input:

· img: a binary image {0, 1}

· detail: the number of Fourier Descriptors to be not set to zero; the larger value the more detail of the contour (high-frequency components) will be kept; ranging from 1 to the half total number of Fourier Descriptors (pixels on the contour).

· range: the number of Fourier Descriptors to be modified; ranging from 0 to (Detail value – 1); the larger value the greater change is applied to the contour.

· magnitude: the magnitude of modification to the Fourier Descriptors to be changed according to the Range; the larger value the greater change is applied to the contour.

· GUI_use: control the type of input and output of this function. If true, the input is mask image and output is the boundary of mask; if false, the input and output are mask images; Boolean.

o true: mask image -> mask boundary

o false: mask image -> mask image

For technical details of these parameters, please see in the “2.4 Fourier Descriptor” section of the paper:

Output: A binary image includes the modified mask (synthetic segmentation) by changing the Fourier Descriptors of the mask, same size as the input image.

Note: due to the method of change includes random seeds, for each time running the function, the result could be different. For more details about the randomness, please see in the “2.4 Fourier Descriptor” section of the paper (https://doi.org/10.1117/12.2653650).

Affine transformation: mod_affine

This function is used to change the mask of user’s input using affine transformations, which includes: resizing (changing the height and width ratio), shifting, and rotation.

Usage: syn = mod_affine(img, resize, rotate, shift, GUI_use)

Input:

· img: a binary image {0, 1}

· resize: the ratio of size change on x-axis (vertical) and y-axis (horizontal). Input format: [rx, ry].

· rotation: the degree of rotation by the center of the mask; it is an angle value ranging from 0 to 359 degree (360 = 0 degree).

· shift: moving the mask by its center with the vector (X, Y), the number of pixels. Input format: [sx, sy].

· GUI_use: control the type of input and output of this function. If true, the input is mask image and output is the boundary of mask; if false, the input and output are mask images; Boolean.

o true: mask image -> mask boundary

o false: mask image -> mask image

For technical details of these parameters, please see in the “2.1 Affine Transformation” section of the paper:

Output: A binary image includes the modified mask (synthetic segmentation) by using affine transformations, same size as the input image.

Note: the “resize” and “shift” are dual-value parameters requiring two values for input; the format is like: [x, y].

An example of segmentation evaluation

Here we show an example for calculating the 24 metrics of segmentation images. 10 pairs of truth mask and segmentation images need to be evaluated. In the same folder of the MATLAB code, the file name of the i-th truth mask is “TruthMask_i.png”, and the file name of the i-th segmentation image is “Segmentation_i.png”.

Implement by GUI

1. Open Metrics Computation window (click the “Computing Segmentation Metrics” on the main options window).

2. Click the “Segmentation contourS” in the “User Input (Many)” panel to select 10 segmentation images.

3. Click the “Reference contourS” in the “User Input (Many)” panel to select the 10 truth masks.

4. Click the “Performance Metrics” buttons to start computing the 24 metrics values for each pair of segmentations and reference masks. Once calculation finished, it will ask you to save the results in a CSV file by your selection.

Implement by Command (Code)

The MATLAB code:

M={};

for i= 1: 10

truth_i = imread(['TruthMask_', num2str(i), '.png']);

seg_i = imread(['Segmentation_', num2str(i), '.png']);

M{i} = seg_measures(truth_i, seg_i, true);

end

The result M{} is a cell includes 10 struct data. For example, M{6}.DICE is the Dice Coefficient of “TruthMask_6.png” and “Segmentation_6.png”.

An example of segmentation synthesis

Here we show an example to generate 100 synthetic segmentation images (changing contour by Fourier Descriptor) from a truth mask. In the same folder of the MATLAB code, the file name of truth mask is “TruthMask.png”.

Implement by GUI

1. Open Synthesis of Contour window (click the “Synthesis of Segmentation Contours” on the main options window).

2. Click the “Load Truth Mask” button to load the truth mask “TruthMask.png”.

3. Click “Fourier” button to open the Fourier Descriptor window.

4. In the Fourier Descriptor window, set parameters like detail = 5, range = 3, and magnitude = 10. Then, click “Apply to original” button to see its modified result.

5. Since to generate more than one synthetic segmentation image, we use the Batch Processing function. Click the “Batch Processing” button in the main “Segmentation Viewer” window to open the Batch Processing window.

6. Click “Add params to Batch Processing” on the Fourier Descriptor window, you will see the three parameters are added into the Modifications list on the Batch Processing window. Note: on the Batch Processing window, the parameters: Detail and Range for Fourier Descriptor (FD), will be presented and saved in percentage (%) forms. Here for more details.

7. Make 100 copies of the truth mask and save them in a folder, like “TruthMask_copy1.png”, “TruthMask_copy2.png”, …, “TruthMask_copy100.png”.

8. Click the “Apply to User Input (many)” button, then select the 100 copies of the truth mask. Operations on the Modifications list will be applied in their orders to the selected masks by users. The modified (synthetic) mask will be saved in PNG files under folder: “…/MISS-tool/Saved/Batch_saved/”, by adding “_syn” in their file names. Like “TruthMask_copy1_syn.png”, “TruthMask_copy2_syn.png”, …, “TruthMask_copy100_syn.png”.

Implement by Command (Code)

The MATLAB code:

truth_img = imread('TruthMask.png');

% modify the mask by Fourier Descriptor; it contains randomness

detail = 5; % < total pixels

range = 3; % <= detail

magnitude = 10; % >=0

Syn_masks= cell(100,1);

for k = 1:100

Syn_masks{k} = mod_fourier(truth_img, detail, range, magnitude, false);

end

The result Syn_masks{} is a cell includes 100 segmentation images. For example, Syn_masks{i} is the i-th (synthetic) segmentation image.