4.1.1 Study Raw Data

Study Raw Data is the raw MRMC reading scores. It include 2 parts: 1. Study Description, 2. List of ROC ratings.

Study Description can include any information as free text, or no information at all. iMRMC looks for four lines corresponding to the size of the experiment.

N0 = number of cases without disease

N1 = number of cases with disease

NR = number of readers

NM = number of modalities

The list of ROC ratings begins with "BEGIN DATA:'. It has a row for each score—each reader, case, modality combination observed—in any order. It also has rows specifying the truth state of each case (given by the reference gold-standard reader = truth). iMRMC can handle data that is not fully-crossed for any number of modalities, though only two of those modalities may be used at a time for variance analysis.

For rows recording reader scores, there are four fields in the following order:

reader I.D. (string)

The reader I.D. can be any string (or number) to identify a study reader. Make sure that this reader I.D. appears the same way each time or the program will think that there are multiple readers. For example, iMRMC believes that “novice0247”, “novice 0247”, “NOVICE0247” are three different readers. Do not change the I.D. of a reader when recording his or her scores from a different modality.

Note: Do not use “-1” as that indicates a row recording the truth state.

case I.D. (string)

The case I.D. can be any string (or number) to identify the case imaged. Make sure that this case appears the same way each time or the program will think that there are multiple cases. For example, iMRMC believes that “TeacherMale15”, “Male Teacher15”, “TeacherM15” are three different cases. Do not change the I.D. of a case when recording its scores from a different modality.

modality I.D. (string)

The modality I.D. can be any string (or number) to identify the imaging modality. Some examples could be: “film”, “digital”, “digital+CAD”, “MRI”. Again be careful how you type the modality I.D.

score (integer or float)

The score is the level of confidence, likelihood, rank, or severity of disease that the reader gives to the case for the modality indicated. The score can have any scale and can be negative : percent, fraction, integer (1 to 7, 0 to 100, -20 to +20), or floating point. Low scores correspond to truth state 0 (normal or no disease) and high scores correspond to truth state 1 (abnormal or disease). If low scores correspond to truth state 1 and high scores correspond to truth state 0, AUC calculations and ROC curve displays will be incorrect.

For rows recording truth states, there are also four fields in the following order:

Truth flag (-1 or truth)

Rows recording case truth states are denoted with a -1 or truth in the first field.

Case ID (Integer)

Every case I.D. scored by a reader (appears in a row recording a reader score described above) must have one and only one row recording the truth state. Each case must be defined as 0 (normal or no disease) or 1 (abnormal or disease).

Arbitrary String (String)

This string can be anything and it needs to be something. It is currently unused by iMRMC. You can use “0” as a reminder of what the row is recording.

Truth state (0 or 1)

The truth state 0 indicates normal or no disease. The truth state 1 indicates abnormal or disease.

Higher reader scores should indicate higher likelihood or confidence of disease and low ratings should indicate lesser likelihood or confidence of disease.

FIELDS MUST BE SEPARATED BY COMMAS!*

For example, one input file could look like the following: