Home             Utilities       Documentation -> ROC Curve Docs       FAQ       About

About ROC Curves  

Input data and generation of the ROC curve  

Using the ROC curve module  

About ROC curves

A ROC curve provides a graphical representation of the relationship between the true-positive and false-positive prediction rate of a model. The y-axis corresponds to the sensitivity of the model, i.e. how well the model is able to predict true positives (real cleavages) from sites that are not cleaved, and the y-coordinates are calculated as:

The x-axis corresponds to the specificity (expressed on the curve as 1-specificity), i.e. the ability of the model to identify true negatives. An increase in specificity (i.e. a decrease along the X-axis) results in an increase in sensitivity. The x-coordinates are calculated as:

The greater the sensitivity at high specificity values (i.e. high y-axis values at low X-axis values) the better the model. A numerical measure of the accuracy of the model can be obtained from the area under the curve, where an area of 1.0 signifies near perfect accuracy, while an area of less than 0.5 indicates that the model is worse than just random. The quantitative-qualitative relationship between area and accuracy follows a fairly linear pattern, such that the following could be used as a guide:

• 0.9-1: Excellent
• 0.8-0.9: Very good
• 0.7-0.8: Good
• 0.6-0.7: Average
• 0.5-0.6: Poor

Top

Input data and generation of the ROC curve

This module takes as input a set of PoPS scores and whether each score belongs to a real cleavage site (true positive, indicated using `+') or to a site that is known not to be cleaved (true negative, indicated using `-'). The file contains three columns, with the following headers on the first line: CleavageSite, TrueCleavage? and PoPSScore. An example input file is:

In this (very small) example, there are 5 true negatives (sites that are not cleaved) and one true positive (site that is cleaved). There is no provision for comments in the file, i.e. the file should only contain the data for the ROC curve. The headers must be included, but are not case-sensitive (upper- or lower-case can be used), but each header MUST be a single word, for example `CleavageSite' (no space), not `Cleavage Site' (space between the two words).

The above example produces the following ROC curve (download the example input file here):

Click here for larger version

The red line is the ROC curve for the input data, the gray line is the reference line. The output is discussed in detail in the section below.

The ROC curve is generated using the following steps:

1. Find all unique pairs of scores in the input file.
2. For each unique pair, calculate the threshold.

The ROC curve module uses an empirical technique to calculate the ROC curve, which does not use an assumed likelihood estimator to fit a smooth curve between the sample points. Instead, a set of thresholds are calculated from each unique pair of PoPS scores. The threshold values are simply the mid-point between each (unique) pair of scores in the input file. Note that for the first and last elements in the set of scores, the pair is defined as (Score_0-1, Score_0) and (Score_L, Score_L+1) where Score_0 and Score_L denote the first and last elements respectively.
3. For each threshold, classify the input scores.

The thresholds are used to classify the scores according to the following predicate rule:

If the score is greater than or equal to the threshold value, then it is classified as positive, otherwise it is classified as negative.

4. For each threshold/classification, calculate the normalised TP, TN, FP and FN using the true cleavage site indicator.

Because the true cleavage state (i.e. cleaved or not cleaved) for each score is specified, the true-positive (TP), true-negative (TN), false-positive (FP) and false-negative (FN) values can be calculated.
5. Calculate the specificity and sensitivity, adding each point to the ROC curve plot.

The normalised TP, TN, FP and FN values are used to calculate the specificity (X-coordinates) and sensitivity (Y-coordinates) of each point on the curve, using the equations described above.
6. Calculate the area under the curve.

The area under the curve is calculated using the trapezoid rule:

Using the ROC curve module

Top

Load a ROC data file.

To load a ROC data file, click on the "Load a data file button", and select the appropriate file. Note that the input file must follow the format described above.

Manipulating the output.

On loading a data file, the data will be plotted in the main window, and the name of the file will appear in the "Graph" list box at the bottom left of the module. To remove a ROC curve from the main graph, click on the name of the curve in the "Graph" list and use the ">>" arrow to transfer it to the "Don't Graph" list. The curve will automatically be removed from the graph in the main window. The curve can be added back into the main window by clicking on the name of the curve in the "Don't Graph" list and using the "<<" button to transfer it back to the "Graph" list.

Each curve is automatically named using the input file name, and assigned a colour and marker type. The curve settings can be edited by double-clicking on the name of the curve (in either the "Graph" or "Don't Graph" lists). The following window will appear, allowing the settings of curve name, colour, and line and marker styles to be changed:

This dialog allows you to set the name, colour, and line and marker style of the curve.

As mentioned above, the performance of the model can be interepreted using the area under the curve. This is displayed in the legend:

The graph contains a reference line, drawn in gray, which indicates the 50% mark for every X/Y coordinate on the curve. When the graph falls below this mark, the model is performining poorly, worse than just random.

Finally, the zoom buttons enlarge ("zoom +") or shrink ("zoom -") the graph within the graphing window.

Top

Top       Home             Utilities       Documentation -> ROC Curve Docs       FAQ       About