Close
SciStat
 

Comparison of ROC curves

Description

Use Comparison of ROC curves to test the statistical significance of the difference between the areas under different dependent ROC curves (derived from the same cases).

Required input

  • Variables: identify the variables under study.
  • Classification: select a dichotomous variable indicating diagnosis (0=negative, 1=positive). If diagnosis is coded differently than using the values 0 and 1, you can use the IF function to transform the codes into 0 and 1 values, e.g. IF(RESULT="pos",1,0).
  • Filter: (optionally) a filter to include only a subgroup of cases (e.g. AGE>21, SEX="Male").

Methodology

  • DeLong et al.: use the method of DeLong et al. (1988) for the calculation of the Standard Error of the Area Under the Curve (recommended).
  • Hanley & McNeil: use the method of Hanley & McNeil (1982) for the calculation of the Standard Error of the Area Under the Curve.
  • Binomial exact Confidence Interval for the AUC: calculate exact Binomial Confidence Intervals for the Area Under the Curves (AUC) (recommended). If this option is not selected, the Confidence Intervals for the AUCs are calculated as AUC ± 1.96 SE (Standard Error). This option does not apply to the difference between two AUCs.

Graph

  • Create graph
  • Mark points corresponding to criterion values

Results

The results for Comparison of ROC curves include:

1. The results for all variables

  • Area under the ROC curve (AUC), with standard error: this value can be interpreted as follows: an area of 0.84, for example, means that a randomly selected individual from the positive group has a test value larger than that for a randomly chosen individual from the negative group in 84% of the time (Zweig & Campbell, 1993). When the variable under study cannot distinguish between the two groups, i.e. where there is no difference between the two distributions, the area will be equal to 0.5 (the ROC curve will coincide with the diagonal). When there is a perfect separation of the values of the two groups, i.e. there is no overlapping of the distributions, the area under the ROC curve equals 1 (the ROC curve will reach the upper left corner of the plot).
  • 95% confidence interval: the 95% confidence interval for the area can be used to test the hypothesis that the theoretical area is 0.5. If the confidence interval does not include the 0.5 value, then there is evidence that the laboratory test does have an ability to distinguish between the two groups.

2. Pairwise comparison of the Area under the curve for all variables

  • Difference between the areas with standard error: the difference between the two areas under the curve.
  • 95% confidence interval: the 95% confidence interval for this difference.
  • Significance level: probability of the hypothesis that the difference between the two AUCs is 0.

Graph

This graphs displays 2 or more ROC curves, allowing visual comparison.

In a ROC curve the true positive rate (Sensitivity) is plotted in function of the false positive rate (100-Specificity) for different cut-off points. Each point on the ROC plot represents a sensitivity/specificity pair corresponding to a particular decision threshold. A test with perfect discrimination (no overlap in the two distributions) has a ROC plot that passes through the upper left corner (100% sensitivity, 100% specificity). Therefore the closer the ROC plot is to the upper left corner, the higher the overall accuracy of the test (Zweig & Campbell, 1993).

Literature

  • DeLong ER, DeLong DM, Clarke-Pearson DL (1988): Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 44:837-845.
  • Hanley JA, Hajian-Tilaki KO (1997) Sampling variability of nonparametric estimates of the areas under receiver operating characteristic curves: an update. Academic Rediology 4:49-58.
  • Hanley JA, McNeil BJ (1982) The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143:29-36.
  • Hanley JA, McNeil BJ (1983) A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology 148:839-843.
  • Zweig MH, Campbell G (1993) Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine. Clinical Chemistry 39:561-577.

See also

Link

Go to Comparison of ROC curves.