Evaluating Predictive Models: Computing and Interpreting the c Statistic

Overview

Automation of predictive model selection has become the alchemy of today's Business Intelligence (BI), with BI practitioners hoping to transform jargon and acronyms into gold. While statistical modeling experts disparage 'Black-box' methods, business analysts tend to ignore theory and traditional model diagnostics, and rely on 'Lift' or 'Gain' charts; that is, evidence of improved accuracy of predictions of outcomes for groups, often represented graphically by Area Under Curve (AUC) of signal detection Receiver Operating Characteristic (ROC) plots. Even where interest in competitive advantage overrides concern for meaningful model parameter estimates, a concordance c statistic offers a simpler and more intuitive measure of accuracy of model predictions. SAS/Stat PROCs LOGISTIC and SURVEYLOGISTIC generate a c statistic.