3.1. Added Predictive Ability for Logistic Models
3.1.1. Integrated Discrimination Improvement
IDI has been defined by Pencina et al. (2) as:
Equation 1.
Where IS is integral sensitivity over all cutoff values and IP is integral “one minus specificity,” “new” refers to the prediction model incorporating the new biomarker and “old” refers to the prediction model that does not. The arguments in the first presentation assumed nested models. However, IDI is applicable to situation where aim is to compare any two models with possibly different predicators and different analytic techniques, as far as predicated probabilities are calibrated to the same incidence or prevalence rate (12). Pencina et al. (2) provided the following estimator for IDI:
Equation 2.
P̅ is the average estimated probability of an event. An average is taken for people in sample who experienced an event (“event”) and for those who did not (“nonevents”). Rearranging terms we obtain:
Equation 3.
Which is the difference in discrimination slope proposed by Caputo et al. (13). The magnitude of discrimination slopes and their difference is tends to be small (14), 15). This could be more conspicuous when the incidence or prevalence of the event of interest is relatively low (16). Considering the definition mentioned above, one could define relative IDI as the increase in discrimination slopes divided by the slope of the old model. As such, relative IDI could be estimated as follows:
Equation 4.
Hereafter, we refer to IDI as absolute IDI.
3.1. 2. Net Reclassification Improvement
To obtain NRI, predicted probability based on the basic (old or without new biomarker or new risk factor) and enhanced (new or with new biomarker or new risk factor) are classified into three categories (17); these two cross-classification are then cross-tabulated. The reclassification of people who develop and those who do not develop events is to be considered separately. Any ‘upwards’ movement across classes for those with the event (i.e. event group) implies improvement; whereas, any ‘downwards’ movement across classes indicates worse reclassification. The interpretation is opposite for those who do not develop event (the event nonevent group). The NRI will be a sum differences in proportion of individuals moved up minus proportion of those who moved down among event subjects, and the proportion of individuals moved down minus proportion of those who moved up among nonevent subjects. The NRI as such quantifies the improvement in reclassification. If assign 1 for each upwards movement and -1 for each downwards movement, and 0 for no movement in categories, the NRI can be estimated as:
Equation 5.
where v (i) is the above-defined movement indicator.
In general, it is not recommended to use more than three categories unless they are already established and there are rigorous justifications to do so. For situation where finer portioning is needed, Pencina et al. (17) have suggested the cutpoint-free NRI. The definition of cutpoint-free NRI remains consistent with cutpoint-based formula defined above with the only difference in the meaning of the upwards and downwards reclassification (17).
3.2. Commands
3.2.1. “Adpred” Command
3.2.1.1. Syntax
Adpred depvar oldrisk newrisk, cutpoint (numlist).
Depvar represents dependent binary outcome variable.
Oldrisk is the variable that represents the risk calculated based on the baseline model.
Newrisk is the variable that represents the risk calculated based on the enhanced model.
Numlist is the list of risk cutoff points for cutpoint-based NRI.
3.2.1.2. Description
“adpred” calculates absolute and relative IDI, as well as cutpoint-based and cutpoint-free NRI. For cutpoint-based NRI to be calculated cutoff-points of risk should be specified by users.
3.2.1.3. Option
“Cut (numlist)” gives the numbers that present cutpoints of risk based on the old model at which new model is to be evaluated.
LEAVE A COMMENT HERE: