Skip to contents

Introduction

Dynamic risk prediction models, such as joint models, are increasingly used to estimate patient-specific risks and support clinical decision-making. Assessing the quality of these predictions is therefore essential. Common performance measures, including the Brier score, logarithmic score, and AUC, evaluate predictive accuracy but do not directly reflect the clinical consequences of decisions based on those predictions.

This vignette focuses on measures of clinical utility, including the net benefit (NB) and expected cost (EC), which have been adapted for the dynamic prediction setting. These measures account for the consequences of false positive and false negative decisions. We also present the integrated weighted expected cost (IWEC), a metric that considers clinical utility while achieving strict properness. An additional advantage of the IWEC is that it does not depend on a single decision threshold; instead, it utilizes a weight function that incorporates clinical knowledge in a more flexible manner. Another interesting feature of the IWEC is that it characterizes the class of strictly proper rules. The Brier score and the logarithmic score can be obtained as particular cases, as will be shown below with numerical examples. Additionally, we present a weighted decision curve that offers a graphical assessment of model performance.

The package implements both inverse probability weighting (IPW) and model-based estimation approaches for left-truncated and right-censored data (LTRC). The aim of this vignette is to introduce the available methods and demonstrate their use through practical examples with publicly available data.

Methods

We consider a study with n subjects. For subject i, let T_i^* denote the true event time, L_i the left-truncation time, C_i the right-censoring time, T_i=\min(T_i^*, C_i) the observed time, \delta_i=\mathbf{1}\{T_i^*\leq C_i\} the event indicator and \mathcal{H}_i(t) the available history at time t (including baseline and time-dependent covariates). We are interested in the subject-specific probability of experiencing the event within a time interval (t, t+s], given the information until the landmark time t, where s is a fixed prediction horizon. We define the dynamic predictions as the cumulative risk probabilities \pi_i(s\mid t)=\Pr(T_i^*\leq t + s\mid T_i^*>t, \mathcal{H}_i(t)).

For a given decision threshold c\in(0,1), we define the dynamic net benefit as

\begin{split} \text{NB}_c\big(\pi_i(s\mid t),D_i(t,s)\big) = & \mathbf{1}\{\pi_i(s\mid t)\geq c\} D_i(t,s)\\ &-\mathbf{1}\{\pi_i(s\mid t)\geq c\}\{1- D_i(t,s)\}\frac{c}{1-c}, \end{split}

where D_i(t,s)=\mathbf{1}\{t<T_i^*\leq t+s\} is a binary indicator that is 1 if the event is observed within (t,t+s], and 0 otherwise. For a given decision threshold c\in(0,1), we define the dynamic expected cost as

\begin{split} \text{EC}_c\big(\pi_i(s\mid t),D_i(t,s)\big) = & c\mathbf{1}\{\pi_i(s\mid t)\geq c\} \{1-D_i(t,s)\}\\ &+(1-c)\mathbf{1}\{\pi_i(s\mid t)< c\}D_i(t,s). \end{split}

To avoid reliance on a single threshold and achieve strict properness, we define the dynamic integrated weighted expected cost as

\text{IWEC}_w\big\{\pi_i(s\mid t),D_i(t,s)\big\} = \int_0^1\text{EC}_c\big\{\pi_i(s\mid t),D_i(t,s)\big\}w(c)dc,

where w(\cdot) is a positive and measurable weight function over all decision thresholds c\in(0,1). In the tvIWEC() function included in JMbayes2, we implemented this metric using the beta family of weight functions, defined by the two-parameter form w(c;\texttt{shape1}, \texttt{shape2})=c^{\texttt{shape1}-1}(1-c)^{\texttt{shape2}-1}/B(\texttt{shape1}, \texttt{shape2}), where \texttt{shape1}, \texttt{shape2}\geq 0 are shape parameters and B(\texttt{shape1},\texttt{shape2})=1 if \texttt{shape1}=\texttt{shape2}=0, and otherwise denote the beta function.

Examples

Joint models evaluation

For illustration, we use the pbc2 dataset available in JMbayes2. The pbc2 includes just right-censored data. Since we are interested in exemplify how the functions presented in this vignette work with left-truncated and right-censored (LTRC) data, we first generate left-truncation times for the data:

set.seed(1234)
pbc2$lt <- unlist(by(pbc2, pbc2$id, function(x) rep(runif(1, 0, x$years), nrow(x))))
pbc2$event <- as.numeric(pbc2$status != "alive")
pbc2.id$lt <- pbc2$lt[match(pbc2.id$id, pbc2$id)]

Then, we fit the Cox model for the time to either transplantation or death, using then left-truncated and right-censored data. We included the baseline covariates age and drug:

pbc2.id$event <- as.numeric(pbc2.id$status != "alive")
CoxFit <- coxph(Surv(lt, years, event) ~ drug + age, data = pbc2.id)

For the longitudinal submodel, we consider the longitudinal outcome serBilir, which is a continuous outcome. We allow for nonlinear subject-specific time effects via natural cubic splines:

fm1 <- lme(log(serBilir) ~ ns(year, 3) * sex, data = pbc2,
           random = ~ ns(year, 3) | id, control = lmeControl(opt = 'optim'))

Now, we fit the joint model that will produce the dynamic predictions to be evaluated:

jm1 <- jm(CoxFit, fm1, time_var = "year")

The net benefit quantifies the quality of decisions based on the dynamic predictions given a decision threshold c\in (0,1). We consider the landmark time Tstart=4 and the prediction horizon Dt=3. In addition we consider the decision threshold c=0.3.

tvNB(jm1, newdata = pbc2, Tstart = 4, Dt = 3, decision_threshold = 0.3)
#> 
#> Accuracy Metric for the Joint Model 'jm1'
#> 
#> Estimated Net Benefit with decision threshold 0.3: 0.1751
#> At time: 7
#> For the 125 subjects at risk at time 4
#> Number of subjects with an event in [4, 7): 36
#> Number of subjects with a censored time in [4, 7): 29
#> Accounting for left truncation and right censoring using 
#>  model-based weights

A similar clinical utility metric is the expected cost. Using the same landmark time, prediction horizon and decision threshold as above, now we use the IPW approach for the metric estimation:

tvEC(jm1, newdata = pbc2, Tstart = 4, Dt = 3, decision_threshold = 0.3, 
     type_weights = "IPCW")
#> 
#> Accuracy Metric for the Joint Model 'jm1'
#> 
#> Estimated Expected Cost with decision threshold 0.3: 0.0944
#> At time: 7
#> For the 125 subjects at risk at time 4
#> Number of subjects with an event in [4, 7): 36
#> Number of subjects with a censored time in [4, 7): 29
#> Accounting for left truncation and right censoring using 
#>  nonparametric inverse probability weights

One might be interested in assessing clinical utility within a span of relevant decision thresholds, say c\in(0.2, 0.375). The standard method to consider net benefit (or expected cost) across multiple decision thresholds, is decision curve analysis. We display a decision curve for the net benefit that allows us to compare the joint model jm1 with the default strategies of treat all individuals or none:

dc1 <- decision_curves(jm1, newdata = pbc2, Tstart = 4, Dt = 3, type = "NB")
plot(dc1)

We observe in the previous plot that jm1 yields greater, and then better, NB estimates in most of decision thresholds. In particular, in the interval of relevant decision thresholds (0.2, 0.375), we see that using the joint model to decide on patient treatment is better than both treat all or none patients.

However, decision curves are descriptive tools that do not provide a single summary measure for model comparison. To address this limitation, while using a metric that is strictly proper, we can use the integrated weighted expected cost (IWEC). The IWEC allows us to flexibly accommodate the clinical context through a weight function. We implemented the beta family of weight functions, which is completely specified by the shape parameters shape1 and shape2. Instead of passing directly the shape parameters, the user can pass the interval of relevant decision thresholds c\in(0.2, 0.375) (interv = c(0.2, 0.375)), the mode (we set mod = 0.3) and the mass of the function to be allocated within the interval (we set mass = 0.85). By passing these arguments, shape1 and shape2 are internally computed.

tvIWEC(jm1, newdata = pbc2, Tstart = 4, Dt = 3,
       mod = 0.3, interv = c(0.2, 0.375), mass = 0.85)
#> 
#> Accuracy Metric for the Joint Model 'jm1'
#> 
#> Estimated Integrated Weighted Expected Cost
#>  with shape parameters 18.9 and 42.7: 0.0993
#> At time: 7
#> For the 125 subjects at risk at time 4
#> Number of subjects with an event in [4, 7): 36
#> Number of subjects with a censored time in [4, 7): 29
#> Accounting for left truncation and right censoring using 
#>  model-based weights

In addition, we can plot a weighted decision curve, which displays the weighted expected cost and also the area under the weighted decision cuves that corresponds to the IWEC:

wdc1 <- decision_curves(jm1, newdata = pbc2, Tstart = 4, Dt = 3, 
                        type = "weightedEC", mod = 0.3, interv = c(0.2, 0.375), 
                        mass = 0.85)
plot(wdc1, legend_names = c("Joint Model 1", "All", "None"))

At the top part of the plot we can see the shape of the weight function used. We see that the joint model yields smaller, so better, weighted EC estimates within the interval of decision thresholds of interest. In addition, the IWEC is also better. Hence, we can conclude that in the clinical context of interest, jm1 is a clinical useful model.

Now, imagine that we consider multiple candidate joint models. We can compare them via NB, EC, IWEC and also using weighted and unweighted decision curves. Let us fit a different joint model, we consider the prothrombin for the longitudinal submodel:

fm2 <- lme(prothrombin ~ ns(year, 2) * sex, data = pbc2,
           random = ~ ns(year, 2) | id, control = lmeControl(opt = 'optim'))

We then fit the joint model:

jm2 <- jm(CoxFit, fm2, time_var = "year")

For illustration, we now change the clinical context and we assume the interval of interest for decision thresholds is c\in(0.125, 0.275). We now consider as a landmark time Tstart = 6 and prediction horizon Dt = 2. For the NB and EC we will fix c = 0.2:

tvNB(jm2, newdata = pbc2, Tstart = 6, Thoriz = 8,
     decision_threshold = 0.2, type_weights = "IPCW")
#> 
#> Accuracy Metric for the Joint Model 'jm2'
#> 
#> Estimated Net Benefit with decision threshold 0.2: 0.054
#> At time: 8
#> For the 121 subjects at risk at time 6
#> Number of subjects with an event in [6, 8): 19
#> Number of subjects with a censored time in [6, 8): 33
#> Accounting for left truncation and right censoring using 
#>  nonparametric inverse probability weights

tvEC(jm2, newdata = pbc2, Tstart = 6, Thoriz = 8,
     decision_threshold = 0.2, type_weights = "IPCW")
#> 
#> Accuracy Metric for the Joint Model 'jm2'
#> 
#> Estimated Expected Cost with decision threshold 0.2: 0.099
#> At time: 8
#> For the 121 subjects at risk at time 6
#> Number of subjects with an event in [6, 8): 19
#> Number of subjects with a censored time in [6, 8): 33
#> Accounting for left truncation and right censoring using 
#>  nonparametric inverse probability weights

For the decision curve of the expected cost, by default the weight function is displayed to show the relative importance of all decision thresholds. So, let’s assume that we want to assign a high relevance to the interval c\in(0.125, 0.275), say mass=0.95 (the default). The decision curve for the expected cost, including both jm1 and jm2 is:

dc_ec1 <- decision_curves(jm1, newdata = pbc2, Tstart = 6, Dt = 2, 
                          type_weights = "IPCW", mod = 0.2, 
                          interv = c(0.125, 0.275))

dc_ec2 <- decision_curves(jm2, newdata = pbc2, Tstart = 6, Dt = 2, 
                          type_weights = "IPCW", mod = 0.2, 
                          interv = c(0.125, 0.275))

plot(dc_ec1, dc_ec2, legend_names = c("JM1 (serBilir)", "JM2 (prothrombin)", 
                                      "Treat all", "Treat none"),
     col = c("black", "purple2", "lightgreen", "brown2"))

We observe that the joint model using the serum bilirubin overcomes the default strategies for thresholds from 0 to 0.6. The joint model using the prothrombin is just improving the default strategies within (0.18, 0.4), approximately.

We can compute the IWEC for both joint models, using the weight function above. For the first joint model (the one using serBilir):

tvIWEC(jm1, newdata = pbc2, Tstart = 6, Thoriz = 8, 
       mod = 0.2, interv = c(0.125, 0.275))
#> 
#> Accuracy Metric for the Joint Model 'jm1'
#> 
#> Estimated Integrated Weighted Expected Cost
#>  with shape parameters 23.1 and 89.4: 0.0766
#> At time: 8
#> For the 121 subjects at risk at time 6
#> Number of subjects with an event in [6, 8): 19
#> Number of subjects with a censored time in [6, 8): 33
#> Accounting for left truncation and right censoring using 
#>  model-based weights

And, for jm2 (joint model using prothrombin):

tvIWEC(jm2, newdata = pbc2, Tstart = 6, Thoriz = 8, 
       mod = 0.2, interv = c(0.125, 0.275))
#> 
#> Accuracy Metric for the Joint Model 'jm2'
#> 
#> Estimated Integrated Weighted Expected Cost
#>  with shape parameters 23.1 and 89.4: 0.124
#> At time: 8
#> For the 121 subjects at risk at time 6
#> Number of subjects with an event in [6, 8): 19
#> Number of subjects with a censored time in [6, 8): 33
#> Accounting for left truncation and right censoring using 
#>  model-based weights

We can compare jm1, jm2 as well as the default strategies with the weighted decision curve. In this case we consider a finer grid of decision thresholds to plot the decision curve.

thr_rng <- seq(0.001, 0.999, length.out = 100)
wdc1 <- decision_curves(jm1, newdata = pbc2, Tstart = 6, Dt = 2, 
                        type = "weightedEC", mod = 0.2, 
                        interv = c(0.125, 0.275), 
                        decision_threshold_range = thr_rng)

wdc2 <- decision_curves(jm2, newdata = pbc2, Tstart = 6, Dt = 2, 
                        type = "weightedEC", 
                        mod = 0.2, interv = c(0.125, 0.275),
                        decision_threshold_range = thr_rng)
plot(wdc1, wdc2)

In this plot, we see that both joint models overcome the default strategies when incorporating the relative relevance of the decision thresholds via the weight function w(c; 23.11, 89.45). However, the joint model using the serum bilirubin yields better estimates per each relevant decision threshold as well as with respect the IWEC.

Cox proportional hazards models evaluation

Notably, the implementation supports both joint models (jm class objects) and Cox proportional hazards models (coxph class objects). This is, all the functions above can also be used with Cox models. Let us consider the Cox model used in the survival process of the joint models fitted above. We can assess predictive performance via the net benefit:

tvNB(CoxFit, newdata = pbc2.id, Tstart = 5, Thoriz = 7.5, 
     decision_threshold = 0.125, type_weights = "IPCW")
#> 
#> Accuracy Metric for the Cox Model 'CoxFit'
#> 
#> Estimated Net Benefit with decision threshold 0.125: 0.1328
#> At time: 7.5
#> For the 134 subjects at risk at time 5
#> Number of subjects with an event in [5, 7.5): 28
#> Number of subjects with a censored time in [5, 7.5): 41
#> Accounting for left truncation and right censoring using 
#>  nonparametric inverse probability weights

Similarly, we can use the expected cost:

tvEC(CoxFit, newdata = pbc2.id, Tstart = 6, Dt = 3,
     decision_threshold = 0.45, name_model = "Proportional Hazards Model")
#> 
#> Accuracy Metric for the Cox Model 'Proportional Hazards Model'
#> 
#> Estimated Expected Cost with decision threshold 0.45: 0.1783
#> At time: 9
#> For the 121 subjects at risk at time 6
#> Number of subjects with an event in [6, 9): 29
#> Number of subjects with a censored time in [6, 9): 45
#> Accounting for left truncation and right censoring using 
#>  model-based weights

The IWEC also supports Cox models:

tvIWEC(CoxFit, newdata = pbc2.id, Tstart = 6, Thoriz = 8,
       shape1 = 2, shape2 = 2)
#> 
#> Accuracy Metric for the Cox Model 'CoxFit'
#> 
#> Estimated Integrated Weighted Expected Cost
#>  with shape parameters 2.0 and 2.0: 0.0877
#> At time: 8
#> For the 121 subjects at risk at time 6
#> Number of subjects with an event in [6, 8): 19
#> Number of subjects with a censored time in [6, 8): 33
#> Accounting for left truncation and right censoring using 
#>  model-based weights

Decision curves can be easily displayed also with Cox models:

cox_wdg <- decision_curves(CoxFit, newdata = pbc2.id, Tstart = 6, Thoriz = 8,
                           type = "weightedEC", shape1 = 15, shape2 = 25)

plot(cox_wdg, legend_names = c("Cox model", "Treat all", "Treat none"), 
     legend_loc = "topleft", lty = c(1, 2, 2), xlim = c(0, 0.65))

The IWEC reduces to the 1/2 times the Brier score when shape1 = 1 and shape2 = 1:

iwec1_1 <- tvIWEC(CoxFit, newdata = pbc2.id, Tstart = 2, Dt = 5, 
                  shape1 = 1, shape2 = 1, type_weights = "IPCW")

bs_expl <- tvBrier(CoxFit, newdata = pbc2.id, Tstart = 2, Dt = 5,
                   type_weights = "IPCW")

cat("IWEC when shape1 = shape2 = 1:", 2 * iwec1_1$metric, "\n")
#> IWEC when shape1 = shape2 = 1: 0.2364726
cat("Brier score:", bs_expl$Brier)
#> Brier score: 0.2364726

In addition, the logarithmic score is also a particular case of the IWEC that is obtained when shape1 = 0 and shape2 = 0.

General dynamic predictions evaluation

In addition, because these metrics can be applied to dynamic predictions obtained from any kind of model, we implemented a general ‘default’ method that works by using the vector of dynamic predictions, the observed times, the event indicator and the left-truncated times. Let’s see some examples of this.

First, we compute dynamic predictions for the time window (5, 8] using the Cox model above:

sfit_5 <- summary(survfit(CoxFit, newdata = pbc2.id), times = 5)
sfit_8 <- summary(survfit(CoxFit, newdata = pbc2.id), times = 8)
predictions <- 
    1.0 - (as.matrix(sfit_8$surv)[1L, ] / as.matrix(sfit_5$surv)[1L, ])

Now, we compute the net benefit with decision threshold 0.375, we use the IPW estimator:

tvNB(predictions, Time = pbc2.id$years, LT_Time = pbc2.id$lt, 
     event = pbc2.id$event, decision_threshold = 0.375, Tstart = 5, 
     Thoriz = 8, type_weights = "IPCW")
#> 
#> Accuracy Metric for the predictions 'predictions',
#>  observed times 'pbc2.id$years', event indicator 'pbc2.id$event'
#>  and left-truncated times 'pbc2.id$lt'
#> 
#> Estimated Net Benefit with decision threshold 0.375: -0.0049
#> At time: 8
#> For the 134 subjects at risk at time 5
#> Number of subjects with an event in [5, 8): 31
#> Number of subjects with a censored time in [5, 8): 47
#> Accounting for left truncation and right censoring using 
#>  nonparametric inverse probability weights

For the default method, if we want to use the model-based estimator, we must pass the preds_MB argument, which contains the weights corresponding to the censored individuals under the model-based approach. See an example using the same Cox model and time window below. Now, we compute the IWEC with a weight function that has its mode in 0.33, and allocates 70\% of its mass within (0.125, 0.46).

ind_censoring <- pbc2.id$years > 5 & pbc2.id$lt < 5 & 
    pbc2.id$years < 8 & pbc2.id$event == 0

sfit2_8 <- summary(survfit(CoxFit, newdata = pbc2.id[ind_censoring, ]),
                 times = 8)
sfit_cens <- summary(survfit(CoxFit, newdata = pbc2.id[ind_censoring, ]),
                     times = pbc2.id$years[ind_censoring])

preds_MB <- rep(NA, length(pbc2.id$years))
preds_MB[ind_censoring] <- 
    1.0 - (as.matrix(sfit2_8$surv)[1L, ] / as.matrix(sfit_cens$surv)[1L, ])

tvIWEC(predictions, Time = pbc2.id$years, event = pbc2.id$event, 
       LT_Time = pbc2.id$lt, Tstart = 5, Dt = 3, type_weights = "model-based", 
       preds_MB = preds_MB, mod = 0.33, interv = c(0.125, 0.46), mass = 0.7)
#> 
#> Accuracy Metric for the predictions 'predictions',
#>  observed times 'pbc2.id$years', event indicator 'pbc2.id$event'
#>  and left-truncated times 'pbc2.id$lt'
#> 
#> Estimated Integrated Weighted Expected Cost
#>  with shape parameters 3.5 and 6.0: 0.1528
#> At time: 8
#> For the 134 subjects at risk at time 5
#> Number of subjects with an event in [5, 8): 31
#> Number of subjects with a censored time in [5, 8): 47
#> Accounting for left truncation and right censoring using 
#>  model-based weights

To clarify the explanation regarding the preds_MB argument: we need to provide a vector that matches the length of the Time, event, and LT_Time vectors. This new vector should contain NA values for all entries, except for those corresponding to censored individuals, which should hold the model-based weights.

For the decision curves the ‘default’ method work the same way:

dc_default <- 
    decision_curves(predictions, Time = pbc2.id$years, event =  pbc2.id$event,
                    LT_Time =  pbc2.id$lt, Tstart = 5, Dt = 3, 
                    type_weights = "IPCW", type = "NB")
plot(dc_default)