Training and evaluating POWERUP models

fit_powerup_models() fits one XGBoost regression model for each selected perturbation. Model evaluation is based on held-out folds from the reference training data. The prediction samples in user_matrix are not used to calculate validation metrics.

Prepare a training set

The bundled example contains 300 reference samples and four dependency responses. Here we train CTNNB1 only.

expression_path <- system.file("extdata", "powerup_example_gene_expression.csv", package = "powerup")
dependency_path <- system.file("extdata", "powerup_example_gene_dependency.csv", package = "powerup")
user_path <- system.file("extdata", "powerup_example_user_matrix.csv", package = "powerup")

example_expression <- read.csv(expression_path, check.names = FALSE)
example_dependency <- read.csv(dependency_path, check.names = FALSE)
example_user <- read.csv(user_path, check.names = FALSE)

prepared <- prepare_powerup_data(
  gene_expression = example_expression,
  response = example_dependency,
  targets = "CTNNB1",
  user_matrix = example_user
)
#> [powerup] target resolution summary requestedUnique=1 resolvedUnique=1 metadataNameMatches=0 canonicalFallbackMatches=1 looseMetadataFallbackResolved=0 exactMetadataMultipleMatches=0 unresolvedSkipped=0
#> [powerup] top-variable feature selection source=reference_matrix reason=user_selected_reference_source reference_samples=300 n_ranked_reference_features=300 n_user_available_features=300

The example data are a demonstration subset of DepMap Public 26Q1. See Example data and provenance for the source data, construction, and acknowledgements.

Repeated cross-validation

By default, POWERUP uses 3-fold cross-validation repeated three times. Folds are stratified on the continuous response. Each fold trains on its analysis samples and predicts its held-out assessment samples.

models <- fit_powerup_models(prepared, seed = 123L)

Each cross-validation fit can run for up to nrounds boosting rounds. Early stopping uses the held-out fold with 200 rounds of patience, and POWERUP records the best iteration from each fit. The final full-data model uses the median of those best-iteration counts rather than early stopping on the full training data.

The main validation metrics are calculated separately in each held-out fold:

Metric	Meaning
Pearson r	Correlation between predicted and observed responses
R2	Held-out coefficient of determination
RMSE	Root mean squared error on the response scale
Sensitivity, specificity, FPR, PPV, NPV, accuracy	Classification-style summaries after applying response_cutoff

summarize_models() reports their means across the repeated folds.

summarize_models(models)
#> # A tibble: 1 × 13
#>   brd   skipped     r    R2  rmse d_sensitivity d_specificity  d_FPR d_PPV d_NPV
#>   <chr> <lgl>   <dbl> <dbl> <dbl>         <dbl>         <dbl>  <dbl> <dbl> <dbl>
#> 1 CTNN… FALSE   0.853 0.716 0.196         0.832         0.973 0.0267 0.895 0.956
#> # ℹ 3 more variables: d_accuracy <dbl>, n_terms <int>, mean_pred_sd <dbl>

Model retention

A model is retained only when its mean validation correlation and mean validation R-squared are finite and mean R-squared is at least min_score. The default min_score is 0.01. Models that fail this check are marked as skipped and are not carried into prediction or SHAP analysis.

Response cutoff

response_cutoff defines the event used for classification-style metrics and downstream event probabilities. When decreasing = TRUE, values at or below the cutoff define the event; otherwise values at or above the cutoff define it.

Prediction uncertainty

POWERUP estimates prediction uncertainty from cross-validation residuals rather than from the final model’s in-sample residuals. Repeated cross-validation provides out-of-fold predictions for the training samples. For each out-of-fold prediction, POWERUP calculates squared error and fits a second XGBoost model to log squared error.

The uncertainty model uses the molecular features together with the corresponding response prediction. Repeated observations of the same training sample are weighted so that additional cross-validation repeats do not give that sample more total weight. A separate cross-fitting step predicts log variance for held-out samples. POWERUP then computes a single calibration factor so that the weighted 80th percentile of absolute standardized out-of-fold errors matches the central 80% Gaussian interval. The final uncertainty model is fit to all out-of-fold residual rows, and its sample-specific prediction SDs are multiplied by this calibration factor.

For the final response model, this produces pred_mean, pred_sd, and approximate Gaussian 95% prediction intervals. The model object records the uncertainty family as gaussian_oof_calibrated_log_variance.

After predictions are added to new samples, use the long prediction summary to inspect these quantities directly:

models <- add_powerup_predictions(models, prepared)
predictions <- summarize_predictions(models, format = "long")
predictions[, c(
  "sample", "perturbation", "pred_mean", "pred_sd",
  "pred_pi_lower_95", "pred_pi_upper_95", "prob_target_event"
)]

Prediction SD describes the uncertainty estimated by this fitted procedure. It does not include every source of biological, measurement, or dataset-shift uncertainty.

Shuffled-response controls

Set shuffle = TRUE to fit a response-shuffled control using the same training machinery. Set the seed argument when the shuffled analysis must be reproduced.

null_models <- fit_powerup_models(prepared, shuffle = TRUE, seed = 123L)

Shuffled-response models are controls for evaluating performance expected after breaking the correspondence between molecular features and the training response; they are not substitutes for held-out validation of the unshuffled model.

Training resources

n_threads sets the number of threads passed to XGBoost.

Record a reproducible fit

For a reported analysis, keep the package version and seed together with the prepared sample set, selected features, perturbations, response_cutoff, decreasing, nfolds, nrepeats, nrounds, min_score, XGBoost overrides, and thread count. Each returned model records the requested seed and its derived perturbation-specific target_seed.

Prediction outputs are covered in Interpreting POWERUP results, while SHAP interpretation is covered in Explaining POWERUP predictions.

Session information

sessionInfo()
#> R version 4.4.2 (2024-10-31)
#> Platform: aarch64-apple-darwin20
#> Running under: macOS Sequoia 15.7.3
#> 
#> Matrix products: default
#> BLAS:   /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/lib/libRblas.0.dylib 
#> LAPACK: /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/lib/libRlapack.dylib;  LAPACK version 3.12.0
#> 
#> locale:
#> [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
#> 
#> time zone: America/New_York
#> tzcode source: internal
#> 
#> attached base packages:
#> [1] stats     graphics  grDevices utils     datasets  methods   base     
#> 
#> other attached packages:
#> [1] powerup_1.0.95
#> 
#> loaded via a namespace (and not attached):
#>  [1] Matrix_1.7-3      jsonlite_2.0.0    dplyr_1.1.4       compiler_4.4.2   
#>  [5] tidyselect_1.2.1  parallel_4.4.2    tidyr_1.3.1       jquerylib_0.1.4  
#>  [9] globals_0.16.3    systemfonts_1.2.2 textshaping_1.0.0 yaml_2.3.10      
#> [13] fastmap_1.2.0     lattice_0.22-6    R6_2.6.1          generics_0.1.3   
#> [17] knitr_1.50        htmlwidgets_1.6.4 future_1.34.0     tibble_3.3.0     
#> [21] desc_1.4.3        bslib_0.9.0       pillar_1.10.1     rlang_1.1.6      
#> [25] utf8_1.2.4        cachem_1.1.0      xfun_0.51         fs_1.6.5         
#> [29] sass_0.4.9        cli_3.6.5         pkgdown_2.2.0     withr_3.0.2      
#> [33] magrittr_2.0.3    grid_4.4.2        digest_0.6.37     xgboost_3.2.1.1  
#> [37] lifecycle_1.0.4   vctrs_0.6.5       data.table_1.17.0 evaluate_1.0.3   
#> [41] glue_1.8.0        rsample_1.2.1     listenv_0.9.1     furrr_0.3.1      
#> [45] codetools_0.2-20  ragg_1.5.1        parallelly_1.42.0 rmarkdown_2.29   
#> [49] purrr_1.1.0       tools_4.4.2       pkgconfig_2.0.3   htmltools_0.5.8.1