Why Multiverse Meta-Analysis?

What You’ll Need

Basic R knowledge is helpful, but I’ll explain each step. You’ll need:

• R (version 4.0 or higher)

• RStudio (recommended but not required)

• About 30 minutes

Step 1: Install the Package

First, we need to install metaMultiverse for running the multiverse analysis. We’ll get the data directly from the Metapsy API (no package installation needed for that!).

# Install devtools if you don't have it
if (!require("devtools")) install.packages("devtools")

# Install metaMultiverse from GitHub
devtools::install_github("cyplessen/metaMultiverse", 
                         force = TRUE, 
                         upgrade = "never")

# You could also use remotes instead of devtools:
if (!require("remotes"))
  install.packages("remotes")

remotes::install_github(
  "metapsy-project/metapsyTools")

library(metaMultiverse)
library(metapsyTools)

library(dplyr)     # for data manipulation
library(jsonlite)  # for reading API data
library(knitr)     # for formatted tables

Step 2: Get the Data from Metapsy

Metapsy maintains databases of psychotherapy trials across different mental health conditions. For this guide, you can either use their API to download data on psychotherapy for depression, I have already installed it so I do not abuse their service too much :)

You could also download their data as a .csv file and load it into R as below:

data <- read.csv2("data-guide.csv") 

# You can get the depression psychotherapy database via API like this:
# The API requires 'shorthand' parameter and optional 'version' (defaults to 'latest')
# api_url <- "http://api.metapsy.org/v1/get_data?shorthand=depression-psyctr&version=latest"
# api_response <- fromJSON(api_url)
# 
# # Extract the data
# data <- as.data.frame(api_response$data)
# 
# # Take a look at what we have
skimr::skim(data)

This dataset contains information from randomized controlled trials comparing psychotherapy to control conditions for depression. Each row is a comparison from a study.

Note: The API has a rate limit, so please don’t make rapid repeated requests. For this tutorial, you only need to run this once - the data will be stored in your R session.

Validate Data Set

Validate your data set with the metapsyTools::checkDataFormat() function, as this ensures that the metaMultiverse package runs smoothly. See the documentation for the metaPsy data standard and how your data needs to be structured here.

data <- data %>% 
  # Validate data structure for internal functions metaMultiverse
  metaMultiverse::check_data_multiverse()  %>% 
  metapsyTools::checkDataFormat(
    must.contain = c(
      "study", 
      "condition_arm1",
      "condition_arm2",
      "yi",
      "vi"),
    variable.class = list(
      vi = "numeric",
      vi = "numeric"))

Generated es_id column using row numbers (1 to 900).

Converted .g and .g_se to yi and vi for metaMultiverse compatibility.

Warning in metaMultiverse::check_data_multiverse(.): Found 34 unreasonably
large d detected (|d| > 2.5). Check if SD and SE were confused when calculating
SMD.

250 studies contribute only one effect size, 230 studies contribute multiple effect sizes.

Data validation passed. Dataset is ready for multiverse analysis.

- [OK] Data set contains all variables in 'must.contain'.

- [OK] 'vi' has desired class numeric.
- [OK] 'vi' has desired class numeric.

Step 3: Understand the Data Structure

The dataset includes:

• Effect sizes (g, g_se): How effective was the treatment?

• Study characteristics: Sample size, year, country

• Treatment details: Type of therapy, format, number of sessions

• Population info: Age group, recruitment setting, comorbidity

• Risk of bias: Quality ratings for each study

Let’s look at a few key variables:

# See what types of psychotherapy are included
condition_arm1_table <- as.data.frame(table(data$condition_arm1))
kable(condition_arm1_table,
      col.names = c("Intervention Type", "Count"),
      caption = "Types of psychotherapy interventions in the dataset")

# See what types of control conditions are included
condition_arm2_table <- as.data.frame(table(data$condition_arm2))
kable(condition_arm2_table,
      col.names = c("Control Condition", "Count"),
      caption = "Types of control conditions in the dataset")

Step 4: Define Your Multiverse

Now comes the interesting part: defining which analytical decisions to vary. We’ll use the E/U/N framework (Del Giudice & Gangestad, 2021):

multiverse_specs_example_1 <- data %>%
  define_factors(
    Age = "age_group|U",
    Risk_of_Bias = list(
      "rob",
      decision = "U",
      groups = list(
        low_only = "4",
        low_moderate = c("4", "3"),
        all_studies = c("4", "3", "2", "1", "0")
      )
    )
  )

[OK] Factor setup complete
========================================================
[*] Age (simple)
   Column: age_group | Decision: U (Uncertain - will create multiverse options)
   Levels: adul, old, olderold, yadul, adol&yadul, Not specified (+ all combined)

[*] Risk_of_Bias (custom)
   Column: rob | Decision: U (Uncertain - will create multiverse options)
   Groups:
     - low_only: 4
     - low_moderate: 4, 3
     - all_studies: 4, 3, 2, 1, 0

========================================================
Total: 2 factors (1 simple, 1 custom)

# Display the factor setup as a formatted table
kable(multiverse_specs_example_1$factors,
      caption = "Defined factors for multiverse analysis")

Step 5: Create Analysis Specifications

Now we tell the package to create all possible combinations of our decisions, paired with different meta-analytic methods:

# Create all combinations
multiverse_full_example_1 <- multiverse_specs_example_1 %>%
  create_multiverse_specifications(
    
    # Try different statistical models
    ma_methods = c("reml", "p-uniform", "waap", "rve"),
    
    # How to handle multiple comparisons from same study
    dependencies = c("select_max", "aggregate", "modeled")
  )

[*] Multiverse Specifications Created
========================================================
  126 specifications
  1 multiverse(s)
  2 factors included
  4 methods x 3 dependencies

# Display the number of specifications created
specs_summary <- data.frame(
  Metric = "Total specifications created",
  Value = nrow(multiverse_full_example_1$specifications)
)
kable(specs_summary,
      caption = "Multiverse specification summary")

This creates dozens or hundreds of unique meta-analyses, each with different combinations of inclusion criteria and statistical methods.

Step 6: Run the Multiverse Analysis

Now we run all these analyses. This might take a few minutes:

# Run all analyses
results_example_1 <- run_multiverse_analysis(multiverse_full_example_1)

# Create a formatted summary of the multiverse analysis results
n_failed <- results_example_1$n_attempted - results_example_1$n_successful

analysis_summary <- data.frame(
  Metric = c("Total specifications", "Successful", "Failed", "Success rate"),
  Value = c(
    as.character(results_example_1$n_attempted),
    as.character(results_example_1$n_successful),
    as.character(n_failed),
    paste0(round(100 * results_example_1$n_successful / results_example_1$n_attempted, 1), "%")
  )
)

kable(analysis_summary,
      caption = "Multiverse analysis execution summary")

The package runs each meta-analysis and stores the results: effect size, confidence interval, p-value, and heterogeneity statistics.

Step 7: Visualize the Results

The real power comes from visualization. A specification curve shows how effect sizes vary across all your analytical choices:

# Plot specification curve
plot_spec_curve(results_example_1)

This plot shows:

• Top panel: Effect size for each analysis (sorted by magnitude)

• Bottom panels: Which analytical choices were used for each analysis

You can immediately see:

• How much do results vary?

• Are some choices driving the results?

• Is the effect robust across specifications?

Another useful visualization is the Vibration of Effects (VoE) plot:

# Explore relationship between effect size and significance
plot_voe(results_example_1)

This shows the relationship between effect sizes and p-values across your multiverse.

multiverse_specs_example_2 <- data %>%
  define_factors(
    Age = "age_group|U",

    # Type N: Each quality threshold is a separate research question
    Risk_of_Bias = list(
      "rob",
      decision = "N",  # Non-equivalent: creates separate multiverses
      groups = list(
        low_only = "4",                          # Only highest quality
        low_moderate = c("4", "3"),              # High + moderate quality
        all_studies = c("4", "3", "2", "1", "0") # All studies
      )
    )
  ) %>%
  create_multiverse_specifications(
    ma_methods = c("reml", "p-uniform", "waap", "rve"),
    dependencies = c("select_max", "aggregate", "modeled")
  ) %>%
  run_multiverse_analysis()

# Display the multiverse structure as a formatted table
multiverse_structure <- as.data.frame(table(multiverse_specs_example_2$results$multiverse_id))
kable(multiverse_structure,
      col.names = c("Multiverse ID", "Number of Specifications"),
      caption = "Separate multiverses by study quality threshold")

# Plot specification curve showing all three multiverses
plot_spec_curve(multiverse_specs_example_2)

# VoE plot across multiverses
plot_voe(multiverse_specs_example_2)

Step 8: Interpret Your Results

Ask yourself:

1. How much do results vary? If all analyses point in the same direction with similar magnitudes, your conclusion is robust. If effect sizes range from positive to negative, your conclusion depends heavily on analytical choices.

2. Which choices matter most? Look at the bottom panels of the specification curve. Do certain inclusion criteria consistently produce larger or smaller effects?

3. Statistical significance: Do most analyses show significant effects, or does significance depend on which models you choose?

What This Tells Us

Multiverse meta-analysis doesn’t give you “the answer”—it gives you transparency about how confident you should be in your answer. If your conclusion holds across most reasonable analytical choices, you can be more confident. If it’s sensitive to specific decisions, that’s important to report.

This approach moves us from “the effect size is X” to “across plausible specifications, effect sizes range from Y to Z, with most falling around X.”

Going Further

This tutorial covered the practical workflow for multiverse meta-analysis. The metaMultiverse package can do much more:

• Include bias-adjustment methods (PET-PEESE, selection models)

• Explore moderator effects across specifications

• Export results for custom visualizations

• Use the interactive Shiny app for exploration

• Advanced custom factor groupings

• Bayesian meta-analytic methods

Resources

Package:

• metaMultiverse GitHub

• Package Documentation

Data Source:

• Metapsy Database

• Metapsy Documentation

Key References:

• Del Giudice, M., & Gangestad, S. W. (2021). A traveler’s guide to the multiverse. Advances in Methods and Practices in Psychological Science, 4(1), 1-15. DOI

• Steegen, S., et al. (2016). Increasing transparency through a multiverse analysis. Perspectives on Psychological Science, 11(5), 702-712. DOI

• Voracek, M., Kossmeier, M., & Tran, U. S. (2019). Which data to meta-analyze, and how? Zeitschrift für Psychologie, 227(1), 64-82. DOI

Have questions or run into issues? Open an issue on GitHub or reach out.

Research Focus

The work continues examining how analytical choices shape conclusions about treatment effectiveness, but with a stronger focus on individual differences. Questions like: Do treatments work differently for different people? Can we identify patterns that help match patients to interventions? How do gender and other characteristics moderate treatment effects?

This fits naturally with multiverse meta-analysis approaches—both are about understanding when and for whom our conclusions actually hold. My previous work examined analytical variability at the study level (psychotherapy effectiveness, digital interventions). Now extending this to variability at the individual level.

I’ll keep sharing code and tools on GitHub and posting updates here. If you’re working on personalization, gender-sensitive interventions, or meta-science questions, let’s talk.

The essays

The collection

Alongside the essays there is a living, annotated collection of the literature — updated as the field moves, with notes on which analytic paths each paper varied and which decisions turned out to matter.

It is organised by where in the research pipeline the forking happens:

Layer 0 — Rationale and critiques. Should we multiverse at all? The case for running every reasonable analysis, and the strongest arguments against — that there is only one correct analysis, that multiverses manufacture doubt, that garbage specifications make garbage curves.

Layer 1 — Psychometrics. How we measure what we measure. The forking starts before any data are analysed: which instrument, which scoring theory, which measurement model — and whether that choice changes the method or the question itself.

Layer 2 — Primary studies. How we gather evidence. One trial, one dataset, and the many defensible ways to analyse it: covariates, exclusions, models, and what to do about the people who vanish halfway through.

Layer 3 — Meta-analysis. How we combine knowledge. Which studies get in, which effect size, which model, which of the many corrections for publication bias — the layer where my own multiverse work lives.

Layer 4 — Meta-meta. How we combine the combinations. Umbrella reviews, meta-meta-analyses, and what happens when the forking compounds across an entire literature.

The collection is certainly incomplete. Suggestions are genuinely welcome.