We adhered to methods that have been reported in prior research works [12, 14,15,16,17]. All analysis was conducted using the R statistical environment [18], and the code and original data to reproduce this work is made publicly available at the following online repository: https://doi.org/10.5683/SP2/BEVNRG.
2.1 Data Extraction and Preparation
Meta-analyses were identified through a PubMed search conducted in April 2021 using the following search terms: [“endothelial function” (Title/Abstract)] AND [meta-analy*(Title/Abstract)]. Meta-analyses were screened for eligibility, and the included meta-analyses are accessible as supplementary material. Effect sizes, and their corresponding lower and upper limits, were extracted from any meta-analysis that reported a standardized mean difference effect size (SMD; Cohen’s d or Hedges’ g) for endothelial function assessed via changes in vasomotion or changes in blood flow. Meta-analyses that included outcomes related to pulse-wave velocity or augmentation index were excluded as these outcomes, though correlated with measures of endothelial function, also represent measures of arterial stiffness and structure and therefore beyond the scope of this work [19]. We reported the absolute value of the effect sizes as the objective of our investigation was to query the magnitude, and not direction, of the reported effects for the effect size distribution analyses [12, 15].
Group sample sizes and the region of vascular assessment (microvascular or macrovascular) were extracted from descriptive tables or figures in the meta-analysis. In the event that this information was missing or ambiguous, the details were either retrieved from the primary article or coded as unavailable. We also recorded the category of the studied biological process that corresponded to the effect size extracted from the meta-analyses to appreciate the range of investigation topics included in the data. The biological process categories were determined according to the title and topic of the meta-analysis.
A Hedges’ g correction was applied to provide a conservative estimation of Cohen’s d values; particularly for those based upon small sample sizes or predicated upon biased sample size estimates [6]. Cohen’s d values were converted to Hedges’ g using a formula derived from Lakens [9], where n1 and n2 denote group sample sizes:
$$Hedges^{\prime}g = Cohen^{\prime}s ~d \times \left( {1 - \frac{3}{{4\left( {n_{1} + n_{2} } \right) - 9}}} \right)$$
We also solved for Cohen’s d from the above equation to enable us to convert effect sizes that were originally reported as Hedges’ g in meta-analyses to Cohen’s d, such that all effect sizes could be expressed as either a Cohen’s d or Hedges’ g effect statistic. If a Hedges’ g effect size was reported with no indication of sample size, then the statistic was kept as Hedges’ g in our analysis [14].
Duplicate effect sizes were coded as such and subsequently removed from analysis. Effect sizes larger than eight were deemed to be unrealistically large and classified as outliers, and therefore removed from our analysis to mitigate the influence of outlying data on effect size distributions. We chose to classify effect sizes larger than eight as outliers because this value corresponded to the largest effect size reported among 6,447 effect sizes from social psychology [14].
2.2 Statistical Analyses
Cohen’s original work defined that small and large effect sizes are equidistant from the medium effect size [12, 20]. Thus, we rank-ordered and plotted effect sizes from smallest to largest using a frequency distribution and found the effect sizes at the 25th, 50th, and 75th percentiles. We then subdivided the data according to the assessment of vessel type (macrovascular or microvascular), and according to the modality used to measure endothelial function [ultrasonography, plethysmography (peripheral arterial tonometry was included in this category due to its classification as a plethysmographic method [21, 22]), laser doppler, or other measures such as capillaroscopy and near-infrared spectroscopy] and repeated the aforementioned analysis. The ‘psych’ package [23] was used to describe the skew and kurtosis of each distribution.
We examined the bivariate relationships between effect size magnitude and year, and effect size magnitude and the logarithm of sample size. We plotted the reported effect sizes as a function of the year in which they were published to visualize any changes in the magnitude of effect sizes over time, as measurement methods and guidelines for endothelial function change over time [24, 25]. A locally estimated scatterplot smoothing curve with a smoothing parameter span set to ƒ = 0.75 was employed to visualize the relationship between effect size magnitude and year of publication. Locally estimated scatterplot smoothing was used to detect any fluctuation in effect size magnitude associated with the publication of guidelines for the assessment of endothelial function using flow-mediated dilation (in 2011, see [24]; and in 2019, see [25]). Some meta-analyses reported multiple effect sizes from the same study and thereby include correlated observations that violate assumptions of independence. To mitigate the undue influence of correlated data, we computed regression coefficients (β) and adjusted standard errors (SEadj) using vce(cluster) syntax on STATA (version 17; StataCorp, College Station, TX, USA) as well as a clustered bootstrap 95% confidence intervals (95% CI) based on 10,000 samples using the ‘jtools’ [26] and ‘ClusterBootstrap’ [27] packages available in the R environment.
A series of a priori power calculations for independent and paired samples t tests, powered to detect the small, medium, and large effect sizes for endothelial function research, were computed using the ‘pwr’ package [28]. A visualization of the sample sizes required to reliably detect a given effect size, for a range of statistical power, for independent samples and paired t tests was constructed. Each power calculation was determined assuming two-tailed analyses and α set to 0.05.
The ‘metameta’ package [29] was used to visualize the median power of all meta-analyses that are included in our work. The evidential value of each meta-analysis was displayed across a range of possible true effect sizes from δ = 0.1 to δ = 1.0, as well as for the observed summary effect size reported in the meta-analysis.