On Power Calculation Based on Effect Size in Clinical Research

Yinuo Zhang^* and Shein-Chung Chow
^*Correspondence: Yinuo Zhang, Department of Biostatistics and Bioinformatics, Duke University School of Medicine, Durham, NC 27710, USA, Email: ,

Keywords

Treatment effect • Effect size • Power analysis for sample size calculation

Introduction

Power calculation, also known as sample size estimation, is a crucial step in the design of clinical trials. It involves determining the number of participants required to detect a statistically significant effect of the treatment under investigation, given a specified level of significance and desired power [1]. The importance of proper sample size selection in clinical trials cannot be overstated. A study with an inadequate sample size may lack the statistical power to detect clinically meaningful treatment effects, leading to inconclusive or misleading results. Conversely, an excessively large sample size may result in undue exposure of participants to experimental treatments, increased costs, and potential ethical concerns [2]. In practice, power calculation can be a challenging task, especially when the investigational treatment is new and/or when there is limited information available regarding specific parameters of the test treatment under study. In such cases, the traditional approach of using the mean and standard deviation for power calculation may not be applicable or reliable.

Power calculation based on effect size offers a viable solution to this challenge. By focusing on the standardized effect size, a dimensionless measure of the magnitude of the treatment effect, the problem is reduced from estimating two parameters (mean and standard deviation) to estimating a single parameter (standardized effect size). This approach has several advantages:

a. It simplifies the calculation process, making it more efficient and requiring less information about the specific parameters of the study.

b. It can be applied to various study endpoints, such as relative change, absolute change, responder analysis and survival analysis, providing flexibility in the analysis of clinical trial data [3].

c. It facilitates comparison and synthesis of results across different studies and outcome measures, as effect sizes provide a standardized metric [4].

In this study, we aim to provide a systematic review and analysis of how power calculation based on standardized effect size can aid in sample size estimation for different study endpoints and in various clinical trial settings. By exploring the practical applications and implications of this approach, we seek to contribute to the understanding and implementation of efficient and reliable power calculations in clinical research. In the below session will delve into the formulation of hypotheses and sample size determination using standardized effect size in clinical trials, laying the groundwork for our methodology. In below session numerical analysis provides insights into estimating treatment effects for various endpoints, including relative change, absolute change, absolute risk, relative risk, exponential model and Cox's proportional model. We engage in a thorough discussion, critically examining the findings from our numerical analysis and considering their implications for clinical trial design and analysis. This includes a discussion on the limitations and potential areas for future research. Finally, it presents our concluding remarks, summarizing the study's key findings and their implications for the field of clinical research. We reflect on how our study contributes to a better understanding of the use of standardized effect sizes in power calculations and sample size estimation, offering recommendations for researchers looking to implement these methods in their clinical trials.

Methodology

Formulation of hypotheses and sample size determination using standardized effect size in clinical trials

To ensure the simplicity and broad applicability of our methodological framework, several foundational assumptions have been established. Firstly, the model presupposes the absence of interaction or confounding variables, which may otherwise distort the estimated effects of the treatments under consideration. Secondly, the design of choice is a two-sided, one-sample parallel configuration. This design is particularly advantageous for regulatory approval processes, such as those employed by the Food and Drug Administration (FDA), due to its robustness and the comprehensive nature of the comparative insights it provides [3]. Thirdly, we assume an equal allocation of subjects across treatment arms, which facilitates a balanced comparison and enhances the statistical power of the trial. Lastly, the assumption of equal variance across groups is adopted, ensuring that any observed differences in treatment effects can be attributed with greater confidence to the treatments themselves rather than to underlying variability in the populations studied. These assumptions are integral to the integrity of the statistical analysis and are reflective of standard practices within clinical trial methodology.

Continuous (Comparing means) and binary (Comparing proportions)

Test for equality: The trial aims to test if there is a difference as compared to an active control. The hypothesis often considered is

Where is the true mean difference between a test drug (µ_T) and active control agent (µ_R). Without loss of generality, consider ε > 0 as an indication of improvement (worsening) of the test drug as compared to the active control agent. The general power calculation for the required sample size per group N is given by:

Equation (1.1) [3]

Where α is the significance level and β is the probability of making type II errors. For simplicity, we assume . Based on Cohen’s d (Standardized mean difference), Equation (1.1) can be rewritten as:

Test for non-inferiority/superiority: The trial aims to test the non-inferiority/superiority as compared to an active control. The hypothesis often considered is:

The test of noninferiority and superiority can be unified with the above hypothesis,

where ε= µ_T - µ_R and δ is the noninferiority or superiority margin. With a, the rejection of the null hypothesis shows the superiority of the treatment group; With , the rejection of the null hypothesis shows the no inferiority of the treatment group. The general power calculation for the required sample size per group N is given by:

Equation (1.2) [3]

Equation (1.2) can be rewritten as:

Test for equivalence: The trial aims to test the equivalence as compared to an active control. The hypothesis often considered is

The general power calculation for the required sample size per group N is given by:

Equation (1.3) [3]

Equation (1.3) can be rewritten as:

Binary: The derivation of binary endpoints proceeds analogously to that of continuous endpoints. For binary outcomes, the effect size (ε) is defined as the difference in proportions, such that , where p and p₀ represent the probabilities of success in the treatment and a reference value, respectively.

Time to event (Cox’s proportional hazards model)

Test for equality: In practical applications, it is most common to evaluate two treatments by considering only the treatment indicator as the solitary covariate without making adjustments for additional covariates [3], to test for equality of two survival curves, the following hypotheses are usually considered:

Where . The general power calculation for the required sample size per group N is given by:

Equation 2.1 [3]

Where d is the probability of observing an event and p_i is the proportion of patients in the i^th treatment group. We assume d = 0.8 and p₁=p₂=0.5 for simplicity, Equation (2.1) can be rewritten as:

Test for Non-inferiority/superiority: To test for noninferiority and superiority of two survival curves, the following hypotheses are usually considered:

The general power calculation for the required sample size per group N is given by:

Equation 2.2 [3]

Equation (2.2) can be rewritten as:

Test for equivalence: To test for equivalence of two survival curves, the following hypotheses are usually considered:

The general power calculation for the required sample size per group N is given by:

Equation 2.3 [3]

Equation (2.3) can be rewritten as:

Table 1 provides a summary of the requisite formulas for computing sample size, predicated upon the establishment of a clinically meaningful difference and the quantification of the standardized effect size.

Numerical analysis

In the numerical analysis of our study, we explore the intricacies of estimating the treatment effect for different endpoints, specifically delving into the nuances of relative change, absolute change, absolute risk, relative risk, exponential model and cox’s proportional model and their impact on the determination of sample size in clinical trials. Our findings underscore the distinct nature of these metrics and illustrate that they are not readily translatable into one another, further reinforcing the necessity of a standardized measure effect size.

Continuous endpoint: The simulation was grounded on following assumptions: a significance level (α) of 0.05 for a 95% confidence threshold, a Type II error rate (β) of 0.20 for an 80% power, a standard deviation of outcomes (σ) set at 15, and a mean outcome of 150 in the control group.

In Table 2, we present the outcomes of our simulation for the test for equality. This Table illustrates the effects of varying treatment magnitudes on both absolute and relative change effect sizes, as well as their corresponding sample sizes. This approach serves as an example, highlighting the unique characteristics of absolute and relative changes and how they do not necessarily translate directly into one another. Similar procedures can be applied to reproduce other tests, providing insights into the distinct impacts of treatments across different metrics.

Table 3 and 6 display effect sizes for estimating sample sizes within the range of 0.2 to 0.8, aligning with the FDA's recognized range of effectiveness [5]. Table 3 provides uniform sample size calculations for both continuous and binary endpoints applying the formula from Table 1.

Binary endpoint: We assume that a baseline event probability in the control group (p₀) at 0.1, representing the probability of the event occurring without treatment intervention with a significance level (α) of 0.05 and a Type II error rate (β) of 0.20.

The Odds Ratio (OR) is a critical statistic for assessing relative risk in binary outcome studies. It is a non-negative value, typically ranging up to 4, where an OR of 1, corresponding to equal probabilities of an event in treatment and control groups, indicates no effect. An OR between 1 and 4 suggests a higher likelihood of the event occurring in the treatment group compared to the control group [3] (Table 4).

The effect size based on the odds ratio is often expressed in a logarithmic form, primarily to symmetrize the distribution and facilitate the application of standard statistical methods. The formula to compute the effect size for the odds ratio is as follows:

Time-to-event endpoint: We assume a baseline hazard rate, λ₁(t), of 0.5 for the control group, indicating the risk of event occurrence without treatment. For the treatment group, the hazard rate, λ₂(t), is conceptualized as 0.5+"treatment effect". These assumptions underpin the comparative analysis between control and treatment groups.

In our comparison, we utilized the Exponential Model, which posits a consistent hazard rate over time. Let be the difference between the hazard rates of a control and a test drug, which is the effect size here [3] (Tables 5 and 6).

Results and Discussion

The numerical analysis highlights the intricate relationships between effect size metrics and their distinct characteristics across different study endpoints. While absolute change, relative change, absolute risk and relative risk represent alternative ways to quantify treatment effects, our findings underscore their incommensurable nature, even when applied to the same dataset. This divergence becomes more pronounced as the treatment effect increases, underscoring the limitations of relying solely on a single metric to assess efficacy comprehensively. Interestingly, our analysis suggests a potential avenue for translating continuous and binary outcomes through a common effect size metric. By standardizing the treatment effect using appropriate scaling factors (e.g., standard deviation for continuous outcomes, baseline event rate for binary outcomes), it may be possible to derive a unified effect size measure that facilitates comparisons across diverse endpoints. This approach could prove valuable in meta-analyses and evidence synthesis efforts, where combining results from studies with varying outcome types is often necessary.

However, the translation between continuous, binary, and time-to-event outcomes presents additional challenges. While the exponential model and Cox's proportional model both capture the treatment effect on event occurrence, their effect size interpretations and underlying assumptions differ. Exploring methods to bridge these gaps, perhaps through the use of common hazard-based metrics or advanced modeling techniques, could be a fruitful area for future research. It is important to acknowledge the limitations of our analysis. The simulations were based on specific assumptions, such as equal variances across groups, absence of confounding variables, and equal allocation of subjects. Deviations from these assumptions in real-world scenarios may impact the accuracy of sample size calculations and effect size estimates. Additionally, our analysis focused on a subset of effect size metrics and study endpoints; other metrics or more complex study designs may warrant further investigation.

Conclusion

This study underscores the pivotal role of effect sizes in power calculations and sample size estimation for clinical trials. By highlighting the distinct nature of various effect size metrics and the challenges associated with translating findings across different endpoint types, our analysis reinforces the value of a standardized effect size measure as a unifying framework. The potential for translating continuous and binary outcomes through a common effect size metric represents a promising avenue for future exploration. Developing robust methods for such translations could facilitate more comprehensive meta-analyses and evidence synthesis efforts, ultimately enhancing our understanding of treatment efficacy across diverse clinical contexts. While the translation between continuous, binary, and time-to-event outcomes presents additional complexities, continued research into advanced modeling techniques and common hazard-based metrics may yield valuable solutions. Addressing these challenges could further refine the accuracy of sample size determination and facilitate more meaningful comparisons across clinical trials employing different endpoint types.

As clinical research continues to evolve, the insights gained from this study contribute to a deeper understanding of the complexities involved in power calculations and sample size estimation. By highlighting the importance of a standardized effect size measure and the potential for translating outcomes, this work informs more robust and efficient clinical trial design and analysis, ultimately advancing our ability to develop and evaluate safe and effective medical interventions.

Acknowledgement

None.

Conflict of Interest

The authors declare that there is no conflict of interests regarding the publication of this article.

References

1. Friedman, Lawrence M., Curt D. Furberg, David L. DeMets and David M. Reboussin, et al. “Fundamentals of clinical trials.” springer, (2015).

Google Scholar

2. Bacchetti, Peter, Leslie E. Wolf, Mark R. Segal and Charles E. McCulloch. "Ethics and sample size." Am J Epidemiol 161 (2005): 105-110.

Google Scholar, Crossref, Indexed at

3. Chow, Shein-Chung, Jun Shao, Hansheng Wang and Yuliya Lokhnygina. “Sample size calculations in clinical research.” Sample size calculations in clinical research. chapman and hall/CRC, (2017).

Google Scholar, Indexed at

4. Nakagawa, Shinichi and Innes C. Cuthill. "Effect size, confidence interval and statistical significance: A practical guide for biologists." Biol Rev 82 (2007): 591-605.

Google Scholar, Crossref, Indexed at

5. Keefe, Richard SE, Helena C. Kraemer, Robert S. Epstein and Ellen Frank, et al. “Defining a clinically meaningful effect for the design and interpretation of randomized controlled trials.” Innov Clin Neurosci 10 (2013): (5-6 Suppl A), 4S–19S.

Google Scholar, Indexed at