Statistical Evaluation of the Validity of Real-World Data and Real-World Evidence

Yuankang Zhao^* and Shein-Chung Chow
^*Correspondence: Yuankang Zhao, Department of Biostatistics and Bioinformatics, Duke University School of Medicine, Durham, USA, Email:

Keywords

Real-world data (RWD) • Real-world evidence (RWE) • Pragmatic clinical trial • Selection bias • Reproducibility probability

Introduction

Real-world data (RWD) refers to the data relating to patient health status or the delivery of health care routinely collected from various sources (FDA) [1]. The source of RWD includes electronic health record (EHR), medical claims databases, products and diseases registries, data from randomized clinical trials, and so on. As indicated in the US Food and Drug Administration (FDA) draft guidance on Framework for FDA’s Real-World Evidence Program, real-world evidence (RWE) refers to the clinical evidence about the usage and potential benefits or risks of a medical product derived from an analysis of RWD (FDA) [2]. In practice, RWE is often generated by different designs or investigations, such as pragmatic clinical trial (PCT) and prospective or retrospective observational studies.

Although randomized controlled trials (RCTs) are the gold standard for evaluating the safety and efficacy of pharmaceutical drugs, RCTs are conducted under specific or controlled environment, which do not reflect real clinical practice [3]. In practice, RCTs limit generalizability due to strict inclusion and exclusion criteria, as well as high costs and long duration, causes people to consider RWE as alternative clinical evidence [4]. Not only RWE can make up for the drawbacks, but it also can provide treatment effects evidence in more diverse applied settings due to the massive volume of data, as well as provide evidence of some rare disease drug development due to the data availability in multicenter trial and EHR. Therefore, the challenge faced by biostatisticians is how to generate robust RWE from RWD and integrate it into drug development and regulatory review; in other words, map RWE to Substantial Evidence [5].

In order to map RWE to substantial evidence (current regulatory standard and can only be obtained through the conduct of RCTs), researchers often focus on the following aspects. First, it is to determine the difference in evidence provided by RWD and data collected by RCTs (i.e., gap analysis between RWE and substantial evidence). Second, it is to evaluate whether the RWD is robust and representative of the target population (i.e., data relevancy and selection bias) [6]. In addition, data quality or data reliability is the most crucial part of evaluating RWD because of the volume and multiple sources nature of RWD. Finally, it is to assess whether historical data is suitable for data borrowing (e.g., the use of Bayesian inference) for efficient quantitative analysis in order to meet the current regulatory standard [7]. In this article, we will focus on the validity of RWD/RWE (i.e., the presence of selection bias and information bias).

As indicated by Chow SC [7], a mathematical model with statistical analysis was performed, which considers the biased positive studies proportion, providing a systematic way to evaluate the validity of RWD from the regulatory perspective [7]. The most recent guidance on RWD clearly indicates that the “relevant impacts of unstructured data on data quality should be documented in the protocol and analysis plan”. As indicated in FDA (2021), unstructured data refer to the data within EHR, either as free text data fields (such as physician notes) or as other non-standardized information in computer documents (such as PDF-based radiology reports). All these data need further processing (such as the deep learning algorithm with a significant amount of human aid) to extract valid clinical information, which inevitably brings bias. In this article, we study the validity (in terms of selection bias) of unstructured data and divide datasets into four classifications as shown in (Figure 1). Therefore, we will give a new model with further statistical analysis and simulation, as well as apply our model in RWD setting to prove the valuable insight from regulatory perspectives. In the Section 2, the statistical method for studying the validity of real-world data is briefly outlined. This section includes the estimation of the selection bias using three proposed reproducibility probability-based approaches. In the Section 3, simulation studies were conducted to examine the robustness and efficiency of the proposed methods. In the Section 4, the limitations and relative advantages of the proposed methods are compared with other bias adjustment methods. Section 5 provides some concluding remarks.

Statistical methods

The validity of real world data: Let μ_rwd be the true mean of the target population study’s target patient group, μ_s and μ_ns be the true means of data sets from positive and negative studies done in the same target population. Also, let μ_s and μ_ns be the true means of data sets of structured data and unstructured data. In addition, let ρ be the true proportion of structured data conducted on the target population, and let r₁ and r₂ be the true proportions of positive result within the structured data and unstructured data, respectively. Assume that there is no treatment-by-centre and treatment-by-study interaction in multicentre studies for illustrating. Following (Figure 1), we have

Usually, μ_N<δ<μ_P , where δ is the clinical meaningful treatment effect. In other words, the estimate effect of a study larger than δ would be considers a positive study. Besides, usually in practice, we would expect .In particular, if r_i = 1 , then μ would degrade to μ = μ_P .

Then, for illustration purpose, we assume that all the studies included in the RWD are parallel design studies, and all positive and no positive studies share same sample size, denoting n_P and n_N.x_ij denote the clinical response of the patient i in the jth positive study, i=1,…,n_P and j=1,…,sn, where n is the number of studies included in the RWD. Besides, denote the clinical response of the patient i in the jth no positive study, i=1,…,n_N and j=1,…,(1-s)n. If ρ and r_i are given, the bias of mean of RWD, μ_rwd is

(4)

Where

Based on the equation 4, if we assign ρ and r_i certain value, we can gain the bias as shown in Table 1. From Table 1, we can find that the different ρ and r_i correspond different selection bias.

In practice, however, r_i are unknown. For a given RWD, estimation of r_i is based on the number of positive studies in RWD. Similarly, the estimation of ρ (ie. Pˆ, ) is based on the number of structural data in RWD. However, i r usually be overestimated, since positive data are more likely published and included in historical data (S.C Chow, 2020). Therefore, we have

Therefore, the bias of mean of RWD, μ_rwd is given by

For simplicity, if we neglect Δi.Δj, where i≠j, then the bias of mean of RWD can be simplified to

(5)

Considering the power calculation, by (1), the variance of μ_rwd is given by

(6)

If we take derivative above, then

In practice, the variance in positive studies is larger than in negative studies, since difference between controlled group and referred group is larger in positive study. If the gap of sizes of positive and no positive studies is not huge, we have . Thus, is an increasing function of r₁ and r₂ . Furthermore, the power of the RWD can be calculated by the probability:

Based on the data availability of RWD.

Estimation of u_P - u_N

Let (L_P,U_P) and (L_N,U_N) be the (1− a)100% for u_P and u_N. Under the assumption of normality, we have

According to Chow’s assumption (2020), when selection bias does exist, it is reasonable to assume the positive studies and no positive studies are different. In this case, we assume u_P > u_N and (L_P,U_P) and (L_N,U_N) would not have intersection. At some extreme case [7], L_P is close to U_N . Then, we have

Therefore, the distance of u_P and u_N can be calculated as the estimation of u_P − u_N , we have

Reproducibility probability: The definition of reproducibility probability is the estimated power (EP) of a future trial using the information from previous trials. In theory, different trials are independent; the probability of achieving a statistically significant result from the new trial would be identical with previous studies if these trials apply the same study design and hypothesis, regardless of the outcome of previous trials. When H_A is true, the probability of gaining a significant result is the power of the test [8,9]

Where H_A is the alternative hypothesis and θ is an unknown parameter or a vector of parameter (Shao & Chow, 2002). Even though trials are independent to each other, it reasonable to use the previous trials information in RWD to refer a later trial. Reproducibility probability could be estimated through (i) the estimated power (EP) approach, (ii) the confidence bound (CB) approach, which is more a conservative approach than EP approach and (iii) Bayesian approach, a more sensible approach to obtain reproducibility [9].

To illustrate EP approach, consider a control group and reference group trial with unequal variances. Let x_ij be the jth subject in the ith group (i = 1,2) and distributed as N(u_i,σ_i²) respectively, in which Therefore, assuming n₁ and n₂ are large, statistics T is shown as

Then, T asymptotically has normal distribution N(θ,1), where

Therefore, the reproducibility probability can be calculated by replacing θ by its estimate statistics T

The EP approach would provide an optimistic result that the adjustment of bias might be inadequate when applied in our proposed approach. The CB method would yield a more cautious estimation of reproducibility probability.

Considering the trial setting mentioned before, the CB approach considers a 95^th percent lower confidence bound as the reproducibility probability. Therefore, CB approach provides the estimation of reproducibility probability as follow equation.

Furthermore, the Bayesian approach provides a clear definition of reproducibility. To be more specific, we assume the unknown parameter θ is a random vector with priori distribution π(θ), which is known. Then, we can define reproducibility probability is the conditional probability of |T|>c in the future trial, which can be shown as

where T is T statistics based on future data set and π(θ|x) is the posterior density of θ given x.

Considering the same setting in EP approach part, if the variance, σ ² , is known, then reproducibility probability is

The proportion of positive studies in RWD, ˆr , is highly likely overestimated, since positive studies are more likely collected. In other words, ˆr is likely larger than the true proportion r.

According to the definition of reproducibility probability, r can be estimated by reproducibility probability of observing a significant result based on the mean and variance of response in RWD. The probability can be represented as follow equation:

The interpretation of the reproducibility probability under this setting is that we expect to gain p × 100 significant studies if the similar experimental setting trial conduct 100 times based on the observed mean response ( μˆ_B) and standard deviation ( σˆ_B ) in RWD. To be more specific, with RWD, we have

where are the pooled means of controlled and reference group in RWD, n₁ and n₀ are the sizes of controlled and reference group of pooled RWD, are the estimated variances of the controlled and reference group, respectively. Then, the T statistic is shown as

where n_m1 and n_m0 are the median sample sizes of controlled and reference group in studies included in RWD. Therefore, the bias of positive proportion, Δi. can be represent as

Simulation study

In this part, we used the method proposed in chapter 2 to adjust bias in simulation studies of different scenarios. The performance of methods is evaluated by comparing the adjusted mean and the true mean of the target population. For simplicity, we assume that the structural data is from equal sample size randomized clinical trials and the unstructured data is from equal sample size observational cohort studies. All these data are generated from the same population. As for the target population, we assume that each trial or study contains two groups of patients, the treatment group and the control group. Each group's responses follow the normal distribution N(2,25) and N(0,25). In order to achieve 80% power to detect a clinically meaningful difference of 2, 100 subjects in each group are needed based on sample size calculation. Furthermore, to achieve 50%, 60%, 70%, 80%, 90% power in the trial, we need sample size of 49, 63, 79, 100, 133 per group respectively. We generated 1000 studies to simulate the target population for each sample size. To evaluate the performance of the proposed methods, we define relative bias as , where μˆ_rwd is the estimated treatment effect based on real-world data and μ is the true treatment effect, which is 2 in this simulation. We also define the relative adjusted bias as , where ε is the estimated bias based on the proposed method. Four scenarios are considered to assess the robustness of the adjustment method. The first scenario compares performance between EP (estimated power) method, CB (confidence bound) method, and Bayesian method in structural data under the previous setting when only structural data presented. The second scenario compares the performance of the three methods when negative data are absent in structural data. The third scenario tests the robustness when structural data and unstructured data presents inconsistent positive proportion. Finally, we test the robustness when inconsistent positive proportion presents and imbalance structural data proportion occurs.

In the first scenario, we assessed the relative bias and adjusted bias based on three adjustment methods by different power and r, the proportion of positive studies, which is summarized by (Table 2) and (Figure 2). As expected, when the gap between the proportion of positive studies and power, the theoretical proportion of positive studies, increases, the selection bias increases. As the most conservative adjusted method, EP method can slightly adjust selection bias compared with the other two methods. When the bias larger than 15%, the bias can reduce 20% by the EP method. On the other hand, when the bias less than 5%, the adjustment is limited, causing the adjusted bias is nearly the same as original bias. As for CB approach, the adjustment is significantly larger than the adjustment made by EP approach. When the selection bias is larger than 10%, a significant adjustment in right direction can be made by the EP approach. However, when the bias is smaller than 10%, the CB approach can lead the bias toward the wrong direction and even show an absolute bias larger than the original bias. Finally, the Bayesian approach give us a more effective adjustment result. The overall adjustment pattern is similar to the EP approach but more effective. To be more specific, when the bias larger than 15%, the bias could reduce 40% by the EP method. When the bias is less than 5%, the Bayesian approach significantly leads to more adjustment compared with EP approach. In addition, compared with the CB approach, the Bayesian approach always makes adjustment toward the right direction.

Second scenario, we consider the situation that only positive studies exist in the structural data. In other words, the r₁ , proportion of positive study is fixed to 1 in this scenario. As we can see in Table 3 and (Figure 3), bias decrease when power increase, and the result of the EP approach and Bayesian approach is similar to the pattern in scenario 1. More specifically, when bias is more negligible, the EP and Bayesian approaches lead less adjustment. In addition, the EP approach provides a more conservative adjustment compared with the Bayesian approach. As for CB approach, similar as scenario 1, when bias is larger than 10%, the bias can be significantly reduced. However, the CB approach could lead adjustment toward the wrong direction when bias smaller than 10%.

In the third scenario, the power set to 0.6, ρ set to 0.5, and the rest of setting is the same as previous scenario, and we consider the situation that the structural data and unstructured data provide inconsistent positive proportion. To be more specific, we consider that the structural data provide high proportion of positive result and the unstructured data provided low proportion of positive result. In this context, according to Table 4, when the two proportion are away from each other and the mean of two proportion away from the setting power, 0.6, the bias decreases. Similar situation observed in scenario 1 and 2. The EP approach and Bayesian approach provide effective bias adjustments toward the right direction, and the Bayesian approach provide slightly more adjustment compared with EP approach for most of the case. As for CB approach, most of the adjustment towards wrong direction, which in line with the observation about CB approach under 10% bias from scenario 1 and 2. However, we do notice that the adjustment based on Bayesian method also toward to wrong direction slightly when bias is less than 5%. In conclusion, the EP approach and Bayesian approach provide effective adjustment under the inconsistent positive study proportion situation, but CB approach lead the adjustment toward wrong direction in this scenario.

In the final scenario, we consider both imbalance of structural data and inconsistent positive proportion occurs. The r₁ set to 0.4, and r₂ set to 0.8, the power set to 0.6, and the rest of setting is the same as scenario 1. As we can see in Table 5, when the ρ, the structural data proportion, increase (i.e., Balance of structural and unstructured data), the bias reduced. Under the imbalanced structural data proportion, both EP approach and Bayesian Approach shows capability to reduce bias. But for CB approach, it adjusts the bias toward the wrong direction and even increase the bias when the bias less than 5%.

In summary, three proposed methods can adjust the bias in real world data. EP approach provides the most conservative and stable adjustment. The Bayesian method provides more adjustment toward the right direction with robustness in most the extreme cases. In addition, the CB approach leads to the most aggressive adjustment of these three methods. However, it shows less stability and robustness when the power is larger than the proportion of positive studies in structural data or the bias of real-world data is less than 10%.

Discussion

Several limitations were found in the simulation study. First, when the proportion of positive studies is close to the designed power, the adjustment led by the EP approach and Bayesian approach is limited. In some cases, the Bayesian method even enlarges the bias. This is because the reproducibility probability-based approaches can make over-adjustments when bias is not obvious. Another reason is that the estimation of is over-estimated when bias is negligible. The second limitation is that CB approach provides adjustment towards the wrong direction when bias is less than 10%. One explanation is that the lower confidence bound provided by CB approach is too conservative in estimating the real positive proportion, r, when bias is less than 10%. In addition, the over-estimation of can also lead adjustment failure of CB approach.

Previously, compared the substantial evidence and real-world evidence and pointed out that the bias in the substantial evidence is minimized but selection bias exists in the real-world evidence, and proposed a reproducibility probability based bias adjustment approach. Compared to the method in our paper, we added the Bayesian method to adjust bias, which has been proved as an effective and robust adjustment method and applied the method in a new architecture of real-world data, which divided the data into structural and unstructured data. In reality, meta-analysis has been used to estimate non-inferiority margin or treatment effect based on historical data, which is similar to use real-world data to estimate the treatment effect. The parameter, can also be estimated by meta-analysis even the individual data is unavailable [10]. Therefore, the combination of meta-analysis and our method can also adjust the bias in real world data. However, either our proposed method or meta-analysis method has the assumption that there is no treatment-by-study interaction and treatment-by-center interaction. Therefore, further investigation of these two interactions can be included in future work.

Regarding the bias in RWD or historical data, various methods have been proposed to identify and quantify the bias. An adjusted rank correlation test has been proposed to identify the bias in a meta-analysis based on historical data [11]. The test shows fairly high power for meta-analysis when studies number are large, which also suitable for RWD bias detection. Egger proposed a test of asymmetry funnel plot that can predict the discordance of outcomes in meta-analysis and evaluated the prevalence of bias in metaanalyses [12]. Furthermore, several methods are proposed to control the bias to acquire inferiority margin or treatment effect from historical data. Chow and Shao proposed a method for selecting non-inferiority margin with statistical assurance. However, these methods only detected the bias but did not adjust the bias in historical data. As for bias adjustment method in RWD proposed a structure that generates robust RWE from RWD via adjusting confounding bias based on causal inference [13]. In addition, several suggestions study designs were proposed to adjust the selection bias in RWD. In our method, not only do we define and quantify the selection bias in RWD, but also we give adjustment methods quantitively [14,15].

Conclusion

Real-World data often consist of positive or negative studies and the data may be structured or unstructured. In this case, the validity of RWD is a concern for providing evaluation of safety and efficacy of the test treatment under investigation. The validity of RWD or Real-World Evidence is important, especially when it is intended to support a regulatory submission. In this paper, we discussed the selection bias adjustment of real-world data based on reproducibility probability approaches. We defined the selection bias in real-world data under the structural and unstructured data setting. Based on the form of bias, three reproducibility probability-based approaches have been introduced to estimate the real proportion of positive studies in the structural and unstructured data. The reproducibility probability-based approach provides effective bias adjustment when the proportion of positive studies is different as designed power. EP approach and Bayesian approach provide robust and effective bias adjustment in most of cases, and the CB approach provide huge and effective adjustment only when bias is larger than 10%.

References

1. FDA. "Real‐world data: Assessing electronic health records and medical claims data to support regulatory decision‐making for drug and biological products."Pharmacoepidemiol Drug Saf (2021).

Google Scholar, Crossref, Indexed at

2. FDA. "Framework for FDA’s real-world evidence program." Food and Drug Administration (2018).

Google Scholar, Indexed at

3. Kabisch, Maria, Christian Ruckes, Monika Seibert-Grafe and Maria Blettner. "Randomized controlled trials: Part 17 of a series on evaluation of scientific publications."Dtsch Arztebl Int 108 (2011): 663.

Google Scholar, Crossref

4. Franklin, Jessica M. and Sebastian Schneeweiss. "When and how can real world data analyses substitute for randomized controlled trials?."Clin Pharm Therap 102 (2017): 924-933.

Google Scholar, Crossref, Indexed at

5. Fang, Yixin, Hongwei Wang and Weili He. "A statistical roadmap for journey from real-world data to real-world evidence."Ther Innov Regul Sci 54 (2020): 749-757.

Google Scholar, Crossref, Indexed at

6. Song, Fuyu, Chenxuan Zang, Xinyi Ma and Sifan Hu, et al. "The use of real-world data/evidence in regulatory submissions."Contemp Clin Trials 109 (2021): 106521.

Google Scholar, Crossref, Indexed at

7. Chow, Shein-Chung.Innovative methods for rare disease drug development. Chapman and Hall/CRC (2020).

Google Scholar, Crossref, Indexed at

8. Goodman, Steven N. "A comment on replication, p‐values and evidence."Stat Med11 (1992): 875-879.

Google Scholar, Crossref, Indexed at

9. Shao, Jun and Shein‐Chung Chow. "Reproducibility probability in clinical trials."Stat Med 21 (2002): 1727-1742.

Google Scholar, Crossref, Indexed at

10. Hozo, Stela Pudar, Benjamin Djulbegovic and Iztok Hozo. "Estimating the mean and variance from the median, range, and the size of a sample."BMC Med Res Methodol 5 (2005): 1-10.

Google Scholar, Crossref, Indexed at

11. Begg, Colin B. and Madhuchhanda Mazumdar. "Operating characteristics of a rank correlation test for publication bias."Biometrics(1994): 1088-1101.

Google Scholar, Crossref, Indexed at

12. Egger, Matthias, George Davey Smith, Martin Schneider and Christoph Minder. "Bias in meta-analysis detected by a simple, graphical test."Bmj315 (1997): 629-634.

Google Scholar, Crossref, Indexed at

13. Levenson, Mark, Weili He, Jie Chen and Yixin Fang, et al. "Biostatistical considerations when using RWD and RWE in clinical studies for regulatory purposes: A landscape assessment."Stat Biopharm Res (2021): 1-11.

Google Scholar, Crossref, Indexed at

14. Sheikhalishahi, Seyedmostafa, Riccardo Miotto, Joel T. Dudley and Alberto Lavelli, et al. "Natural language processing of clinical notes on chronic diseases: Systematic review."J Med Internet Res 7 (2019): e12239.

Google Scholar, Crossref, Indexed at

15. Miguel-Alvarez, Marina, Alejandro Santos-Lozano, Fabian Sanchis-Gomar and Carmen Fiuza-Luces, et al. "Non-steroidal anti-inflammatory drugs as a treatment for Alzheimer’s disease: A systematic review and meta-analysis of treatment effect."Drugs Aging32 (2015): 139-147.

Google Scholar, Crossref, Indexed at