Results Summary
What was the research about?
Clinical trials study the effects of medical treatments, like how safe they are and how well they work. But most clinical trials don’t get all the data they need from patients. Patients may not answer all questions on a survey, or they may drop out of a study after it has started. The missing data can affect researchers’ ability to detect the effects of treatments.
To address the problem of missing data, researchers can make different guesses based on why and how data are missing. Then they can look at results for each guess. If results based on different guesses are similar, researchers can have more confidence that the study results are accurate. In this study, the research team created new methods to do these tests and developed software that runs these tests.
What were the results?
The research team created software and tested it by analyzing missing data from three clinical trials. For example, one clinical trial looked at a medicine for bipolar disorder. More than half of patients withdrew from the trial, creating missing data. Using the software to run tests, the team found that missing data didn’t change the results of the trial.
What did the research team do?
The research team created new software to run the tests and made the software available online for free. The team also used this software on data from previous clinical trials to see if study results would differ depending on how the team handles missing data.
An advisory panel of 15 people in the fields of statistics, software development, and medicine helped solve technical problems when creating the software.
What were the limits of the study?
The software doesn’t work with data that have outliers. Outliers fall far outside the normal range of other data within a study. In addition, data from studies may not be available to the public, which makes it hard to test the software further.
Future research could create software that works with data with outliers.
How can people use the results?
Researchers can consider using the software to see if missing data affect their results.
Professional Abstract
Objective
To develop and test new statistical methods and software for global sensitivity analyses in clinical trials with missing data
| Design Element | Description |
|---|---|
| Design | Theoretical development and empirical analysis |
| Data Sources and Data Sets | 3 clinical trials with missing patient-reported outcomes data |
| Analytic Approach |
|
| Outcomes | Software for conducting global sensitivity analyses in clinical trials with monotone and non-monotone missing data |
Sensitivity analyses allow researchers to have greater confidence in their results if their study has missing data. These analyses involve fitting a series of models using varying assumptions and then evaluating how these assumptions would influence treatment effect estimates. If results are similar across different models, researchers have more confidence that results are robust and not seriously biased by missing data. Currently few sensitivity analysis methods can address either
- Monotone missing data, or data that are missing after first missed assessment
- Non-monotone missing data, or data that are missing at one assessment but observed at a later assessment
Before this study, user-friendly software to implement sensitivity analyses didn’t exist.
In this study, the research team wanted to develop and test new statistical methods and software for global sensitivity analyses in clinical trials with monotone and non-monotone missing data.
An advisory panel of 15 experts in biostatistics, software development, evidence-based medicine, and patient-reported outcomes helped address technical issues when creating the software.
Results
The research team created new methods and software in SAS and R for conducting sensitivity analyses to address studies with monotone and non-monotone missing data. The programs are free to use and open source.
The research team conducted case studies of three clinical trials using the software. For example, one clinical trial compared the effect on patient-reported quality of life of each of two different strengths of quetiapine versus placebo in treating people with bipolar disorder. Many participants withdrew from the trial, which created monotone missing data. Only about 60% of patients from each study arm had complete quality of life data. The sensitivity analyses showed that across various ways of treating missing data, patient-reported quality of life was still statistically different between study arms.
Limitations
The methods and software developed in this study do not work well for data sets with outliers. In addition, it was difficult to test the software further because data sets from studies often are not available for public use.
Conclusions and Relevance
Researchers can use the software developed in this study to examine how sensitive research study results are to missing data.
Future Research Needs
Future research could develop methods and software that are robust to outliers to conduct sensitivity analyses on randomized controlled trials with missing data.
Final Research Report
View this project's final research report.
Journal Citations
Related Journal Citations
Peer-Review Summary
Peer review of PCORI-funded research helps make sure the report presents complete, balanced, and useful information about the research. It also assesses how the project addressed PCORI’s Methodology Standards. During peer review, experts read a draft report of the research and provide comments about the report. These experts may include a scientist focused on the research topic, a specialist in research methods, a patient or caregiver, and a healthcare professional. These reviewers cannot have conflicts of interest with the study.
The peer reviewers point out where the draft report may need revision. For example, they may suggest ways to improve descriptions of the conduct of the study or to clarify the connection between results and conclusions. Sometimes, awardees revise their draft reports twice or more to address all of the reviewers’ comments.
Peer reviewers commented, and the researchers made changes or provided responses. The comments and responses included the following:
- Reviewers expressed a concern about some of the methodological assumptions related to the statistical models developed in this study. The researchers disagreed that the assumptions could not be met, and therefore, there were major limitations on the study. However, they did acknowledge in the report that since these assumptions could impose restrictions on the data, they should be subjected to goodness-of-fit tests to make sure that the proposed approach is suitable.
- Noting that the report lists advisory board members and gives selected examples of how they engaged in the study, reviewers suggested the authors also provide examples of how each stakeholder group engaged. The researchers indicated that the advisory board engaged on an ad hoc basis. They reported successful and unsuccessful engagement in the report. For instance, the researchers worked with industry stakeholders on the advisory board to identify datasets that could be used in this study but ultimately used datasets provided through nonadvisory board industry connections.
- Given that the authors received very little feedback about their new software, reviewers asked how the researchers planned to obtain feedback. The reviewers also asked the researchers to provide more support for their statements that the lack of uptake was due to a lack of incentives and a lack of understanding of new methods. The researchers explained that they were not planning on seeking further feedback at this time. They explained that their speculation about the reasons for lack of uptake came from a senior FDA official who indicated that staff did not have the time and resources to train to use the new software tool. The researchers also compared this lack of uptake to other recommended best practices in analyzing and reporting study results which have not been widely adopted, saying that these best practices would need to be requirements in order to improve the uptake.