Results Summary
What was the research about?
Organizations that fund research often seek input on which topics are important to study. Patients can take part in setting priorities for research to help make sure that future studies focus on the topics that matter most to them. Researchers can collect patients’ views on which research topics are most important in different ways.
In this study, the research team compared three methods of research priority setting:
- Online crowd voting, where patients submit, discuss, and vote on ideas online
- Focus groups with nominal group technique, where patients come up with ideas on their own and then discuss those ideas in a group with a moderator
- Modified Delphi method, where patients give input through a series of mailed surveys
The research team looked at research topic rankings and experiences with each method among patients with low back pain.
What were the results?
Across the three methods, patients came up with similar lists of priorities. The highest-rated research topics included causes of low back pain, ways to improve communication between doctors and patients, and self-care strategies.
Of the three methods, patients who took part in focus groups rated their experience highest. They liked having in-person discussions with other patients and the research team.
Who was in the study?
The study included 151 patients ages 65 and older with low back pain. Of these, 78 percent were white, 14 percent were black, and 5 percent were Asian. The average age was 76, and 60 percent were women. All patients received care from large healthcare systems in Michigan and Northern California.
What did the research team do?
In the first part of the study, the research team came up with a list of research topics. To do so, they asked patients who were part of a low back pain registry or part of a large online community to take a survey. The survey asked patients about important topics for research. The survey also gave patients in the registry the choice to take part in one of the three research priority-setting methods.
The research team assigned patients to one of the three methods by chance, taking their preference among the methods into account. Patients in all three groups received the list of topics from the first part of the study.
In the online crowd voting group, 38 patients discussed and voted on topics over six weeks. In the focus groups, 39 patients, in groups of up to 10, produced topic lists after discussion.
In the Delphi method group, 74 patients received a survey where they rated the importance of each topic. The research team summarized the patient feedback. Then the team created a second survey with the new topic rankings. Patients reviewed the new rankings and could change their own rankings if they wanted.
At the end of each method, all patients received a survey asking them to rate their experience with their priority-setting method. The survey also asked what they liked the most and least about their method.
A patient advisor was a member of the research team. A patient advisory group also gave input throughout the study.
What were the limits of the study?
All patients in the study were older adults who received care in health systems that managed all aspects of their care. Their preferred research topics may be different from those of other groups with low back pain or with other health problems. Patients’ preferences for priority-setting methods may be related to traits, such as race and gender. Such traits, rather than the methods themselves, may have affected patients’ responses to survey questions.
Future research could compare priority-setting methods with more diverse groups of patients.
How can people use the results?
Organizations that fund research can use these results when considering how to get patient input on important topics for research.
Professional Abstract
Objective
To compare three interactive research prioritization methods—online crowd voting, focus groups with nominal group technique, and modified Delphi method—in terms of priority ranking and participant evaluations among patients 65 and older with low back pain
Study Design
Design Elements | Description |
---|---|
Design | Cross-sectional |
Population | Patients ages 65 and older with low back pain |
Interventions/ Comparators |
|
Outcomes | Level of agreement in research priorities across research prioritization methods, participant evaluation of engagement experience |
Timeframe | Not applicable |
This cross-sectional study examined the level of agreement in research priorities generated by three common, interactive research prioritization methods and compared participant evaluations of each method.
In the first phase of the project, people who were part of a large registry of patients ages 65 and older with low back pain or part of a large crowdsourcing platform generated a list of research topics. The second phase of the project included 151 participants who were part of this registry and received care from large healthcare systems in Michigan and Northern California. Of these, 78% were white, 14% were black, and 5% were Asian. The average age was 76, and 60% of participants were female.
Researchers randomly assigned participants to one of three groups:
- Online crowd voting, which used an online platform where participants submit ideas, vote on existing ideas, and interact with others through online discussion
- Focus groups with nominal group technique, which combined individual idea generation with moderated, interactive group discussion
- Modified Delphi method, which used a series of mailed surveys to obtain input from respondents
To address potential barriers to participation, researchers used preference-based randomization, asking participants to rank the methods in order of preference and weighting their group assignment probability by preference.
Participants in all three groups received the list of research topics from phase 1 and prioritized them. In the online crowd voting group, 38 participants discussed and voted on priorities over six weeks. In the focus groups, 39 participants, split into six groups, produced priority lists after multiple rounds of discussion.
In the Delphi method group, 74 participants received a paper-based survey and rated the importance of each priority on a five-point Likert scale, indicating which priorities they thought were the most and least important. After receiving these responses, researchers sent participants a summary of the group’s rankings and feedback. Participants reviewed the rankings and revised their priority list, if desired.
All participants received an evaluation survey that asked about their experience with and the perceived effectiveness of the prioritization method.
A patient advisor was a member of the research team. In addition, a patient advisory group provided input throughout the study.
Results
Across the three methods, participants generated similar lists of research priorities. Highly rated topics included causes of low back pain, improving physician-patient communication, and self-care strategies.
In the evaluation survey, focus group participants rated their experience highest compared with participants in the other methods. Focus group participants cited their interactions with other participants and the research team as the parts they liked most about the activity.
Limitations
All participants were adults aged 65 and older with low back pain who received care in integrated healthcare systems. Results may differ for patients with different backgrounds or other health conditions. During randomization, participants’ preferences for a certain engagement method may have been associated with other characteristics that could have affected prioritization or evaluations of experience.
Conclusions and Relevance
Common methods for research prioritization yielded similar priorities but different participant experiences. Research funders can use these results when considering methods to elicit patient input on priorities for research funding.
Future Research Needs
Future research could compare prioritization methods with diverse patient populations.
Final Research Report
View this project's final research report.
Journal Citations
Results of This Project
Related Journal Citations
Peer-Review Summary
Peer review of PCORI-funded research helps make sure the report presents complete, balanced, and useful information about the research. It also assesses how the project addressed PCORI’s Methodology Standards. During peer review, experts read a draft report of the research and provide comments about the report. These experts may include a scientist focused on the research topic, a specialist in research methods, a patient or caregiver, and a healthcare professional. These reviewers cannot have conflicts of interest with the study.
The peer reviewers point out where the draft report may need revision. For example, they may suggest ways to improve descriptions of the conduct of the study or to clarify the connection between results and conclusions. Sometimes, awardees revise their draft reports twice or more to address all of the reviewers’ comments.
Peer reviewers commented, and the researchers made changes or provided responses. The comments and responses included the following:
- Reviewers commented that taking participants’ preferences into account when assigning them to one of three prioritization sessions in Phase 2 of the study, rather than assigning them completely randomly, made it less clear whether differences between groups were because of preferences or due to chance. The researchers acknowledged this concern and expanded discussion of this limitation in the report. They also added a table that compares participant characteristics in Phase 1 of the study by first preference.
- Reviewers noted that in addition to the preferences of patient participants, the relative workload and cost associated with different methods of soliciting patient input would be helpful for researchers to know. The researchers agreed these are important pragmatic considerations. The researchers could not evaluate the costs of different methods, but they added discussion about the time and resources required for each method.
Conflict of Interest Disclosures
Project Information
Key Dates
Study Registration Information
Note: This project was initially titled "Comparing Engagement Techniques for Incorporating Patient Input in Research Prioritization."