User login
Partnering with stakeholders using an example patient-reported outcomes project
Recently, researchers have been challenged to design methods that ensure that key constituents are partners in research, and not simply participants. Here we describe some innovative approaches we used to engage stakeholders. The approaches are drawn from a patient-centered outcomes research project, focusing on the graphic display of patient-reported outcomes (PROs) data. PROs represent patients’ perspectives on the impact of health, disease, and treatment, without interpretation by a clinician or anyone else. PROs include, among other things, patients’ assessments of their symptoms, their level of physical and psychosocial functioning, and health-related quality-of-life.1
As a first example of the key role of stakeholders in this project, input from cancer patients and clinicians, drawn from previous research, motivated us to ask whether there might be a “better way” to display PRO data when used to inform clinical practice. Specifically, even though cancer patients and clinicians endorse the importance of PRO data to promote patient-centered care, both groups report challenges using PROs in practice because of difficulty understanding what the PRO scores mean (eg, what is a good score or a bad score?; for individual patients, which scores should clinicians be concerned about?; for clinical trial PROs, what differences in PRO scores between treatments are clinically important?). The challenges in interpreting PRO data result in part from a large number of PRO measures (eg, one database includes more than 1,000 instruments)2 and no standards across PRO measures regarding how they are scored and scaled, or in how the data are presented.3 For example, on some PRO measures, higher scores represent better outcomes; on some PRO measures, lower scores represent better outcomes; and on some PRO measures, whether higher or lower scores represent better outcomes depends on the domain being measured. Further, some measures are scaled 0-100, with the extremes representing the best/worst scores possible, whereas others are normed to, for example, a population average of 50. Because of this variation, a score of 70 can have a completely different meaning depending on the PRO measure (or domain within a measure). As noted above, previous research has documented that this variation limits patients’ and clinicians’ understanding of the PRO scores, creating an important barrier to their use in practice.4-5
To address this stakeholder-driven research question, we undertook a three-part study to identify approaches for PRO data display that can be easily interpreted, regardless of scoring or scaling conventions, with the overall goal of improving patient and clinician understanding and use of PROs in oncology clinical practice. Part 1 of the study identified attributes of graphic displays of PRO data that are helpful and confusing.6 Part 2 involved developing improved PRO data presentation approaches.7 Part 3 evaluated the accuracy-of-interpretation and clarity of the developed approaches.8-10 The methods and findings of the three-part study are reported elsewhere;6-10 here, we describe the various approaches employed to engage stakeholders throughout the project.
As described above, the first reflection of stakeholder input was in the research question we asked. We then sought to identify the key stakeholder groups and ensure that they participated in each stage of the project. The relevant stakeholder groups we identified were: patients and their caregivers; health care providers (eg, oncologists, oncology nurses) who need to understand PRO data for their own consideration and for discussion with patients; and PRO researchers who develop, validate, and apply PRO measures.
Having identified these three key stakeholder groups, we sought to obtain broad representation of their perspectives. For example, we ensured that our investigative team included a cancer survivor, a cancer care provider, and PRO researchers. To supplement the stakeholder input from the investigative team, we formed a nine-member Stakeholder Advisory Board, with multiple representatives from each key constituency. We also aimed to be as broad as possible in the populations sampled for data collection. For example, we extended beyond the Johns Hopkins cancer center to include the Johns Hopkins Clinical Research Network, a consortium of academic and community health systems across the mid-Atlantic United States. Beyond the in-person data collection across the region, our study also included an internet survey of cancer patients/survivors, cancer care providers, and PRO researchers from across the United States and internationally. Taken together, these approaches improve the diversity of our sample and, thereby, the generalizability of our findings.
In addition to obtaining broad perspectives across stakeholder groups, we created genuine partnerships with the stakeholders to inform every aspect of the project. As described above, the study itself was motivated by feedback from cancer patients and clinicians regarding the challenges they experienced when trying to interpret PRO scores, and we therefore ensured that each stakeholder group contributed to the study’s design. Stakeholders also played a critical role in the conduct of the study. For example, in the first part of the study, we conducted one-on-one interviews with 50 cancer patients and 20 cancer clinicians to obtain their insights regarding attributes of current approaches for presenting PRO data that are helpful and confusing.6 At the completion of each interview, we asked participants whether they would be interested in partnering with the researchers in developing improved presentation formats in the next phase of the project. These volunteers were organized into work groups that reviewed the findings from the initial round of interviews with the investigative team, provided suggestions regarding candidate formats that could be used to improve presentation approaches, and helped pilot the internet survey.7 In this way, research participants had the opportunity to evolve into research partners, providing critically important input throughout the process.
The implementation and dissemination of findings is another area in which stakeholder partnership is particularly valuable. For example, several of our stakeholder partners have an advocacy background, which can be quite useful for conveying the project’s results in a compelling way. Other stakeholders, such as journal editors, are in a position to act directly to implement the study findings by, for example, adding best practices for presenting PRO data to their journal’s author instructions. Notably, some of the skills stakeholders bring come in addition to their role as stakeholders. For example, one of our patient stakeholders has a background in marketing, and this marketing expertise (completely separate from his patient experience) has helped the research team think about how to present data to broad audiences in a meaningful way.
In summary, this project has implemented stakeholder-driven approaches to address an important barrier to patient-centered cancer care. Several key lessons in stakeholder engagement have emerged from this experience. It is important to identify the key constituencies early on in the process. Involving stakeholders from the start enables them to play important roles in every aspect of the study, starting with study design conception. There are also innovative ways to integrate stakeholders in study conduct, such as our work groups of research participants who volunteered to partner with the research team to develop improved data presentation approaches. Implementation and dissemination is another area where stakeholders, based on their background and connections, can play a critical role. Throughout the process, it is valuable to challenge the project to obtain perspectives from as broad a range of stakeholders as possible. Finally, stakeholders have expertise beyond their stakeholder roles, and these skills can be quite valuable to the overall research agenda. In this project, our partnership with stakeholders has helped improve the presentation of PRO data to patients and providers, thereby improving the patient-centeredness of cancer care.
Acknowledgments
The PRO Data Presentation Stakeholder Advisory Board includes Neil K Aaronson, PhD (Netherlands Cancer Institute, Amsterdam); Patricia A Ganz, MD (University of California-Los Angeles and Jonsson Comprehensive Cancer Center, Los Angeles, CA); Ravin Garg, MD (Anne Arundel Medical Center, Annapolis, MD); Michael Fisch, MD (MD Anderson Cancer Center, Houston, TX); Vanessa Hoffman, MPH (Bladder Cancer Advocacy Network, Washington, DC); Bryce B Reeve, PhD (University of North Carolina at Chapel Hill and Lineberger Comprehensive Cancer Center, Chapel Hill, NC); Eden Stotsky-Himelfarb (Sidney Kimmel Comprehensive Cancer Center at Johns Hopkins, Baltimore, MD); Ellen Stovall (National Coalition for Cancer Survivorship, Washington, DC [posthumous]); Matthew Zachary (Stupid Cancer, New York, NY).
The authors thank The Johns Hopkins Clinical Research Network site investigators and staff and, in particular, the patients and clinicians who participated in this project.
Supported by a Patient-Centered Outcomes Research Institute (PCORI) Award (R-1410-24904). All statements in this report, including its findings and conclusions, are solely those of the authors and do not necessarily represent the views of PCORI, its board of governors or methodology committee. Drs Snyder and Smith are members of the Sidney Kimmel Comprehensive Cancer Center at Johns Hopkins (P30 CA 006973). The funders had no role in the study design; data collection, analysis, or interpretation; writing; or decision to submit.
1. Acquadro C, Berzon R, Dubois D, et al. Incorporating the patient’s perspective into drug development and communication: an ad hoc task force report of the Patient-Reported Outcomes (PRO) Harmonization Group meeting at the Food and Drug Administration, February 16, 2001. Value Health. 2003;6(5):522-531.
2. PROQOLID, the Patient-Reported Outcome and Quality of Life Instruments Database. https://eprovide.mapi-trust.org/. Accessed November 10, 2016.
3. Brundage MD, Snyder CF. Patient-reported outcomes in clinical practice: using standards to break down barriers. Clin Invest. 2012;2(4):343-346.
4. Brundage M, Bass B, Jolie R, et al. A knowledge translation challenge: clinical use of quality of life data from cancer clinical trials. Qual Life Res. 2011;20(7):979-985.
5. Snyder CF, Jensen R, Courtin SO, et al. PatientViewpoint: a website for patient-reported outcomes assessment. Qual Life Res. 2009;18(7):793-800.
6. Brundage M, Smith KC, Little EA, Bantug ET, Snyder CF. PRO Data Presentation Stakeholder Advisory Board. Communicating patient-reported outcome scores using graphic formats: results from a mixed-methods evaluation. Qual Life Res. 2015;24(10):2457-2472.
7. Smith KC, Brundage MD, Tolbert E, Little EA, Bantug ET, Snyder C. PRO Data Presentation Stakeholder Advisory Board. Engaging stakeholders to improve presentation of patient-reported outcomes data in clinical practice. Support Care Cancer. 2016;24(10):4149-4157.
8. Snyder CF, Smith KC, Bantug ET, Tolbert EE, Blackford AL, Brundage MD. PRO Data Presentation Stakeholder Advisory Board. What do these scores mean? Presenting patient-reported outcomes data to patients and clinicians to improve interpretability. Cancer. 2017;123(10):1848-1859.
9. Brundage M, Blackford A, Tolbert E, Smith K, Bantug E, Snyder C. PRO Data Presentation Stakeholder Advisory Board. Presenting comparative study PRO results to clinicians and researchers: beyond the eye of the beholder. Qual Life Res. 2017 Nov 2 [Epub ahead of print].
10. Tolbert E, Snyder C, Bantug E, Blackford A, Brundage M. PRO Data Presentation Stakeholder Advisory Board. Graphing group-level data from research studies for presentation to patients in educational materials and decision aids. Qual Life Res. 2016;25(suppl 1):17.
Recently, researchers have been challenged to design methods that ensure that key constituents are partners in research, and not simply participants. Here we describe some innovative approaches we used to engage stakeholders. The approaches are drawn from a patient-centered outcomes research project, focusing on the graphic display of patient-reported outcomes (PROs) data. PROs represent patients’ perspectives on the impact of health, disease, and treatment, without interpretation by a clinician or anyone else. PROs include, among other things, patients’ assessments of their symptoms, their level of physical and psychosocial functioning, and health-related quality-of-life.1
As a first example of the key role of stakeholders in this project, input from cancer patients and clinicians, drawn from previous research, motivated us to ask whether there might be a “better way” to display PRO data when used to inform clinical practice. Specifically, even though cancer patients and clinicians endorse the importance of PRO data to promote patient-centered care, both groups report challenges using PROs in practice because of difficulty understanding what the PRO scores mean (eg, what is a good score or a bad score?; for individual patients, which scores should clinicians be concerned about?; for clinical trial PROs, what differences in PRO scores between treatments are clinically important?). The challenges in interpreting PRO data result in part from a large number of PRO measures (eg, one database includes more than 1,000 instruments)2 and no standards across PRO measures regarding how they are scored and scaled, or in how the data are presented.3 For example, on some PRO measures, higher scores represent better outcomes; on some PRO measures, lower scores represent better outcomes; and on some PRO measures, whether higher or lower scores represent better outcomes depends on the domain being measured. Further, some measures are scaled 0-100, with the extremes representing the best/worst scores possible, whereas others are normed to, for example, a population average of 50. Because of this variation, a score of 70 can have a completely different meaning depending on the PRO measure (or domain within a measure). As noted above, previous research has documented that this variation limits patients’ and clinicians’ understanding of the PRO scores, creating an important barrier to their use in practice.4-5
To address this stakeholder-driven research question, we undertook a three-part study to identify approaches for PRO data display that can be easily interpreted, regardless of scoring or scaling conventions, with the overall goal of improving patient and clinician understanding and use of PROs in oncology clinical practice. Part 1 of the study identified attributes of graphic displays of PRO data that are helpful and confusing.6 Part 2 involved developing improved PRO data presentation approaches.7 Part 3 evaluated the accuracy-of-interpretation and clarity of the developed approaches.8-10 The methods and findings of the three-part study are reported elsewhere;6-10 here, we describe the various approaches employed to engage stakeholders throughout the project.
As described above, the first reflection of stakeholder input was in the research question we asked. We then sought to identify the key stakeholder groups and ensure that they participated in each stage of the project. The relevant stakeholder groups we identified were: patients and their caregivers; health care providers (eg, oncologists, oncology nurses) who need to understand PRO data for their own consideration and for discussion with patients; and PRO researchers who develop, validate, and apply PRO measures.
Having identified these three key stakeholder groups, we sought to obtain broad representation of their perspectives. For example, we ensured that our investigative team included a cancer survivor, a cancer care provider, and PRO researchers. To supplement the stakeholder input from the investigative team, we formed a nine-member Stakeholder Advisory Board, with multiple representatives from each key constituency. We also aimed to be as broad as possible in the populations sampled for data collection. For example, we extended beyond the Johns Hopkins cancer center to include the Johns Hopkins Clinical Research Network, a consortium of academic and community health systems across the mid-Atlantic United States. Beyond the in-person data collection across the region, our study also included an internet survey of cancer patients/survivors, cancer care providers, and PRO researchers from across the United States and internationally. Taken together, these approaches improve the diversity of our sample and, thereby, the generalizability of our findings.
In addition to obtaining broad perspectives across stakeholder groups, we created genuine partnerships with the stakeholders to inform every aspect of the project. As described above, the study itself was motivated by feedback from cancer patients and clinicians regarding the challenges they experienced when trying to interpret PRO scores, and we therefore ensured that each stakeholder group contributed to the study’s design. Stakeholders also played a critical role in the conduct of the study. For example, in the first part of the study, we conducted one-on-one interviews with 50 cancer patients and 20 cancer clinicians to obtain their insights regarding attributes of current approaches for presenting PRO data that are helpful and confusing.6 At the completion of each interview, we asked participants whether they would be interested in partnering with the researchers in developing improved presentation formats in the next phase of the project. These volunteers were organized into work groups that reviewed the findings from the initial round of interviews with the investigative team, provided suggestions regarding candidate formats that could be used to improve presentation approaches, and helped pilot the internet survey.7 In this way, research participants had the opportunity to evolve into research partners, providing critically important input throughout the process.
The implementation and dissemination of findings is another area in which stakeholder partnership is particularly valuable. For example, several of our stakeholder partners have an advocacy background, which can be quite useful for conveying the project’s results in a compelling way. Other stakeholders, such as journal editors, are in a position to act directly to implement the study findings by, for example, adding best practices for presenting PRO data to their journal’s author instructions. Notably, some of the skills stakeholders bring come in addition to their role as stakeholders. For example, one of our patient stakeholders has a background in marketing, and this marketing expertise (completely separate from his patient experience) has helped the research team think about how to present data to broad audiences in a meaningful way.
In summary, this project has implemented stakeholder-driven approaches to address an important barrier to patient-centered cancer care. Several key lessons in stakeholder engagement have emerged from this experience. It is important to identify the key constituencies early on in the process. Involving stakeholders from the start enables them to play important roles in every aspect of the study, starting with study design conception. There are also innovative ways to integrate stakeholders in study conduct, such as our work groups of research participants who volunteered to partner with the research team to develop improved data presentation approaches. Implementation and dissemination is another area where stakeholders, based on their background and connections, can play a critical role. Throughout the process, it is valuable to challenge the project to obtain perspectives from as broad a range of stakeholders as possible. Finally, stakeholders have expertise beyond their stakeholder roles, and these skills can be quite valuable to the overall research agenda. In this project, our partnership with stakeholders has helped improve the presentation of PRO data to patients and providers, thereby improving the patient-centeredness of cancer care.
Acknowledgments
The PRO Data Presentation Stakeholder Advisory Board includes Neil K Aaronson, PhD (Netherlands Cancer Institute, Amsterdam); Patricia A Ganz, MD (University of California-Los Angeles and Jonsson Comprehensive Cancer Center, Los Angeles, CA); Ravin Garg, MD (Anne Arundel Medical Center, Annapolis, MD); Michael Fisch, MD (MD Anderson Cancer Center, Houston, TX); Vanessa Hoffman, MPH (Bladder Cancer Advocacy Network, Washington, DC); Bryce B Reeve, PhD (University of North Carolina at Chapel Hill and Lineberger Comprehensive Cancer Center, Chapel Hill, NC); Eden Stotsky-Himelfarb (Sidney Kimmel Comprehensive Cancer Center at Johns Hopkins, Baltimore, MD); Ellen Stovall (National Coalition for Cancer Survivorship, Washington, DC [posthumous]); Matthew Zachary (Stupid Cancer, New York, NY).
The authors thank The Johns Hopkins Clinical Research Network site investigators and staff and, in particular, the patients and clinicians who participated in this project.
Supported by a Patient-Centered Outcomes Research Institute (PCORI) Award (R-1410-24904). All statements in this report, including its findings and conclusions, are solely those of the authors and do not necessarily represent the views of PCORI, its board of governors or methodology committee. Drs Snyder and Smith are members of the Sidney Kimmel Comprehensive Cancer Center at Johns Hopkins (P30 CA 006973). The funders had no role in the study design; data collection, analysis, or interpretation; writing; or decision to submit.
Recently, researchers have been challenged to design methods that ensure that key constituents are partners in research, and not simply participants. Here we describe some innovative approaches we used to engage stakeholders. The approaches are drawn from a patient-centered outcomes research project, focusing on the graphic display of patient-reported outcomes (PROs) data. PROs represent patients’ perspectives on the impact of health, disease, and treatment, without interpretation by a clinician or anyone else. PROs include, among other things, patients’ assessments of their symptoms, their level of physical and psychosocial functioning, and health-related quality-of-life.1
As a first example of the key role of stakeholders in this project, input from cancer patients and clinicians, drawn from previous research, motivated us to ask whether there might be a “better way” to display PRO data when used to inform clinical practice. Specifically, even though cancer patients and clinicians endorse the importance of PRO data to promote patient-centered care, both groups report challenges using PROs in practice because of difficulty understanding what the PRO scores mean (eg, what is a good score or a bad score?; for individual patients, which scores should clinicians be concerned about?; for clinical trial PROs, what differences in PRO scores between treatments are clinically important?). The challenges in interpreting PRO data result in part from a large number of PRO measures (eg, one database includes more than 1,000 instruments)2 and no standards across PRO measures regarding how they are scored and scaled, or in how the data are presented.3 For example, on some PRO measures, higher scores represent better outcomes; on some PRO measures, lower scores represent better outcomes; and on some PRO measures, whether higher or lower scores represent better outcomes depends on the domain being measured. Further, some measures are scaled 0-100, with the extremes representing the best/worst scores possible, whereas others are normed to, for example, a population average of 50. Because of this variation, a score of 70 can have a completely different meaning depending on the PRO measure (or domain within a measure). As noted above, previous research has documented that this variation limits patients’ and clinicians’ understanding of the PRO scores, creating an important barrier to their use in practice.4-5
To address this stakeholder-driven research question, we undertook a three-part study to identify approaches for PRO data display that can be easily interpreted, regardless of scoring or scaling conventions, with the overall goal of improving patient and clinician understanding and use of PROs in oncology clinical practice. Part 1 of the study identified attributes of graphic displays of PRO data that are helpful and confusing.6 Part 2 involved developing improved PRO data presentation approaches.7 Part 3 evaluated the accuracy-of-interpretation and clarity of the developed approaches.8-10 The methods and findings of the three-part study are reported elsewhere;6-10 here, we describe the various approaches employed to engage stakeholders throughout the project.
As described above, the first reflection of stakeholder input was in the research question we asked. We then sought to identify the key stakeholder groups and ensure that they participated in each stage of the project. The relevant stakeholder groups we identified were: patients and their caregivers; health care providers (eg, oncologists, oncology nurses) who need to understand PRO data for their own consideration and for discussion with patients; and PRO researchers who develop, validate, and apply PRO measures.
Having identified these three key stakeholder groups, we sought to obtain broad representation of their perspectives. For example, we ensured that our investigative team included a cancer survivor, a cancer care provider, and PRO researchers. To supplement the stakeholder input from the investigative team, we formed a nine-member Stakeholder Advisory Board, with multiple representatives from each key constituency. We also aimed to be as broad as possible in the populations sampled for data collection. For example, we extended beyond the Johns Hopkins cancer center to include the Johns Hopkins Clinical Research Network, a consortium of academic and community health systems across the mid-Atlantic United States. Beyond the in-person data collection across the region, our study also included an internet survey of cancer patients/survivors, cancer care providers, and PRO researchers from across the United States and internationally. Taken together, these approaches improve the diversity of our sample and, thereby, the generalizability of our findings.
In addition to obtaining broad perspectives across stakeholder groups, we created genuine partnerships with the stakeholders to inform every aspect of the project. As described above, the study itself was motivated by feedback from cancer patients and clinicians regarding the challenges they experienced when trying to interpret PRO scores, and we therefore ensured that each stakeholder group contributed to the study’s design. Stakeholders also played a critical role in the conduct of the study. For example, in the first part of the study, we conducted one-on-one interviews with 50 cancer patients and 20 cancer clinicians to obtain their insights regarding attributes of current approaches for presenting PRO data that are helpful and confusing.6 At the completion of each interview, we asked participants whether they would be interested in partnering with the researchers in developing improved presentation formats in the next phase of the project. These volunteers were organized into work groups that reviewed the findings from the initial round of interviews with the investigative team, provided suggestions regarding candidate formats that could be used to improve presentation approaches, and helped pilot the internet survey.7 In this way, research participants had the opportunity to evolve into research partners, providing critically important input throughout the process.
The implementation and dissemination of findings is another area in which stakeholder partnership is particularly valuable. For example, several of our stakeholder partners have an advocacy background, which can be quite useful for conveying the project’s results in a compelling way. Other stakeholders, such as journal editors, are in a position to act directly to implement the study findings by, for example, adding best practices for presenting PRO data to their journal’s author instructions. Notably, some of the skills stakeholders bring come in addition to their role as stakeholders. For example, one of our patient stakeholders has a background in marketing, and this marketing expertise (completely separate from his patient experience) has helped the research team think about how to present data to broad audiences in a meaningful way.
In summary, this project has implemented stakeholder-driven approaches to address an important barrier to patient-centered cancer care. Several key lessons in stakeholder engagement have emerged from this experience. It is important to identify the key constituencies early on in the process. Involving stakeholders from the start enables them to play important roles in every aspect of the study, starting with study design conception. There are also innovative ways to integrate stakeholders in study conduct, such as our work groups of research participants who volunteered to partner with the research team to develop improved data presentation approaches. Implementation and dissemination is another area where stakeholders, based on their background and connections, can play a critical role. Throughout the process, it is valuable to challenge the project to obtain perspectives from as broad a range of stakeholders as possible. Finally, stakeholders have expertise beyond their stakeholder roles, and these skills can be quite valuable to the overall research agenda. In this project, our partnership with stakeholders has helped improve the presentation of PRO data to patients and providers, thereby improving the patient-centeredness of cancer care.
Acknowledgments
The PRO Data Presentation Stakeholder Advisory Board includes Neil K Aaronson, PhD (Netherlands Cancer Institute, Amsterdam); Patricia A Ganz, MD (University of California-Los Angeles and Jonsson Comprehensive Cancer Center, Los Angeles, CA); Ravin Garg, MD (Anne Arundel Medical Center, Annapolis, MD); Michael Fisch, MD (MD Anderson Cancer Center, Houston, TX); Vanessa Hoffman, MPH (Bladder Cancer Advocacy Network, Washington, DC); Bryce B Reeve, PhD (University of North Carolina at Chapel Hill and Lineberger Comprehensive Cancer Center, Chapel Hill, NC); Eden Stotsky-Himelfarb (Sidney Kimmel Comprehensive Cancer Center at Johns Hopkins, Baltimore, MD); Ellen Stovall (National Coalition for Cancer Survivorship, Washington, DC [posthumous]); Matthew Zachary (Stupid Cancer, New York, NY).
The authors thank The Johns Hopkins Clinical Research Network site investigators and staff and, in particular, the patients and clinicians who participated in this project.
Supported by a Patient-Centered Outcomes Research Institute (PCORI) Award (R-1410-24904). All statements in this report, including its findings and conclusions, are solely those of the authors and do not necessarily represent the views of PCORI, its board of governors or methodology committee. Drs Snyder and Smith are members of the Sidney Kimmel Comprehensive Cancer Center at Johns Hopkins (P30 CA 006973). The funders had no role in the study design; data collection, analysis, or interpretation; writing; or decision to submit.
1. Acquadro C, Berzon R, Dubois D, et al. Incorporating the patient’s perspective into drug development and communication: an ad hoc task force report of the Patient-Reported Outcomes (PRO) Harmonization Group meeting at the Food and Drug Administration, February 16, 2001. Value Health. 2003;6(5):522-531.
2. PROQOLID, the Patient-Reported Outcome and Quality of Life Instruments Database. https://eprovide.mapi-trust.org/. Accessed November 10, 2016.
3. Brundage MD, Snyder CF. Patient-reported outcomes in clinical practice: using standards to break down barriers. Clin Invest. 2012;2(4):343-346.
4. Brundage M, Bass B, Jolie R, et al. A knowledge translation challenge: clinical use of quality of life data from cancer clinical trials. Qual Life Res. 2011;20(7):979-985.
5. Snyder CF, Jensen R, Courtin SO, et al. PatientViewpoint: a website for patient-reported outcomes assessment. Qual Life Res. 2009;18(7):793-800.
6. Brundage M, Smith KC, Little EA, Bantug ET, Snyder CF. PRO Data Presentation Stakeholder Advisory Board. Communicating patient-reported outcome scores using graphic formats: results from a mixed-methods evaluation. Qual Life Res. 2015;24(10):2457-2472.
7. Smith KC, Brundage MD, Tolbert E, Little EA, Bantug ET, Snyder C. PRO Data Presentation Stakeholder Advisory Board. Engaging stakeholders to improve presentation of patient-reported outcomes data in clinical practice. Support Care Cancer. 2016;24(10):4149-4157.
8. Snyder CF, Smith KC, Bantug ET, Tolbert EE, Blackford AL, Brundage MD. PRO Data Presentation Stakeholder Advisory Board. What do these scores mean? Presenting patient-reported outcomes data to patients and clinicians to improve interpretability. Cancer. 2017;123(10):1848-1859.
9. Brundage M, Blackford A, Tolbert E, Smith K, Bantug E, Snyder C. PRO Data Presentation Stakeholder Advisory Board. Presenting comparative study PRO results to clinicians and researchers: beyond the eye of the beholder. Qual Life Res. 2017 Nov 2 [Epub ahead of print].
10. Tolbert E, Snyder C, Bantug E, Blackford A, Brundage M. PRO Data Presentation Stakeholder Advisory Board. Graphing group-level data from research studies for presentation to patients in educational materials and decision aids. Qual Life Res. 2016;25(suppl 1):17.
1. Acquadro C, Berzon R, Dubois D, et al. Incorporating the patient’s perspective into drug development and communication: an ad hoc task force report of the Patient-Reported Outcomes (PRO) Harmonization Group meeting at the Food and Drug Administration, February 16, 2001. Value Health. 2003;6(5):522-531.
2. PROQOLID, the Patient-Reported Outcome and Quality of Life Instruments Database. https://eprovide.mapi-trust.org/. Accessed November 10, 2016.
3. Brundage MD, Snyder CF. Patient-reported outcomes in clinical practice: using standards to break down barriers. Clin Invest. 2012;2(4):343-346.
4. Brundage M, Bass B, Jolie R, et al. A knowledge translation challenge: clinical use of quality of life data from cancer clinical trials. Qual Life Res. 2011;20(7):979-985.
5. Snyder CF, Jensen R, Courtin SO, et al. PatientViewpoint: a website for patient-reported outcomes assessment. Qual Life Res. 2009;18(7):793-800.
6. Brundage M, Smith KC, Little EA, Bantug ET, Snyder CF. PRO Data Presentation Stakeholder Advisory Board. Communicating patient-reported outcome scores using graphic formats: results from a mixed-methods evaluation. Qual Life Res. 2015;24(10):2457-2472.
7. Smith KC, Brundage MD, Tolbert E, Little EA, Bantug ET, Snyder C. PRO Data Presentation Stakeholder Advisory Board. Engaging stakeholders to improve presentation of patient-reported outcomes data in clinical practice. Support Care Cancer. 2016;24(10):4149-4157.
8. Snyder CF, Smith KC, Bantug ET, Tolbert EE, Blackford AL, Brundage MD. PRO Data Presentation Stakeholder Advisory Board. What do these scores mean? Presenting patient-reported outcomes data to patients and clinicians to improve interpretability. Cancer. 2017;123(10):1848-1859.
9. Brundage M, Blackford A, Tolbert E, Smith K, Bantug E, Snyder C. PRO Data Presentation Stakeholder Advisory Board. Presenting comparative study PRO results to clinicians and researchers: beyond the eye of the beholder. Qual Life Res. 2017 Nov 2 [Epub ahead of print].
10. Tolbert E, Snyder C, Bantug E, Blackford A, Brundage M. PRO Data Presentation Stakeholder Advisory Board. Graphing group-level data from research studies for presentation to patients in educational materials and decision aids. Qual Life Res. 2016;25(suppl 1):17.