Developing visit usage recommendations for outpatient rehabilitation after total knee arthroplasty: A Delphi study ================================================================================================================== * Jeremy Graber * Laura Churchill * Tamara Struessel * Shane O’Malley * Michael Bade * Jennifer Stevens-Lapsley ## Abstract **Objective** No consensus recommendations exist for rehabilitation visit usage (i.e., visit timing, visit frequency, duration of care) after total knee arthroplasty. In the absence of guiding evidence, rehabilitation visit usage varies considerably in the US, especially in the outpatient setting. We sought to develop expert consensus for visit usage recommendations in outpatient rehabilitation after total knee arthroplasty. **Methods** We used a 2-round Delphi process to develop expert consensus for visit usage recommendations. The Delphi panel consisted of surgeons, physical therapists, and advanced practice providers (N=29) with clinical and research expertise in recovery after total knee arthroplasty. **Results** The panel reached consensus on eight visit frequency recommendations for use in the first three months after total knee arthroplasty. These recommendations are responsive to patients’ time since surgery and their recovery status relative to expected. Twelve additional complementary recommendations were developed to be used with the visit frequency recommendations. **Conclusions** We used the Delphi process to develop expert recommendations for visit usage in outpatient rehabilitation after total knee arthroplasty. We envision these recommendations may help to facilitate a more preference-sensitive approach to postoperative rehabilitation. Keywords * total knee arthroplasty * rehabilitation * utilization * usage * preference-sensitive care * Delphi ## Introduction No consensus recommendations exist for rehabilitation visit usage (i.e., visit timing, visit frequency, care duration) after total knee arthroplasty (TKA). Recent systematic reviews and clinical practice guidelines for post-TKA rehabilitation have provided no specific visit usage recommendations due to lack of supporting evidence—especially in the outpatient setting. 1,2 In the absence of guiding evidence, outpatient rehabilitation visit usage varies considerably in the US. Recent TKA rehabilitation studies have observed standard deviations ranging from 5-23 in the number of outpatient visits patients attend after TKA. 3-7 Warren et al. also found regional differences among Medicare beneficiaries in the average timing and duration of outpatient rehabilitation. 8 This practice variation may undermine the overall quality and cost-effectiveness of outpatient rehabilitation, 9 which is especially relevant given payors’ emphasis on reducing TKA costs. 10,11 Given the lack of available evidence, expert opinion could be useful for developing preliminary visit usage recommendations. In 2014, Westby et al. used the Delphi process to build expert consensus around best practices for TKA rehabilitation. 12 This study produced many key practice recommendations, but it did not develop consensus around visit usage in post-acute rehabilitation. The lack of consensus may have been partially due to (1) the wide scope of included rehabilitation topics, which might have limited the focus on visit usage and (2) the opinion among participants that rehabilitation should be individualized. 12 Individuals do have unique needs, goals, and expectations after TKA, 13,14 which suggests a preference-sensitive approach to visit usage may be ideal. 15,16 Preference-sensitive care occurs when well-informed patients make health care decisions in line with their individual preferences. 16-18 Some patients with TKA may prefer extensive rehabilitation to help them achieve ambitious goals, while others may prefer to recover mostly independently. However, because no evidence currently exists to help patients make informed decisions, visit usage is more likely driven by the capacity of local health care systems. 17-20 This type of supply-sensitive care fosters overutilization and ignores patients’ individual needs and preferences. Therefore, patients and clinicians could benefit from expert recommendations to anchor preference-sensitive decisions for visit usage in outpatient rehabilitation after TKA. In this study, we used the Delphi method to develop expert consensus for visit usage recommendations after TKA. We focused specifically on the optimal visit timing, visit frequency, and duration of outpatient rehabilitation; we hypothesized this focused approach would avoid the challenges experienced in previous studies for developing consensus. Our overall goal was to create evidence that can be used to facilitate preference-sensitive visit usage decisions. ## Methods ### Panelist recruitment We sought to enroll a heterogenous Delphi panel consisting of physical therapists, orthopedic surgeons, and advanced practice providers (e.g., physician assistants) with both clinical and research expertise in TKA recovery. We recruited participants from our own professional networks and through author lists of recently published literature in TKA rehabilitation. We limited our recruitment to individuals based in the United States because practice patterns can vary widely between countries. Individuals were eligible to participate if they had > 5 years of TKA-related experience and met one of the following volume criteria: (1) physical therapist who sees > 10 patients with TKA/year in the outpatient setting, (2) orthopedic surgeon who performs > 50 TKAs per year, or (3) advanced practice provider who sees > 50 patients with TKA/year. Additionally, clinicians from these professions who did not meet the volume criteria were eligible to participate if they had > 5 years of experience conducting and publishing TKA-related research. We aimed to enroll 30 participants with representation from each eligible profession; we chose this sample size as previous Delphi studies have observed stability with as few as 23 participants. 21 After enrollment, we sent participants personalized email reminders for each Delphi round to maximize response rates. ### Recommendation Development Phase Before recruiting panelists, we developed a candidate list of visit usage recommendations using our collective expertise and existing TKA rehabilitation literature. We iteratively piloted these candidate recommendations and all survey materials with members of our research team and broader institutional network with relevant expertise in TKA rehabilitation. We used pilot feedback to revise recommendations, generate ideas for new recommendations, and ensure all survey materials were worded clearly and without bias. Using this process, we developed two categories of recommendations: (1) visit frequency recommendations and (2) complementary recommendations, which were designed to be used with the visit frequency recommendations. None of the individuals involved in the development phase participated on the Delphi panel. ### Visit frequency recommendations We framed the visit frequency recommendations around efficient visit usage, which we defined as the minimum frequency of supervised visits needed to provide adequate care. We used two main strategies to develop these recommendations. First, we anchored each recommendation to a specific timeframe after surgery (i.e., postoperative month 1, 2, or 3). We used this strategy because visit frequency recommendations may depend on postoperative tissue healing. 12 Also, clinicians typically re-evaluate rehabilitation treatment plans on monthly intervals, and patients often discharge from rehabilitation within three months after surgery. 3,7,8 Second, we anchored each recommendation according to patients’ observed recovery status relative to their expected recovery (i.e., patient is demonstrating a slow, typical, or fast recovery relative to expected). We used this strategy because patients are expected to recover differently based on their individual characteristics, 1 and monitoring recovery against an expected value can be a useful decision making strategy. 22 Together, these strategies resulted in the development of nine separate combinations of recovery month + recovery status (e.g., postoperative month 1, recovering slower than expected). For each of these combinations, we asked panelists to rate their level of agreement with six different visit frequency options (0x/month, 1x/month, 2x/month, 1x/week, 2x/week, 3x/week). Overall, panelists considered 54 different visit frequency recommendations (nine combinations of recovery month + recovery status with six frequency options each). See Box 1 for an illustration of the structure used to create these recommendations. Box 1. ### General structure used to create 54 different candidate visit frequency recommendations View this table: [Table1](http://medrxiv.org/content/early/2023/01/27/2022.11.29.22282593/T1) ### Complementary recommendations We designed the complementary recommendations to be used with the visit frequency recommendations. They queried panelists’ opinions on (1) the optimal timing for initiating and stopping outpatient rehabilitation, (2) important range of motion recovery thresholds, (3) the approximate proportion of patients who demonstrate slow, typical, or fast recovery relative to expected, and (4) the safety and effectiveness of telerehabilitation for TKA. #### Delphi Structure We compiled the list of candidate recommendations into surveys for panelists to consider in Round 1 of the Delphi process. Panelists were asked to rate their level of agreement or disagreement with the visit frequency recommendations using a Likert scale ranging from 1 (strongly disagree) to 9 (strongly agree) (see Box 1). We used the RAND UCLA method to define consensus, 23 where recommendations with a median response of > 7/9 and less than 30% of responses in the 1-3 range were considered to have reached consensus. 24,25 Because panelists separately rated six different visit frequency options for each combination of recovery month + recovery status, more than one frequency within a combination could potentially meet our definition of consensus. In this case, we considered the frequency with the higher me an response to have reached consensus. A few of the complementary recommendations were scored using the Likert scale and method described above, but most of them required numeric responses. We did not employ a formal definition of consensus for these numerically scored recommendations because they were meant to be supportive—not definitive. Instead, we calculated the mean response for these recommendations during the final Delphi round. In addition to Likert and numeric responses, the Delphi survey also included open-ended text boxes to record panelists’ comments after each recommendation. Panelists were encouraged to provide the rationale behind their response, their opinions on specific recommendations, and suggestions for new/revised recommendations. We conducted additional rounds as needed to develop consensus. During each subsequent round, we included all previous recommendations that had not reached consensus, and we revised and added new recommendations based on panelist feedback. We also provided panelists with (1) a list of recommendations that previously reached consensus, (2) a comparison between the individual’s response and the group’s response for each recommendation, and (3) a qualitative summary of the group’s comments for recommendations from the previous round. To determine whether additional rounds were needed, we examined the response stability for each recommendation that had not reached consensus. Specifically, we compared the absolute difference in the coefficient of variation (CV, standard deviation / mean) between rounds and considered values < 0.2 to be indicative of stability. 26,27 We also monitored the number and content of comments between rounds; we considered fewer comments with no major change in content as further evidence of response stability. 28 We discontinued the Delphi process once all recommendations had either reached consensus or demonstrated stability in the previous two rounds. See Figure 1 below for a summary of the Delphi round structure. ![Figure 1.](http://medrxiv.org/http://medrxiv.stage.highwire.org/content/medrxiv/early/2023/01/27/2022.11.29.22282593/F1.medium.gif) [Figure 1.](http://medrxiv.org/content/early/2023/01/27/2022.11.29.22282593/F1) Figure 1. Visual summary of study structure All study procedures were approved by the Colorado Multiple Institutional Review Board. We administered all surveys electronically via REDCap electronic data capture tools hosted at the University of Colorado. 29,30 We assured panelists their responses and identities would be anonymous throughout the study. ## Results We invited 49 individuals to participate, and 30 panelists enrolled with 29 completing two Delphi rounds. The panel had a median of 17 years of TKA experience and included individuals from 11 US states and 22 unique zip codes. The panel included representation from the Department of Veterans Affairs (n=6), university settings (n=6), non-profit organizations (n=9), private practice (n=5), and other practice settings (n=3). Twelve out of the 18 participating physical therapists held either a PhD or board-certified specialization. Additional details regarding the panel’s experience and participation by round are available in Figure 2 and Table 1, respectively. We did not conduct additional rounds beyond Round 2 because all recommendations that had not reached consensus met our definition of stability between Round 1 and 2: (1) absolute difference in CV < 0.2, (2) reduced number of comments, (3) no major changes in content identified in comments. ![Figure 2.](http://medrxiv.org/http://medrxiv.stage.highwire.org/content/medrxiv/early/2023/01/27/2022.11.29.22282593/F2.medium.gif) [Figure 2.](http://medrxiv.org/content/early/2023/01/27/2022.11.29.22282593/F2) Figure 2. Flow chart depicting panelist recruitment and participation View this table: [Table 1.](http://medrxiv.org/content/early/2023/01/27/2022.11.29.22282593/T2) Table 1. Summary of panel’s TKA-related experience. All values presented as median (range) unless otherwise noted View this table: [Table 2.](http://medrxiv.org/content/early/2023/01/27/2022.11.29.22282593/T3) Table 2. TKA rehabilitation visit frequency recommendations by patient’s postoperative month and recovery status ### Visit frequency recommendations for outpatient TKA rehabilitation The panel reached consensus in Round 1 for visit frequency recommendations in eight of the nine combinations of recovery month + recovery status. The panel did not reach consensus on a recommendation for patients recovering faster than expected in postoperative month 3 during either round. In both rounds, many panelists commented that a fast-recovering individual’s need for rehabilitation at postoperative month 3 is highly contingent upon their postoperative goals. Other panelists commented that fast-recovering patients should already be discharged by postoperative month 3. All complementary recommendations that used Likert scoring reached consensus. The panel agreed that outpatient rehabilitation should be initiated within 1 week following TKA, and that telerehabilitation is safe and effective for patients demonstrating a typical or fast recovery, but not for patients demonstrating a slow recovery. The panelists also provided stable numeric responses for (1) the optimal timing for discharge from outpatient rehabilitation, (2) the proportion of patients who demonstrate a slow, typical, or fast recovery, and (3) important knee flexion and extension range of motion thresholds. The complementary recommendations are displayed in Table 3 along with the format (Likert or numerical) used to score them. View this table: [Table 3.](http://medrxiv.org/content/early/2023/01/27/2022.11.29.22282593/T4) Table 3. Complementary recommendations designed to be used with visit frequency recommendations ## Discussion The panelists reached consensus on recommendations for the frequency, timing, and duration of outpatient rehabilitation during the first 3 months after TKA. We envision that patients and clinicians can use these expert recommendations as the starting point for preference-sensitive visit usage decisions. Their clinical utility may best be illustrated by example (see Box 2). Box 2. ### Example case of using recommendations to facilitate preference sensitive visit usage Patient A was recently re-evaluated by their physical therapist 8 weeks after their TKA (start of month 3). Patient A is pleased with their recovery because their pain is much improved, and they have met their personal goal of returning to a walking program. Patient A’s physical therapist informs them their recovery has progressed typically thus far. Their physical therapist suggests that most patients are recommended to be seen 1x/week at this point after surgery, and discharge is typically recommended around week 10. Patient A considers this information with respect to their goals and preferences and decides to return for one additional visit at week 10. They would prefer to rehabilitate independently until then because they feel confident with their home exercise routine and have already achieved their primary goal. The case above demonstrates a simplified example of how these recommendations could facilitate preference-sensitive care. Patient A used the best available evidence (expert opinion in this case) to make a decision about visit usage in line with their own preferences. 19 This type of preference-sensitive approach requires supporting evidence, which the results of this study provide, to ensure patients are well informed in their preferences. Conversely, supply sensitive care occurs in the absence of medical theory or evidence, and visit usage is driven primarily by the capacity of the local healthcare system. 17-20 Current post-TKA rehabilitation practice exemplifies supply-sensitive care, where visit usage varies considerably by clinician, facility, and region in the absence of guiding evidence. 3-8 Shifting towards a more preference-sensitive approach to rehabilitation could meaningfully improve patient outcomes; patients with knee osteoarthritis who make informed, preference-sensitive care decisions have reported higher quality of life, function, and satisfaction compared to patients who do not. 31-33 It may also reduce overall costs, 17 which could help ensure that outpatient rehabilitation remains a valued and reimbursable service after TKA. The clinician’s role in facilitating preference-sensitive care using this study’s recommendations—or any medical evidence—should not be overlooked. To facilitate preference-sensitive care, clinicians must engage patients in the decision-making process, 17 attempt to maintain equipoise, 16 and avoid rigidly applying evidence. 34 Clinicians may face barriers to consistently practicing this way in outpatient rehabilitation after TKA. 15 Therefore, the recommendations from this study may be most effective when combined with clinician training35 or patient-facing support programs36,37 to facilitate preference-sensitive decisions for visit usage. Clinicians must also use their clinical judgement when using the recommendations from this study. They need to determine whether a patient is demonstrating a slow, typical, or fast recovery relative to expected; the complementary recommendations were designed to aid in this determination. For example, a clinician may determine a patient’s recovery to be slower than expected if they are not on pace to achieve an important range of motion threshold (e.g., around 114 degrees of knee flexion based on this study’s results). Additional examples of how the complementary recommendations can be applied with the visit frequency recommendations are provided in Table 3. However, clinicians must consider numerous additional factors when assessing an individual’s recovery such as their preoperative prognosis, 1 pain, wound healing status, 12 and individual goals/expectations. 13,14 Clinicians and patients should work collaboratively to assess recovery and consider using these recommendations as a starting point for visit usage decisions. Clinicians should also ask patients about external factors that may influence their preferences such as familial support, access/transportation to rehabilitation, and insurance coverage. 12,15 Clinicians regularly make recommendations in consideration of these interplaying factors, but they should discuss their rationale with patients to facilitate preference-sensitive decisions. The panelists in this study did not reach consensus on a visit frequency recommendation for individuals recovering faster than expected in postoperative month 3. It appeared unlikely that additional rounds would lead to consensus because panelists’ responses and comments were stable between rounds. The panelists appeared to be split among two groups based on their comments. One group felt these patients should already be discharged from rehabilitation, while the other group felt the recommended rehabilitation frequency for these patients should depend on the ambitiousness of their goals (which would be consistent with preference-sensitive care). Although no consensus was reached for a specific visit frequency, the complementary recommendations suggest that clinicians should consider discussing discharge with fast-recovering patients around 8 weeks after surgery. This study does have a few limitations. Perhaps most notably, patients with TKA were not included on the Delphi panel. We chose not to include patients because we felt it was important for panelists to have experience with a wide range of cases given the heterogeneity of the TKA population. We also envisioned that an individual patient’s input on visit usage would be most valuable when applied to their own care (i.e., preference-sensitive care). Regardless, future work should examine patients’ perceptions of the acceptability and usefulness of the recommendations developed in this study. This study also has notable strengths. We enrolled an experienced panel with considerable diversity in terms of profession, geography, and practice setting. To strengthen the validity of our findings, we used an iterative development phase, established predefined criteria for defining consensus and for stopping the Delphi process, provided participants with both quantitative and qualitative feedback between rounds, and incorporated panelist feedback into subsequent rounds. 38,39 ## Conclusion We used the Delphi method to develop recommendations foroutpatient rehabilitation usage after TKA. These recommendations can be used by clinicians and patients to facilitate a preference-sensitive approach to rehabilitation usage, which may improve the quality and efficiency of care. ## Data Availability Data produced in the present study may be available upon reasonable request to the authors. ## Footnotes * The manuscript underwent minor edits in response to peer review. The manuscript title has also been slightly altered. ## Abbreviations (TKA) : total knee arthroplasty (CV) : coefficient of variation * Received November 29, 2022. * Revision received January 27, 2023. * Accepted January 27, 2023. * © 2023, Posted by Cold Spring Harbor Laboratory This pre-print is available under a Creative Commons License (Attribution-NonCommercial-NoDerivs 4.0 International), CC BY-NC-ND 4.0, as described at [http://creativecommons.org/licenses/by-nc-nd/4.0/](http://creativecommons.org/licenses/by-nc-nd/4.0/) ## References 1. 1.Jette DU, Hunter SJ, Burkett L, et al. Physical Therapist Management of Total Knee Arthroplasty. Phys Ther. 2020;100(9):1603–1631. doi:10.1093/ptj/pzaa099. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1093/ptj/pzaa099&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F01%2F27%2F2022.11.29.22282593.atom) 2. 2.Konnyu KJ, Thoma LM, Bhuma MR, et al. Prehabilitation and Rehabilitation for Major Joint Replacement. Rockville (MD): Agency for Healthcare Research and Quality (US); November 2021. 3. 3.Oatis CA, Li W, DiRusso JM, et al. Variations in Delivery and Exercise Content of Physical Therapy Rehabilitation Following Total Knee Replacement Surgery: A Cross-Sectional Observation Study. Int J Phys Med Rehabil. 2014;Suppl 5:002. doi:10.4172/2329-9096.S5-002 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.4172/2329-9096.S5-002&link_type=DOI) 4. 4.Oatis CA, Johnson JK, DeWan T, Donahue K, Li W, Franklin PD. Characteristics of Usual Physical Therapy Post-Total Knee Replacement and Their Associations With Functional Outcomes. Arthritis Care Res (Hoboken). 2019;71(9):1171–1177. doi:10.1002/acr.23761 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1002/acr.23761&link_type=DOI) 5. 5.Mitchell JM, Reschovsky JD, Reicherter EA. Use of Physical Therapy Following Total Knee Replacement Surgery: Implications of Orthopedic Surgeons’ Ownership of Physical Therapy Services. Health Serv Res. 2016;51(5):1838–1857. doi:10.1111/1475-6773.12465 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1111/1475-6773.12465&link_type=DOI) 6. 6.Hung A, Li Y, Keefe FJ, et al. Ninety-day and one-year healthcare utilization and costs after knee arthroplasty. Osteoarthritis Cartilage. 2019;27(10):1462–1469. doi:10.1016/j.joca.2019.05.019 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.joca.2019.05.019&link_type=DOI) 7. 7.Brennan GP, Fritz JM, Houck LT, Hunter SJ. Outpatient rehabilitation care process factors and clinical outcomes among patients discharged home following unilateral total knee arthroplasty. J Arthroplasty. 2015;30(5):885–890. doi:10.1016/j.arth.2014.12.013 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.arth.2014.12.013&link_type=DOI) 8. 8.Warren M, Shireman TI. Geographic Variability in Discharge Setting and Outpatient Postacute Physical Therapy After Total Knee Arthroplasty: A Retrospective Cohort Study. Phys Ther. 2018;98(10):855–864. doi:10.1093/ptj/pzy077 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1093/ptj/pzy077&link_type=DOI) 9. 9.Fatoye F, Yeowell G, Wright JM, Gebrye T. Clinical and cost-effectiveness of physiotherapy interventions following total knee replacement: a systematic review and meta-analysis. Arch Orthop Trauma Surg. 2021;141(10):1761–1778. doi:10.1007/s00402-021-03784-5 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1007/s00402-021-03784-5&link_type=DOI) 10. 10.Centers for Medicare and Medicaid Services. Medicare Program: Comprehensive Care for Joint Replacement Model Three-Year Extension and Changes Episode Definition and Pricing; Medicare and Medicaid Programs; Policies and Regulatory Revisions in Response to the COVID-19 Public Health Emergency. Updated July 9, 2021. Accessed January 27, 2023. [https://www.govinfo.gov/content/pkg/FR-2021-05-03/pdf/2021-09097.pdf](https://www.govinfo.gov/content/pkg/FR-2021-05-03/pdf/2021-09097.pdf). Accessed January 27, 2023. 11. 11.Siddiqi A, White PB, Mistry JB, et al. Effect of Bundled Payments and Health Care Reform as Alternative Payment Models in Total Joint Arthroplasty: A Clinical Review. J Arthroplasty. 2017;32(8):2590–2597. doi:10.1016/j.arth.2017.03.027 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.arth.2017.03.027&link_type=DOI) 12. 12.Westby MD, Brittain A, Backman CL. Expert consensus on best practices for post-acute rehabilitation after total hip and knee arthroplasty: a Canada and United States Delphi study. Arthritis Care Res (Hoboken). 2014;66(3):411–423. doi:10.1002/acr.22164 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1002/acr.22164&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=24023047&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F01%2F27%2F2022.11.29.22282593.atom) 13. 13.Weiss JM, Noble PC, Conditt MA, et al. What functional activities are important to patients with knee replacements?. Clin Orthop Relat Res. 2002;(404):172–188. doi:10.1097/00003086-200211000-00030 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1097/00003086-200211000-00030&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=12439258&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F01%2F27%2F2022.11.29.22282593.atom) 14. 14.Tilbury C, Haanstra TM, Leichtenberg CS, et al. Unfulfilled Expectations After Total Hip and Knee Arthroplasty Surgery: There Is a Need for Better Preoperative Patient Information and Education. J Arthroplasty. 2016;31(10):2139–2145. doi:10.1016/j.arth.2016.02.061 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.arth.2016.02.061&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F01%2F27%2F2022.11.29.22282593.atom) 15. 15.Graber J, Lockhart S, Matlock DD, Stevens-Lapsley J, Kittelson AJ. “ This is not negotiable. You need to do this…” : A directed content analysis of decision making in rehabilitation after knee arthroplasty. J Eval Clin Pract. 2022;28(1):99–107. doi:10.1111/jep.13591 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1111/jep.13591&link_type=DOI) 16. 16.Elwyn G, Frosch D, Rollnick S. Dual equipoise shared decision making: definiti ons for decision and behaviour support interventions. Implement Sci. 2009;4:75. Published 2009 Nov 18. doi:10.1186/1748-5908-4-75 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1186/1748-5908-4-75&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=19922647&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F01%2F27%2F2022.11.29.22282593.atom) 17. 17.Wennberg JE, Fisher ES, Skinner JS. Geography and the debate over medicare reform. Health Affairs. 2004:W96. 18. 18.Wennberg JE. Unwarranted variations in healthcare delivery: implications for academic medical centres. BMJ. 2002;325(7370):961–964. doi:10.1136/bmj.325.7370.961 [FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMjUvNzM3MC85NjEiO3M6NDoiYXRvbSI7czo1MDoiL21lZHJ4aXYvZWFybHkvMjAyMy8wMS8yNy8yMDIyLjExLjI5LjIyMjgyNTkzLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 19. 19.Fisher ES, Wennberg JE. Health care quality, geographic variations, and the challenge of supply-sensitive care. Perspect Biol Med. 2003;46(1):69–79. doi:10.1353/pbm.2003.0004 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1353/pbm.2003.0004&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=12582271&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F01%2F27%2F2022.11.29.22282593.atom) 20. 20.Wennberg JE, Brownlee S, Fisher ES, Skinner JS, Weinstein JN. An Agenda for Change: Improving Quality and Curbing Health Care Spending: Opportunities for the Congress and the Obama Administration. Lebanon (NH): The Dartmouth Institute for Health Policy and Clinical Practice; December 1, 2008. 21. 21.Akins RB, Tolson H, Cole BR. Stability of response characteristics of a Delphi panel: application of bootstrap data expansion. BMC Med Res Methodol. 2005;5:37. Published 2005 Dec 1. doi:10.1186/1471-2288-5-37 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1186/1471-2288-5-37&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=16321161&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F01%2F27%2F2022.11.29.22282593.atom) 22. 22.Glasziou PP, Irwig L, Aronson JK, editors. Evidence-based medical monitoring: from principles to practice. John Wiley & Sons; 2008 Apr 30. 23. 23.Brook RH. The RAND/UCLA appropriateness method. RAND; 1995. 24. 24.Campbell SM, Cantrill JA, Roberts D. Prescribing indicators for UK general practice: Delphi consultation study. BMJ. 2000;321(7258):425–428. doi:10.1136/bmj.321.7258.425 [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMjEvNzI1OC80MjUiO3M6NDoiYXRvbSI7czo1MDoiL21lZHJ4aXYvZWFybHkvMjAyMy8wMS8yNy8yMDIyLjExLjI5LjIyMjgyNTkzLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 25. 25.Brook RH, Chassin MR, Fink A, Solomon DH, Kosecoff J, Park RE. A method for the detailed assessment of the appropriateness of medical technologies. Int J Technol Assess Health Care. 1986;2(1):53–63. doi:10.1017/s0266462300002774 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1017/S0266462300002774&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=10300718&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F01%2F27%2F2022.11.29.22282593.atom) 26. 26.Kalaian SA, Kasim RM. Terminating sequential Delphi survey data collection. Practical assessment, research & evaluation. 2012 Jan;17(5):n5. 27. 27.Shah HA, Kalaian SA. Which is the best parametric statistical method for analyzing Delphi data?. Journal of Modern Applied Statistical Methods. 2009;8(1):20. 28. 28.Holey EA, Feeley JL, Dixon J, Whittaker VJ. An exploration of the use of simple statistics to measure consensus and stability in Delphi studies. BMC Med Res Methodol. 2007;7:52. Published 2007 Nov 29. doi:10.1186/1471-2288-7-52 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1186/1471-2288-7-52&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=18045508&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F01%2F27%2F2022.11.29.22282593.atom) 29. 29.Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42(2):377–381. doi:10.1016/j.jbi.2008.08.010 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.jbi.2008.08.010&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=18929686&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F01%2F27%2F2022.11.29.22282593.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000264958800018&link_type=ISI) 30. 30.Harris PA, Taylor R, Minor BL, et al. The REDCap consortium: Building an international community of software platform partners. J Biomed Inform. 2019;95:103208. doi:10.1016/j.jbi.2019.103208 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.jbi.2019.103208&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=31078660&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F01%2F27%2F2022.11.29.22282593.atom) 31. 31.Sepucha KR, Atlas SJ, Chang Y, et al. Informed, Patient-Centered Decisions Associated with Better Health Outcomes in Orthopedics: Prospective Cohort Study. Med Decis Making. 2018;38(8):1018–1026. doi:10.1177/0272989X18801308 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1177/0272989X18801308&link_type=DOI) 32. 32.Sepucha KR, Vo H, Chang Y, et al. Shared Decision-Making Is Associated with Better Outcomes in Patients with Knee But Not Hip Osteoarthritis: The DECIDE-OA Randomized Study. J Bone Joint Surg Am. 2022;104(1):62–69. doi:10.2106/JBJS.21.00064 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.2106/JBJS.21.00064&link_type=DOI) 33. 33.Chrenka EA, Solberg LI, Asche SE, et al. Is Shared Decision-making Associated with Better Patient-reported Outcomes? A Longitudinal Study of Patients Undergoing Total Joint Arthroplasty. Clin Orthop Relat Res. 2022;480(1):82–91. doi:10.1097/CORR.0000000000001920 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1097/CORR.0000000000001920&link_type=DOI) 34. 34.Guyatt GH, Oxman AD, Vist GE, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336(7650):924–926. doi:10.1136/bmj.39489.470347.AD [FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMzYvNzY1MC85MjQiO3M6NDoiYXRvbSI7czo1MDoiL21lZHJ4aXYvZWFybHkvMjAyMy8wMS8yNy8yMDIyLjExLjI5LjIyMjgyNTkzLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 35. 35.Lawford BJ, Delany C, Bennell KL, Bills C, Gale J, Hinman RS. Training Physical Therapists in Person-Centered Practice for People With Osteoarthritis: A Qualitative Case Study. Arthritis Care Res (Hoboken). 2018;70(4):558–570. doi:10.1002/acr.23314 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1002/acr.23314&link_type=DOI) 36. 36.Veroff D, Marr A, Wennberg DE. Enhanced support for shared decision making reduced costs of care for patients with preference-sensitive conditions. Health Aff (Millwood). 2013;32(2):285–293. doi:10.1377/hlthaff.2011.0941 [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6OToiaGVhbHRoYWZmIjtzOjU6InJlc2lkIjtzOjg6IjMyLzIvMjg1IjtzOjQ6ImF0b20iO3M6NTA6Ii9tZWRyeGl2L2Vhcmx5LzIwMjMvMDEvMjcvMjAyMi4xMS4yOS4yMjI4MjU5My5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 37. 37.Wennberg DE, Marr A, Lang L, O’Malley S, Bennett G. A randomized trial of a telephone care-management strategy. N Engl J Med. 2010;363(13):1245–1255. doi:10.1056/NEJMsa0902321 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1056/NEJMsa0902321&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=20860506&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F01%2F27%2F2022.11.29.22282593.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000282050000008&link_type=ISI) 38. 38.Boulkedid R, Abdoul H, Loustau M, Sibony O, Alberti C. Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review. PLoS One. 2011;6(6):e20476. doi:10.1371/journal.pone.0020476 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1371/journal.pone.0020476&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=21694759&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F01%2F27%2F2022.11.29.22282593.atom) 39. 39.Jünger S, Payne SA, Brine J, Radbruch L, Brearley SG. Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: Recommendations based on a methodological systematic review. Palliat Med. 2017;31(8):684–706. doi:10.1177/0269216317690685 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1177/0269216317690685&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F01%2F27%2F2022.11.29.22282593.atom)