User login
Barriers to Implementation of Telehealth Pre-anesthesia Evaluation Visits in the Department of Veterans Affairs
Days or weeks before a scheduled surgical or invasive procedure involving anesthesia, evaluations are conducted to assess a patient’s condition and risk, optimize their status, and prepare them for their procedure. A comprehensive pre-anesthesia evaluation visit includes a history of present illness, the evaluation of comorbidities and medication use, the assessment of health habits such as alcohol and tobacco use, functional capacity and nutritional assessments, and the identification of social support deficiencies that may influence recovery. It also includes a focused physical examination and laboratory and other ancillary testing as needed and may include optimization interventions such as anemia management or prehabilitation. Conducting pre-anesthesia evaluations before surgery has been shown to reduce delays and cancellations, unnecessary preprocedure testing, hospital length of stay, and in-hospital mortality.1-4
The pre-anesthesia evaluation is usually conducted in person, although other modalities have been in use for several years and have accelerated since the advent of the COVID-19 pandemic. Specifically, audio-only telephone visits are used in many settings to conduct abbreviated forms of a pre-anesthesia evaluation, typically for less-invasive procedures. When patients are evaluated over the telephone, the physical examination and testing are deferred until the day of the procedure. Another modality is the use of synchronous video telehealth. Emerging evidence for the use of video-based care in anesthesiology provides encouraging results. Several institutions have proven the technological feasibility of performing preoperative evaluations via video.5,6 Compared with in-person evaluations, these visits seem to have similar surgery cancellation rates, improved patient satisfaction, and reduced wait times and costs.7-9
As part of a quality improvement project, we studied the use of telehealth for pre-anesthesia evaluations within the US Department of Veterans Affairs (VA). An internal review found overall low utilization of these modalities before the COVID-19 pandemic that accelerated toward telehealth during the pandemic: The largest uptake was with telephone visits. Given the increasing adoption of telehealth for pre-anesthesia evaluations and the marked preference for telephone over video modalities among VA practitioners during the COVID-19 pandemic, we sought to understand the barriers and facilitators to the adoption of telephone- and video-based pre-anesthesia evaluation visits within the VA.
Methods
Our objective was to assess health care practitioners’ (HCPs) preferences regarding pre-anesthesia evaluation modalities (in-person, telephone, or video), and the perceived advantages and barriers to adoption for each modality. We followed the Strengthening the Reporting of Observational studies in Epidemiology (STROBE) guideline and Checklist for statistical Assessment of Medical Papers (CHAMP) statement.10,11 The survey was deemed a quality improvement activity that was exempt from institutional review board oversight by the VA National Anesthesia Program Office and the VA Office of Connected Care.
A survey was distributed to all VA anesthesiology service chiefs via email between April 27, 2022, and May 3, 2022. Three emails were sent to each participant (initial invitation and 2 reminders). The respondents were asked to identify themselves by facility and role and to indicate whether their anesthesiology service performed any pre-anesthesia evaluations, including any telephone- or video-based evaluations; and whether their service has a dedicated pre-anesthesia evaluation clinic.
A second set of questions referred to the use of telephone- and video-based preprocedure evaluations. The questions were based on branch logic and depended on the respondent’s answers concerning their use of telephone- and video-based evaluations. Questions included statements about perceived barriers to the adoption of these pre-anesthesia evaluation modalities. Each item was rated on a 5-point Likert scale, (completely disagree [1] to completely agree [5]). A third section measured acceptability and feasibility of video using the validated Acceptability of Intervention Measure (AIM) and Feasibility of Intervention Measure (FIM)questionnaires.12 These instruments are 4-item measures of implementation outcomes that are often considered indicators of implementation success.13Acceptability is the perception among implementation stakeholders that a given treatment, service, practice, or innovation is agreeable, palatable, or satisfactory. Feasibility is defined as the extent to which a new treatment or an innovation can be successfully used or carried out within a given agency or setting.13 The criterion for acceptability is personal, meaning that different HCPs may have differing needs, preferences, and expectations regarding the same intervention. The criterion for feasibility is practical. An intervention may be considered to be feasible if the required tasks can be performed easily or conveniently. Finally, 2 open-ended questions allowed respondents to identify the most important factor that allowed the implementation of telehealth for pre-anesthesia evaluations in their service, and provide comments about the use of telehealth for pre-anesthesia evaluations at the VA. All questions were developed by the authors except for the 2 implementation measure instruments.
The survey was administered using an electronic survey platform (Qualtrics, version April 2022) and sent by email alongside a brief introductory video. Participation was voluntary and anonymous, as no personal information was collected. Responses were attributed to each facility, using the self-declared affiliation. When an affiliation was not provided, we deduced it using the latitude/longitude of the respondent, a feature included in the survey software. No incentives were provided. Data were stored and maintained in a secure VA server. All completed surveys were included. Some facilities had > 1 complete response, and all were included. Facilities that provided > 1 response and where responses were discordant, we clarified with the facility service chief. Incomplete responses were excluded from the analysis.
Statistics
For this analysis, the 2 positive sentiment responses (agree and completely agree) and the 2 negative sentiment responses (disagree and completely disagree) in the Likert scale were collapsed into single categories (good and poor, respectively). The neither agree nor disagree responses were coded as neutral. Our analysis began with a visual exploration of all variables to evaluate the frequency, percentage, and near-zero variance for categorical variables.14 Near-zero variance occurs when a categorical variable has a low frequency of unique values over the sample size (ie, the variable is almost constant), and we addressed it by combining different variable categorizations. We handled missing values through imputation algorithms followed by sensitivity analyses to verify whether our results were stable with and without imputation. We performed comparisons for the exploratory analysis using P values for one-way analysis of variance tests for numeric variables and χ2tests for categorical variables. We considered P values < .05 to be statistically significant. We also used correlation matrices and plots as exploratory analysis tools to better understand all items’ correlations. We used Pearson, polychoric, and polyserial correlation tests as appropriate for numeric, ordinal, and logical items.
Our modeling strategy involved a series of generalized linear models (GLMs) with a Gaussian family, ie, multiple linear regression models, to assess the association between (1) facilities’ preferences regarding pre-anesthesia evaluation modalities; (2) advantages between modalities; and (3) barriers to the adoption of telehealth and the ability to perform different pre-anesthesia evaluation-related tasks. In addition, we used backward deletion to reach the most parsimonious model based on a series of likelihood-ratio tests comparing nested models. Results are reported as predicted means with 95% confidence intervals, with results being interpreted as significant when any 2 predicted means do not overlap between different estimates along with P for trends < .001. We performed all analyses using the R language.15
Results
Of 109 surveyed facilities, 50 (46%) responded to the survey. The final study sample included 67 responses, and 55 were included in the analysis. Twelve responses were excluded from the analysis as they were either incomplete or test responses. Three facilities had > 1 complete response (2 facilities had 2 responses and 1 facility had 4 responses), and these were all included in the analysis.
Thirty-six locations were complex inpatient facilities, and 32 (89%) had pre-anesthesia evaluation clinics (Table 1).
The ability to obtain a history of present illness was rated good/very good via telephone for 34 respondents (92%) and 25 for video (86%). Assessing comorbidities and health habits was rated good/very good via telephone for 32 respondents (89%) and 31 respondents (86%), respectively, and via video for 24 respondents (83%) and 23 respondents (79%), respectively (Figure 1).
To compare differences between the 2 remote pre-anesthesia evaluation modalities, we created GLMs evaluating the association between each modality and the perceived ability to perform the tasks. For GLMs, we transformed the values of the categories into numerical (ie, 1, poor; 2, neutral; 3, good). Compared with telephone, video was rated more favorably regarding the assessment of nutritional status (mean, 2.1; 95% CI, 1.8-2.3 vs mean, 2.4; 95% CI, 2.2-2.7; P = .04) (eAppendix 1, available at doi:10.12788/fp.0387). No other significant differences in ratings existed between the 2 remote pre-anesthesia evaluation modalities.
The most significant barriers (cited as significant or very significant in the survey) included the inability to perform a physical examination, which was noted by 13 respondents (72%) and 15 respondents (60%) for telephone and video, respectively. The inability to obtain vital signs was rated as a significant barrier for telephone by 12 respondents (67%) and for video by 15 respondents (60%)(Figure 2).
The average FIM score was 3.7, with the highest score among respondents who used both phone and video (Table 2). The average AIM score was 3.4, with the highest score among respondents who used both telehealth modalities. The internal consistency of the implementation measures was excellent (Cronbach’s α 0.95 and 0.975 for FIM and AIM, respectively).
Discussion
We surveyed 109 anesthesiology services across the VA regarding barriers to implementing telephone- and video-based pre-anesthesia evaluation visits. We found that 12 (23%) of the 50 anesthesiology services responding to this survey still conduct the totality of their pre-anesthesia evaluations in person. This represents an opportunity to further disseminate the appropriate use of telehealth and potentially reduce travel time, costs, and low-value testing, as it is well established that remote pre-anesthesia evaluations for low-risk procedures are safe and effective.6
We also found no difference between telephone and video regarding users’ perceived ability to perform any of the basic pre-anesthesia evaluation tasks except for assessing patients’ nutritional status, which was rated as easier using video than telephone. According to those not using telephone and/or video, the biggest barriers to implementation of telehealth visits were the inability to obtain vital signs and to perform a physical examination. This finding was unexpected, as facilities that conduct remote evaluations typically defer these tasks to the day of surgery, a practice that has been well established and shown to be safe and efficient. Respondents also identified patient-level factors (eg, patient preference, lack of telephone or computer) as significant barriers. Finally, feasibility ratings were higher than acceptability ratings with regards to the implementation of telehealth.
In 2004, the first use of telehealth for pre-anesthesia evaluations was reported by Wong and colleagues.16 Since then, several case series and a literature review have documented the efficacy, safety, and patient and HCP satisfaction with the use of telehealth for pre-anesthesia evaluations. A study by Mullen-Fortino and colleagues showed reduced visit times when telehealth was used for pre-anesthesia evaluation.8 Another study at VA hospitals showed that 88% of veterans reported that telemedicine saved them time and money.17 A report of 35 patients in rural Australia reported 98% satisfaction with the video quality of the visit, 95% perceived efficacy, and 87% preference for telehealth compared with driving to be seen in person.18 These reports conflict with the perceptions of the respondents of our survey, who identified patient preference as an important barrier to adoption of telehealth. Given these findings, research is needed on veterans’ perceptions on the use of telehealth modalities for pre-anesthesia evaluations; if their perceptions are similarly favorable, it will be important to communicate this information to HCPs and leadership, which may help increase subsequent telehealth adoption.
Despite the reported safety, efficacy, and high satisfaction of video visits among anesthesiology teams conducting pre-anesthesia evaluations, its use remains low at VA. We have found that most facilities in the VA system chose telephone platforms during the COVID-19 pandemic. One possibility is that the adoption of video modalities among pre-anesthesia evaluation clinics in the VA system is resource intensive or difficult from the HCP’s perspective. When combined with the lack of perceived advantages over telephone as we found in our survey, most practitioners resort to the technologically less demanding and more familiar telephone platform. The results from FIM and AIM support this. While both telephone and video have high feasibility scores, acceptability scores are lower for video, even among those currently using this technology. Our findings do not rule out the utility of video-based care in perioperative medicine. Rather than a yes/no proposition, future studies need to establish the precise indications for video for pre-anesthesia evaluations; that is, situations where video visits offer an advantage over telephone. For example, video could be used to deliver preoperative optimization therapies, such as supervised exercise or mental health interventions or to guide the achievement of certain milestones before surgery in patients with chronic conditions, such as target glucose values or the treatment of anemia. Future studies should explore the perceived benefits of video over telephone among centers offering these more advanced optimization interventions.
Limitations
We received responses from a subset of VA anesthesiology services; therefore, they may not be representative of the entire VA system. Facilities designated by the VA as inpatient complex were overrepresented (72% of our sample vs 50% of the total facilities nationally), and ambulatory centers (those designed by the VA as ambulatory procedural center with basic or advanced capabilities) were underrepresented (2% of our sample vs 22% nationally). Despite this, the response rate was high, and no geographic area appeared to be underrepresented. In addition, we surveyed pre-anesthesia evaluation facilities led by anesthesiologists, and the results may not be representative of the preferences of HCPs working in nonanesthesiology led pre-anesthesia evaluation clinics. Finally, just 11 facilities used both telephone and video; therefore, a true direct comparison between these 2 platforms was limited. The VA serves a unique patient population, and the findings may not be completely applicable to the non-VA population.
Conclusions
We found no significant perceived advantages of video over telephone in the ability to conduct routine pre-anesthesia evaluations among a sample of anesthesiology HCPs in the VA except for the perceived ability to assess nutritional status. HCPs with no telehealth experience cited the inability to perform a physical examination and obtain vital signs as the most significant barriers to implementation. Respondents not using telephone cited concerns about safety. Video visits in this clinical setting had additional perceived barriers to implementation, such as lack of information technology and staff support and patient-level barriers. Video had lower acceptability by HCPs. Given findings that pre-anesthesia evaluations can be conducted effectively via telehealth and have high levels of patient satisfaction, future work should focus on increasing uptake of these remote modalities. Additionally, research on the most appropriate uses of video visits within perioperative care is also needed.
1. Starsnic MA, Guarnieri DM, Norris MC. Efficacy and financial benefit of an anesthesiologist-directed university preadmission evaluation center. J Clin Anesth. 1997;9(4):299-305. doi:10.1016/s0952-8180(97)00007-x
2. Kristoffersen EW, Opsal A, Tveit TO, Berg RC, Fossum M. Effectiveness of pre-anaesthetic assessment clinic: a systematic review of randomised and non-randomised prospective controlled studies. BMJ Open. 2022;12(5):e054206. doi:10.1136/bmjopen-2021-054206
3. Ferschl MB, Tung A, Sweitzer B, Huo D, Glick DB. Preoperative clinic visits reduce operating room cancellations and delays. Anesthesiology. 2005;103(4):855-9. doi:10.1097/00000542-200510000-00025
4. Blitz JD, Kendale SM, Jain SK, Cuff GE, Kim JT, Rosenberg AD. preoperative evaluation clinic visit is associated with decreased risk of in-hospital postoperative mortality. Anesthesiology. 2016;125(2):280-294. doi:10.1097/ALN.0000000000001193
5. Dilisio RP, Dilisio AJ, Weiner MM. Preoperative virtual screening examination of the airway. J Clin Anesth. 2014;26(4):315-317. doi:10.1016/j.jclinane.2013.12.010
6. Kamdar NV, Huverserian A, Jalilian L, et al. Development, implementation, and evaluation of a telemedicine preoperative evaluation initiative at a major academic medical center. Anesth Analg. 2020;131(6):1647-1656. doi:10.1213/ANE.0000000000005208
7. Azizad O, Joshi GP. Telemedicine for preanesthesia evaluation: review of current literature and recommendations for future implementation. Curr Opin Anaesthesiol. 2021;34(6):672-677. doi:10.1097/ACO.0000000000001064
8. Mullen-Fortino M, Rising KL, Duckworth J, Gwynn V, Sites FD, Hollander JE. Presurgical assessment using telemedicine technology: impact on efficiency, effectiveness, and patient experience of care. Telemed J E Health. 2019;25(2):137-142. doi:10.1089/tmj.2017.0133
9. Zhang K, Rashid-Kolvear M, Waseem R, Englesakis M, Chung F. Virtual preoperative assessment in surgical patients: a systematic review and meta-analysis. J Clin Anesth. 2021;75:110540. doi:10.1016/j.jclinane.2021.110540
10. Mansournia MA, Collins GS, Nielsen RO, et al. A CHecklist for statistical Assessment of Medical Papers (the CHAMP statement): explanation and elaboration. Br J Sports Med. 2021;55(18):1009-1017. doi:10.1136/bjsports-2020-103652
11. von Elm E, Altman DG, Egger M, et al. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: guidelines for reporting observational studies. Int J Surg. 2014;12(12):1495-1499. doi:10.1016/j.ijsu.2014.07.013
12. Weiner BJ, Lewis CC, Stanick C, et al. Psychometric assessment of three newly developed implementation outcome measures. Implement Sci. 2017;12(1):108. doi:10.1186/s13012-017-0635-3
13. Proctor E, Silmere H, Raghavan R, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. 2011;38(2):65-76. doi:10.1007/s10488-010-0319-7
14. Kuhn M, Johnson K. Applied Predictive Modeling. Springer; 2013.
15. Team RC. A language and environment for statistical computing. 2018. Accessed December 16, 2022. https://www.R-project.org
16. Wong DT, Kamming D, Salenieks ME, Go K, Kohm C, Chung F. Preadmission anesthesia consultation using telemedicine technology: a pilot study. Anesthesiology. 2004;100(6):1605-1607. doi:10.1097/00000542-200406000-00038
17. Zetterman CV, Sweitzer BJ, Webb B, Barak-Bernhagen MA, Boedeker BH. Validation of a virtual preoperative evaluation clinic: a pilot study. Stud Health Technol Inform. 2011;163:737-739. doi: 10.3233/978-1-60750-706-2-737
18. Roberts S, Spain B, Hicks C, London J, Tay S. Telemedicine in the Northern Territory: an assessment of patient perceptions in the preoperative anaesthetic clinic. Aust J Rural Health. 2015;23(3):136-141. doi:10.1111/ajr.12140
Days or weeks before a scheduled surgical or invasive procedure involving anesthesia, evaluations are conducted to assess a patient’s condition and risk, optimize their status, and prepare them for their procedure. A comprehensive pre-anesthesia evaluation visit includes a history of present illness, the evaluation of comorbidities and medication use, the assessment of health habits such as alcohol and tobacco use, functional capacity and nutritional assessments, and the identification of social support deficiencies that may influence recovery. It also includes a focused physical examination and laboratory and other ancillary testing as needed and may include optimization interventions such as anemia management or prehabilitation. Conducting pre-anesthesia evaluations before surgery has been shown to reduce delays and cancellations, unnecessary preprocedure testing, hospital length of stay, and in-hospital mortality.1-4
The pre-anesthesia evaluation is usually conducted in person, although other modalities have been in use for several years and have accelerated since the advent of the COVID-19 pandemic. Specifically, audio-only telephone visits are used in many settings to conduct abbreviated forms of a pre-anesthesia evaluation, typically for less-invasive procedures. When patients are evaluated over the telephone, the physical examination and testing are deferred until the day of the procedure. Another modality is the use of synchronous video telehealth. Emerging evidence for the use of video-based care in anesthesiology provides encouraging results. Several institutions have proven the technological feasibility of performing preoperative evaluations via video.5,6 Compared with in-person evaluations, these visits seem to have similar surgery cancellation rates, improved patient satisfaction, and reduced wait times and costs.7-9
As part of a quality improvement project, we studied the use of telehealth for pre-anesthesia evaluations within the US Department of Veterans Affairs (VA). An internal review found overall low utilization of these modalities before the COVID-19 pandemic that accelerated toward telehealth during the pandemic: The largest uptake was with telephone visits. Given the increasing adoption of telehealth for pre-anesthesia evaluations and the marked preference for telephone over video modalities among VA practitioners during the COVID-19 pandemic, we sought to understand the barriers and facilitators to the adoption of telephone- and video-based pre-anesthesia evaluation visits within the VA.
Methods
Our objective was to assess health care practitioners’ (HCPs) preferences regarding pre-anesthesia evaluation modalities (in-person, telephone, or video), and the perceived advantages and barriers to adoption for each modality. We followed the Strengthening the Reporting of Observational studies in Epidemiology (STROBE) guideline and Checklist for statistical Assessment of Medical Papers (CHAMP) statement.10,11 The survey was deemed a quality improvement activity that was exempt from institutional review board oversight by the VA National Anesthesia Program Office and the VA Office of Connected Care.
A survey was distributed to all VA anesthesiology service chiefs via email between April 27, 2022, and May 3, 2022. Three emails were sent to each participant (initial invitation and 2 reminders). The respondents were asked to identify themselves by facility and role and to indicate whether their anesthesiology service performed any pre-anesthesia evaluations, including any telephone- or video-based evaluations; and whether their service has a dedicated pre-anesthesia evaluation clinic.
A second set of questions referred to the use of telephone- and video-based preprocedure evaluations. The questions were based on branch logic and depended on the respondent’s answers concerning their use of telephone- and video-based evaluations. Questions included statements about perceived barriers to the adoption of these pre-anesthesia evaluation modalities. Each item was rated on a 5-point Likert scale, (completely disagree [1] to completely agree [5]). A third section measured acceptability and feasibility of video using the validated Acceptability of Intervention Measure (AIM) and Feasibility of Intervention Measure (FIM)questionnaires.12 These instruments are 4-item measures of implementation outcomes that are often considered indicators of implementation success.13Acceptability is the perception among implementation stakeholders that a given treatment, service, practice, or innovation is agreeable, palatable, or satisfactory. Feasibility is defined as the extent to which a new treatment or an innovation can be successfully used or carried out within a given agency or setting.13 The criterion for acceptability is personal, meaning that different HCPs may have differing needs, preferences, and expectations regarding the same intervention. The criterion for feasibility is practical. An intervention may be considered to be feasible if the required tasks can be performed easily or conveniently. Finally, 2 open-ended questions allowed respondents to identify the most important factor that allowed the implementation of telehealth for pre-anesthesia evaluations in their service, and provide comments about the use of telehealth for pre-anesthesia evaluations at the VA. All questions were developed by the authors except for the 2 implementation measure instruments.
The survey was administered using an electronic survey platform (Qualtrics, version April 2022) and sent by email alongside a brief introductory video. Participation was voluntary and anonymous, as no personal information was collected. Responses were attributed to each facility, using the self-declared affiliation. When an affiliation was not provided, we deduced it using the latitude/longitude of the respondent, a feature included in the survey software. No incentives were provided. Data were stored and maintained in a secure VA server. All completed surveys were included. Some facilities had > 1 complete response, and all were included. Facilities that provided > 1 response and where responses were discordant, we clarified with the facility service chief. Incomplete responses were excluded from the analysis.
Statistics
For this analysis, the 2 positive sentiment responses (agree and completely agree) and the 2 negative sentiment responses (disagree and completely disagree) in the Likert scale were collapsed into single categories (good and poor, respectively). The neither agree nor disagree responses were coded as neutral. Our analysis began with a visual exploration of all variables to evaluate the frequency, percentage, and near-zero variance for categorical variables.14 Near-zero variance occurs when a categorical variable has a low frequency of unique values over the sample size (ie, the variable is almost constant), and we addressed it by combining different variable categorizations. We handled missing values through imputation algorithms followed by sensitivity analyses to verify whether our results were stable with and without imputation. We performed comparisons for the exploratory analysis using P values for one-way analysis of variance tests for numeric variables and χ2tests for categorical variables. We considered P values < .05 to be statistically significant. We also used correlation matrices and plots as exploratory analysis tools to better understand all items’ correlations. We used Pearson, polychoric, and polyserial correlation tests as appropriate for numeric, ordinal, and logical items.
Our modeling strategy involved a series of generalized linear models (GLMs) with a Gaussian family, ie, multiple linear regression models, to assess the association between (1) facilities’ preferences regarding pre-anesthesia evaluation modalities; (2) advantages between modalities; and (3) barriers to the adoption of telehealth and the ability to perform different pre-anesthesia evaluation-related tasks. In addition, we used backward deletion to reach the most parsimonious model based on a series of likelihood-ratio tests comparing nested models. Results are reported as predicted means with 95% confidence intervals, with results being interpreted as significant when any 2 predicted means do not overlap between different estimates along with P for trends < .001. We performed all analyses using the R language.15
Results
Of 109 surveyed facilities, 50 (46%) responded to the survey. The final study sample included 67 responses, and 55 were included in the analysis. Twelve responses were excluded from the analysis as they were either incomplete or test responses. Three facilities had > 1 complete response (2 facilities had 2 responses and 1 facility had 4 responses), and these were all included in the analysis.
Thirty-six locations were complex inpatient facilities, and 32 (89%) had pre-anesthesia evaluation clinics (Table 1).
The ability to obtain a history of present illness was rated good/very good via telephone for 34 respondents (92%) and 25 for video (86%). Assessing comorbidities and health habits was rated good/very good via telephone for 32 respondents (89%) and 31 respondents (86%), respectively, and via video for 24 respondents (83%) and 23 respondents (79%), respectively (Figure 1).
To compare differences between the 2 remote pre-anesthesia evaluation modalities, we created GLMs evaluating the association between each modality and the perceived ability to perform the tasks. For GLMs, we transformed the values of the categories into numerical (ie, 1, poor; 2, neutral; 3, good). Compared with telephone, video was rated more favorably regarding the assessment of nutritional status (mean, 2.1; 95% CI, 1.8-2.3 vs mean, 2.4; 95% CI, 2.2-2.7; P = .04) (eAppendix 1, available at doi:10.12788/fp.0387). No other significant differences in ratings existed between the 2 remote pre-anesthesia evaluation modalities.
The most significant barriers (cited as significant or very significant in the survey) included the inability to perform a physical examination, which was noted by 13 respondents (72%) and 15 respondents (60%) for telephone and video, respectively. The inability to obtain vital signs was rated as a significant barrier for telephone by 12 respondents (67%) and for video by 15 respondents (60%)(Figure 2).
The average FIM score was 3.7, with the highest score among respondents who used both phone and video (Table 2). The average AIM score was 3.4, with the highest score among respondents who used both telehealth modalities. The internal consistency of the implementation measures was excellent (Cronbach’s α 0.95 and 0.975 for FIM and AIM, respectively).
Discussion
We surveyed 109 anesthesiology services across the VA regarding barriers to implementing telephone- and video-based pre-anesthesia evaluation visits. We found that 12 (23%) of the 50 anesthesiology services responding to this survey still conduct the totality of their pre-anesthesia evaluations in person. This represents an opportunity to further disseminate the appropriate use of telehealth and potentially reduce travel time, costs, and low-value testing, as it is well established that remote pre-anesthesia evaluations for low-risk procedures are safe and effective.6
We also found no difference between telephone and video regarding users’ perceived ability to perform any of the basic pre-anesthesia evaluation tasks except for assessing patients’ nutritional status, which was rated as easier using video than telephone. According to those not using telephone and/or video, the biggest barriers to implementation of telehealth visits were the inability to obtain vital signs and to perform a physical examination. This finding was unexpected, as facilities that conduct remote evaluations typically defer these tasks to the day of surgery, a practice that has been well established and shown to be safe and efficient. Respondents also identified patient-level factors (eg, patient preference, lack of telephone or computer) as significant barriers. Finally, feasibility ratings were higher than acceptability ratings with regards to the implementation of telehealth.
In 2004, the first use of telehealth for pre-anesthesia evaluations was reported by Wong and colleagues.16 Since then, several case series and a literature review have documented the efficacy, safety, and patient and HCP satisfaction with the use of telehealth for pre-anesthesia evaluations. A study by Mullen-Fortino and colleagues showed reduced visit times when telehealth was used for pre-anesthesia evaluation.8 Another study at VA hospitals showed that 88% of veterans reported that telemedicine saved them time and money.17 A report of 35 patients in rural Australia reported 98% satisfaction with the video quality of the visit, 95% perceived efficacy, and 87% preference for telehealth compared with driving to be seen in person.18 These reports conflict with the perceptions of the respondents of our survey, who identified patient preference as an important barrier to adoption of telehealth. Given these findings, research is needed on veterans’ perceptions on the use of telehealth modalities for pre-anesthesia evaluations; if their perceptions are similarly favorable, it will be important to communicate this information to HCPs and leadership, which may help increase subsequent telehealth adoption.
Despite the reported safety, efficacy, and high satisfaction of video visits among anesthesiology teams conducting pre-anesthesia evaluations, its use remains low at VA. We have found that most facilities in the VA system chose telephone platforms during the COVID-19 pandemic. One possibility is that the adoption of video modalities among pre-anesthesia evaluation clinics in the VA system is resource intensive or difficult from the HCP’s perspective. When combined with the lack of perceived advantages over telephone as we found in our survey, most practitioners resort to the technologically less demanding and more familiar telephone platform. The results from FIM and AIM support this. While both telephone and video have high feasibility scores, acceptability scores are lower for video, even among those currently using this technology. Our findings do not rule out the utility of video-based care in perioperative medicine. Rather than a yes/no proposition, future studies need to establish the precise indications for video for pre-anesthesia evaluations; that is, situations where video visits offer an advantage over telephone. For example, video could be used to deliver preoperative optimization therapies, such as supervised exercise or mental health interventions or to guide the achievement of certain milestones before surgery in patients with chronic conditions, such as target glucose values or the treatment of anemia. Future studies should explore the perceived benefits of video over telephone among centers offering these more advanced optimization interventions.
Limitations
We received responses from a subset of VA anesthesiology services; therefore, they may not be representative of the entire VA system. Facilities designated by the VA as inpatient complex were overrepresented (72% of our sample vs 50% of the total facilities nationally), and ambulatory centers (those designed by the VA as ambulatory procedural center with basic or advanced capabilities) were underrepresented (2% of our sample vs 22% nationally). Despite this, the response rate was high, and no geographic area appeared to be underrepresented. In addition, we surveyed pre-anesthesia evaluation facilities led by anesthesiologists, and the results may not be representative of the preferences of HCPs working in nonanesthesiology led pre-anesthesia evaluation clinics. Finally, just 11 facilities used both telephone and video; therefore, a true direct comparison between these 2 platforms was limited. The VA serves a unique patient population, and the findings may not be completely applicable to the non-VA population.
Conclusions
We found no significant perceived advantages of video over telephone in the ability to conduct routine pre-anesthesia evaluations among a sample of anesthesiology HCPs in the VA except for the perceived ability to assess nutritional status. HCPs with no telehealth experience cited the inability to perform a physical examination and obtain vital signs as the most significant barriers to implementation. Respondents not using telephone cited concerns about safety. Video visits in this clinical setting had additional perceived barriers to implementation, such as lack of information technology and staff support and patient-level barriers. Video had lower acceptability by HCPs. Given findings that pre-anesthesia evaluations can be conducted effectively via telehealth and have high levels of patient satisfaction, future work should focus on increasing uptake of these remote modalities. Additionally, research on the most appropriate uses of video visits within perioperative care is also needed.
Days or weeks before a scheduled surgical or invasive procedure involving anesthesia, evaluations are conducted to assess a patient’s condition and risk, optimize their status, and prepare them for their procedure. A comprehensive pre-anesthesia evaluation visit includes a history of present illness, the evaluation of comorbidities and medication use, the assessment of health habits such as alcohol and tobacco use, functional capacity and nutritional assessments, and the identification of social support deficiencies that may influence recovery. It also includes a focused physical examination and laboratory and other ancillary testing as needed and may include optimization interventions such as anemia management or prehabilitation. Conducting pre-anesthesia evaluations before surgery has been shown to reduce delays and cancellations, unnecessary preprocedure testing, hospital length of stay, and in-hospital mortality.1-4
The pre-anesthesia evaluation is usually conducted in person, although other modalities have been in use for several years and have accelerated since the advent of the COVID-19 pandemic. Specifically, audio-only telephone visits are used in many settings to conduct abbreviated forms of a pre-anesthesia evaluation, typically for less-invasive procedures. When patients are evaluated over the telephone, the physical examination and testing are deferred until the day of the procedure. Another modality is the use of synchronous video telehealth. Emerging evidence for the use of video-based care in anesthesiology provides encouraging results. Several institutions have proven the technological feasibility of performing preoperative evaluations via video.5,6 Compared with in-person evaluations, these visits seem to have similar surgery cancellation rates, improved patient satisfaction, and reduced wait times and costs.7-9
As part of a quality improvement project, we studied the use of telehealth for pre-anesthesia evaluations within the US Department of Veterans Affairs (VA). An internal review found overall low utilization of these modalities before the COVID-19 pandemic that accelerated toward telehealth during the pandemic: The largest uptake was with telephone visits. Given the increasing adoption of telehealth for pre-anesthesia evaluations and the marked preference for telephone over video modalities among VA practitioners during the COVID-19 pandemic, we sought to understand the barriers and facilitators to the adoption of telephone- and video-based pre-anesthesia evaluation visits within the VA.
Methods
Our objective was to assess health care practitioners’ (HCPs) preferences regarding pre-anesthesia evaluation modalities (in-person, telephone, or video), and the perceived advantages and barriers to adoption for each modality. We followed the Strengthening the Reporting of Observational studies in Epidemiology (STROBE) guideline and Checklist for statistical Assessment of Medical Papers (CHAMP) statement.10,11 The survey was deemed a quality improvement activity that was exempt from institutional review board oversight by the VA National Anesthesia Program Office and the VA Office of Connected Care.
A survey was distributed to all VA anesthesiology service chiefs via email between April 27, 2022, and May 3, 2022. Three emails were sent to each participant (initial invitation and 2 reminders). The respondents were asked to identify themselves by facility and role and to indicate whether their anesthesiology service performed any pre-anesthesia evaluations, including any telephone- or video-based evaluations; and whether their service has a dedicated pre-anesthesia evaluation clinic.
A second set of questions referred to the use of telephone- and video-based preprocedure evaluations. The questions were based on branch logic and depended on the respondent’s answers concerning their use of telephone- and video-based evaluations. Questions included statements about perceived barriers to the adoption of these pre-anesthesia evaluation modalities. Each item was rated on a 5-point Likert scale, (completely disagree [1] to completely agree [5]). A third section measured acceptability and feasibility of video using the validated Acceptability of Intervention Measure (AIM) and Feasibility of Intervention Measure (FIM)questionnaires.12 These instruments are 4-item measures of implementation outcomes that are often considered indicators of implementation success.13Acceptability is the perception among implementation stakeholders that a given treatment, service, practice, or innovation is agreeable, palatable, or satisfactory. Feasibility is defined as the extent to which a new treatment or an innovation can be successfully used or carried out within a given agency or setting.13 The criterion for acceptability is personal, meaning that different HCPs may have differing needs, preferences, and expectations regarding the same intervention. The criterion for feasibility is practical. An intervention may be considered to be feasible if the required tasks can be performed easily or conveniently. Finally, 2 open-ended questions allowed respondents to identify the most important factor that allowed the implementation of telehealth for pre-anesthesia evaluations in their service, and provide comments about the use of telehealth for pre-anesthesia evaluations at the VA. All questions were developed by the authors except for the 2 implementation measure instruments.
The survey was administered using an electronic survey platform (Qualtrics, version April 2022) and sent by email alongside a brief introductory video. Participation was voluntary and anonymous, as no personal information was collected. Responses were attributed to each facility, using the self-declared affiliation. When an affiliation was not provided, we deduced it using the latitude/longitude of the respondent, a feature included in the survey software. No incentives were provided. Data were stored and maintained in a secure VA server. All completed surveys were included. Some facilities had > 1 complete response, and all were included. Facilities that provided > 1 response and where responses were discordant, we clarified with the facility service chief. Incomplete responses were excluded from the analysis.
Statistics
For this analysis, the 2 positive sentiment responses (agree and completely agree) and the 2 negative sentiment responses (disagree and completely disagree) in the Likert scale were collapsed into single categories (good and poor, respectively). The neither agree nor disagree responses were coded as neutral. Our analysis began with a visual exploration of all variables to evaluate the frequency, percentage, and near-zero variance for categorical variables.14 Near-zero variance occurs when a categorical variable has a low frequency of unique values over the sample size (ie, the variable is almost constant), and we addressed it by combining different variable categorizations. We handled missing values through imputation algorithms followed by sensitivity analyses to verify whether our results were stable with and without imputation. We performed comparisons for the exploratory analysis using P values for one-way analysis of variance tests for numeric variables and χ2tests for categorical variables. We considered P values < .05 to be statistically significant. We also used correlation matrices and plots as exploratory analysis tools to better understand all items’ correlations. We used Pearson, polychoric, and polyserial correlation tests as appropriate for numeric, ordinal, and logical items.
Our modeling strategy involved a series of generalized linear models (GLMs) with a Gaussian family, ie, multiple linear regression models, to assess the association between (1) facilities’ preferences regarding pre-anesthesia evaluation modalities; (2) advantages between modalities; and (3) barriers to the adoption of telehealth and the ability to perform different pre-anesthesia evaluation-related tasks. In addition, we used backward deletion to reach the most parsimonious model based on a series of likelihood-ratio tests comparing nested models. Results are reported as predicted means with 95% confidence intervals, with results being interpreted as significant when any 2 predicted means do not overlap between different estimates along with P for trends < .001. We performed all analyses using the R language.15
Results
Of 109 surveyed facilities, 50 (46%) responded to the survey. The final study sample included 67 responses, and 55 were included in the analysis. Twelve responses were excluded from the analysis as they were either incomplete or test responses. Three facilities had > 1 complete response (2 facilities had 2 responses and 1 facility had 4 responses), and these were all included in the analysis.
Thirty-six locations were complex inpatient facilities, and 32 (89%) had pre-anesthesia evaluation clinics (Table 1).
The ability to obtain a history of present illness was rated good/very good via telephone for 34 respondents (92%) and 25 for video (86%). Assessing comorbidities and health habits was rated good/very good via telephone for 32 respondents (89%) and 31 respondents (86%), respectively, and via video for 24 respondents (83%) and 23 respondents (79%), respectively (Figure 1).
To compare differences between the 2 remote pre-anesthesia evaluation modalities, we created GLMs evaluating the association between each modality and the perceived ability to perform the tasks. For GLMs, we transformed the values of the categories into numerical (ie, 1, poor; 2, neutral; 3, good). Compared with telephone, video was rated more favorably regarding the assessment of nutritional status (mean, 2.1; 95% CI, 1.8-2.3 vs mean, 2.4; 95% CI, 2.2-2.7; P = .04) (eAppendix 1, available at doi:10.12788/fp.0387). No other significant differences in ratings existed between the 2 remote pre-anesthesia evaluation modalities.
The most significant barriers (cited as significant or very significant in the survey) included the inability to perform a physical examination, which was noted by 13 respondents (72%) and 15 respondents (60%) for telephone and video, respectively. The inability to obtain vital signs was rated as a significant barrier for telephone by 12 respondents (67%) and for video by 15 respondents (60%)(Figure 2).
The average FIM score was 3.7, with the highest score among respondents who used both phone and video (Table 2). The average AIM score was 3.4, with the highest score among respondents who used both telehealth modalities. The internal consistency of the implementation measures was excellent (Cronbach’s α 0.95 and 0.975 for FIM and AIM, respectively).
Discussion
We surveyed 109 anesthesiology services across the VA regarding barriers to implementing telephone- and video-based pre-anesthesia evaluation visits. We found that 12 (23%) of the 50 anesthesiology services responding to this survey still conduct the totality of their pre-anesthesia evaluations in person. This represents an opportunity to further disseminate the appropriate use of telehealth and potentially reduce travel time, costs, and low-value testing, as it is well established that remote pre-anesthesia evaluations for low-risk procedures are safe and effective.6
We also found no difference between telephone and video regarding users’ perceived ability to perform any of the basic pre-anesthesia evaluation tasks except for assessing patients’ nutritional status, which was rated as easier using video than telephone. According to those not using telephone and/or video, the biggest barriers to implementation of telehealth visits were the inability to obtain vital signs and to perform a physical examination. This finding was unexpected, as facilities that conduct remote evaluations typically defer these tasks to the day of surgery, a practice that has been well established and shown to be safe and efficient. Respondents also identified patient-level factors (eg, patient preference, lack of telephone or computer) as significant barriers. Finally, feasibility ratings were higher than acceptability ratings with regards to the implementation of telehealth.
In 2004, the first use of telehealth for pre-anesthesia evaluations was reported by Wong and colleagues.16 Since then, several case series and a literature review have documented the efficacy, safety, and patient and HCP satisfaction with the use of telehealth for pre-anesthesia evaluations. A study by Mullen-Fortino and colleagues showed reduced visit times when telehealth was used for pre-anesthesia evaluation.8 Another study at VA hospitals showed that 88% of veterans reported that telemedicine saved them time and money.17 A report of 35 patients in rural Australia reported 98% satisfaction with the video quality of the visit, 95% perceived efficacy, and 87% preference for telehealth compared with driving to be seen in person.18 These reports conflict with the perceptions of the respondents of our survey, who identified patient preference as an important barrier to adoption of telehealth. Given these findings, research is needed on veterans’ perceptions on the use of telehealth modalities for pre-anesthesia evaluations; if their perceptions are similarly favorable, it will be important to communicate this information to HCPs and leadership, which may help increase subsequent telehealth adoption.
Despite the reported safety, efficacy, and high satisfaction of video visits among anesthesiology teams conducting pre-anesthesia evaluations, its use remains low at VA. We have found that most facilities in the VA system chose telephone platforms during the COVID-19 pandemic. One possibility is that the adoption of video modalities among pre-anesthesia evaluation clinics in the VA system is resource intensive or difficult from the HCP’s perspective. When combined with the lack of perceived advantages over telephone as we found in our survey, most practitioners resort to the technologically less demanding and more familiar telephone platform. The results from FIM and AIM support this. While both telephone and video have high feasibility scores, acceptability scores are lower for video, even among those currently using this technology. Our findings do not rule out the utility of video-based care in perioperative medicine. Rather than a yes/no proposition, future studies need to establish the precise indications for video for pre-anesthesia evaluations; that is, situations where video visits offer an advantage over telephone. For example, video could be used to deliver preoperative optimization therapies, such as supervised exercise or mental health interventions or to guide the achievement of certain milestones before surgery in patients with chronic conditions, such as target glucose values or the treatment of anemia. Future studies should explore the perceived benefits of video over telephone among centers offering these more advanced optimization interventions.
Limitations
We received responses from a subset of VA anesthesiology services; therefore, they may not be representative of the entire VA system. Facilities designated by the VA as inpatient complex were overrepresented (72% of our sample vs 50% of the total facilities nationally), and ambulatory centers (those designed by the VA as ambulatory procedural center with basic or advanced capabilities) were underrepresented (2% of our sample vs 22% nationally). Despite this, the response rate was high, and no geographic area appeared to be underrepresented. In addition, we surveyed pre-anesthesia evaluation facilities led by anesthesiologists, and the results may not be representative of the preferences of HCPs working in nonanesthesiology led pre-anesthesia evaluation clinics. Finally, just 11 facilities used both telephone and video; therefore, a true direct comparison between these 2 platforms was limited. The VA serves a unique patient population, and the findings may not be completely applicable to the non-VA population.
Conclusions
We found no significant perceived advantages of video over telephone in the ability to conduct routine pre-anesthesia evaluations among a sample of anesthesiology HCPs in the VA except for the perceived ability to assess nutritional status. HCPs with no telehealth experience cited the inability to perform a physical examination and obtain vital signs as the most significant barriers to implementation. Respondents not using telephone cited concerns about safety. Video visits in this clinical setting had additional perceived barriers to implementation, such as lack of information technology and staff support and patient-level barriers. Video had lower acceptability by HCPs. Given findings that pre-anesthesia evaluations can be conducted effectively via telehealth and have high levels of patient satisfaction, future work should focus on increasing uptake of these remote modalities. Additionally, research on the most appropriate uses of video visits within perioperative care is also needed.
1. Starsnic MA, Guarnieri DM, Norris MC. Efficacy and financial benefit of an anesthesiologist-directed university preadmission evaluation center. J Clin Anesth. 1997;9(4):299-305. doi:10.1016/s0952-8180(97)00007-x
2. Kristoffersen EW, Opsal A, Tveit TO, Berg RC, Fossum M. Effectiveness of pre-anaesthetic assessment clinic: a systematic review of randomised and non-randomised prospective controlled studies. BMJ Open. 2022;12(5):e054206. doi:10.1136/bmjopen-2021-054206
3. Ferschl MB, Tung A, Sweitzer B, Huo D, Glick DB. Preoperative clinic visits reduce operating room cancellations and delays. Anesthesiology. 2005;103(4):855-9. doi:10.1097/00000542-200510000-00025
4. Blitz JD, Kendale SM, Jain SK, Cuff GE, Kim JT, Rosenberg AD. preoperative evaluation clinic visit is associated with decreased risk of in-hospital postoperative mortality. Anesthesiology. 2016;125(2):280-294. doi:10.1097/ALN.0000000000001193
5. Dilisio RP, Dilisio AJ, Weiner MM. Preoperative virtual screening examination of the airway. J Clin Anesth. 2014;26(4):315-317. doi:10.1016/j.jclinane.2013.12.010
6. Kamdar NV, Huverserian A, Jalilian L, et al. Development, implementation, and evaluation of a telemedicine preoperative evaluation initiative at a major academic medical center. Anesth Analg. 2020;131(6):1647-1656. doi:10.1213/ANE.0000000000005208
7. Azizad O, Joshi GP. Telemedicine for preanesthesia evaluation: review of current literature and recommendations for future implementation. Curr Opin Anaesthesiol. 2021;34(6):672-677. doi:10.1097/ACO.0000000000001064
8. Mullen-Fortino M, Rising KL, Duckworth J, Gwynn V, Sites FD, Hollander JE. Presurgical assessment using telemedicine technology: impact on efficiency, effectiveness, and patient experience of care. Telemed J E Health. 2019;25(2):137-142. doi:10.1089/tmj.2017.0133
9. Zhang K, Rashid-Kolvear M, Waseem R, Englesakis M, Chung F. Virtual preoperative assessment in surgical patients: a systematic review and meta-analysis. J Clin Anesth. 2021;75:110540. doi:10.1016/j.jclinane.2021.110540
10. Mansournia MA, Collins GS, Nielsen RO, et al. A CHecklist for statistical Assessment of Medical Papers (the CHAMP statement): explanation and elaboration. Br J Sports Med. 2021;55(18):1009-1017. doi:10.1136/bjsports-2020-103652
11. von Elm E, Altman DG, Egger M, et al. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: guidelines for reporting observational studies. Int J Surg. 2014;12(12):1495-1499. doi:10.1016/j.ijsu.2014.07.013
12. Weiner BJ, Lewis CC, Stanick C, et al. Psychometric assessment of three newly developed implementation outcome measures. Implement Sci. 2017;12(1):108. doi:10.1186/s13012-017-0635-3
13. Proctor E, Silmere H, Raghavan R, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. 2011;38(2):65-76. doi:10.1007/s10488-010-0319-7
14. Kuhn M, Johnson K. Applied Predictive Modeling. Springer; 2013.
15. Team RC. A language and environment for statistical computing. 2018. Accessed December 16, 2022. https://www.R-project.org
16. Wong DT, Kamming D, Salenieks ME, Go K, Kohm C, Chung F. Preadmission anesthesia consultation using telemedicine technology: a pilot study. Anesthesiology. 2004;100(6):1605-1607. doi:10.1097/00000542-200406000-00038
17. Zetterman CV, Sweitzer BJ, Webb B, Barak-Bernhagen MA, Boedeker BH. Validation of a virtual preoperative evaluation clinic: a pilot study. Stud Health Technol Inform. 2011;163:737-739. doi: 10.3233/978-1-60750-706-2-737
18. Roberts S, Spain B, Hicks C, London J, Tay S. Telemedicine in the Northern Territory: an assessment of patient perceptions in the preoperative anaesthetic clinic. Aust J Rural Health. 2015;23(3):136-141. doi:10.1111/ajr.12140
1. Starsnic MA, Guarnieri DM, Norris MC. Efficacy and financial benefit of an anesthesiologist-directed university preadmission evaluation center. J Clin Anesth. 1997;9(4):299-305. doi:10.1016/s0952-8180(97)00007-x
2. Kristoffersen EW, Opsal A, Tveit TO, Berg RC, Fossum M. Effectiveness of pre-anaesthetic assessment clinic: a systematic review of randomised and non-randomised prospective controlled studies. BMJ Open. 2022;12(5):e054206. doi:10.1136/bmjopen-2021-054206
3. Ferschl MB, Tung A, Sweitzer B, Huo D, Glick DB. Preoperative clinic visits reduce operating room cancellations and delays. Anesthesiology. 2005;103(4):855-9. doi:10.1097/00000542-200510000-00025
4. Blitz JD, Kendale SM, Jain SK, Cuff GE, Kim JT, Rosenberg AD. preoperative evaluation clinic visit is associated with decreased risk of in-hospital postoperative mortality. Anesthesiology. 2016;125(2):280-294. doi:10.1097/ALN.0000000000001193
5. Dilisio RP, Dilisio AJ, Weiner MM. Preoperative virtual screening examination of the airway. J Clin Anesth. 2014;26(4):315-317. doi:10.1016/j.jclinane.2013.12.010
6. Kamdar NV, Huverserian A, Jalilian L, et al. Development, implementation, and evaluation of a telemedicine preoperative evaluation initiative at a major academic medical center. Anesth Analg. 2020;131(6):1647-1656. doi:10.1213/ANE.0000000000005208
7. Azizad O, Joshi GP. Telemedicine for preanesthesia evaluation: review of current literature and recommendations for future implementation. Curr Opin Anaesthesiol. 2021;34(6):672-677. doi:10.1097/ACO.0000000000001064
8. Mullen-Fortino M, Rising KL, Duckworth J, Gwynn V, Sites FD, Hollander JE. Presurgical assessment using telemedicine technology: impact on efficiency, effectiveness, and patient experience of care. Telemed J E Health. 2019;25(2):137-142. doi:10.1089/tmj.2017.0133
9. Zhang K, Rashid-Kolvear M, Waseem R, Englesakis M, Chung F. Virtual preoperative assessment in surgical patients: a systematic review and meta-analysis. J Clin Anesth. 2021;75:110540. doi:10.1016/j.jclinane.2021.110540
10. Mansournia MA, Collins GS, Nielsen RO, et al. A CHecklist for statistical Assessment of Medical Papers (the CHAMP statement): explanation and elaboration. Br J Sports Med. 2021;55(18):1009-1017. doi:10.1136/bjsports-2020-103652
11. von Elm E, Altman DG, Egger M, et al. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: guidelines for reporting observational studies. Int J Surg. 2014;12(12):1495-1499. doi:10.1016/j.ijsu.2014.07.013
12. Weiner BJ, Lewis CC, Stanick C, et al. Psychometric assessment of three newly developed implementation outcome measures. Implement Sci. 2017;12(1):108. doi:10.1186/s13012-017-0635-3
13. Proctor E, Silmere H, Raghavan R, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. 2011;38(2):65-76. doi:10.1007/s10488-010-0319-7
14. Kuhn M, Johnson K. Applied Predictive Modeling. Springer; 2013.
15. Team RC. A language and environment for statistical computing. 2018. Accessed December 16, 2022. https://www.R-project.org
16. Wong DT, Kamming D, Salenieks ME, Go K, Kohm C, Chung F. Preadmission anesthesia consultation using telemedicine technology: a pilot study. Anesthesiology. 2004;100(6):1605-1607. doi:10.1097/00000542-200406000-00038
17. Zetterman CV, Sweitzer BJ, Webb B, Barak-Bernhagen MA, Boedeker BH. Validation of a virtual preoperative evaluation clinic: a pilot study. Stud Health Technol Inform. 2011;163:737-739. doi: 10.3233/978-1-60750-706-2-737
18. Roberts S, Spain B, Hicks C, London J, Tay S. Telemedicine in the Northern Territory: an assessment of patient perceptions in the preoperative anaesthetic clinic. Aust J Rural Health. 2015;23(3):136-141. doi:10.1111/ajr.12140