You are here

  • Home
  • Archive
  • Volume 14, Issue 1
  • Effect of an algorithm for automatic placing of standardised test order sets on low-value appointments and attendance rates at four Spanish teaching hospitals: an interrupted time series analysis

Article Text

Article menu
Health services research
Original research
Effect of an algorithm for automatic placing of standardised test order sets on low-value appointments and attendance rates at four Spanish teaching hospitals: an interrupted time series analysis
  1. Juan Antonio Álvaro de la Parra1,
  2. Marta del Olmo Rodríguez1,2,
  3. Cristina Caramés Sánchez1,2,
  4. Ángel Blanco1,
  5. Bernadette Pfang2,3,
  6. Sagrario Mayoralas-Alises4,
  7. Jose Fernandez-Ferro2,5,
  8. Emilio Calvo2,6,
  9. Óscar Gómez Martín2,3,
  10. Jesús Fernández Tabera2,7,
  11. Carmen Plaza Nohales2,8,
  12. Carlota Nieto3,
  13. Jorge Short Apellaniz2,3
  1. 1Quirónsalud, Madrid, Spain
  2. 2Instituto de Investigacion Sanitaria de la Fundación Jiménez Díaz, Madrid, Spain
  3. 3Hospital Universitario Fundación Jiménez Díaz, Madrid, Spain
  4. 4Hospital Quirónsalud San Jose, Madrid, Spain
  5. 5Neurology Department, Hospital Universitario Rey Juan Carlos, Mostoles, Spain
  6. 6Orthopaedic Surgery and Traumatology, Hospital Universitario Fundación Jiménez Díaz, Madrid, Spain
  7. 7Villalba General University Hospital, Collado Villalba, Spain
  8. 8Hospital Universitario Rey Juan Carlos, Mostoles, Spain
  1. Correspondence to Dr Jorge Short Apellaniz; jorge.short{at}quironsalud.es

Abstract

Objective Reducing backlogs for elective care is a priority for healthcare systems. We conducted an interrupted time series analysis demonstrating the effect of an algorithm for placing automatic test order sets prior to first specialist appointment on avoidable follow-up appointments and attendance rates.

Design Interrupted time series analysis.

Setting 4 academic hospitals from Madrid, Spain.

Participants Patients referred from primary care attending 10 033 470 outpatient appointments from 16 clinical specialties during a 6-year period (1 January 2018 to 30 June 2023).

Intervention An algorithm using natural language processing was launched in May 2021. Test order sets developed for 257 presenting complaints from 16 clinical specialties were placed automatically before first specialist appointments to increase rates of diagnosis and initiation of treatment with discharge back to primary care.

Primary and secondary outcome measures Primary outcomes included rate of diagnosis and discharge to primary care and follow-up to first appointment index. The secondary outcome was trend in ‘did not attend’ rates.

Results Since May 2021, a total of 1 175 814 automatic test orders have been placed. Significant changes in trend of diagnosis and discharge to primary care at first appointment (p=0.005, 95% CI 0.5 to 2.9) and ‘did not attend’ rates (p=0.006, 95% CI −0.1 to −0.8) and an estimated attributable reduction of 11 306 avoidable follow-up appointments per month were observed.

Conclusion An algorithm for placing automatic standardised test order sets can reduce low-value follow-up appointments by allowing specialists to confirm diagnoses and initiate treatment at first appointment, also leading to early discharge to primary care and a reduction in ‘did not attend’ rates. This initiative points to an improved process for outpatient diagnosis and treatment, delivering healthcare more effectively and efficiently.

  • Natural Language Processing
  • Clinical Decision-Making
  • Protocols & guidelines
  • Quality in health care

Data availability statement

Data are available upon reasonable request. Technical appendix, statistical code and data set are available from the last author (JSA) upon reasonable request.

http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

https://doi.org/10.1136/bmjopen-2023-081158

Statistics from Altmetric.com

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

STRENGTHS AND LIMITATIONS OF THIS STUDY

  • Interrupted time series is the most robust quasiexperimental model when randomisation is not possible.

  • Over 10 000 000 outpatient appointments included in the analysis.

  • The 6-year study period provides enough data points in the pre-implementation and post implementation groups (26 in each group) for a robust statistical analysis.

  • A year of data corresponding to the first three waves of the COVID-19 pandemic in Spain was censored from analysis.

  • The inclusion of a reference group sharing similar characteristics to the intervention group controls for time-varying confounders.

Introduction

Tackling backlogs for outpatient care has become a priority for many public healthcare systems in the aftermath of the COVID-19 pandemic. Multiple initiatives to optimise the use of outpatient care have been reported, including telemedicine-based strategies, referral programmes and patient-initiated follow-up.1–4 Follow-up appointments account for up to 66% of outpatient visits,5 and strategies aimed at reducing unnecessary follow-up may decrease backlogs, improve patient experience and attendance rates.6–8

The definition of ‘low-value care’ includes medical services that provide little or no clinical benefit or may even cause harm to patients.9 In line with this definition and initiatives such as the ‘Choosing Wisely’ campaign,10 reducing avoidable outpatient appointments has implications for reducing backlogs and for improving value-related aspects of care, such as better distribution of healthcare resources, improved access to specialist care and reduced disruption to patients’ daily routines, thus improving quality of life. The use of technological support in clinical decision-making has shown potential to increase appropriateness of test orders and reduce low-value care.11 12 However, to the best of our knowledge, large-scale implementation of algorithms for placing automatic test orders prior to outpatient appointments has not been reported.

A network of four teaching hospitals in Madrid, Spain, launched an algorithm for placing automatic standardised test orders on referral from primary to specialist outpatient care in May 2021. Since then, automatic test orders have been placed for 1 175 814 first appointments (FA) from 16 different specialties. We hypothesised that the use of the algorithm would lead to an increase in the number of patients receiving diagnosis and treatment plans at the FA, with higher rates of discharge back to primary care and a reduction in low-value follow-up appointments and ‘did not attend’ rates.

Methods

Design, participants and care setting

We used an interrupted time series (ITS) analysis quasiexperimental design to study changes in outpatient discharge rates, follow-up appointments and attendance rates after the implementation of an algorithm placing automatic standardised test orders for patients attending first outpatient appointments. Implementation took place in May 2021. Monthly observational data from January 2018 to June 2023 gathered as part of routine hospital management processes were included. We included a reference group comprising three specialties which did not participate in the programme to control for other external changes occurring during the study period.

Centres participating in the intervention included four teaching hospitals (Fundación Jiménez Díaz University Hospital, Infanta Elena University Hospital, General Villalba University Hospital and Rey Juan Carlos University Hospital) from the Quirónsalud 4-H Network, a publicly funded, privately run healthcare network in Madrid, Spain. The network’s catchment area covers over 1 million inhabitants. Each year, more than 500 000 first specialist appointments take place at the four hospitals’ outpatient clinics, which have seen a steady increase in activity over the last decade.

The Spanish NHS relies heavily on primary care physicians (PCPs) as the first port of call for patient care. While PCPs can refer patients to specialists without restrictions, they have limited access to diagnostic tests, which are usually ordered by specialists and performed in hospital facilities. Often, after anamnesis and physical examination, PCPs will refer patients to specialist care for an FA with a referral note requesting confirmation of a diagnostic suspicion. Specialists must reperform anamnesis and physical examination, and then order diagnostic tests to confirm or rule out a diagnosis; patients must wait until an appointment is available for tests to be performed, and then return to the hospital outpatient clinic for a follow-up appointment to be informed of the diagnosis and treatment plan and potentially discharged back to primary care, thus ‘closing the loop’ between primary and specialty care.13 For the four centres participating in the intervention, around 80% of specialist referrals come from PCPs (other specialists from the network account for 15% of referrals, and referrals from the hospitals’ emergency departments make up the remaining 5%). Additionally, adherence to current guidelines for diagnostic workup is often poor, leading to incomplete test results in some cases and inappropriate testing in others,14 and potentially causing diagnostic delays and unnecessary follow-up appointments.

In 2019, faced with specialist backlogs due to the increasing demand, members of hospital management and clinical leaders began to design a data-driven initiative to improve the rate of confirmed/ruled-out diagnoses at FA through implementation of automatic test orders before FA. In line with other international initiatives for best practice in healthcare stewardship,15 16 we hypothesised that if all necessary diagnostic tests could be carried out prior to FA, we would be able to increase the number of confirmed/ruled-out diagnoses and treatment plans made at FA, leading to higher outpatient discharge rates and reducing low-value follow-up appointments and ‘did not attend’ rates (figure 1). Due to the onset of the COVID-19 pandemic, large-scale launch of the initiative was delayed until May 2021.

Figure 1
Figure 1

Process map depicting current specialty outpatient care workflow and targeted area for improvement (shaded in green) to reduce unnecessary follow-up appointments.

Data sharing

Technical appendix, statistical code and data set are available from the last author (JSA).

Patient and public involvement

No patients or members of the public were involved in the design of this initiative, although patient quality of life was considered by aiming to reduce the number of unnecessary outpatient visits. Patient experience in outpatient care is recorded as part of routine quality monitoring using net promoter scores (NPS). However, specific NPS campaigns to evaluate patient experience of this new approach to delivering outpatient care have yet to be launched.

Intervention

We developed a decision-making algorithm for placing automatic standardised test order sets prior to FA using natural language processing (NLP) (figure 2). Protocols for standardised test order sets were developed for different presenting complaints from 16 clinical specialties, including cardiology, general surgery, maxillofacial surgery, dermatology, gastroenterology, endocrinology, gynaecology, clinical haematology, internal medicine, nephrology, pulmonology, neurology, rehabilitation, rheumatology, orthopaedic surgery and urology. Specialties were chosen based on volume of patients in outpatient care, follow-up appointment rates and scope for protocolising tests prior to the FA.

Figure 2
Figure 2

Process map showing the process of referral from primary care to specialty outpatient care before and after the implementation of the algorithm. PCP, primary care physician; SCAE, Solicitud de Cita en Atención Especializada (Appointment Request for Specialized Care).

A multidisciplinary team, including hospital managers, hospital-based specialist physicians, PCPs (data scientists), administrative staff, radiologists and laboratory staff, was formed to design and implement the project.

A data-driven approach to standardised test order sets

Protocols for automatic standardised test order sets were developed using a data-driven approach. We analysed outpatient appointments for 16 clinical specialties over a 12-month period using diagnostic coding to group diagnoses by their most common presenting complaints (signs and symptoms). Once a list of the most frequent presenting complaints and their corresponding diagnoses was compiled, we analysed test orders placed by specialists for each complaint. Test orders placed for an individual presenting complaint in more than 90% of appointments were considered for inclusion. We then designed 257 protocols for standardised test order sets prior to FA, based on presenting complaint and/or suspected diagnosis, demographic variables such as age and sex, and clinical variables such as previous injury, associated symptoms and comorbidity (figure 3). Each protocol was then sent to designated experts from the corresponding clinical specialty for review, modification and final validation. Protocols are periodically reviewed by laboratory staff, radiology staff and clinical leaders, with several modifications being made since the project was first implemented.

Figure 3
Figure 3

Example of a standardised test order set for a patient referred from primary care to a first cardiology outpatient appointment for suspected atrial fibrillation.

Algorithm implementation and administrative workflow

In Madrid, Spain, PCP petitions for specialist referral are placed electronically and sent to a centralised application (Solicitud de Cita en Atención Especializada). Before implementing the project, as other hospital networks, administrative staff from the participating centres booked FAs on a first-come, first-served basis (with some exceptions, such as suspicion of malignancy17) in which the content of the referral note is not taken into account. To implement the algorithm, we developed an NLP pipeline to perform automatic analysis of each referral note. The NLP pipeline was designed following a deterministic model: we formed monographic teams to decide which clinical terms from referral notes should be assigned to each prior test order protocol, and then extracted data from the referral database to feed an automatic annotator, creating specific dictionaries for each specialty which were then validated by clinical experts. Referral notes are analysed by the algorithm, which places standardised test order sets automatically.

To offer reliable and explainable results, the classification logic was implemented based on text labelling, considering predefined clinical criteria. This strategy allowed us to define rules based on standardised concepts adaptable to the needs of the project, for example, including new criteria through rule modification without having to relabel the original texts. A cross-protocol priority matrix was implemented for each patient. The results of the classification are reviewed periodically, considering false positives (which are unusual due to the deterministic nature of the process) and false negatives, which serve to improve the dictionary. On the other hand, the study of unclassified cases helps identify possible categories that would be interesting to add to the project.

After classification, administrative staff contact patients to schedule test orders and FA, ensuring that all test results are available for specialist review before the appointment takes place. The number of administrative staff assigned to the project is proportional to overall outpatient activity; as of July 2023, a total of 31 full-time administrative staff have been reassigned from general administrative workflow. To streamline the FA coordination process, we assigned designated administrative personnel to a ‘contact centre’. These workers are given specific training regarding situations such as those test orders requiring informed consent and unclassified referral notes. For protocols including tests which pose a risk of potentially serious adverse effects, such as imaging tests with intravenous contrast, contact centre staff flag the patients for a brief call with a clinician, who provides information, answers doubts and obtains informed consent. When referral notes do not include sufficient information to be classified by the NLP algorithm, contact centre staff call the patient and perform a brief survey following predetermined scripts to determine if a protocol for test order sets can be applied. Clinical experts are available for consultation in those cases in which administrative staff have doubts regarding choice of protocol. PCPs were informed of the initiative before and during the implementation period through online meetings coordinated by the network’s care continuity team. Information focused on clinical and operational aspects of the quality improvement project, although members of the information technology department were available to answer technical questions that could arise.

Measurements

The primary endpoint was defined as the trend in follow-up outpatient appointment rates. To evaluate this outcome, we designed two composite variables: the rate of patients receiving diagnosis and treatment plans at FA with discharge back to primary care, and the follow-up to FA index. The rate of discharge to primary care was calculated as the number of patients with a diagnosis or treatment plan discharged back to primary care at the first outpatient appointment divided by the total number of FAs, multiplied by 100. The follow-up to FA index was calculated by dividing the overall number of follow-up appointments by FAs. The secondary endpoint was defined as the trend in ‘did not attend’ rates, calculated by dividing the overall number of ‘did not attends’ by the number of follow-up appointments, multiplied by 100. Data were gathered at monthly intervals as part of routine hospital management processes, which enabled us to implement statistical control for seasonality.

Additional endpoints were included based on data which did not form part of routine quality monitoring. The percentage of patients declining to participate in the new model of healthcare delivery was collected for the intervention period. Also, as a potential adverse effect of our initiative could be an increase in test order rates, we collected the number of overall test orders for the following categories: laboratory test orders (including microbiological tests), imaging test orders (plain X-rays, MRI scans and CT scans) and electromyogram orders. These data were recorded on a semestral basis, and so we compared the first semester of the pre-implementation period (January 2018 to June 2018) and the last semester of the postintervention period (January 2023 to June 2023) to correct for seasonality. To compensate for fluctuations in the number of outpatient appointments, we measured changes in test order rates using a composite indicator, the test order to outpatient appointment index (total number of test orders for a given period divided by total number of outpatient appointments for a given period, multiplied by 100).

Analysis

We performed ITS analyses for both primary and secondary endpoints using the method described by Penfold and Zhang.18 We set the interruption on 1 May 2021, coinciding with the large-scale implementation of the algorithm across all 16 specialties. Due to COVID-19-related disruptions in normal outpatient activity, with suspension of all unnecessary elective care including the majority of first outpatient appointments, we censored the period from March 2020 to March 2021, which accounted for the first three waves of the pandemic in Spain.19 20 After censoring the COVID-19 period, our ITS analysis included a total of 52 time points (26 preintervention and 26 postintervention). Each ITS model included a linear monthly trend term for the preintervention period (‘Time’), a binary indicator variable for the months after algorithm implementation (‘Intervention’), capturing any level change in the outcome postintervention, and a linear monthly trend term only for the post implementation period (‘Time-After’). These three terms allowed us to evaluate changes in the level and slope of the outcome after intervention. The 12-month COVID-19 period was censored from analysis by coding the intervention variable as missing during these months. The model was estimated through conditional maximum likelihood estimation using the AutoReg function from the package StatsModels v0.14.0,21 and p values and 95% CIs were calculated using the same package. The number of autocorrelation parameters was reduced using backward elimination to fit the most parsimonious model. Goodness of fit was assessed using the Akaike information criteria score and by visual inspection of residuals around predicted regression lines (standardised residual, histogram plus estimated density and normal Q-Q). We also graphed differences between the pre-implementation and post implementation periods using linear regression to present counterfactual effect. We imputed changes in number of patients discharged to primary care with a diagnosis or treatment plan and ‘did not attend’ rates during the post implementation period using the time series coefficient for each statistically significant change in trend, dividing by the total number of time points in the post implementation period to calculate absolute differences per month. To estimate attributable change in low-value follow-up appointments for the intervention group, we simulated the effect of ‘non-intervention’ by extrapolating the significant change in trend from the control group to postintervention data from the intervention group. To infer the number of avoided appointments in the intervention group, we then subtracted the observed number of patients in the postintervention period from the number of patients in the simulated ‘non-intervention’ group. To calculate absolute differences per month, we divided the result by the total number of time points in the postintervention group. The non-statistically significant improvement observed the implementation group was not taken into account when imputing attributable reduction in follow-up appointments to avoid overestimating the effects of the project implementation. After the first three waves of the COVID-19 pandemic, normal elective care was resumed, with a tendency towards increased virtual care delivery and reduced in-person visits where possible.22 To control for time-varying confounding due to these and other external factors, we included a reference group comprising three specialties (allergology, cardiovascular surgery and ophthalmology) which were not included in the algorithm and were subject to the same external factors as the study group specialties. Data for this group were collected during the same period as the study group, and trends in primary and secondary outcomes were analysed using the same method.

Results

A total of 10 033 470 outpatient appointments took place during the 6-year study period, of which 7 326 548 were follow-up appointments and 2 706 922 were FAs. Analysis of the overall series showed a steady increase in FAs during 2018–2019 (online supplemental figure 1). The COVID-19 pandemic (2020–2021) caused a disruption in outpatient activity, with a return to prepandemic levels in 2022 and an increase in outpatient appointments above prepandemic levels during 2023. Since the implementation of the algorithm in May 2021 to June 2023, automatic test orders have been placed for 1 175 814 FAs from 16 different specialties, and 282 692 patients (26.11%) have been diagnosed and discharged back to primary care with appropriate treatment plans at FA, compared with 226 291 patients (20.51%) in the preintervention period.

Figure 4 presents the observed time series for primary and secondary outcomes, with fitted linear regression lines showing preintervention and postintervention trends and the extrapolated preintervention trends demonstrating counterfactual differences. Statistically significant improvements due to algorithm implementation were observed for endpoints in the study group compared with the reference group regarding diagnosis and discharge rates (1.7%, p=0.005, 95% CI 0.5% to 2.9%) and ‘did not attend’ rates (−0.4%, p=0.044, 95% CI −0.1% to −0.8%). Using the change in trend attributable to the algorithm, we computed the absolute increase in patients discharged to primary care with a diagnosis and/or treatment plan during the post implementation period as 19 989 appointments (769 per month). The absolute reduction of ‘did not attends’ attributable to the algorithm was calculated as 26 888, signifying a reduction of 1034 ‘did not attends’ per month The follow-up to FA index presented a non-significant reduction (improvement) in the study group and showed a significant increasing trend in the control group (0.25, p=0.007, 95% CI 0.07 to 0.44). We estimated an attributable reduction of 293 954 follow-up appointments during the intervention period, signifying a reduction of 11 306 unnecessary follow-up appointments per month.

Figure 4
Figure 4

Interrupted time series analyses evaluating the effect of an algorithm for placing automatic standardised test order sets prior to first specialist appointment on low-value follow-up appointments and attendance rates. For each plot, the black circles are the raw data for each month, the grey circles are the censored data from the first three waves of the COVID-19 pandemic, the solid black lines are the fitted regression lines for pre-implementation and post-implementation periods, and the dashed line is the projected trend (counterfactual) assuming there was no intervention. Rates of patients receiving diagnosis and treatment plans at first appointment with discharge back to primary care is depicted in panel A (intervention group) and panel B (control group). Changes in the follow-up to first appointment index are presented in panel C (intervention group) and panel D (control group). Changes in ‘did not attend’ rates are depicted in panel E (intervention group) and panel F (control group).

Regarding additional endpoints, less than 1% of patients declined to participate in the new model of care delivery. We also compared overall test order rates between the first and last 6-month intervals of the study period using data from the first half of the year in both cases to correct for seasonality. Overall, test order rates decreased by 2.67. Test order rates for plain X-rays showed a slight reduction between 2018 and 2023, as did electromyogram and laboratory test order rates, with a slight increase in MRI scan and CT scan order rates (table 1).

View this table:
Table 1

Test order/outpatient appointment index for the first and last semesters of the study period (January to June 2018 and January to June 2023)

Discussion

This study presents the results of an ITS analysing the effects of an algorithm placing automatic standardised test order sets prior to first outpatient appointments. During the study period, we found a significant increase in the number of patients diagnosed and discharged back to primary care with appropriate treatment plans at FA. An estimated attributable reduction of 11 306 avoidable follow-up appointments per month was observed. A significant reduction in ‘did not attend’ rates was also observed, with an estimated attributable average reduction of 1034 ‘did not attends’ per month. Overall test order rates did not increase during the study period, and less than 1% of patients declined to participate in the new model of healthcare delivery.

One of the major strengths of our study is the use of ITS, the most robust quasiexperimental methodology when randomisation is not possible. The inclusion of a reference group controls for the influence of external factors which could bias our results. The extended study period (6 years) guarantees sufficient data points to provide a robust analysis. Limitations of the study include the absence of patient experience data, since this variable was not included specifically as part of routine data collection, and the lack of patient participation during algorithm design. However, the fact that less than 1% of patients declined to participate in the new care model points to high patient acceptance. We believe that by reducing time to diagnosis through the implementation of automatic standardised test order sets, patients receive treatment earlier, improving health and experience. Another limitation is the fact that, as the number of test orders was not included as part of routine data collection, we were unable to include this variable as part of the primary analysis. However, we were able to correct for seasonality by including the first semester of each year from the preintervention and postintervention periods and also corrected for the number of outpatient appointments in each period. A slight increase was observed in CT scan and MRI scan orders (+0.89% and +1.32%, respectively). This may be due, in part, to the increased complexity of patients referred to our four centres during the study period. However, we cannot exclude the possibility that this increase may be partially due to the implementation of protocolised test order sets. No increase in intravenous contrast-related complications was notified during the study period. Finally, both clinicians and patients were aware of the project implementation, which potentially may have led to clinician bias when discharging patients to primary care.

Process standardisation and automation in healthcare has been demonstrated to improve quality of care,23 24 patient safety,25 26 patient experience27 28 and healthcare savings.29–31 Protocols for test orders during diagnostic workup are common, with many clinical guidelines featuring algorithms for confirming or ruling out clinical conditions based on presenting signs and symptoms, along with a series of other variables such as duration of symptoms, comorbidities, age and sex.32–35 However, reports of fully automated test order sets in clinical practice are few and are mostly experimental and limited to emergency care.36–38 These studies indicate that automated protocols for test ordering may improve metrics such as emergency room length of stay but may increase overall test costs. In our report, test order rates for specialist outpatient appointments have shown a slight overall decrease between 2018 and 2023, indicating that the implementation of the algorithm has not caused a rise in overall test orders and their associated costs. To the best of our knowledge, this is the first report of large-scale standardisation and automation of test order sets for common presenting complaints prior to FA. Our results demonstrate an increase in diagnosis and initiation of treatment at FA, pointing to scope for improvement regarding appointment and clinician time-related opportunity costs. On a practical level, this does translate as improved use of resources, reduced backlog for specialist appointments and cost savings and improved patient experience due to time savings and less travel.

Recent publications highlight the importance of minimising unnecessary time spent at healthcare facilities to improve patient experience, especially in patients with chronic conditions.28 Unnecessary healthcare interventions are a global concern, with many international and national initiatives underway to reinforce appropriate use of health resources.4 Our project is in line with others such as the ‘Choosing Wisely Campaign’, which seeks to improve healthcare by reducing those medical interventions which do not add value for patients or may even cause harm.15 The use of technology including artificial intelligence-based approaches to standardise and automatise test orders has potential to reduce low-value care and should be considered when seeking to optimise care.

Sustainability is a current priority for healthcare policymakers and reducing backlogs for elective care is a challenge that both public and private healthcare systems face in the aftermath of the COVID-19 pandemic. Given the current shortage of healthcare professionals, optimal distribution of resources such as specialty care is vital to ensure sustainable systems. Our initiative demonstrates an increase in discharge back to primary care and a reduction in ‘did not attend’ rates. The 11 306 extra appointment slots per month gained by avoiding unnecessary follow-up appointments are used entirely for FAs, increasing access to timely care. The four centres participating in this project currently present an overall time from referral to FA of 11.88 days (±0.89), compared with an average of 60.08 (±19.22) for other non-participating public centres in Madrid, Spain.

Given the positive impact the project appears to have had on patients and service provision, we have begun to implement the initiative in the remaining 43 Spanish hospitals belonging to the Quirónsalud network. The main difficulties for implementation included technical issues regarding algorithm integration with underlying software which have been solved with the network-wide adoption of the Casiopea electronic health record. On the other hand, the implementation team has organised various meetings between clinical experts from the four original centres participating in the initiative and healthcare professionals from the other 43 hospitals, in order to answer questions and overcome resistance to change (a common obstacle for digital transformation and quality improvement initiatives39 40).

Our study opens the door to several areas for future research, including the effect of the algorithm on the earlier diagnosis of serious conditions such as cancer, and the way that healthcare professionals and patients perceive this new form of delivering healthcare. Further studies are being planned to answer these questions.

Conclusion

An algorithm for placing automatic standardised test order sets reduces low-value follow-up appointments by allowing specialists to confirm diagnoses and initiate treatment at FA. The algorithm also reduces ‘did not attend’ rates. This initiative points to an improved process for outpatient diagnosis and treatment, delivering healthcare more effectively and efficiently. Further studies are necessary to explore aspects such as patient and healthcare professional experience with the new model of outpatient healthcare delivery.

Data availability statement

Data are available upon reasonable request. Technical appendix, statistical code and data set are available from the last author (JSA) upon reasonable request.

Ethics statements

Patient consent for publication

Ethics approval

The project used only non-identified data and, as part of a quality improvement initiative, did not require approval from the Fundación Jiménez Díaz Institutional Ethics Committee. Although informed consent was not formally required due to the nature of the project, verbal informed consent was obtained from participating patients. Standards for Quality Improvement Reporting Excellence (SQUIRE) 2.0 reporting guidelines were followed when writing the report.

Acknowledgments

We thank Antonio Herrero from the Quirónsalud 4-H Network’s Big Data Department for his contributions to the project, and Juan José Serrano from the Quirónsalud 4-H Network’s BI department for help with the data collection process. We thank Marco Antonio Villegas García, PhD, for his help with data analysis. We also thank the members of the network’s cardiology, general surgery, maxillofacial surgery, dermatology, gastroenterology, endocrinology, gynaecology, clinical haematology, internal medicine, nephrology, pulmonology, neurology, rehabilitation, rheumatology, orthopaedic surgery and urology departments for reviewing test order protocols and providing clinical support. We acknowledge the members of the network’s admissions department and administrative staff for their role in the project’s success.

References

  1. ↵
    1. Abu-Ramadan MA
    . Making better use of scarce resources: the palestinian experience, 1995-1999. J Ambul Care Manage 2002;25:63–9. doi:10.1097/00004479-200207000-00009
    OpenUrlPubMed
  2. ↵
    1. Akbari A,
    2. Mayhew A,
    3. Al-Alawi MA, et al
    . Interventions to improve outpatient referrals from primary care to secondary care. Cochrane Database Syst Rev 2008;2008:CD005471. doi:10.1002/14651858.CD005471.pub2
  3. ↵
    1. Jaatinen PT,
    2. Aarnio P,
    3. Remes J, et al
    . Teleconsultation as a replacement for referral to an outpatient clinic. J Telemed Telecare 2002;8:102–6. doi:10.1258/1357633021937550
    OpenUrlCrossRefPubMed
  4. ↵
    1. Kühlein T,
    2. Macdonald H,
    3. Kramer B, et al
    . Overdiagnosis and too much medicine in a world of crises. BMJ 2023;382:1865. doi:10.1136/bmj.p1865
    OpenUrl
  5. ↵
    1. Institute for Government
    . Performance tracker 2022/23: spring update - hospitals. Available: https://www.instituteforgovernment.org.uk/performance-tracker-2022-23/hospitals [Accessed 11 Aug 2023].
  6. ↵
    1. McLean S,
    2. Gee M,
    3. Booth A, et al
    . Patterns and influences on health-care attendance behaviour: a narrative overview of key themes and issues. 2014. Available: https://www.ncbi.nlm.nih.gov/books/NBK260108/
  7. ↵
    1. Newton C,
    2. Beaver K,
    3. Clegg A
    . Patient initiated follow-up in cancer patients: a systematic review. Front Oncol 2022;12:954854. doi:10.3389/fonc.2022.954854
  8. ↵
    1. Whear R,
    2. Thompson-Coon J,
    3. Rogers M, et al
    . Patient-initiated appointment systems for adults with chronic conditions in secondary care. Cochrane Database Syst Rev 2020;4:CD010763. doi:10.1002/14651858.CD010763.pub2
  9. ↵
    1. Sorenson C,
    2. Japinga M,
    3. Crook H, et al
    . Building A better health care system post-COVID-19: steps for reducing low-value and wasteful care. NEJM Catal 2020;1. doi:10.1056/CAT.20.0368
  10. ↵
    1. Cassel CK,
    2. Guest JA
    . Choosing wisely: helping physicians and patients make smart decisions about their care. JAMA 2012;307:1801–2. doi:10.1001/jama.2012.476
    OpenUrlCrossRefPubMedWeb of Science
  11. ↵
    1. Heekin AM,
    2. Kontor J,
    3. Sax HC, et al
    . Choosing wisely clinical decision support adherence and associated inpatient outcomes. Am J Manag Care 2018;24:361–6.
    OpenUrl
  12. ↵
    1. Doyle J,
    2. Abraham S,
    3. Feeney L, et al
    . Clinical decision support for high-cost imaging: a randomized clinical trial. PLoS One 2019;14:e0213373. doi:10.1371/journal.pone.0213373
  13. ↵
    1. Patel MP,
    2. Schettini P,
    3. O’Leary CP, et al
    . Closing the referral loop: an analysis of primary care referrals to specialists in a large health system. J Gen Intern Med 2018;33:715–21. doi:10.1007/s11606-018-4392-z
    OpenUrlCrossRefPubMed
  14. ↵
    1. O’Sullivan JW,
    2. Albasri A,
    3. Nicholson BD, et al
    . Overtesting and Undertesting in primary care: a systematic review and meta-analysis. BMJ Open 2018;8:e018557. doi:10.1136/bmjopen-2017-018557
  15. ↵
    1. Levinson W,
    2. Kallewaard M,
    3. Bhatia RS, et al
    . 'Choosing wisely’: a growing international campaign. BMJ Qual Saf 2015;24:167–74. doi:10.1136/bmjqs-2014-003821
    OpenUrlAbstract/FREE Full Text
  16. ↵
    1. van Dulmen SA,
    2. Tran NH,
    3. Wiersma T, et al
    . Identifying and prioritizing do-not-do recommendations in dutch primary care. BMC Prim Care 2022;23:141. doi:10.1186/s12875-022-01713-y
    OpenUrl
  17. ↵
    1. Arroyo Hernández M,
    2. Casas Ramos P,
    3. Saldaña Díaz A, et al
    . The computer application «SCAE» in the early diagnosis of musculoskeletal cancer in the healthcare area of the hospital universitario 12 de octubre. Rev Esp Cir Ortop Traumatol 2022;66:290–7. doi:10.1016/j.recot.2021.05.006
    OpenUrl
  18. ↵
    1. Penfold RB,
    2. Zhang F
    . Use of interrupted time series analysis in evaluating health care quality improvements. Acad Pediatr 2013;13:S38–44. doi:10.1016/j.acap.2013.08.002
    OpenUrlCrossRefPubMed
  19. ↵
    1. Soriano V,
    2. Ganado-Pinilla P,
    3. Sanchez-Santos M, et al
    . Main differences between the first and second waves of COVID-19 in Madrid, Spain. Int J Infect Dis 2021;105:374–6. doi:10.1016/j.ijid.2021.02.115
    OpenUrlCrossRefPubMed
  20. ↵
    1. Soriano V,
    2. de Mendoza C,
    3. Gómez-Gallego F, et al
    . Third wave of COVID-19 in Madrid, Spain. Int J Infect Dis 2021;107:212–4. doi:10.1016/j.ijid.2021.04.074
    OpenUrl
  21. ↵
    1. Seabold S,
    2. Perktold J
    . Statsmodels: econometric and statistical modeling with python. Proceedings of the 9th python in science conference; 2010:92–6
  22. ↵
    1. Short Apellaniz J,
    2. Álvaro de la Parra JA,
    3. Gomez-Meana A, et al
    . Leveraging telemedicine to reduce ED overcrowding: the quirónsalud virtual urgent care program. NEJM Catalyst 2023;4. doi:10.1056/CAT.22.0422
  23. ↵
    1. Lavelle J,
    2. Schast A,
    3. Keren R
    . Standardizing care processes and improving quality using pathways and continuous quality improvement. Curr Treat Options Peds 2015;1:347–58. doi:10.1007/s40746-015-0026-4
    OpenUrl
  24. ↵
    1. Bozic KJ,
    2. Maselli J,
    3. Pekow PS, et al
    . The influence of procedure volumes and standardization of care on quality and efficiency in total joint replacement surgery. J Bone Joint Surg Am 2010;92:2643–52. doi:10.2106/JBJS.I.01477
    OpenUrlAbstract/FREE Full Text
  25. ↵
    1. Rozich JD,
    2. Howard RJ,
    3. Justeson JM, et al
    . Standardization as a mechanism to improve safety in health care. Jt Comm J Qual Saf 2004;30:5–14. doi:10.1016/s1549-3741(04)30001-8
    OpenUrlPubMed
  26. ↵
    1. Leotsakos A,
    2. Zheng H,
    3. Croteau R, et al
    . Standardization in patient safety: the WHO high 5S project. Int J Qual Health Care 2014;26:109–16. doi:10.1093/intqhc/mzu010
    OpenUrlCrossRefPubMed
  27. ↵
    1. Queenan CC,
    2. Angst CM,
    3. Devaraj S
    . Doctors’ Orders—If they’re electronic, do they improve patient satisfaction? A complements/substitutes perspective. J of Ops Management 2011;29:639–49. doi:10.1016/j.jom.2011.03.001
    OpenUrl
  28. ↵
    1. Caramés Sánchez C,
    2. Álvaro de la Parra JA,
    3. Dómine M, et al
    . The HOPE project: improving cancer patient experience and clinical outcomes through an integrated practice unit and digital transformation. NEJM Catal 2023;4. doi:10.1056/CAT.22.0414/ASSET/23A3EACC-D7C3-4872-88EF-AFBB3B1757E8/ASSETS/GRAPHIC/CAT.22.0414-HERO.JPG
  29. ↵
    1. Guzman MJ,
    2. Gitelis ME,
    3. Linn JG, et al
    . A model of cost reduction and standardization: improved cost savings while maintaining the quality of care. Dis Colon Rectum 2015;58:1104–7. doi:10.1097/DCR.0000000000000463
    OpenUrlPubMed
  30. ↵
    1. Friedman KG,
    2. Fulton DR
    . Reducing cost through standardization. Curr Treat Options Peds 2016;2:296–310. doi:10.1007/s40746-016-0068-2
    OpenUrl
  31. ↵
    1. Healy WL,
    2. Kirven FM,
    3. Iorio R, et al
    . Implant standardization for total hip Arthroplasty: an implant selection and a cost reduction program. J Arthroplasty 1995;10:177–83. doi:10.1016/s0883-5403(05)80124-x
    OpenUrlCrossRefPubMed
  32. ↵
    1. Usman OA,
    2. Usman AA,
    3. Ward MA
    . Comparison of SIRS, qSOFA, and NEWS for the early identification of sepsis in the emergency department. Am J Emerg Med 2019;37:1490–7. doi:10.1016/j.ajem.2018.10.058
    OpenUrlCrossRefPubMed
  33. ↵
    1. Lerner LB,
    2. McVary KT,
    3. Barry MJ, et al
    . Management of lower urinary tract symptoms attributed to benign prostatic hyperplasia: AUA GUIDELINE PART I—initial work-up and medical management. J Urol 2021;206:806–17. doi:10.1097/JU.0000000000002183
    OpenUrlCrossRef
  34. ↵
    1. Mourad O,
    2. Palda V,
    3. Detsky AS
    . A comprehensive evidence-based approach to fever of unknown origin. Arch Intern Med 2003;163:545–51. doi:10.1001/archinte.163.5.545
    OpenUrlCrossRefPubMedWeb of Science
  35. ↵
    1. Osmon DR,
    2. Berbari EF,
    3. Berendt AR, et al
    . Executive summary: diagnosis and management of prosthetic joint infection: clinical practice guidelines by the infectious diseases society of America. Clin Infect Dis 2013;56:1–10. doi:10.1093/cid/cis966
    OpenUrlCrossRefPubMed
  36. ↵
    1. Baum A,
    2. Carr BG,
    3. Perman SM, et al
    . Advanced triage protocol: the role of an automated lactate order in expediting rapid identification of patients at risk of sepsis in the emergency department. Crit Care Explor 2022;4:e0736. doi:10.1097/CCE.0000000000000736
  37. ↵
    1. Hunter-Zinck HS,
    2. Peck JS,
    3. Strout TD, et al
    . Predicting emergency department orders with multilabel machine learning techniques and simulating effects on length of stay. J Am Med Inform Assoc 2019;26:1427–36. doi:10.1093/jamia/ocz171
    OpenUrl
  38. ↵
    1. Singh D,
    2. Nagaraj S,
    3. Mashouri P, et al
    . Achieving clinical automation in emergency medicine with machine learning medical directives. SSRN Journal 2021;12. doi:10.2139/ssrn.3857668
  39. ↵
    1. Iyanna S,
    2. Kaur P,
    3. Ractham P, et al
    . Digital transformation of healthcare sector. What is impeding adoption and continued usage of technology-driven innovations by end-users. J Bus Res 2022;153:150–61. doi:10.1016/j.jbusres.2022.08.007
    OpenUrlCrossRef
  40. ↵
    1. Shekelle PG
    . Why don’t physicians enthusiastically support quality improvement programmes Qual Saf Health Care 2002;11:6. doi:10.1136/qhc.11.1.6
    OpenUrlFREE Full Text

Supplementary materials

  • Supplementary Data

    This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

Footnotes

  • Twitter @ShortJorge

  • Contributors JAÁdlP, MdOR, CCS, ÁB, SM-A, JF-F, EC, ÓGM, JFT, CPN, CN and JSA were responsible for the conception and design of the project. BP, MdOR and JSA were responsible for data interpretation. BP, MdOR and JSA were responsible for drafting the work. All other authors reviewed it critically for important intellectual content. JSA is the guarantor of this work and accepts full responsibility for the work, had full access to the data, and was responsible for the decision to publish. All authors gave their final approval of the version to be published and agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

  • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

  • Competing interests All authors declare being employees of the Quirónsalud Hospital Network.

  • Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.