Effects of virtual reality OSCE on nursing students’ education: a study protocol for systematic review and meta-analysis
==========================================================================================================================

* Ping Liu
* Xuan Dong
* Fei Liu
* Haixia Fu

## Abstract

**Introduction** Virtual objective structured clinical examination (OSCE) has been shown to influence the performance of nursing students. However, its specific effects, particularly students’ competence, stress, anxiety, confidence, satisfaction with virtual reality OSCE and examiners’ satisfaction, remain unclear.

**Method and analysis** This study aims to assess the effects of virtual reality OSCE on nursing students’ education. The study follows the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocol guidelines. A literature search is performed on electronic databases, namely, PubMed, Web of Science, CINAHL, EBSCO, EMBASE and the Cochrane Library. The inclusion criteria adhere to the PICOS principle, encompassing nursing students, including those studying in school and those engaged in hospital internship. This review includes studies on the use of virtual reality OSCE as an assessment tool, compared with traditional clinical examinations, such as in-person OSCE. The outcome assessments encompass (1) competence, (2) stress, (3) anxiety, (4) confidence, (5) student satisfaction with virtual reality OSCE and (6) examiners’ satisfaction. These studies are designed as randomised controlled trials (RCTs) or quasi-experimental research. The search time is from the inception of each database to 30 June 2023, without language restriction. Studies for inclusion are screened by two reviewers for data extraction dependently. Any dispute is resolved through discussion. Unresolved disputes are decided by consulting a third author. For the risk of bias (ROB) assessment, the Cochrane ROB tool for RCTs and the risk of bias in non-randomised studies of intervention tool are used. Moreover, RevMan V.5.3 is used for meta-analysis.

**Ethics and dissemination** This study protocol does not include any clinical research and thus does not require ethical approval. Research findings are published in a peer-reviewed journal.

**PROSPERO registration number** CRD42023437685.

*   Virtual Reality
*   Nurses
*   MEDICAL EDUCATION & TRAINING

### STRENGTHS AND LIMITATIONS OF THIS STUDY

*   We strictly follow the recommendations of the Cochrane handbook for systematic reviews of interventions to ensure a rigorous research process.

*   The search algorithm will be developed by an experienced librarian to ensure the comprehensiveness of the literature retrieval and processing.

*   The limited quantity and quality of the original research may reduce the reliability of the evidence.

## Introduction

The objective structured clinical examination (OSCE) provides an objective, orderly and organised assessment framework, in which medical schools, hospitals, or medical or examination institutions can add corresponding assessment contents and methods according to their teaching and examination syllabus.1 This method tests the clinical abilities of nurses or nursing students by simulating clinical scenarios.2 It is also a clinical ability assessment method that emphasises knowledge, skills and attitude.3 Candidates conduct practical tests through a series of predesigned exam stations, including standardised patients (SP), practical operations on medical simulators, collection of clinical data and document retrieval.4 The exam station is divided into long and short stations, with a duration ranging from 5 min to 20 min; candidates are evaluated by the examiner or SP.5 

However, given that the OSCE requires a person-in-person offline operation, some objective factors, such as the restrictions of the COVID-19 epidemic some time ago,6–8 the development of virtual reality (VR) technology in the field of nursing education,9–11 and the increasingly popular cross-regional and multicentre joint training,12 13 exist. As a result, the traditional offline OSCE cannot satisfy the requirements of modern nursing education.

VR OSCE refers to the implementation of traditional OSCE on VR devices and the use of VR technology.14 More applications are developed in the fields of medical and nursing education.15–17 Compared with traditional OSCE, VR OSCE has some significant advantages. It can be carried out without physical distance limitations, thereby allowing participation from long distances, multiple locations and simultaneous engagement, making it highly accessible.18 Modern VR devices are more popular among young people.19 Meanwhile, research has shown that these devices can increase participants’ confidence, allowing them to perform better in exams.20 With the rapid development of VR OSCE, its application in the field of nursing education is gradually increasing, playing an important and irreplaceable role in the assessment of nursing students.21 22 

Nevertheless, the effects of VR OSCE as an assessment method for nursing students are controversial. Some studies showed that VR OSCE can improve confidence and competence among nursing students,23 24 whereas others presented no significant growth.17 Although VR OSCE demonstrates potential in improving the assessment of nursing students, adequate evidence confirming the effects of VR OSCE as an assessment method for nursing students is lacking.

To the best of our knowledge, a meta-analysis of the effects of VR OSCE on the education performance of nursing students has not yet been carried out. A systematic review23 has reported the implementation of VR OSCE, strengthening confidence in the virtual environment. However, their study population includes health professionals, rather than nursing students exclusively. Another systematic review25 has reported that OSCE is a more credible assessment format than the virtual style in evaluating the clinical competence of nursing students. Therefore, assessing the effects of VR OSCE for nursing students is urgently necessary. In this study, we aim to systematically evaluate the effectiveness of VR OSCE as an assessment method, particularly in terms of students’ competence, stress, anxiety, confidence, satisfaction with VR OSCE and examiners’ satisfaction.

## Methods

### Aim

This study aims to assess the effects of VR OSCE on nursing students’ education.

### Registration

This protocol study is conducted in accordance with the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA) guidelines26 and has been registered in the International Prospective Register of Systematic Reviews (PROSPERO) with registration number CRD42023437685.

### Search strategy

Electronic data search is carried out on PubMed, Web of Science, CINAHL, EBSCO, EMBASE and the Cochrane Library. In addition, references of included papers are searched to identify additional eligible studies. For studies without full text or with missing original data, the original authors are contacted. Finally, the research that contains sufficient information to assess eligibility for inclusion criteria is also included.

We establish the search strategy by using preretrieval PubMed. Search terms related to “virtual” are as follows: “virtual reality” OR “virtual” OR “online” OR “digital” OR “remote” OR “electronic” OR “video” OR “web”. The Boolean operator “OR” is used to combine these terms with different syntaxes adapted to each original database.

The keywords used to capture the concept of “OSCE” include “OSCE” OR “objective structured clinical examination” OR “clinical simulation in nursing” OR “high fidelity simulation training” OR “clinical examination” OR “clinical assessment” OR “clinical skill assessment” OR “clinical competence” OR “clinical performance”. We use the Boolean operator “OR” to combine these search terms with different syntaxes adapted to each database.

Search terms related to “nursing students” are “students, nursing” OR “nursing student*” OR “pupil nurse*” OR “nurse intern” OR “nursing staff” OR “nurse education” Similarly, the Boolean operator “OR” is used to combine the search terms with different syntaxes adapted to each database.

We use the Boolean operator “‘AND” to combine the three search terms, namely, “virtual,” “OSCE” and “nursing students”. The search time is from the inception of each database to 30 June 2023, and no language restriction is considered. References of included studies are searched for additional identification. For studies without original data, we attempt to contact the original authors to obtain the required information. The search algorithm will be developed by an experienced librarian to ensure the comprehensiveness of the literature retrieval and processing. The search strategy is shown in online supplemental appendix.

### Supplementary data

[[bmjopen-2023-082847supp001.pdf]](pending:yes) 

### Eligibility criteria

#### Population

Nursing students comprising those studying in school and engaging in hospital internships are included.

#### Intervention

Studies on the use of VR OSCE as an assessment tool.

#### Comparator

Studies on the use of traditional clinical examinations, such as in-person OSCE as assessment tools.

#### Outcome

We assess the outcome list as follows: (1) competence, (2) stress, (3) anxiety, (4) confidence, (5) students’ satisfaction with VR OSCE and (6) examiners’ satisfaction. Competence refers to the ability of an individual to complete a task appropriately.27 It can be assessed using different instruments, such as the nurse competency scale.28 Stress is a cognitive and behavioural experience composed of psychological stress source and psychological stress response.29 It can be assessed using instruments, such as the Perceived Stress Scale.30 Anxiety is a restless emotion caused by excessive concerns about the safety of family members or one’s own life, future and destiny.31 It can be assessed using different instruments, such as state–trait anxiety inventory32 and Self-rating Anxiety Scale.33 Confidence refers to a psychological characteristic that reflects an individual’s level of trust in his ability to successfully complete a certain activity; it is a positive and effective expression of self-worth, self-respect and self-awareness, as well as a psychological state.34 It can be assessed using student satisfaction and self-confidence in the learning scale.35 Satisfaction is a psychological state that refers to a person’s subjective evaluation of the quality of a relationship.36 It can be assessed using the Simulated Clinical Experience Satisfaction Scale,37 Clinical Learning Environment, Supervision and Nurse Teacher Scale38 and some self-made scales.39 Data extracted by additional scales can also be applied to this study.

#### Study design

This study includes randomised controlled trials (RCTs) and quasi-experimental studies, focusing on VR OSCE groups versus traditional clinical examination groups.

### Exclusion criteria

The exclusion criteria are as follows: (1) outcome measures are inappropriate and relevant data cannot be obtained from original authors; (2) animal experiments, reviews, notes, editorials or errata articles and (3) duplicate published literature.

### Study selection and data extraction

#### Study selection

Preliminary search results are downloaded from the software ‘EndNote V.X9’. First, on accessing titles and abstracts using the function ‘Find duplicates’ of the software, we delete duplicate articles by comparing titles and authors. Second, we enter the manual screening stage, where the preliminary screening allows removing documents that do not satisfy the requirements by reading the titles and abstracts of included articles. Third, we download the remaining documents to obtain, read the full texts and then remove documents that do not satisfy the requirements. Fourth, for documents with missing texts or original data, we attempt to contact the authors to obtain information. If such an attempt is still unsuccessful, then we delete the related documents and provide reasons. Finally, references of included documents in the final study are reviewed and assessed for additional research that may satisfy the inclusion criteria. Two of the present authors (PL and XD) independently conducted the literature retrieval. Disputes are resolved through discussion, and unresolved issues are decided on by consulting the research director (HF). The selection process is conducted according to the PRISMA flow chart.

#### Data extraction

Data for extraction include the following information: (1) Basic information of each study, including author, publication year and country (or region); (2) Participant characteristics: sample size, grouping and sample size of each group, mean age and gender; (3) Intervention method characteristics: study design, specific intervention and control methods, VR intervention duration, and comparator; (4) Research results: result measurement method, data type, statistical data and results. Outcome data are expressed as mean±SD (M±SD). If data are provided in other formats, such as median range or median IQR, then M±SD values are calculated following the recommendations of the Cochrane Handbook for Systematic Reviews of Interventions.40 (5) Other information includes support from funding institutions and potential conflicts of interest. Two of the authors (PL and XD) conduct data extraction independently, and any dispute is settled by discussion. Unresolved disputes are decided on by consulting the third author (HF). The data extraction method is to manually fill in the Excel table, the extracted data are input into the software RevMan V.5.3 for meta-analysis.

#### Quality assessment of included studies

For randomised trials, we use the Cochrane risk-of-bias (ROB) tool41 to evaluate the bias risk of RCT. Seven criteria are included, namely, random sequence generation, allocation concealment, participant and personnel blinding, outcome assessment blinding, incomplete data outcome, selective outcome reporting, and other biases. Risk bias level is classified as high, unclear and low. We select the risk of bias in non-randomised studies of intervention42 tool for non-RCT studies to evaluate the ROB. The ROB includes issues related to confounding, participant selection, intervention classification, deviation from intended intervention, missing data, outcome measurement, reported result selection and overall bias.

The research quality will be assessed by applying GRADE (Grading of Recommendations Assessment, Development and Evaluation) approach,43 and calculating the between-rater agreement coefficient. The kappa coefficients will be classified according to the study of Landis and Koch44 as follows: 0.0–0.20=slight agreement, 0.21–0.40=fair agreement, 0.41–0.60=moderate agreement, 0.61–0.80=substantial agreement and 0.81–1.00=nearly perfect agreement.45 

The two authors (PL and XD) independently evaluate the ROB for each included study and evidence quality. Any dispute is resolved through discussion. Unresolved disputes are decided on by consulting the third author (HF).

### Data synthesis and statistical analysis

#### Data synthesis

SPSS V.22.0 and RevMan V.5.3 software will be used for statistical analysis. For continuous data, if the measurement methods used in each study are the same, then we select the weighted mean difference model for statistical analysis; otherwise, the standardised mean difference model is preferred. For dichotomous data, the OR value is calculated. All effective quantities are expressed with 95% CI. A p<0.05 indicates a statistically significant difference.

#### Heterogeneity assessment

I2 test is used to assess the heterogeneity level. According to the Cochrane handbook, large heterogeneity exists when I2>50%. If p>0.1 and I2<50%, then a fixed effect model is used; otherwise, if p<0.1 and I2>50%, then the random effect model is applied. If conditions permit, then we collect quantitative data for meta-analysis; otherwise, data are presented in narrative form. Sensitivity and subgroup analyses are performed to explain possible heterogeneity sources.

#### Subgroup analysis

If significant heterogeneity (I2>50%) is found and the heterogeneity source cannot be detected through sensitivity analysis, then a subgroup analysis is conducted. Grouping analysis can be applied to basic research information, subject characteristics, intervention methods, intervention duration, sample size or other aspects.

#### Sensitivity analysis

When heterogeneity is large, the leave-one-out method is used to determine whether it is caused by a certain study. For example, we remove one study to determine whether heterogeneity decreases. This method is used to test each study to find the possible heterogeneity source.

#### Publication bias assessment

For 10 or more studies available for meta-analysis, we use funnel plot to measure the publication bias level. Specifically, the method evaluates whether the funnel plot is symmetrical through visual inspection using the Egger’s test with a significance level of 5%.46 If less than 10 items exist, then we determine whether publication bias exists according to the characteristics of the included studies.

#### Evidence quality

The quality of each evidence is assessed using the GRADE rating scale.47 We classify the quality as high, moderate, low or very low according to the consideration of ROB, inconsistency (heterogeneity), indirectness, imprecision and publication bias.48 The results start with ‘high’-quality evidence and are then degraded according to the problems in each field. Results can also be enhanced when the evidence shows that all possible confounding factors and other deviations increase confidence in the estimated effect. The two authors (PL and XD) score each area of comparison and resolve differences through consensus. Unresolved disputes are decided on by consulting the third author (HF).

#### Expected dates for research

The literature search is from 30 June 2023 to 31 January 2024, data extraction is from 1 February 2024 to 31 March 2024, quality evaluation is from 1 April 2024 to 30 April 2024, the meta-analysis is from 1 May 2024 to 30 June 2024 and evidence quality evaluation is from 1 July 2024 to 31 July 2024.

#### Patient and public involvement

Our study will not involve or did not involve patients or the public in the design, execution or planning for reporting and dissemination.

#### Future directions and clinical implications

With the continuous improvement and development of VR technology, it has been applied in clinical research and has achieved satisfactory treatment results.49 Previous studies showed that VR plays a certain role in the treatment of psychological conditions in ICU patients, but the specific efficacy remains controversial.39 50 We further analyse which aspects of VR have positive therapeutic effects on the psychological conditions of ICU patients, which aspects have no therapeutic effects, and which have adverse effects. We also explore the possible causes and reasons. How to maximise the advantages of VR in clinical intervention will become the future development direction. This concept has clinical significance for providing more scientific intervention plans and a theoretical basis for the application of VR in the treatment of psychological disorders in ICU patients.

### Ethics and dissemination

This protocol study does not carry out clinical research and thus does not require ethical approval. Research findings will be published in a peer-reviewed journal.

### Supplementary data

[[bmjopen-2023-082847supp002.pdf]](pending:yes) 

## Ethics statements

### Patient consent for publication

Not applicable.

## Footnotes

*   Contributors PL contributed to the first rudimentary draft and revision of the manuscript. XD contributed to the data collection and data analysis. FL contributed to the conception, first draft and revision of the manuscript. HF had primary responsibility for the overall content as the guarantor.

*   Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

*   Competing interests None declared.

*   Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

*   Provenance and peer review Not commissioned; externally peer reviewed.

*   Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

[http://creativecommons.org/licenses/by-nc/4.0/](http://creativecommons.org/licenses/by-nc/4.0/)
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: [http://creativecommons.org/licenses/by-nc/4.0/](http://creativecommons.org/licenses/by-nc/4.0/).

## References

1.   Tavakol M , O’Brien D . Does widening participation status affect undergraduate medical student performance; a meta-analysis of knowledge-based assessments and OSCE over a 5-year period. Med Teach 2022;44:1421. [doi:10.1080/0142159X.2021.2011848](http://dx.doi.org/10.1080/0142159X.2021.2011848) 
    
    

2.   Bobos P , Pouliopoulou DV , Harriss A , et al . A systematic review and meta-analysis of measurement properties of objective structured clinical examinations used in physical therapy Licensure and a structured review of Licensure practices in countries with well-developed regulation systems. PLoS One 2021;16:e0255696. [doi:10.1371/journal.pone.0255696](http://dx.doi.org/10.1371/journal.pone.0255696) 
    
    

3.   White K . To OSCE or not to OSCE BMJ 2023;381:1081. [doi:10.1136/bmj.p1081](http://dx.doi.org/10.1136/bmj.p1081) 
    
    

4.   Smee S , Coetzee K , Bartman I , et al . OSCE standard setting: three borderline group methods. Med Sci Educ 2022;32:1439–45. [doi:10.1007/s40670-022-01667-x](http://dx.doi.org/10.1007/s40670-022-01667-x) 
    
    

5.   Gilani S , Pankhania K , Aruketty M , et al . Twelve tips to organise a mock OSCE. Med Teach 2022;44:26–31. [doi:10.1080/0142159X.2021.1887465](http://dx.doi.org/10.1080/0142159X.2021.1887465) 
    
    

6.   Saad SL , Richmond C , Jones K , et al . Virtual OSCE delivery and quality assurance during a pandemic: implications for the future. Front Med (Lausanne) 2022;9:844884. [doi:10.3389/fmed.2022.844884](http://dx.doi.org/10.3389/fmed.2022.844884) 
    
    

7.   Deville RL , Fellers CM , Howard ML . Lessons learned pivoting to a virtual OSCE: Pharmacy faculty and student perspectives. Currents in Pharmacy Teaching and Learning 2021;13:1498–502. [doi:10.1016/j.cptl.2021.06.046](http://dx.doi.org/10.1016/j.cptl.2021.06.046) 
    
    

8.   Blythe J , Patel NSA , Spiring W , et al . “Undertaking a high stakes virtual OSCE ("VOSCE") during COVID-19”. BMC Med Educ 2021;21:221. [doi:10.1186/s12909-021-02660-5](http://dx.doi.org/10.1186/s12909-021-02660-5) 
    
    

9.   Chen F-Q , Leng Y-F , Ge J-F , et al . Effectiveness of virtual reality in nursing education: meta-analysis. J Med Internet Res 2020;22:e18290. [doi:10.2196/18290](http://dx.doi.org/10.2196/18290) 
    
    

10.  Zhang W , Lozynska I , Li W , et al . Benefits and barriers of Holistic nursing training by high-Fidelity simulation in obstetrics. Comput Math Methods Med 2022;2022:1848849. [doi:10.1155/2022/1848849](http://dx.doi.org/10.1155/2022/1848849) 
    
    

11.  Felix RB , Rao A , Khalid M , et al . Adjunctive virtual reality pain relief following traumatic injury: protocol for a randomised within-subjects clinical trial. BMJ Open 2021;11:e056030. [doi:10.1136/bmjopen-2021-056030](http://dx.doi.org/10.1136/bmjopen-2021-056030) 
    
    

12.  Celik S , Olgun N , Yilmaz FT , et al . Assessment the effect of diabetes education on self-care behaviors and Glycemic control in the Turkey nursing diabetes education evaluating project (TURNUDEP): a multi-center study. BMC Nurs 2022;21:215. [doi:10.1186/s12912-022-01001-1](http://dx.doi.org/10.1186/s12912-022-01001-1) 
    
    

13.  Zhou H , Wang Y , Cheng L . The mediating effect of self-directed learning in the relationship between caring and resilience among Chinese nursing students: A multi-center cross-sectional study. Nurse Educ Today 2022;119:105598. [doi:10.1016/j.nedt.2022.105598](http://dx.doi.org/10.1016/j.nedt.2022.105598) 
    
    

14.  Updike WH , Cowart K , Woodyard JL , et al . Protecting the integrity of the virtual objective structured clinical examination. Am J Pharm Educ 2021;85:8438. [doi:10.5688/ajpe8438](http://dx.doi.org/10.5688/ajpe8438) 
    
    

15.  Boardman D , Wilhite JA , Adams J , et al . Telemedicine training in the COVID era: revamping a routine OSCE to prepare medicine residents for virtual care. J Med Educ Curric Dev 2021;8. [doi:10.1177/23821205211024076](http://dx.doi.org/10.1177/23821205211024076) 
    
    

16.  Haidet P , Hempel EV , Louw BC , et al . Virtual decisions: using a Telehealth OSCE to enhance Trainees' triage skills. Med Educ 2021;55:659. [doi:10.1111/medu.14493](http://dx.doi.org/10.1111/medu.14493) 
    
    

17.  Roman P , Ruiz-Gonzalez C , Rodriguez-Arrastia M , et al . A serious game for online-based objective structured clinical examination in nursing: A qualitative study. Nurse Educ Today 2022;109:105246. [doi:10.1016/j.nedt.2021.105246](http://dx.doi.org/10.1016/j.nedt.2021.105246) 
    
    

18.  Bevan J , Russell B , Marshall B . A new approach to OSCE preparation - Prosces. BMC Med Educ 2019;19:126. [doi:10.1186/s12909-019-1571-5](http://dx.doi.org/10.1186/s12909-019-1571-5) 
    
    

19.  Hoag JA , Karst J , Bingen K , et al . Distracting through procedural pain and distress using virtual reality and guided imagery in pediatric, adolescent, and young adult patients: randomized controlled trial. J Med Internet Res 2022;24:e30260. [doi:10.2196/30260](http://dx.doi.org/10.2196/30260) 
    
    

20.  Grover S , Pandya M , Ranasinghe C , et al . Assessing the utility of virtual OSCE sessions as an educational tool: a national pilot study. BMC Med Educ 2022;22:178. [doi:10.1186/s12909-022-03248-3](http://dx.doi.org/10.1186/s12909-022-03248-3) 
    
    

21.  Arrogante O , López-Torre EM , Carrión-García L , et al . n.d. High-Fidelity virtual objective structured clinical examinations with standardized patients in nursing students: an innovative proposal during the COVID-19 pandemic. Healthcare 9:355. [doi:10.3390/healthcare9030355](http://dx.doi.org/10.3390/healthcare9030355) 
    
    

22.  Luke S , Petitt E , Tombrella J , et al . Virtual evaluation of clinical competence in nurse practitioner students. Med Sci Educ 2021;31:1267–71. [doi:10.1007/s40670-021-01312-z](http://dx.doi.org/10.1007/s40670-021-01312-z) 
    
    

23.  Chan SCC , Choa G , Kelly J , et al . Implementation of virtual OSCE in health professions education: A systematic review. Med Educ 2023;57:833–43. [doi:10.1111/medu.15089](http://dx.doi.org/10.1111/medu.15089) 
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1111/medu.15089&link_type=DOI) 

24.  Phillips TA , Munn AC , George TP . Assessing the impact of Telehealth objective structured clinical examinations in graduate nursing education. Nurse Educ 2020;45:169–72. [doi:10.1097/NNE.0000000000000729](http://dx.doi.org/10.1097/NNE.0000000000000729) 
    
    

25.  Vincent SC , Arulappan J , Amirtharaj A , et al . Objective structured clinical examination vs traditional clinical examination to evaluate students' clinical competence: A systematic review of nursing faculty and students' perceptions and experiences. Nurse Educ Today 2022;108:105170. [doi:10.1016/j.nedt.2021.105170](http://dx.doi.org/10.1016/j.nedt.2021.105170) 
    
    

26.  Moher D , Shamseer L , Clarke M , et al . Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev 2015;4:1. [doi:10.1186/2046-4053-4-1](http://dx.doi.org/10.1186/2046-4053-4-1) 
    
    

27.  Jager M , den Boeft A , Versteeg-Pieterse A , et al . Observing cultural competence of Healthcare professionals: A systematic review of observational assessment instruments. Patient Educ Couns 2021;104:750–9. [doi:10.1016/j.pec.2020.10.010](http://dx.doi.org/10.1016/j.pec.2020.10.010) 
    
    

28.  Nilsson J , Johansson S , Nordström G , et al . Development and validation of the ambulance nurse competence scale. J Emerg Nurs 2020;46:34–43. [doi:10.1016/j.jen.2019.07.019](http://dx.doi.org/10.1016/j.jen.2019.07.019) 
    
    

29.  Pátek M , Grulich M , Nešvera J . Stress response in Rhodococcus strains. Biotechnol Adv 2021;53:107698. [doi:10.1016/j.biotechadv.2021.107698](http://dx.doi.org/10.1016/j.biotechadv.2021.107698) 
    
    

30.  Simon PD . The 10-item perceived stress scale as a valid measure of stress perception. Asia Pac Psychiatry 2021;13:e12420. [doi:10.1111/appy.12420](http://dx.doi.org/10.1111/appy.12420) 
    
    

31.  Chellappa SL , Aeschbach D . Sleep and anxiety: from mechanisms to interventions. Sleep Med Rev 2022;61:101583. [doi:10.1016/j.smrv.2021.101583](http://dx.doi.org/10.1016/j.smrv.2021.101583) 
    
    

32.  Şahin S , Tokgöz B , Demir G . Effect of lavender Aromatherapy on Arteriovenous Fistula puncture pain and the level of State and trait anxiety in Hemodialysis patients: A randomized controlled trial. Pain Manag Nurs 2021;22:509–15. [doi:10.1016/j.pmn.2021.01.009](http://dx.doi.org/10.1016/j.pmn.2021.01.009) 
    
    

33.  Dunstan DA , Scott N . Norms for Zung’s self-rating anxiety scale. BMC Psychiatry 2020;20:90. [doi:10.1186/s12888-019-2427-6](http://dx.doi.org/10.1186/s12888-019-2427-6) 
    
    

34.  Alpert AB , Moftakhar B . Confidence. J Clin Oncol 2021;39:2410–2. [doi:10.1200/JCO.21.00012](http://dx.doi.org/10.1200/JCO.21.00012) 
    
    

35.  Unver V , Basak T , Watts P , et al . The Reliability and validity of three questionnaires: the student satisfaction and self-confidence in learning scale, simulation design scale, and educational practices questionnaire. Contemp Nurse 2017;53:60–74. [doi:10.1080/10376178.2017.1282319](http://dx.doi.org/10.1080/10376178.2017.1282319) 
    
    

36.  Rossini S , Bulfone G , Vellone E , et al . Nursing students' satisfaction with the curriculum: an integrative review. J Prof Nurs 2021;37:648–61. [doi:10.1016/j.profnurs.2021.02.003](http://dx.doi.org/10.1016/j.profnurs.2021.02.003) 
    
    

37.  Souza C de , Santos WGD , Salgado P de O , et al . Evaluating the satisfaction and self-confidence in nursing students in undergoing simulated clinical experiences. Rev Esc Enferm USP 2020;54:e03583. [doi:10.1590/s1980-220x2018038303583](http://dx.doi.org/10.1590/s1980-220x2018038303583) 
    
    

38.  Guejdad K , Ikrou A , Strandell-Laine C , et al . Clinical learning environment, supervision and nurse teacher (CLES+T) scale: translation and validation of the Arabic version. Nurse Educ Pract 2022;63:103374. [doi:10.1016/j.nepr.2022.103374](http://dx.doi.org/10.1016/j.nepr.2022.103374) 
    
    

39.  Vlake JH , van Bommel J , Wils E-J , et al . Intensive care unit-specific virtual reality for critically ill patients with COVID-19: multicenter randomized controlled trial. J Med Internet Res 2022;24:e32368. [doi:10.2196/32368](http://dx.doi.org/10.2196/32368) 
    
    

40.  Pavel M-C , Casanova R , Estalella L , et al . The effect of preoperative chemotherapy on liver regeneration after portal vein Embolization/ligation or liver resection in patients with colorectal liver metastasis: a systematic review protocol. Syst Rev 2020;9:279. [doi:10.1186/s13643-020-01545-w](http://dx.doi.org/10.1186/s13643-020-01545-w) 
    
    

41.  Sterne JAC , Savović J , Page MJ , et al . Rob 2: a revised tool for assessing risk of bias in randomised trials. BMJ 2019;366:l4898. [doi:10.1136/bmj.l4898](http://dx.doi.org/10.1136/bmj.l4898) 
    
    

42.  Sterne JA , Hernán MA , Reeves BC , et al . ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ 2016;355:i4919. [doi:10.1136/bmj.i4919](http://dx.doi.org/10.1136/bmj.i4919) 
    
    

43.  Guyatt GH , Oxman AD , Sultan S , et al . GRADE guidelines: 9. rating up the quality of evidence. J Clin Epidemiol 2011;64:1311–6. [doi:10.1016/j.jclinepi.2011.06.004](http://dx.doi.org/10.1016/j.jclinepi.2011.06.004) 
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1016/j.jclinepi.2011.06.004&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=21802902&link_type=MED&atom=%2Fbmjopen%2F14%2F5%2Fe082847.atom) 

44.  Landis JR , Koch GG . The measurement of observer agreement for categorical data. Biometrics 1977;33:159. [doi:10.2307/2529310](http://dx.doi.org/10.2307/2529310) 
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.2307/2529310&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=843571&link_type=MED&atom=%2Fbmjopen%2F14%2F5%2Fe082847.atom) 
    
    [Web of Science](http://bmjopen.bmj.com/lookup/external-ref?access_num=A1977CY39700012&link_type=ISI) 

45.  Du S , Liu W , Cai S , et al . The efficacy of E-health in the self-management of chronic low back pain: a meta analysis. Int J Nurs Stud 2020;106:103507. [doi:10.1016/j.ijnurstu.2019.103507](http://dx.doi.org/10.1016/j.ijnurstu.2019.103507) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjopen%2F14%2F5%2Fe082847.atom) 

46.  Sterne JAC , Sutton AJ , Ioannidis JPA , et al . Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ 2011;343. [doi:10.1136/bmj.d4002](http://dx.doi.org/10.1136/bmj.d4002) 
    
    

47.  Yang B , Mustafa RA , Bossuyt PM , et al . GRADE guidelines: 31. assessing the certainty across a body of evidence for comparative test accuracy. J Clin Epidemiol 2021;136:146–56. [doi:10.1016/j.jclinepi.2021.04.001](http://dx.doi.org/10.1016/j.jclinepi.2021.04.001) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjopen%2F14%2F5%2Fe082847.atom) 

48.  Balshem H , Helfand M , Schünemann HJ , et al . GRADE guidelines: 3. rating the quality of evidence. J Clin Epidemiol 2011;64:401–6. [doi:10.1016/j.jclinepi.2010.07.015](http://dx.doi.org/10.1016/j.jclinepi.2010.07.015) 
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1016/j.jclinepi.2010.07.015&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=21208779&link_type=MED&atom=%2Fbmjopen%2F14%2F5%2Fe082847.atom) 
    
    [Web of Science](http://bmjopen.bmj.com/lookup/external-ref?access_num=000288364300009&link_type=ISI) 

49.  Twamley J , Hamer O , Hill J , et al . Exploring the perceptions of former ICU patients and clinical staff on barriers and Facilitators to the implementation of virtual reality exposure therapy: a qualitative study. Nursing in Critical Care 2024;29:313–24. [doi:10.1111/nicc.12868](http://dx.doi.org/10.1111/nicc.12868) 
    
    

50.  Small C , Stone R , Pilsbury J , et al . Virtual restorative environment therapy as an adjunct to pain control during burn dressing changes: study protocol for a randomised controlled trial. Trials 2015;16:329. [doi:10.1186/s13063-015-0878-8](http://dx.doi.org/10.1186/s13063-015-0878-8)