Article Text
PDF
Abstract
Introduction Structured E-parenting Support (STEPS) is a digital application (app) designed to help parents manage behaviour of their children who are referred to mental health services and are waiting for an assessment or treatment. STEPS is currently being evaluated in the Online Parent Training for the Initial Management of Attention-Deficit/Hyperactivity Disorder randomised controlled trial. Alongside the examination of STEPS’ clinical and cost-effectiveness, we are conducting a process evaluation to better understand the contextual factors that may influence study outcomes. The purpose of this protocol is to describe the aims, objectives and methodology of the process evaluation prior to it taking place to add to the fidelity and rigour of the trial process and outcomes. Our goal is to adapt STEPS to optimise its benefits in future applications.
Methods In line with the Medical Research Council guidelines for evaluating complex interventions, the process evaluation will adopt a mixed method design using qualitative data collected from clinicians and parent interviews and app usage data from participants assigned to the intervention arm.
Analysis Qualitative data from semistructured interviews and free text box responses included in trial questionnaires will be analysed thematically using framework analysis to better understand how parents use STEPS, how it works and key factors that could aid or hinder its effective implementation in routine clinical practice.
Ethics The application for ethical approval for the study was submitted to the North West—Liverpool Central Research Ethics Committee and received a favourable opinion on further information on 26 November 2021, reference number 21/NW/0319.
Dissemination The process evaluation aims to explore how a digital app might support parents in managing their child’s behaviour. Implications for policy and research will be explored and the clinical implications of offering the app to a wider audience to address the lack of support to parents as highlighted in this paper. We plan to publish findings in international, peer-reviewed journals as well as present at conferences.
Trial registration number The trial has been prospectively registered on 18 November 2021; ISRCTN816523503. https://www.isrctn.com/ISRCTN16523503.
- Randomized Controlled Trial
- Behavior
- Clinical Trial
- eHealth
- Protocols & guidelines
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
STRENGTHS AND LIMITATIONS OF THIS STUDY
Framework analysis allows for in-depth data analysis using a rigorous and transparent methodology.
Outcomes for quantitative data such as app usage metrics will be integrated with qualitative findings.
Inclusion of members from the patient and public involvement panel to advise on the best practice in working with participants as well as assisting in data analysis and interpretation of the study results.
All eligible participants were invited to partake in interviews, including those who did not complete all timepoints and those who did not download or use the app, to further understand barriers to uptake and usage of the Structured E-parenting Support app.
A potential limitation of this study is the crossover of team members working on both the randomised controlled trial and the process evaluation which may influence the interpretation of the qualitative data.
Introduction
Attention-deficit/hyperactivity disorder (ADHD) is a common neurodevelopmental condition characterised by symptoms of inattention and/or impulsivity–hyperactivity.1 Children referred for ADHD assessment may also present with comorbidities such as symptoms of conduct problems,2 which can negatively impact the family.3 The National Institute for Health and Care Excellence recommends that families should receive support as soon as possible after their referral; however, despite these recommendations, parents frequently endure long waiting times for diagnostic assessment and treatment. The average time between seeking help and receiving an ADHD diagnosis has been estimated as 18.3 months in the UK: the longest average interval compared with other European countries.4 Lengthy waiting times and scarcity of services are the most common barriers to accessing mental health services for children and adolescents as reported by parents.5 Furthermore, these waiting times are likely to get even longer, given consistent rises in the number of referrals to Child and Adolescent Mental Health Services (CAMHS).6
A robust body of research has established the efficacy of parent training as a psychosocial intervention for children and young people.7 Research has found that parent training may reduce conduct problems in children with ADHD.8 Moreover, Daley et al 9 conducted a meta-analysis on behavioural interventions that established improvements in parenting quality as well as a reduction in child ADHD symptoms and conduct problems. However, despite evidence of its efficacy, parent training may not be made available until a diagnosis has been established, leaving parents without support during the lengthy waiting period which can have a detrimental effect on children and their families.10
Considering evidence that parent training can have a positive outcome for both parent and child, and to provide families with much-needed timely and accessible support, we have developed a digital parenting application called Structured E-Parenting Support (STEPS). Research suggests that digital health interventions (DHIs), such as mobile apps, may have great potential to deliver large-scale, cost-effective support.11 However, there is a real need to understand how health and digital research can work together for effective implementation.12
The STEPS app and OPTIMA trial
STEPS has been designed to support parents of children with ADHD-type symptoms that are accompanied by challenging behaviour and who are awaiting clinical diagnostic assessment. Its structure, content and approach are described in online supplemental appendix A. STEPS draws inspiration from some of the principles underpinning the New Forest Parenting Program (NFPP),13 an established face-to-face parent programme based on a long track record of research and clinical practice relating to parenting of child behaviour problems. However, its content, structure and approach, tailored to the digital delivery, are substantively different from the NFPP.
Supplemental material
STEPS is delivered via a mobile app that aims to improve parents’ understanding of their child’s challenging behaviour and increase their perceived self-efficacy to manage such behaviours, as well as facilitate effective parent–child communication. STEPS has one preparatory module, ‘Introduction’, followed by eight separate intervention modules (steps) to be followed in order. Each of the eight steps is designed to take about 20 minutes if completed in one go. The content is delivered via short, prerecorded videos, audio clips and text, and parents can download resources as well as make notes on their own reflections within each of the modules (steps) (table 1).
The STEPS app modules’ titles and aims
In our previous study, parents rated the app’s usability level as very high; the overall STEPS usability score on the System Usability Scale was 94.8 (SD 4.8) out of 100.14 Moreover, feedback received was used to optimise the app in preparation for the trial. For example, we improved and simplified the registration process, improved video playback and added captions to videos.
The efficacy and cost-effectiveness of STEPS are currently being evaluated in the Online Parent Training for The Initial Management of ADHD referrals (OPTIMA) randomised controlled trial (RCT).15 OPTIMA is a two-arm, superiority parallel RCT with an internal pilot.14 Participant recruitment took place from May 2022 to July 2023 and during this time 352 parents were randomly assigned to either the intervention group (access to the STEPS app for 3 months) or the Wait as Usual comparison group (WAU) on completion of baseline measures. Randomisation was carried out online via a secure platform provided by Sealed Envelope in a 1:1 ratio and stratification by trial centre location (London, Nottingham, Southampton) using random permuted blocks procedure with varying block sizes. The randomisation system used a unique identifying number.
Questionnaires are administered via Sealed Envelope, every 3 months at five timepoints. Participants were recruited from mental health services across London, Nottingham, Portsmouth, Southampton and Gloucester, after initial eligibility had been established via a positive screen for high levels of hyperactivity (≥8) and conduct problems (≥4) as measured by the Strengths and Difficulties Questionnaire.16 As part of the screening process, researchers checked whether parents had a phone with an operating system that was compatible with the app and whether parents were sufficiently proficient in English to be able to use the app and understand it. Participation in the study does not impact clinical care the family receives, or the time spent on the waitlist. There are no restrictions on concomitant care, which has been monitored carefully during the trial through the Child and Adolescent Service Use Schedule (CASUS).17 During the study, trial administrators have been available to help parents with any technical issues if the visual download and usage guide, received from the study team, was insufficient.
The primary outcome of the OPTIMA trial is the severity of behaviour problems at 3 months post randomisation compared with WAU care using parent-reported child behaviour problems measured with the eight-item oppositional defiance disorder (ODD) subscale of the Swanson Nolan and Pelham Rating Scale (SNAP-IV).18 For the process evaluation, the mean difference between timepoint one and two of the primary outcome will be measured for the intervention arm only. Data relating to the study outcomes comparing the two groups will be published in separate papers.
Process evaluation aims and objectives
Establishing a methodology by which the process evaluation will adhere to a priori is useful to ensure rigour and improve trial quality. Using Medical Research Council (MRC) guidelines,19 this protocol describes the method for the process evaluation of STEPS within the OPTIMA trial. Furthermore, the Standard Protocol Items: Recommendations for Interventional Trials checklist has been used to provide evidence-based guidance in producing this protocol and is a widely accepted standard for trial protocols.20 Specific objectives are to:
To assess the (a) reach, (b) dose, (c) fidelity, (d) impact and (e) context of the intervention. Table 2 defines the components of process evaluation and shows the methods by which the required information is gathered.
To describe how parents implement STEPS.
To explore parents’ and clinicians’ views concerning the value of STEPS and to describe this in the context of their respective needs.
To explore external factors that may have acted as barriers to, or facilitators of, STEPS uptake and engagement.
To consider the sustainability of the STEPS app beyond the trial and, if shown to be effective, the possible ways it could be incorporated into the clinical pathways.
Evaluation of mechanisms of impact (mediating factors contributing to the outcome) and context (intrapersonal and environmental factors influencing app usage).
The STEPS process evaluation components and methodology
Supplemental material
Following MRC guidelines for process evaluations,19 a logic model has been developed (table 3) to elucidate the mechanisms by which the STEPS intervention will produce an outcome and inform the framework of the qualitative analysis. A logic model can be useful in representing the theory of the intervention and its outcomes and helps to clarify the main aspects of the intervention as well as aid in data collection and analysis.21 The STEPS logic model clarifies the current issues in parent support for those waiting on a diagnosis for their child as well as expands on the implications for STEPS use beyond the study. Moreover, by providing a step-by-step process from developing the research question to understanding how outcomes were achieved, it ensures that researchers adhere to the predetermined process of delivery and analysis.
STEPS logic model
Method
Design
This mixed-method process evaluation integrates qualitative and quantitative data. Qualitative data will be gathered from semistructured interviews with parents and clinicians to explore the implementation of the intervention and the perceived impact of the intervention on parenting and child behaviour as well as expectations about the trial as reported by participants via free text responses on the trial questionnaires. Parallel to this, we will use quantitative data such as demographic data and app usage metrics. Table 2 describes the methods and evaluation of data collection.
Qualitative data collection
Qualitative data collection will include semistructured interviews with parents and clinicians and text gathered from Sealed Envelope asking parents about their trial expectations.
Parent interviews
Participants who meet the following criteria will be invited to interview: (1) have consented to be contacted for interviews via the study consent form (optional consent statement) and (2) have been randomised to the STEPS arm. Participants will be invited to take part in interviews irrespective of whether they engaged with the STEPS app or not. Views of participants who have not completed any of the steps are very important in the context of understanding barriers to usage. We aim to recruit n=50 parents for interviews.
Participants who have consented to be contacted about interviews will be approached by a researcher other than the one who has enrolled them on the trial to avoid unblinding. Selection and allocation of eligible participants are completed by the trial manager and trial administrators. Researchers invite participants via email explaining the interview process. The default method for conducting interviews will be a video/phone call (30–45 min duration). Participants who wish to complete the interview via email will be sent an adapted interview schedule. Offering a range of ways to engage in the interviews will ensure that those who feel unable to speak with a researcher on the phone will also be able to take part to give a breadth of views from parents.
The interview schedule has been developed by a team of experienced qualitative researchers in collaboration with the OPTIMA patient and public involvement group (PPI). Once the team had finalised the interview schedule, the three researchers involved in conducting the interviews piloted the interviews with members from the PPI group and colleagues. Initially up to an hour had been allocated for the interviews but the pilot showed that 30–45 min was adequate time to cover all the questions. Furthermore, the PPI group felt that a decrease in the time required from the parents was more commensurate with the compensation for participation, a £20 Amazon gift voucher. The interview schedules remained dynamic and in the early stage of interviewing, the qualitative team worked together to adapt and add questions.
Questions explore the technical experience of downloading and using the app, views on content and features of the app, such as the STEPS buddies, and feedback on if/how the app influenced aspects of parenting and child behaviour management. They will also ask participants about their thoughts on the effectiveness of STEPS in reducing their child’s behavioural challenges and, if applicable, the perceived mechanisms by which STEPS is effective (see online supplemental appendix B for full interview schedule). Interviews with parents took place between October 2022 and November 2023. All parents who were recruited into the intervention arm were invited to take part in interviews whether they had downloaded the app or not. Invitations were sent out 3 months after randomisation, ensuring parents had the full 3-month usage period of the app. All deidentified transcripts and email responses will be stored in electronic form on a KCL OneDrive for Business and SharePoint location. The original recordings or emails will be deleted from OneDrive for Business after transcription.
Clinician interviews
Clinicians form no active part in the OPTIMA RCT with the study being independent of any clinical input from CAMHS or other healthcare providers. However, to be eligible for participation in the OPTIMA trial, parents must be on a current waitlist for their child to receive clinical support and clinics have been informed of the nature of the OPTIMA RCT. It is therefore important to gain clinical perspectives to effectively evaluate the STEPS app in terms of future directions and implementation. Managers in the clinical services that have supported OPTIMA RCT recruitment will be approached with a request to circulate the clinician information sheet to members of the team. Clinicians interested in taking part are asked to contact the team directly. The clinicians who are interviewed have no active involvement in the trial, the STEPS intervention or the collection of outcome data. Some participants may disclose their use of the STEPS app but the clinician is not asked to probe for this. The purpose of the interviews with clinicians is to get their views about the impact of STEPS, potential factors influencing parent engagement and perceived barriers to effectiveness with the aim of facilitating implementation into clinical services.
Clinician interviews can help add depth to the qualitative data in terms of understanding the clinical context in relation to any outcomes shared by parents about contact with services or receiving an assessment and/or diagnosis. Our aim is to include n=10 interviews from clinicians to give adequate representation across the three sites although if more clinicians come forward to be interviewed, they will be able to partake. Clinicians will all be interviewed via phone/video call and data stored as per the participants’ data above. There is no incentive for clinicians to take part.
Quantitative data collection
To establish intervention adherence, the number of completed STEPS modules will be measured (min=0; max=8), with completion of two modules constituting adherence to the intervention. Other collected app usage events will include the number of started modules, the number of videos watched, the time spent watching videos (in seconds), the number of audio clips listened to and the time spent listening to audio clips (in seconds), the number of reflections recorded, the number of items saved to favourites and the number of accessed text resources. These will be used to provide descriptive information about app usage patterns.
To determine the intervention’s reach, the process evaluation will use data collected from parents at baseline (prerandomisation) via Sealed Envelope, including demographic data about the parent, such as parent’s gender, parent ethnicity, parental education, parent employment status, parent relationship status and family socioeconomic status based on total household income as well as child’s age, sex and ethnicity. To describe the severity of oppositional and defiant disorder symptoms and hyperactivity/impulsivity and inattention symptoms in the sample, the respective subscales from the parent-completed SNAP questionnaire will be used.22 23 The 8 items of the SNAP-IV ODD subscale have excellent internal consistency (α=0.93) and the subscale has been shown to be sensitive to change in clinical trials.24 Furthermore, given that ADHD and ASD often co-occur, parent-rated scores for the Social Communication Questionnaire-Lifetime (SCQ-L) will be included.25 The SCQ-L, used in this study to characterise the sample of participants receiving the intervention, has been found to have good internal consistency (Cronbach α=0.82). A cut-off ≥15 differentiated young people with a clinical diagnosis of ASD from those without ASD (sensitivity=0.70 and specificity=0.67).26 At baseline, parents are asked about their trial expectations. Parents are also asked about previous engagement in parent training (yes/no answer), expectations of receiving parent training (strongly disagree to strongly agree) and expectations of the STEPS app (strongly disagree to strongly agree).
Data analysis
Qualitative data analysis
Our objectives are to explore the reach, dose, fidelity, impact and context of the intervention. Qualitative analysis will use a framework approach,27 utilising NVivo V.14, complemented by quantitative analysis. Framework analysis sits within the broader qualitative methodology of thematic analysis and allows researchers to compare data across cases as well as within cases, ensuring the individual’s view is retained.27 Framework analysis is a flexible but rigorous method used in health research to integrate qualitative data from different informants and sources. It uses inductive or deductive approaches to identify, describe and interpret patterns.28 Three researchers will take part in both interviewing, transcribing and analysing transcripts with two senior members of the research team taking part in verifying a selection of transcripts. PPI members will work with the research team during the interpretation and verification stages of analysis. Specifically, PPI members will individually review a selection of transcripts to verify the researchers’ interpretation of the data and also take part in group meetings to discuss codes and meanings. Although several members of the PPI team have prior experience in qualitative research, 2–3 hours of training on the introduction to qualitative research and how to read and code transcripts will be provided by the research team. Finally, the analysis will be overseen by experts in framework analysis and regular meetings between the researchers analysing the transcripts and the larger qualitative team will ensure fidelity and cohesiveness in the coding process. The team will start by identifying a coding framework that aligns with the objectives of the study. Creating a data set, researchers will map out the codes and start looking for themes and relationships in the data set. As data move from codes to themes, the original research questions as well as existing literature will be referred to and discussed and reviewed within the multidisciplinary team to ensure transparency and avoid bias. The method is appropriate for incorporating data from semistructured interviews, PPI panel discussions and free text box data from questionnaires.
Quantitative data analysis
Descriptive data on the study sample will be presented to include means, SD, medians, ranges, n values and percentages. Quantitative data measuring changes in oppositional behaviour (SNAP-IV ODD) between baseline and 3 months and making within-group comparisons will also help to assess the impact of the app.
Data integration
The qualitative data extracted from interviews with parents and clinicians as well as text box data exploring parents’ expectations about the study will provide the main source of data to explore the aims and objectives of the process evaluation. Alongside this, descriptive data from the online questionnaires will be used, both to provide context to the qualitative data in terms of demographics, but also to help refine the themes emerging from the qualitative data analysis. Mixed methods afford multiple perspectives and seek to converge the findings.29 Researchers will analyse the data synchronously and integrate the outcomes from the different datasets to provide a holistic overview of the results.
Patient and public involvement
The OPTIMA RCT and STEPS app were developed in conjunction with an advisory board made up of parents of children with neurodevelopmental disorders including ADHD. The PPI group was established early on in the overall OPTIMA programme of research prior to the RCT taking place. The group advised the team about how the design and functionality of the app could be optimised as part of the panel group discussions as well as individually in the usability study.14 This was implemented and piloted before the RCT. The PPI group also supported the team in ensuring that the trial procedures were acceptable to the participants and that any participant-facing documents were written in clear and accessible language. Finally, they also helped with the development of the schedules for the parent interviews.
In addition to regular PPI panel meetings throughout the study period, panel members advised on subjects such as how to communicate with parents most effectively, how to structure compensation for participating parents’ time in the study and other study management-related questions. Further, members will be involved in the data analysis process, reading transcripts and taking part in meetings to discuss codes and meanings with OPTIMA researchers.
Ethics and dissemination
All participants in the study consented to take part via e-consent on Sealed Envelope after having received written and oral information about the study including a brief participant information sheet (PIS) with condensed information in an easy-to-understand format and as well as a full PIS for their reference. All parents received a countersigned, by the researcher, copy of their consent form. The study received ethical approval from the North West—Liverpool Central Research Ethics Committee on 26 November 2021, reference number 21/NW/0319. Findings will be published in open-access, peer-reviewed scientific journals as well as be presented at conferences.
Discussion
STEPS is a digital, self-guided app that is currently being evaluated in the OPTIMA RCT.15 To better understand the study outcomes and contextual factors influencing these, we are conducting a process evaluation using qualitative and quantitative data gathered from parents, clinicians, app usage and demographic data. We expect the results to allow us to understand how the app has worked, such as if it worked as intended, with the aim of understanding the implications of the potential wider use of STEPS, especially within a clinical setting. In understanding the strengths and weaknesses of the intervention, how the intervention was delivered and whether the intended audience received the intervention and how the app can be further developed and improved to attain its intended purpose, we aim to provide a cost-effective and self-guided support to parents awaiting clinical assessment and/or diagnosis for their child.
Research suggests that DHIs may have great potential to deliver large-scale, cost-effective support.11 The STEPS app may be able to bridge the gap between lengthy waiting times for a diagnosis of ADHD and the strains of managing difficult child behaviour. Furthermore, the study will contribute to a body of research that aims to understand how digital interventions work and the factors that contribute to their efficacy, with the aim of improving and understanding the practical implication of using STEPS as a viable DHI to be accessed by a wider population.
Strength and limitations
Integrating qualitative and quantitative data provide a comprehensive evaluation of the way in which the intervention has worked. Capturing the lived experience of parents through interviews will give valuable insight into both the mechanisms of how the app works as well as the impact on parenting and child behaviour. The data from the app provide detailed measures of how the app was used by participants and will help to better understand how the app was used (eg, the number of times app was used or the length of time per each app use). Some caution must be exercised when analysing these data in terms of potential errors such as parents opening the app but not actually using it.
Limitations in terms of breadth of participant involvement may occur, for example, participants who do not engage with the study may be less likely to respond to invites to take part in interviews. Participants’ interview invites clearly state that the researchers are interested in all views, including those who did not engage with the STEPS app to ensure as wide reach as possible is attained.
Interviews with clinicians may provide limited data as many parents in the study will not yet have been assessed, even after completing the final 12-month timepoint, meaning that clinicians may have limited feedback/views from the parents regarding the app.
Ethics statements
Patient consent for publication
Acknowledgments
Thank you to Cathy Laver-Bradbury for providing advice on the STEPS concept and content. Special thanks go to Catherine Thompson for her work on an earlier prototype—New Forest On-Line. The authors would like to acknowledge and thank the expert members of the PPI panel. The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health and Social Care.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
X @elleshead
Contributors ES-B led the overall conception and design of the study and is the overall OPTIMA Chief Investigator. EH, KK-A and ES-B led the drafting of this manuscript. KK-A is responsible for coordinating the study. BF and EH are responsible for enrolling participants and administering outcome measures and coordinating recruitment. EH and NL are responsible for participant interviews, transcription and qualitative data analysis. NL provides administrative support for the project and for the patient and public involvement activities. CLH, HK and CG contributed to the conception and design of the study. HK and CG had overall responsibility for qualitative data analysis. All authors have read and approved the final version of this manuscript.
Funding This project is funded by the National Institute for Health Research (NIHR) Programme Grants for Applied Research (RP-PG-0618-20003). The views expressed are those of the authors and not necessarily those of the NIHR or the Department of Health and Social Care. The funding body played no role in the study design and conduct. This work was also supported in part by the NIHR Maudsley Biomedical Research Centre at South London and Maudsley National Health Service Foundation Trust and King’s College London to ESB.
Competing interests STEPS concepts and content were designed by ES-B together with other members of the OPTIMA team; David Daley, Johnny Downs, Jana Kreppner, Hanna Kovshoff and Margaret Thompson. STEPS visual design and digital implementation were completed by TOAD with funding provided to ES-B by the South London& Maudsley NHS Trust. Videos were produced by EyeWitness Productions, funded by Solent NHS Trust. ES-B has received speaker fees, consultancy or research funding from Takeda, Neurotech Solutions, QBtech and Medice. He has received royalties from the New Forest Parenting Programme. BF reports personal fees and nonfinancial support from Takeda and Medice.
Patient and public involvement Patients and/or the public were involved in the design, or conduct, or reporting, or dissemination plans of this research. Refer to the Methods section for further details.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.