Learning from evaluation - full abstract


Gemma Shields, Deborah Buck, Linda Davies
The University of Manchester

Background: Preventing psychosis and effective treatment in first episode psychosis are key priorities in the guidelines available from the National Institute for Health and Care Excellence (NICE). Evidence demonstrates that psychological therapies, early intervention services and antipsychotic medications can improve symptoms, quality of life and functioning for these patients. Our review assessed the evidence base for the cost-effectiveness of health and social care interventions for people at risk of psychosis and for first episode psychosis. Additionally, we aimed to determine the robustness of current evidence and identify gaps in the available evidence.

Methods: Electronic searches (conducted in June 2019) were run on the PsycINFO, MEDLINE and Embase databases to identify relevant economic evaluations published within the last twenty years. To meet the criteria for inclusion, studies had to report a full economic evaluation (synthesising costs and health benefits). Additional inclusion criteria were people at risk of psychosis or experiencing first episode psychosis (no restriction by age); any health or social care provided intervention; and the comparator included could be no intervention (usual care) or an active intervention. Screening, data extraction, and critical appraisal were performed using pre-specified criteria and forms based on the NHS Economic Evaluation Database (EED) handbook and Consolidated Health Economic Evaluation Reporting Standards (CHEERS) checklist for economic evaluations. The protocol was registered on the PROSPERO database (CRD42018108226). Results were summarised qualitatively.

Results: Searching identified 1,452 citations (1,178 following the removal of duplications). After two stages of screening 14 studies met the inclusion criteria and were included in the review. Interventions were varied and included multidisciplinary care, antipsychotics, psychotherapy and assertive outreach; with the most common being cognitive behavioural therapy (5/14 studies). Studies were most commonly integrated clinical and economic randomised controlled trials (9/14) and the majority were conducted in Europe (9/14). Trial follow-up ranged from 24-weeks to 5-years. Eleven studies included a cost-effectiveness analysis, with a wide variety of health benefits used, including relapse, recovery, death, averted psychosis, years with disability and symptom measures. Seven studies included a cost-utility analysis, most commonly (6/7) using the EQ-5D to inform Quality-Adjusted Life Years (QALYs). Incremental Cost Effectiveness Ratios (ICERs) for cost-utility analyses ranged from dominant (cost saving and QALY increasing) to €428,842 per QALY gained. The majority of studies (10/14) had favourable outcomes, such as cost savings or cost-effective (10/14).  However, all studies had limitations and demonstrated uncertainty.

Implications: Although many studies concluded that interventions for people at risk or psychosis or experiencing first episode psychosis are cost-saving or cost-effective, there was considerable uncertainty and a wide range of results across studies. A key limitation is that studies used a broad variety of health benefit measure for cost-effectiveness, which cannot be compared across studies and applied arbitrary thresholds for cost-effectiveness. Decision/policy makers wishing to use the evidence would need to consider whether the results can be generalised to their setting and consider their accepted willingness to pay for specific health gains.  As more evidence becomes available, this review will need to be updated.

Results from the development and testing of the world’s first principles-based fidelity index for peer support in mental health services

Lucy Goldsmith
St George's, University of London

Background: Research suggests that the distinctiveness of peer support compared to other forms of mental health support can be compromised in large healthcare organisations for a number of reasons, including the difficulty of establishing whether the support given was truly peer support. There is good evidence that the distinctiveness of peer support can be articulated as a set of principles relating to implementation of peer worker roles. 

Methods: Five principles underpinning one-to-one peer support in mental health services were identified through a literature review and consultation with an expert panel (Gillard et al., 2017). A fidelity index mapping onto these five principles was developed, with supporting semi-structured interviews for use with peer workers, peer worker team leaders and supported peers.  Guidelines for conducting interviews, and for scoring the index were developed. These were piloted at independent peer support organisations in U.K. National Health Service peer support services and at the seven sites of the ENRICH trial of peer support for psychiatric discharge. Blind double-rating was used to verify inter-rater reliability and the psychometric properties of the scale send were verified. Experts with lived experience were involved throughout the research (coproduction); (Gillard et al., 2012). We established the inter rater reliability (IRR) and internal consistency (IC) of domain and total scores. IRR was assessed using intra-class correlation coefficients (ICC’s) calculated with two-way random effects models, assuming absolute agreement. We used the criteria given in Cicchetti and Domenic (1994) to interpret the acceptability of the estimated ICC’s; poor < 0.40, 0.40 – fair - 0.59, 0.60 – good – 0.74, 0.75 + excellent. IC was assessed by Cronbach’s alpha statistics, and is considered acceptable if >=0.7. 

Results: Across the entire index, the total score had an acceptable internal consistency, Cronbach’s alpha = 0.81. The intra-class correlation coefficients were all 0.67 or above, indicating good inter-rater reliability.

Implications: The fidelity of peer support in mental health services can be measured in terms of principles that articulate the distinctiveness of peer support.  It is acceptable and feasible to develop robust measures of values-based fidelity for complex healthcare interventions where psycho-social and relational mechanisms are core to change processes. Values-based fidelity measures might usefully complement conventional, dosage approaches to trial fidelity. Beyond research purposes, the principles-based fidelity index can support the implementation and evaluation of peer worker roles.


Cicchetti DV (1994) Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6:4, 284–290; | 

Gillard S. et al (2017) Describing a principles-based approach to developing and evaluating peer worker roles as peer support moves into mainstream mental health services, Mental Health and Social Inclusion, 21:3, 133-143.

Gillard S. et al (2012). Patient and Public Involvement in the Coproduction Of Knowledge. Qualitative Health Research, 22: 1126-1137.

A systematic review of economic evaluations of antenatal nutrition and alcohol interventions and their associated implementation interventions.

Zoe Szewczyk(1), Elizabeth Holliday(2), Brittany Dean(2), Clare Collins(2), Penny Reeves(2)
(1)University of Newcastle, (2)The University of Newcastle

Background: Health promotion interventions targeting pregnant women have the potential to improve maternal and infant outcomes in the short term and reduce the burden of disease across the lifespan. 1 Specifically, poor nutrition and alcohol consumption during pregnancy have been identified as risk factors for adverse health outcomes.1 National dietary guidelines provide specific advice about healthy eating during pregnancy.2 Despite these guidelines, suboptimal antenatal nutrition is common in developed countries and may contribute to adverse health outcomes and increased utilisation of health care resources throughout the delivery period and across the lifespan.3 Similarly alcohol consumption during pregnancy can increase the risk of various adverse health and developmental problems for infants and older children. Clinical practice guidelines recommend that pregnant women abstain from alcohol use, since there is no safe level of alcohol consumption during pregnancy.4 Despite these recommendations, alcohol use during pregnancy is common and fetal alcohol spectrum disorder is the most common preventable cause of neurodevelopmental abnormalities in the western world.5 Improving antenatal nutrition and promoting alcohol abstinence are identified mechanisms for reducing subsequent economic and social impacts. However, antenatal health promotion interventions are underused, partly because economic evidence to support investment is limited. Valuing resources required for health promotion interventions and their implementation, in conjunction with their effectiveness, will inform the value for money derived from investment and may improve health service efficiency.

Methods: Assess the extent to which economic evaluations have been applied to (1) antenatal public health interventions targeting nutrition and alcohol intake; (2) implementation-strategies of antenatal public health interventions targeting antenatal nutrition and alcohol intake; and (3) synthesize the evidence to develop recommendations to facilitate and promote the conduct of future evaluations. 

Two separate systematic reviews were conducted to address aims (1) and (2). Both reviews adhered to PRISMA guidelines. 

Results: Review (1) returned 9599 records after duplicates were removed, from which 12 economic evaluations were included. Review (2) returned 136 records after duplicates were removed, with none eligible for inclusion. The articles included in review (1) comprised ten economic evaluations of nutrition interventions and two evaluations of alcohol interventions. 

Review (1) indicated antenatal nutrition and alcohol interventions offer value for money. Methodological quality was high, there was a predominance of modelled evaluations and no evidence regarding intervention implementation, and its associated costs. Review (2) identified no evidence of implementation costs. 

Implications: Health promotion interventions have the potential to reduce health care costs. Assessment of the effectiveness and cost-effectiveness of nutrition and alcohol interventions and their implementation strategies is necessary given competing claims on health budgets. Despite this need, there is an absence of evidence of the total cost and cost-effectiveness of antenatal nutrition and alcohol interventions and their implementation strategies. Greater understanding of the cost-effectiveness of providing health promotion services to pregnant women will help inform decisions about how to derive value from investment in health care.


  1. Barker DJ, Osmond C. Infant mortality, childhood nutrition, and ischaemic heart disease in England and Wales. Lancet. 1986;1(8489):1077-1081.

  2. National Health Service (NHS). Have a healthy diet in pregnancy - Your pregnancy and baby guide. https://www.nhs.uk/conditions/pregnancy-and-baby/healthy-pregnancy-diet/. Published 2017. Updated 27 January 2017. Accessed 22/10/19.

  3. Thangaratinam S, Rogozińska E, Jolly K, et al. Effects of interventions in pregnancy on maternal weight and obstetric outcomes: meta-analysis of randomised evidence. BMJ : British Medical Journal. 2012;344.

  4. World Health Organisation. Guidelines for the identification and management of substance use and substance use disorders in pregnancy. Geneva2014.

  5. Popova S, Lange S, Burd L, Chudley AE, Clarren SK, Rehm J. Cost of fetal alcohol spectrum disorder diagnosis in Canada. PLoS ONE. 2013;8(4):e60434.

How to cost the implementation of major system change: case study using reconfigurations of specialist cancer surgery in part of London, UK

Caroline Clarke(1), Cecilia Vindrola-Padros(2), Claire Levermore(3), Angus Ramsay(2), Georgia Black(2), Kathy Pritchard-Jones(4), John Hines(5), Gillian Smith(6), Axel Bex(6), Muntzer Mughal(3), David Shackley(7), Mariya Melnychuk(2), Steve Morris(8), Naomi Fulop(2), Rachael Hunter(9)
(1)University College London, (2)Department of Applied Health Research, University College London, London, UK, (3)University College London Hospitals NHS Foundation Trust, London, UK, (4)University College London Hospitals NHS Foundation Trust, London, UK; UCL Partners Academic Health Science Network, London, UK, (5)University College London Hospitals NHS Foundation Trust, London, UK; London Cancer, University College London, Cancer Collaborative, London, UK; Bart’s Health, NHS Trust, London, UK, (6)Royal Free London NHS Foundation Trust, London, UK, (7)Greater Manchester Cancer, (hosted by) Christie NHS Foundation Trust; MAHSC (Manchester Academic Health Sciences Centre), Manchester, UK, (8)Primary Care Unit, Department of Public Health and Primary Care, University of Cambridge, (9)Research Department of Primary Care and Population Health, University College London, London, UK

Background: Studies have been published on the impact of major system change (MSC) on care quality and outcomes, but few evaluate implementation costs or include them in cost-effectiveness analysis (CEA). This is despite their potential influence on implementation decisions, and potential large costs, associated with for example change planning, purchasing or repurposing assets, and staff time. We present a case study illustrating our framework and principles for costing MSC implementation, which we have developed previously during work on stroke.

Methods: We outlined MSC implementation stages and identified components, using the new MSC costing framework. We present a case study using the RESPECT-21 mixed-methods evaluation of specialist surgery services reconfiguration for prostate, bladder, renal and oesophago-gastric cancers, focusing on a London region where services for eight cancer pathways were centralised to fewer high-volume units. Health economists collaborated with qualitative researchers, clinicians and managers, identifying key reconfiguration stages and expenditures. Data sources (n=100) included meeting minutes, interviews, and business cases. NHS finance and service managers and clinicians were consulted. Using bottom-up costing, items were identified, and unit costs based on salaries, asset costs and consultancy fees assigned. Itemised costs were adjusted and summed, and discounting approaches explored.

Results: Cost components included: (A) options appraisal, bidding process, external review; (B) stakeholder engagement events; (C) planning/monitoring boards/meetings; and (D) making the change: new assets, facilities, posts. Other considerations included: hospital tariff changes; costs to patients; patient population; and lifetime of changes. Using the framework facilitated data identification and collection. Present value of the total implementation cost, adjusted to 2017-18 prices, was estimated at £5.1 million, of which £1.1 million could be attributed to other concurrent reconfigurations. The total was disaggregated into the following categories: replacing robots (£2.8 million), consultancy fees (£1.4 million), staff time costs (£0.8 million) and other costs (£0.1 million).

Implications: The framework and principles can be used by funders, service providers and commissioners planning MSC, and researchers evaluating MSC. We suggest that health economists should be involved early alongside qualitative and health-service colleagues, as retrospective capture risks information loss. These analyses are challenging; many cost factors are difficult to identify, access and measure, and assumptions regarding time horizon are important, as our case study illustrates. Including implementation costs in CEA might make MSC appear less cost-effective, influencing future decisions. Future work incorporates this implementation cost into the full CEAs of the London Cancer reconfiguration.


Staff experience of a test-and-learn of the Buurtzorg care model in West Suffolk

Jo Maybin, Ethan Williams, Matthew Honeyman, Susie Perks Baker
The King's Fund

Background: The Buurtzorg model of care, developed in the Netherlands in 2006, involves small teams of nursing staff providing personal, social and clinical care to people at home in a particular neighbourhood. There is an emphasis on staff working with clients to access resources in their wider social networks to support them to become more independent. The nurses work in ‘non-hierarchical self-managed' teams and can access support from a coach and a central back office.

Evidence indicates that this care model has achieved high levels of patient and staff satisfaction in the Netherlands without increasing costs, and there has been significant interest in the model in the UK.


A group of local government and NHS organisations in West Suffolk have been undertaking a test-and-learn of the Buurtzorg model in a rural village. This study documented progress with establishing a new service and produced an account of staff experience of transitioning to these new ways of working.


Methods: A longitudinal case-study approach was adopted. Data was generated through in-depth semi-structured interviews with nurses and mangers involved in the test at two time points (n.31 total); observations from a two-hour workshop with those staff; analysis of management documents associated with the test; notes from bi-monthly telephone updates between the review lead and the test’s project coordinator; and analysis of administrative data on activity.


Analysis was inductive. A thematic coding framework was developed from contemporaneous notes made during interviews, observation, and telephone updates. Computer software (Dedoose) was used to support the coding of all interview transcripts and observation notes using the established framework. Inter-coder reliability was tested by triple coding a sub-sample of transcripts.


Results: The staff were successful in establishing a new neighbourhood team which offered health and care support to people in their homes. Clinicians and managers reported that the care was person-centred, holistic and enabled individuals to make significant improvements to their health and independence. Nursing staff reported very high levels of satisfaction with the care they were able to provide. A key factor was staff having the time to listen to patients and carers, and the autonomy and mandate to act on what they learned in those conversations to support the individuals. The caseload remained very small.


But the test faced major challenges. The introduction of non-hierarchical self-management combined with an initial expectation the members of the nursing team (with support from managers) would develop much of the organisational infrastructure and service design for the service was one of the greatest sources of difficulty for staff. Related difficulties with recruiting and retaining clinical staff made it difficult to establish and maintain an effective team. The team did not take on responsibility for all the personal and social care needs for clients as intended, due to a lack of capacity and a lack of clarity about what that work comprises.


Implications: In the context of a national workforce crisis, plans for innovative ways of working need to be developed with available staff, taking account of their number, skills and motivations.

Self-management involves a significant cultural shift for clinicians and managers and requires a host of leadership and management skills which need to be actively taught and supported.

Challenges faced in Evaluating the National Health Services Diabetes Prevention Programme.

Rathi Ravindrarajah, Evangelos Kontopantelis,
University of Manchester, 

Background: The NHS Diabetes Prevention Programme (NDPP) is a joint effort by the NHS and Public Health England to implement a behaviour-change programme to those patients who are at risk of developing Type 2 Diabetes Mellitus (T2DM).

The proportion of the population with T2DM has been rising globally and is an important contributor to mortality, morbidity and health care costs. 1. It has been suggested that currently there are 5 million people in England who are at risk of developing T2DM 2.  People who have raised blood glucose levels but not in the diabetic range are identified as those at risk of developing T2DM and this condition is known as Non-Diabetic Hyperglycaemia (NDH). Previous research has shown that individuals diagnosed with NDH are at a higher risk of developing T2DM 3 and cardiovascular conditions.


Methods: NDPP is delivered by non-NHS providers, but most of the participants are identified and referred by GP practices. We use the Clinical Practice Research Datalink (CPRD), a large administrative primary care data, which is generally representative of the UK population, especially in terms of practices and patient deprivation. We have got access to both primary cares using Vision (GOLD) and EMIS (EMIS) software. It consists of complete data on all aspects of care (diagnoses, referrals, treatments, tests) and has also been linked to Hospital Episode Statistics (HES) and Office for National Statistics (ONS) data, allowing the construction of a complete patient journey through primary and secondary care. NDH Read codes are routinely used in the primary care, also the NHS DPP has specific read codes used by GP to identify the patients who were referred, attended and completed the programme. We will use the identified codes to classify patients as scheme-eligible. The effectiveness of the NHS DPP will focus on the effect of the programme on conversion of NDH to T2DM for, those diagnosed with NDH, patients referred to the programme and those participating in the intervention. We plan to explore the effectiveness by only using data from the post-intervention period and compare NDH to T2DM conversion rate between patients referred to the scheme versus matched patients not referred, within the same practice and practices referring to the scheme versus practices that are not referring.  The primary outcome is conversion to T2DM within a year and the secondary outcomes are death, all-cause hospitalisations, diabetes related hospitalisations, primary care visits, weight, BMI and HbA1C.


Results: We had issues regarding the definition of the NDH from our previous analysis as the guidelines of defining NDH has changed over the years. Not all practices in primary care were referring patients. In addition we discovered that patients were not always invited to the programme via GP. Hence, it would be hard to identify if there were patients in the primary care who might have attended the programme but not been recorded. In both datasets we found individuals who had a referral code as well as a referral declined code (CPRD Aurum: n=3620: CPRD Gold: n= 327)

Conclusions: Although the NHS DPP is based on a strong international evidence base, justifying the commissioning of such a large and complex programme requires rigorous scientific evidence that the programme is achieving benefits beyond current practices. The roll-out of the programme makes formal randomised evaluation problematic and analysis need to be adapted as needed.


  1. Worldwide trends in diabetes since 1980: a pooled analysis of 751 population-based studies with 4.4 million participants. The Lancet 2016;387(10027):1513-30. doi: 10.1016/S0140-6736(16)00618-8

  2. Hudson H. The NHS Heath Check screening and non-diabetic hyperglycaemia. Practice Nursing 2016;27(10):473-80. doi: 10.12968/pnur.2016.27.10.473 

  3. Federation ID. IDF Diabetes Atlas. 8th Edition ed: Brussels, Belgium, 2017.

Funding: This research was funded by the National Institute for Health Research (Health Services and Delivery Research, 16/48/07 – Evaluating the NHS Diabetes Prevention Programme (NHS DPP): the DIPLOMA research Programme (Diabetes Prevention – Long Term Multimethod Assessment)).  The views and opinions expressed in this presentation are those of the author(s) and do not necessarily reflect those of the NHS, the National Institute for Health Research or the Department of Health and Social Care


Implementing pelvic floor muscle training for women with pelvic organ prolapse: a realist evaluation of different delivery models

Purva Abhyankar(1), Joyce Wilkinson(1), Karen Berry(1), Sarah Wane(2), Isabelle Uny(1), Patricia Aitchison(1), Edward Duncan(1), Eileen Calveley(1), Helen Mason(3), Karen Guerrero(4), Douglas Tincello(5), Doreen McClurg(3), Andrew Elders(3), Suzanne Hagen(3), Margaret Maxwell(1)
(1)University of Stirling, (2)University of Northumbria, (3)Glasgow Caledonian University, (4)Queen Elizabeth University Hospital, (5)University of Leicester

Background: Pelvic Floor Muscle Training (PFMT) has been shown to be effective for pelvic organ prolapse in women, but its implementation in routine practice is challenging given the lack of adequate specialist staff. It is important to know if PFMT can be delivered by different staff skill mixes, what barriers and facilitators operate in different contexts, what strategies enable successful implementation and what are the underlying mechanisms of their action. PROPEL intervention was designed to maximise the delivery of effective PFMT in the UK NHS using different staff skill mixes. We conducted a realist evaluation (RE) of this implementation to understand what works, for whom, in what circumstances and why. 

Methods: Informed by the Realist and RE-AIM frameworks, the study used a longitudinal, qualitative, multiple case study design. The study took place in five, purposively selected, diverse NHS sites across the UK and proceeded in three phases to identify, test and refine a theory of change. Data collection took place at 4 time points over an 18 month implementation period using focus groups and semi-structured interviews with a range of stakeholders including service leads/managers, senior practitioners, newly trained staff and women receiving care in the new service models. Data were analysed using hematic framework approach adapted to identify Context, Mechanism and Outcome (CMO) configurations of the Realist approach. 

Results: A heightened awareness of the service need among staff and management was a mechanism for change, particularly in areas where there was a shortage of skilled staff. It enhanced the staff’s readiness to extend the reach of PFMT services in the community setting, train different staff types and initiate ‘workarounds’ to enable triage and referrals to the new services. In contrast, the most established specialist physiotherapist-delivered PFMT service activated feelings of role protection and compromised quality, which restricted the reach of PFMT through alternative models. Staff with some level of prior knowledge in women’s health and adequate organisational support were more comfortable and confident in new role. Implementation was seamless when PFMT delivery was incorporated in newly trained staff’s role and core work. 

Implications: Roll-out of PFMT delivery through different staff skill mixes is possible when it is undertaken by clinicians with an interest in women’s health, and carefully implemented ensuring adequate levels of training and ongoing support from specialists, multi-disciplinary teams and management. Successful implementation of PFMT for prolapse requires a) adequate training tailored to differential needs of skill-mix; b) increased awareness of PFMT among women, GPs and other healthcare practitioners; c) well-coordinated and flexible referral systems; d) wider (multidisciplinary) team support/buy-in for PFMT delivery through different staff skill; e) organisational and managerial support (in terms of resources, training, time, autonomy and staffing) with effective leadership; and f) balancing of likely feelings of role protection with the population needs.

Improving transitions from hospital to home: Using ‘real world’ logic models to derive learning from a feasibility trial of a ‘hybrid intervention’

Thomas Mills(1), Ruth Baxter(1), Rosie Shannen(2), Jenni Murray(2), Rebecca Lawton(2), Jane O'Hara(2)
(1)Bradford Institute of Health Research, (2)BTHFT

Background: ‘Hybrid interventions’, a combination of common and variable elements (Lilford, 2018), are increasingly common in intervention research. Allowing interventions to adapt on delivery via variable elements should enhance their effectiveness but intervention complexity can be a barrier to implementation (Carrol et al, 2007). Very little literature exists regarding the challenges of implementing hybrid interventions in trials. Here, we report some key learning from a cluster randomised controlled feasibility trial that sought to assess a hybrid intervention called ‘Your Care Needs You’ (YCNY). The YNCY intervention was developed to improve the safety and experience of older people as they transition from hospital to home. Its common elements include a patient-friendly booklet, short film and discharge letter, which, when supported by staff, may help prepare patients for discharge, primarily through the mechanism of patient involvement (O’Hara, 2020). It is anticipated that patients will read the booklet and ask questions. Ward staff will, in turn, be supported to respond in ways that will vary in accordance with patients’ unique needs and preferences.

Method: A qualitative assessment of feasibility was conducted in 5 diverse hospital wards across three acute NHS trusts, involving observations and both staff (n-17) and patient (n-10) interviews. Data were analysed using pen portraits (Sheard et al, 2019), thematic analysis techniques (Guest et al, 2011) and a ‘real world’ logic model (Mills et al, 2019), which express how complex interventions work in context. Here, we juxtapose some key findings of the research with the learning points that will be taken forward into the trial, using the logic model to convey how the intervention will be implemented in the trial.  

Results: Staff and patients generally viewed the YCNY intervention favourably but a lack of staff time, unclear staff roles and confusion about how the intervention related to recruitment processes complicated delivery. A tick box approach was sometimes adopted where staff focused on delivering of intervention components rather than on how staff can prepare patients for home. Key learning for the trial relates to the importance of extensive facilitation during the early stages of study set-up and a lengthier embedding period. Facilitators must leverage organisational resources and overcome technological and policy-regulatory barriers early on to create a ward environment that is conducive to the intervention’s delivery. Staff training will also have to be flexible, giving staff clarity over roles while allowing roles to flex by ward. As the logic model expresses (see Figure 1), early set-up work will itself have to ‘flex’ to each Trust environment, constituting a distinct stage of system-level adaptation prior to a stage of patient-level adaptation.

Implications: Significant learning can be derived about how to implement hybrid interventions during the early stages of complex intervention development. ‘Real world’ logic models, which have not yet been used in a trial setting, represent a viable analytic technique for expressing such learning. A key benefit of the approach is how it represents the implementation strategy, interventions and delivery settings in a single model, allowing synergies between them to be assessed.

Figure 1: Logic model of the YCNY intervention



Carrol C, Patterson M, Wood S, Booth A, Rick J and Balain S (2007), A Conceptual Framework for Implementation Fidelity, Implementation Science, 2, 40

Guest G, MacQueen K and Namey E (2011), Applied Thematic Analysis, UK: Sage Publications

Lilford R (2017), Implementation Science at the Crossroads, BMJ Quality & Safety, 27, 4

Mills T, Lawton R and Sheard L (2019), Advancing Complexity Science in Healthcare Research: The Logic of Logic Models, BMC Medical Research Methodology, 19

O’Hara J, Baxter R and Hardicre N (2020), ‘Handing Over the Patient: A FRAM Analysis of Transitional Care Combining Multiple Stakeholder Perspectives, Applied Ergonomics, 85

Sheard L and Marsh C (2019), How to Analyse Longitudinal Data from Multiple Sources in Qualitative Health Research: The Pen Portrait Analytic Technique, BMC Medical Research Methodology, 19