Evaluation

Our approach to undertaking evaluations in health and social care is underpinned by four principles: 

  • The outcome is co-designed: with stakeholders, then results in more robust and relevant findings. 
  • There is meaningful involvement: meaningful, valued and well supported stakeholder involvement is central to the Stakeholder engagement. 
  • Our work together is reflexive and developmental: to give space and time to consider findings as they emerge, reality check assumptions, and offer a critical eye.
  • Building working relationships: to add capacity and value to the existing expertise of the organisations we work with.

HSMC is also the lead for the BRACE Centre (The Birmingham, RAND and Cambridge Evaluation Centre), a NIHR funded centre carrying out rapid evaluations of innovations in health service organisation and delivery.

Examples of evaluations

Quick links: BRACE Evaluation: Acute hospitals managing general practice services Critical appraisal of primary care innovation programme - Chief Registrar Project - Midland Heart Complex Needs Service - Cancer Research UK - Real Life Options - Evaluation of Systems Leadership in Mental Health

BRACE Evaluation: Acute hospitals managing general practice services

The aims of this project is to understand the early impacts of vertical integration: its objectives; how it is being implemented; whether and how vertical integration can drive the redesigning of care pathways; whether and how services offered in primary care settings change as a result; and the impact on the general practice and hospital workforces. Click here for further information.

Critical appraisal of primary care innovation programme 'Pacesetters' in Wales

The Health Services Management Centre (HSMC) has been commissioned by the Primary Care Hub of Public Health Wales, on behalf of Health Board Directors of Primary Community and Mental Health Services, to undertake a critical appraisal of the Pacesetter programme in Wales. This programme has been funded by Welsh Government to stimulate innovation and promote the redesign of primary care services with pace, with a view to learning from those which deliver benefit and share successes across Wales.

There are 24 projects across the seven Health Boards which are exploring the following themes: 

  • Exploring new models for delivering primary care services
  • Creating new roles within the primary care team
  • Building integrated primary care support and capability
  • Developing innovative referral/demand management models

The programme has led to the development of an emerging model for Primary Care in Wales. This model has six key components that align to Ministerial priorities for the transformation of primary care, which include primary care sustainability, more services delivered in the community and improved access to quality care.

The purpose of the critical appraisal is to identify the key components necessary for the transformation of primary care in Wales, and to understand what enables the components to be successfully deployed in Local Health Board contexts. This work will inform further development of the emerging model for Primary Care in Wales.

The appraisal commenced in June 2017 and be completed by February 2018, and will be undertaken in four stages:

  • Stage 1 will involve interviews with national and local stakeholders with knowledge of primary care and an on-line survey of the 24 Pacesetter initiatives
  • Stage 2 will involve a review of research evidence regarding the components and enablers of primary care transformation and interviews with international experts with experience of relevant large-scale change programmes
  • Stage 3 will involve documentary review and interviews with all leads of the Pacesetter initiatives, and in-depth case study work with 4-6 of these initiatives
  • Stage 4 will involve further stakeholder engagement to refine the emerging model of Primary Care

Researchers

Dr Robin Miller Health Services Management Centre, University of Birmingham
Professor Judith Smith Health Services Management Centre, University of Birmingham
Dr Jo Ellins, Health Services Management Centre, University of Birmingham
Professor Jayne Parry, the Institute of Applied Health Research, University of Birmingham

Funder

Primary Care Hub of Public Health Wales, on behalf of Health Board Directors of Primary Community and Mental Health Services. 

Chief Registrar Project

In September 2013, the Royal College of Physicians  published the report from the Future Hospital Commission  Future Hospital: caring for medical patients, and  laid out a vision for how hospital services can adapt to meet the needs of patients now and in the future. The Future Hospital Programme was established to implement the Commission’s vision for the future of medical care in hospital and community settings. 

The Future Hospital Commission recommended the appointment of a chief (medical) registrar in every acute hospital.  The chief registrar will take a leadership and management role with benefits for the individual, the teams they work in and the organisations as a whole.  They will support the junior doctor workforce whilst acting as a key liaison with the Medical Director and Executive Board.  They will play a vital role in quality improvement and change management.

The aims of the Future Hospital Chief Registrar project are to realise the recommendation of the Future Hospital Commission by leading a pilot of the chief registrar role with selected hospitals.

Professor Mark Exworthy and Iain Snelling have been appointed to evaluate the first cohorts of Chief Registrars who were appointed in 2016, and who will be undertaking a development programme led by the Faculty of Medical Leadership and Management.

The evaluation will be completed in 2017.

Midland Heart Complex Needs Service

Midland Heart provide support services for homeless people with complex needs such as substance misuse, criminal behaviour and mental health problems. They wanted to understand the outcomes that these services facilitate for the individual concerned and how this relates to the responsibilities of different commissioners in health and social care. Undertaken with Contact Consultancy, the evaluation involved interviews with people accessing the services, focus groups with staff members and interviews with key stakeholders.

Steve Russell from Midland Heart: “The evaluation has proven to be very beneficial for Midland Heart, both for the organisation and for the two services directly involved in the process. The experienced research team were able to provide an independent view of the impact of the service, informed by their own expertise in this area. Importantly, they were able to make sensible and actionable recommendations which we have already begun to implement. In addition the outcome of the research, the process has been equally important. Staff working in the services felt included and it has helped enable them to better identify and understand how their work makes a difference to the lives of our customers. It is has also provided the space and opportunity for a deeper reflection on the overall service and the interaction between staff, customers and other agencies. Finally, we wanted to use this as a way to build on our own in house experience and the research team were open in allowing our staff to support the project and the outcome.The overall outcome is a high quality and professional piece of research for us to use”

Cancer Research UK

HSMC have undertaken a number of evaluation projects on behalf of Cancer Research UK. These include an evaluation of the organisation and management of cancer surgery services across the UK, in order to understand the key issues and challenges facing those delivering surgical services. Undertaken in conjunction with ICF International Consultancy, this research involved reviewing and synthesising relevant published literature and evidence, with a consideration of international comparison; qualitative interviews with key stakeholders at a national and local level; and a mixed methods survey with key national and international stakeholders.

Copies of the final report are available through this link - http://www.cancerresearchuk.org/sites/default/files/policy_cruk_cancer_surgery_services_feb14.pdf

In the foreword to the report, the President of the Royal College of Surgery states: "This report makes a timely contribution to this hugely important area of medical care, highlighting the complexity of present service delivery and the considerable need for a greater understanding of what good cancer surgery looks like. I urge clinicians, providers, commissioners and patients to read and embrace its recommendations to help improve the care of all those suffering from cancer." 

A second evaluation considered Cancer Services more broadly. This was also undertaken with ICF International Consultancy and builds on a previous report into the early impact of the reforms, produced for Cancer Research UK by HSMC (Cancer Services: Reverse, pause or progress? 2012). The recent work considers key issues including - leadership and accountability in the new system; the effectiveness of the new commissioning architecture created by the reforms; the new challenges or opportunities emerging; and the factors that will sustain improvements in cancer services. The evaluation involved the analysis of cancer datasets and in-depth interviews at a national level, as well as in five Local Area sites across England.

Quoted in the Guardian, Harpal Kumar, Cancer Research's chief executive, said: "The people that have propped up these NHS services tell us in this report that 'enough is enough'. They can't go on like this with no help or support coming over the horizon. And they certainly can't improve services so that our outcomes are up there with the best in the world."

Other examples of media interest in this report can be found at the links below –

http://www.telegraph.co.uk/health/healthnews/11077656/NHS-cancer-services-are-close-to-collapse-it-is-warned.html

http://www.bbc.co.uk/news/health-29107762

http://www.independent.co.uk/news/science/cutbacks-are-harming-nhs-cancer-care-warn-medics-9717403.html

http://www.theguardian.com/society/2014/sep/08/cancer-coalition-nhs-health-service-budgets

Real Life Options

Real Life Options were embarking on an ambitious change programme to improve the quality of life of people with a learning disability living within their care homes. They asked HSMC to help them understand the difference that the changes made to the individuals and to gather lessons for future such programmes. The evaluations included individual interviews with senior managers, focus groups with staff members, observation of people’s quality of life through the Adult Social Care Outcomes Framework and use of the Personalisation Outcomes Evaluation Tool developed by in-control.

Evaluation of Systems Leadership in Mental Health

In January 2016, Bristol Mental Health, as part of the Avon and Wiltshire Partnership Trust, commissioned the Health Services Management Centre to develop an evaluation approach and undertake an independent evaluation of System Leadership in the Bristol Mental Health system.

Systems leadership has been the focus of academic study since at least the mid-20th century although the concept has been gaining increasing momentum in health and social care in recent years, in response to the arguably unprecedented challenges facing public sector commissioners and providers of services. Agencies – and leaders - operating independently have neither the budget nor the human resources to respond to rising levels of expectation and demand; nor do they have sufficient knowledge to solve complex multi-dimensional problems unless they share information and skills with others (The King's Fund Commission on Leadership and Management, 2011) The system requires leaders who are adept at facilitating that sharing process and able to influence practice change (Fillingham & Weir, 2014).

Attention on systems leaders and leadership is growing – see, for example the publication of the NHS Improvement Leadership Development framework in December 2016, which emphasises the importance of systems leadership, particularly in relation to Sustainability and Transformation Plans – and yet there is no clear agreement on what the term connotes. In evaluating the effectiveness of system leadership we used David Welbourn’s definition:

‘The collaborative leadership of a network of people in different places and at different levels in the system creating a shared endeavour and cooperating to make a significant change’ (Welbourn, 2013)

System Leadership had been developing in Bristol Mental Health since a new model of service delivery came into being in late 2014. The new model had been adopted following extensive consultation involving stakeholders from across the BMH system and was designed to ensure that irrespective of how a person came into contact with mental health services, they would be helped to find the support they required. There was an expectation that all pathways should fit together into an integrated mental health system and recognition that service providers needed to work more effectively in partnership to achieve that. The goal was to provide a more responsive and easier to access service, and for mental health services and care pathways to form an integrated whole.

Despite a strong commitment to achieving better system-level integration, the reality is that many users of public services still experience services provided in professional or organisational silos, meaning that navigating between different care providers can be difficult and outcomes varied depending on which part of ‘the system’ a service user or carer is able to access. This is not a challenge confined to mental health and the reasons for it are complex: they include different approaches to managing performance and decision making, the existence of strong group allegiances and the anxieties for staff of stepping outside known and established working practices (Gask, 2010). For people who use mental health services the challenges can be particularly onerous, since their care needs may also include physical health and social care support and because they may lack confidence in asserting their need for support. Re-designing services to deliver integration at system level is therefore challenging, requiring staff to work flexibly across boundaries and professional groups, and calling for confident, skilled leadership focused around a clear and shared purpose. 

Even when there is, as in Bristol, a clearly-defined goal, the approach requires courage, persistence (changing cultures takes time), the ability to reflect, to learn, and to unlearn, and the capacity to foster generative conversations which shift cultures from reactive problem-solving to co-creating the future. It asks considerably more of system leaders than that they retain their individual power and authority whilst working in partnership with others; organisational leaders have to contend with loss – for example, of authority to command and control resources – as they move towards shared decision-making and a long-range vision for the future. At the same time, performance targets and incentives remain firmly organisationally-focused and resolutely short-term.

Our evaluation of system leadership in BMH acknowledged the ambition of the model, which had done much in the first phase to shift mind-sets and thinking amongst key stakeholders. In our interviews with people across the mental health system there were signs that, despite the complexity of the issues surrounding the integration of mental health services, leaders were coming together to lead collaboratively and collectively, developing a common governance infrastructure, shared information systems and revised performance frameworks. Enablers in that process included a focus on place, the willingness of leaders to take risks, and support from a dedicated system leadership team. Fundamentally important was a clear focus on shared values, placing the experience of service users at the heart of the model.

In our work we developed a framework for evaluating system leadership which assesses performance of a system across seven dimensions, including the extent of service user and carer involvement, the degree to which there is shared information to inform decision-making and the quality of relationships and of collective leadership in the system. Using this framework, the BMH system leaders have begun to diagnose the performance of the system, identify priorities for action and development and work collaboratively to design and deliver improvements.

It is important not to under-estimate the scale of the challenge and the time it will take to deliver real cultural change in a system as complex as Bristol Mental Health. If system leadership is to deliver improved outcomes for service users through stronger integration and shared care pathways it will require a relentless focus on the shared vision over the coming months and years. Systems leadership will be a significant capability for health and social care in the coming months and years, and our work to develop systems leadership capability and to evaluate its effectiveness is proving both vital and successful.

If you are interested in finding out more about our approach to developing systems leaders and the evaluation of system leadership, including the model and framework we have developed, please contact Belinda Weir (B.Weir@bham.ac.uk), Director of Leadership at HSMC.