How do we measure becoming more inclusive? National Music Services Working Group on Inclusion, Sept ’21
All of us working in music education need to measure what we do. We set goals in the form of numbers, percentages or outcomes; and we measure against them. We do this for our own business planning, monitoring and evaluation1; for reporting to our funders such as Arts Council England (ACE)2 through the hubs data return; and for demonstrating our value and impact to potential commissioners of services or funders. So how can music services and their hub partners measure becoming more inclusive? What KPIs, outputs and outcomes should we be integrating into our plans for equality, diversity and inclusion? This was the topic for discussion in the National Music Services Working Group on Inclusion in September.
Take a look at our blog from the July ’21 working group:
16 music service leads share ‘what we’ve learned about embedding inclusion in our music services’
In the meeting, we wanted to find out about current concerns, any solutions, and how Changing Tracks could help people move forward on measuring inclusion and tracking change. Not surprisingly, we came away with more questions than answers. So, we’ll be setting up a Task and Finish group (dates TBC) to work on proposals that could inform music services’ work nationally, and might support the refreshed National Plan for Music Education and future versions of the ACE hub data return.
Read on to find out more about the areas the group explored.
What do we mean by ‘measuring inclusion’?
Measuring inclusion in music services can include actions to create change in these different areas:
- young people and families – increasing the diversity and number of participants
- workforce – diversifying the people and their skills
- leadership and governance – strategy, policies, goals, values, culture
- operations – structure, systems/processes, communications
These are the headings in the Youth Music ED&I Action Plan Template that Changing Tracks uses throughout its work.
What are the benefits of music services developing KPIs for inclusion?
- Evidencing good use of the Hub grant to reach a more diverse range of children and young people
- Demonstrating to local authority commissioners of services for children, young people and families, that your work aligns with their agendas and can help them achieve their goals
- Showing school leadership and staff that your work can improve social and personal outcomes for children and young people, eg wellbeing, mental health, alongside musical outcomes.
What are some of the challenges to or questions about measuring inclusion?
- We don’t have a baseline to measure against
- We don’t know what goals are reasonable to set
- How can we get access to data and where from?
- How do we set up our systems to collect this data?
- What data should we collect about young people eg protected characteristics or just FSM data?
- At what level do we measure – school level (% of pupils supported by Pupil Premium (PP), locality level (areas of deprivation)?
- We don’t know what organisational measures to put in place. We’re doing things differently in many areas, but how do we count that?
- How do we measure what percentage of local young people our service reaches? And how, with what activities and level of engagement and continuation? The unpalatable truth is lessons, instruments and groups are too expensive for many families.
Tips and insights: how might we measure becoming more inclusive in music services?
1. Measures can be counterproductive if they require resources from services already at capacity, but much may be available, and possible within existing systems.
2. Who else has data about young people? Talk to them. Eg Local Authorities, schools, Bridge Organisation, Cultural Education Hub, and organisations beyond arts/music. Use data about your local population as a baseline. You can get this from your local authority. You might set goals saying you want to be reflective of the local population plus or minus eg 10%.
3. Start with what data you do have access to eg: uptake of bursaries/remissions for instrument hire and music centres/groups.
4. Using lesson administration systems and Unique Pupil Numbers (available via schools) can help music services track engagement, retention and progression of young people.
5. Use data about and from your staff: eg an annual ED&I survey which may include questions to find out how included and represented they feel; or how they have changed their attitudes to ED&I or their practice.
6. Work with schools: they could help you understand what change they’ve seen in pupils, and to track individual pupils’ progress (social, personal, musical outcomes) over time – including after involvement with a music service – via the ‘Unique Pupil Numbers’. Talk to SENCOs as well as leadership/music staff.
7. Work with your local authority: some have a corporate inclusion lead post, and targeted support teams (eg Educational Psychology), who may be able to help and advise you – particularly if you can show them that you’re a frontline service supporting local communities in a positive way. Many have inclusion training available.
8. Capture data in a way that is meaningful to schools, tutors, funders and participants– In the words of one delegate ‘Don’t White Coat it!’.
9. Involve and train young people and staff in collecting data and writing case studies. Make it as participatory as possible – ask young people and tutors to co-create any questionnaires and processes so they understand it and so it feels like a natural part of any session.
10. Stories and anecdotes are important – sometimes more so than numbers eg, loaning a keyboard made a massive difference to a young person at risk of going into the care system; having a guitar workshop at 9am ensured that a certain group of boys were in school on time. It’s best to combine numbers and stories, quantitative data and qualitative. Could we measure improved attendance at school?
Coming soon …
- a ‘Equality, Diversity and Inclusion KPIs’ tool (collaborative, to be co-created by music services)
- a Task and Finish group on measuring inclusion (dates to follow).
- You may also be interested in our popular ‘Stages and indicators of inclusion in music services and hubs’ resource.
- Follow us on socials and sign up to our enews to be among the first to know about new events, resources and working groups.
 DEFINITIONS: KPI, PERFORMANCE MEASUREMENT, AND EVALUATION
Wikipedia describes a KPI (Key Performance Indicator) as “a type of performance measurement. KPIs evaluate the success of an organisation or of a particular activity (such as projects, programs, products and other initiatives) in which it engages. Often success is simply the repeated, periodic achievement of some levels of operational goal (e.g. zero defects, 10/10 customer satisfaction), and sometimes success is defined in terms of making progress toward strategic goals… There are two categories of measurements for KPIs. Quantitative [and] Qualitative [which] represent … interpretation of personal feelings, tastes, opinions or experiences. An ‘indicator’ can only measure what ‘has’ happened, in the past tense.
Better Evaluation has this to say about the difference between performance measurement and programme evaluation:
The essential purpose of both is to provide information to help improve the effectiveness, efficiency, equity (and/or other evaluative criteria) of public services. While some of the information may overlap, each of the two processes provides some information that the other, typically, does not. Hence, there are distinct advantages to integrate both processes into an agency’s monitoring and evaluation (M&E) approach framework.
For example, performance measurement works on the assumptions that the operational theory of change is correct and will lead to intended outcomes, and, that internally measured data are valid. Program evaluation, with its in-depth assessment and often conducted by those external to the program, can provide key information not only on the extent to which desired outcomes have been achieved, but also why or why not, and to what extent the program has caused (or contributed) to the results.
 The six KPIs agreed between Arts Council England and the Department for Education (DfE), and which hubs and therefore music services are monitored against, are the numbers and percentages of:
1. state-funded schools engaging with their music hub on one or more core role (primary/secondary split)
2. pupils receiving at least one term of whole class ensemble tuition (WCET) (for each year group 1-9)
3. pupils continuing their musical education beyond WCET (split between primary/secondary and by different pupil characteristics)
4. pupils participating in school or hub-led instrumental ensembles and choirs (split between primary/secondary and by different pupil characteristics)
5. pupils who sing regularly in school or hub-led ensembles and choirs (split between primary/secondary and by different pupil characteristics)
6. Percentage of total income that comes from DfE/local authorities/schools/parents/other sources (average and range).
In addition, the return asks a range of ‘narrative’ questions which allow for hubs to provide context and qualitative information.
 For more information about organisational structures and processes, see the McKinsey 7S Framework – referenced on pages 32/33 of The Power of Equality published by Youth Music.