Proving it: how music services can evaluate and evidence the outcomes of music making for vulnerable young people – part 1
Evaluation and evidencing outcomes is an important skill for music services working with young people facing barriers in their learning and their lives. But with so many evaluation models around, where do you start? Lyndall Rosewarne, strategic lead of the Changing Tracks programme, calls for an approach that is authentic and straightforward.
When vulnerable young people have been referred to us, evaluation should enable us to check that their participation is supporting them in ways that are helpful to them, and if not, to adapt what we’re doing. Using evaluation to report back to funders on the learning from the project is important, but should be secondary.
But which outcomes should we focus on? And how can we realistically evidence that change is happening?
Music service tutors are not music therapists – so we don’t necessarily need clinical-standard evaluation
It’s tempting to think we should use clinical-standard, complicated matrices to ‘prove’ the worth of the work. But our focus is on making and learning music: not a therapeutic intervention like music therapy.
Making and learning music supports vulnerable young people because they enjoy it, are challenged by it, and discover they can achieve and learn as a result. It can help with their sense of identity, their wellbeing and ability to cope with life’s challenges. It creates emotional and social connections through which they build trust with adults, form friendships and even find their ‘tribe’, and so redefine themselves.
There are many ways we can evaluate all these different outcomes, but it’s important that our evaluation methods are:
- authentic (not based on claims that we can’t evidence)
- appropriate to the tutors and the young people we work with
- ideally, able to capture longitudinal outcomes (as music services, we may run targeted work to engage young people, but the aim should always be to open up progression routes which keep young people learning in the long term)
- embedded in existing music service systems (from observation and quality monitoring, to student reports and data collection) – otherwise inclusion may risk always being treated like a one-off project
In fact, when inclusive practice becomes truly embedded in music service systems – evidence is naturally authentic and appropriate – because it’s focussed on young people’s lived experience. So by ‘being embedded’ we mean by training tutors in reflective practice and documenting outcomes in case studies; and senior managers in inclusive lesson observation and using reflective questioning with tutors.
What do young people – and those who work with them – want?
Young people tell us that they are tired of interventions, they want to learn music, and to be considered musicians, pure and simple.
Schools, youth services and community partners often have surprisingly simple requirements. They want young people to engage with the activity, experience fun and enjoyment, moments of wellbeing and a sense of having achieved something.
This may be all your evidence needs to show. So the outcomes and indicators we use should be light touch and easily captured as part of the session format.
It’s important to also think about how you capture progression. Music services may run targeted work to engage young people from particular disadvantaged/under represented groups but the aim should always be to open up progression routes which keep those young people learning long term.
Examples of sources of evidence
Sometimes funders or commissioners may ask you to use an evaluation method that makes young people feel they’re having an intervention. They may need to compare your intervention with others on an equal basis so there may be no way around this. But it’s important to feed back to them on how the process is affecting young people, if you feel it’s having a negative impact on them and the programme.
What should we measure?
Most music teachers are comfortable with measuring musical development outcomes along the lines of Time x skill acquired = progress. In a given time they can accurately predict how many new chords the student is likely to learn, or how much progress they will have made in achieving the correct technique for blowing, strumming or pressing down keys.
Social and personal outcomes may seem trickier to define and measure, but if you break these down, they become a lot easier. Here is one of Changing Tracks personal development outcomes for children and young people:
Young people develop resilience through music making activity
· The young person and the tutor/leader develop a relationship of trust, using music to build a place of safety.
· The young person reports using music/music making to express, moderate and change their own emotional state.
· Over time the young person is able to review their own music-making successes and failures in order to learn from experience.
As work would take place over two or three terms, we also wanted an outcome that could encompass significant change. We also knew that resilience chimed with Youth Service thinking and was realistic. While we may not change the challenges the young person is facing, we can be alongside them, and model different ways to respond that validates the young person’s experience.
To measure change you need to start with a baseline measurement. So for musical skill development we might be asking if they have played an instrument before or sung in a group or solo, to get a measure of their starting point.
With resilience, we might observe the level of trust (or distrust) the child brings to the first session (either spoken or through body language). We might discuss how the pupil feels about making mistakes while learning. We might have some information from their SENCO about the way in which the challenges they face impact on their ability to make progress in school. For example, a child who presents as withdrawn may not ask questions or fully engage with lessons and may end up falling behind and feeling inadequate.
What are the indicators of the change?
Once we’d agreed this outcome, we had to consider what we might see from the young person which might indicate that they were developing resilience. Trust seemed to be the key. We have seen that with excellent music leaders, the young people understand they will not be judged, that any views or ideas they give will be welcomed. That making mistakes is an essential part of learning. They are encouraged to lead their own learning and this gives them agency, a feeling of being in control of their destiny. Trust enables young people to try things they might have been wary about, to be vulnerable, to show their real feelings. So this was our first indicator of change.
The second indicator was about the emotional connection in music which enables us to experience our feelings, to be authentic. Evidence often comes from a discussion with the tutor about how music makes them feel (more on evidence in the next section). This both gives them permission to talk about emotions and to begin to use music as a tool to affect mood ie to put on a dance track when they need to motivate themselves, or to channel their anger into a drum solo.
Because much of Changing Tracks delivery lasts between 6 and 30 weeks, sometimes longer, we wanted an indicator for substantial change. Our third indicator describes the young person who has developed in confidence sufficiently to acknowledge mistakes and learn from them, accept criticism, and deal with set-backs constructively. If this is noticed in school or in the family, we can say that our input has had impact.