Picture from ealjournal.org

Earlier this year I wrote (here) about wanting to try out Comparative Judgement with GCSE composition. Since then a collection of 63 example compositions has been assembled from 12 different schools, and 26 music teachers provided judgements via the http://www.nomoremarking.com website.

Teachers were asked to compare pairs of pieces, and choose which they felt was the better composition. A total of 597 judgements was made. No assessment criteria were used: teachers just used their knowledge and experience as musicians and teachers. The nomoremarking.com system then crunched all the judgements to establish a rank order for the compositions, and then used this to award each one mark out of 30.

Judges gave feedback on their experience of applying comparative judgement (CJ) to GCSE compositions, and teachers who had contributed compositions gave feedback on the marks awarded by the process, comparing these to the marks the same compositions had gained using exam board marking criteria.

What the judges said

The majority of judges thought that the 63 compositions presented were representative of the range (in terms of styles and quality) of GCSE compositions as a whole. Some, however, felt that there was a preponderance of mid-range pieces, with not enough really good or really poor ones.

Nearly all the judges reported finding the comparison process easy and relatively quick. They found it liberating to rely solely on musical ‘guild knowledge’ to decide which was the better piece in each pair. Some judges found it difficult to compare pieces that were very different in style. Others reported finding it hard to shake off a feeling of their value judgements being influenced by their well-established knowledge of GCSE assessment criteria, which has become ingrained over time.

In some cases – particularly with film music compositions – it was felt that knowing the candidate’s brief would have helped in making judgements. However, generally, feelings were split between feeling liberated by having no supporting documents, and wanting the clarity that knowledge of candidates’ intentions might bring.

There was an overwhelming appreciation of the opportunity to hear coursework from other schools. Some judges felt reassured that what they are doing in their own schools is along the right lines. Others felt inspired by the different styles and compositional processes they heard, and feel emboldened to try out some new ideas in their teaching.

How the CJ results compared with ‘real’ GCSE marks

The vast majority of teachers who had contributed pieces to the study found that the marks awarded by the CJ process were lower than those that the pieces had gained using exam-board assessment criteria. This was particularly true with the best pieces, with the difference evening out as the quality of the pieces declined.

I think I can explain this (although I am no statistician, so please put me straight if I’ve got something wrong):

• The CJ algorithm takes the data from the comparisons, and then distributes the rank order of pieces onto a bell curve centred on 50%
• GCSE criteria make it relatively difficult to award marks in the bottom couple of bands, provided that the candidate has offered a completed piece of sufficient length
• Therefore (and I have no statistical proof for this) I can imagine that the curve for actual GCSE marks would have its peak above, not at, 50%.
• A contributing factor could be that the standard of pieces of contributed to the study was of a higher general standard than the average anyway.

How could CJ be used by music teachers?

The strongest feeling that came out of participants’ feedback was the benefit they gained from listening to coursework from other schools. Everyone seems to very much appreciate the opportunity to hear what other people are doing.

If you can team up with some other centres (because the whole process is done online, it doesn’t matter whether they are near or far), putting some or all of your GCSE compositions through the CJ process could help you use the hive-mind to establish a valid rank order. You can then fit the established rank order to your exam board’s criteria in order to award marks. Along the way you have the CPD of hearing other centres’ work.

The CJ process is not difficult to administer – all the difficult stuff is done by the nomoremarking.com website (and is free). It just needs one person to coordinate, who needs to anonymise the recordings of the pieces (plus supporting documentation, if you decide to include it) and make them available to all participating judges via Google Drive or similar.

For your interest, here is the composition that came out in the CJ process as the highest-scoring.

I would very much like to thank everyone who has taken an interest in this experiment, whether as a judge or through contributing coursework. The service offered by nomoremarking.com is a remarkable and very clever thing, and it’s pretty amazing that it’s free.