Studying the form

"Second only to Poppleton in teaching quality." "Officially rated the best new university in Loamshire." Yes, it's league table time again!

A love of league tables lies deep in the psyche of a footballing nation, and pandering to the national whim is no bad way to sell newspapers. League tables of universities are constructed from information from a variety of sources, usually combined in a grossly unscientific manner by deadline-chasing journalists.

All of them refer to "official figures" and the most important such figures are those produced by the Quality Assurance Agency from its subject by subject reviews of teaching quality.

As light entertainment, league tables are great fun. As a guide to choosing a university they can be seriously misleading. To produce a single "score", the league tablers combine data gathered for different purposes at different times and so end up with an average that conceals as much as it reveals.

So, what do the "official figures" measure?

The Agency reviews how well different aspects of the performance of a university or college contribute to the achievement of the objectives set for each programme of study. These objectives will differ from institution to institution.


Take, as an example, the
practical skills developed by two programmes in communication and media studies. One is an HND programme in a further education college that aims to develop the skills needed "to work effectively as part of a technical crew". In the same subject area, a masters programme in a major university aims "to provide a research training" to the standards laid down by the Research Councils.

Both are valid objectives, but the programmes differ in level and nature, and are seeking to prepare students for different careers. A judgement that each has done well in meeting its objectives does not mean that the two programmes are equivalent.

How is effectiveness measured? Assessors review six aspects of provision and consider how well each of them contributes to achieving the objectives of the programme. The aspects are curriculum design; teaching, learning and assessment; student progression and achievement; student support and guidance; learning resources; and quality assurance. Each aspect is scored on a scale of 1 to 4. The scores are intended to be read as a profile, reflecting the relative performance of each aspect of provision.


As they are not intended to be added up to give a total out of 24, the scores are unweighted. However, common sense indicates that some are more important than others. For example, if the curriculum is not designed so as to achieve the intended objectives of the programme, no amount of sympathetic student guidance is going to put that right.

A potential student considering a particular programme should do more than look at the total out of 24. Consider each aspect of the profile, and see where the strengths and weaknesses lie. Scores of 4 are good: the aspect in question makes a full contribution to achieving the programme objectives. Any score of 1 is bad: the course provider is simply failing to meet the objectives. A score of 3 marks a substantial contribution to the programme. Some things could be done better, but a student is unlikely to be let down.

Scores of 2 should set the alarm bells ringing. They mean that, while the contribution of the aspect is just about acceptable, significant improvement needs to be made.

It is worth reading the report itself to establish the nature of the problem. The report may cover programmes at several levels and it could be only one of these that gives cause for concern. For example, a mathematics department scored only 2 for curriculum design. A reading of the report would disclose that three types of programme were offered. An access programme was found to be good, as was service teaching provided to science and engineering departments. However, the curriculum for the degree programme was weak, particularly in the final year. A student contemplating a single honours programme in mathematics would be best advised to look elsewhere. But a student needing to acquire the mathematical skills necessary to tackle a science or engineering programme could be well served by that institution.



The numbers in the graded profiles summarise complex judgments, so adding the numbers together to give a score out of 24 can conceal both weaknesses and strengths. One newspaper arbitrarily equates 22 with excellence. An aggregate score of 22 could include a mark of 2 in an important aspect. In turn, the 2 might be an average judgement across several programmes, at different levels, in the same subject. It could cover provision that, far from being excellent, was actually unsatisfactory.

By contrast, a score of 21, ignored by that newspaper, might contain top marks in crucial areas such as curriculum design and the effectiveness of teaching.

So, don't take the league tables too seriously. They are based on figures which are averages of averages of averages. Very few good scores will conceal poor provision, but be aware that it can happen, as the numbers were never intended
to be aggregated to construct
league tables.

Football league tables are simple. Clubs play a single game, and the result can only be win, lose or draw. Universities are complex organisations, each offering hundreds, if not thousands of programmes to meet a great diversity of needs. Each university will have its strengths and weaknesses; quality assurance helps identify these. A potential student should be concerned with matching a programme to their needs, not with a league table position.

Universities teach students to analyse complex data, and to make sound judgments on it. Students learn quickly that such judgments should rest on primary evidence, not on newspaper summaries. They, at least, will know that university league tables should be treated as a source of entertainment, not enlightenment.

John Randall is chief executive of the Quality Assurance Agency. QAA reports are available on www.qaa.ac.uk