At the Society’s Annual Conference last year, one of the most talked-about special sessions explored the returns to higher education using a new data set that provides a record for each student of their school achievement and demographic characteristics, linked to data on their higher education experience, including degree subject and institution and their tax records. The data yield some remarkable results. Anna Vignoles1, one of the participants, explains below.
In an IFS Special Session at the Royal Economic Society Annual Conference in 2019, Jack Britton, Ian Walker and I discussed a recent body of work on the returns to higher education in England. Our team was the first to get to use a new administrative data set that has just become available, the Longitudinal Education Outcomes data or LEO. LEO is undoubtedly powerful data. It provides a record for each student of their school achievement and demographic characteristics (from the National Pupil Database), linked to data on their higher education experience, including degree subject and institution (from the Higher Education Statistics Agency), and their tax records (from the UK tax office, HMRC). The data include all English educated students and we have data for people who were aged 16 in 2002 and observed, in terms of their earnings at least, up to age 29 in 2015/16.
In my talk, I discussed how this fantastic data might be used judiciously to inform us about the economic value of university degrees in England. Later in this post I will return to the crucially important but sometimes forgotten point that higher education is not simply about securing good jobs for students but is also about the myriad of social and cultural benefits that it can potentially give to students, but I begin by focusing only on the economic benefits of degrees. To understand these benefits we need to worry about three important issues.
The importance of the LEO
First, if students are going to make optimal choices, as predicted by economic theory, they need to be well- informed. Higher education is a big investment and changing your mind about your degree course is not easy. So students need good data on the current labour market value of degrees on which to (partially at least) base their decisions.
Second, in England, repayment of the large tuition fee loans incurred by most students is contingent on the level of their subsequent income. The state rather than the individual therefore carries the risk of non-repayment. As the state is subsidising those who do not repay, it is important that we understand how the rates of repayment vary by subject, and therefore, where the state subsidy is going, i.e. to which degree subjects and which institutions. For this, we again need high quality data on the long run earnings of graduates.
Third, higher education is not a free-market, at least in terms of competing on price, even in the English system which charges students around £9k per annum for a first degree. At best it is a quasi-market and English universities have not shown an inclination to compete on price. With a few exceptions, all English universities charge the maximum tuition fee possible. In the absence of market competition on price, regulation is obviously important. This raises the issue of what regulators might use to judge the ‘quality’ or ‘value for money’ of English degrees. One could assess the quality of universities in a number of different ways. For example, in other work we have developed an instrument that seeks to measure the learning gains of students, which could be one useful way to inform regulators about the quality of an institution’s degrees. However, it would also seem, at least on the face of it, that the impact of particular degrees on students’ labour market outcomes might be one of many potentially useful metrics. For this, high quality data on the longer run labour market outcomes of students is also vital.
The new LEO data set therefore helps with all three of these data needs. Here I am going to focus on the third issue, namely what LEO can tell us about the ‘value added’ of different degrees. By value added, I mean the additional earnings that the degree provides, over and above the student’s likely alternative outcome without the degree. In the English context, there is good evidence that students’ alternative labour market prospects will be heavily determined by their prior academic attainment in school. So what we set out to measure in our research is essentially the distance travelled by each student between entry to university with a given set of high school (A level) grades in relation to where they end up, in terms of earnings. The data enable us to control for the individual’s school, region of school, socio-economic background, ethnicity and crucially their prior attainment.
LEO is brand new and its possibilities are being explored. However, using LEO as a metric by which to judge the quality of higher education institutions is highly controversial. Regulation of English higher education has historically been about quality assurance, with some focus on students’ academic outcomes but very little emphasis on students’ real world outcomes. The QA process in the past was also arguably not very high stakes. This can be contrasted to a long standing very high stakes way of measuring research quality in English universities, namely the REF (Research Excellence Framework). The REF and the associated league table rankings of departments and institutions has undoubtedly driven the behaviour of academics and institutions. As a consequence, there has been concern that such a high stakes metric on university research has impacted negatively on the effort put into university teaching. Without a similarly high stakes metric on teaching, the incentives are for academics to focus predominantly on their research. Linked to this issue and with the introduction of higher tuition fees, we have also seen much more vociferous and broader concerns expressed about the value for money of different university degrees.
Regulators have responded to these concerns with a parallel (potentially) high stakes accountability system that is supposed to measure the quality of teaching and the student experience, namely the TEF (Teaching Excellence Framework). TEF has always included some measures of students’ labour market outcomes and in the latest version of the TEF, LEO was used experimentally to provide longer term and arguably higher quality data on students’ earnings outcomes. TEF has been recently reviewed by Dame Shirley Pearce and her report has long been completed but is currently standing in line with other reports postponed by the election. I provided statistical advice to the review and the use of LEO was a major topic of discussion.
What does the data tell us?
So with the significance of the LEO data in mind, what has it actually told us? Jack, Ian and I presented a range of evidence from the new LEO data, particularly around the heterogeneity in returns to different degrees. I won’t reproduce all the results here and the papers are open access at the links below. There are however, some clear messages. Overall earnings returns to a degree are large for women, but somewhat smaller (though still positive) for men. Given the discourse in England about degrees being worthless, this is of course reassuring. The raw earnings differences by subject are very large indeed and even when we model value added, taking into account background characteristics and using an Inverse Probability Weighted Regression, the earnings differences across subjects remain sizeable. Earnings differences across institutions are also sizeable though somewhat smaller when we take account of background characteristics. The bottom line is that, even when comparing students with the same level of prior attainment, their earnings will vary quite dramatically depending on the exact institution they enrol in and the subject they take.
Some pictures might best illustrate the scale of the variation. Figure 1 shows the variation in the raw earnings (black dots) and the conditional earnings (grey dots) for women taking different subjects. Although the wage premia for males are generally lower, a similar pattern of extensive variation by subject is evident for men too. Figure 2 shows the variation in earnings by type of institution, also for women. Again the variation is striking and another key point is that for some students graduating from some subjects and some institutions, there is no return to higher education at all or even a negative return.
Further, in our data we have an indicator of the socio- economic background of students. Using this indicator, we find that graduates from the top fifth of the socio-economic distribution of households earn around 10 per cent more than otherwise similar graduates who took the same degree subject in the same institution. In other words, the value-added of university degrees varies depending on family background. Universities it appears, do not level the playing field when it comes to students’ family background and its influence on their subsequent earnings.
These data are therefore challenging. How useful are they to judge the ‘value added’ of university degrees? Well, wage information of this kind can be used to estimate the private wage gain from HE, but not the social return. So such data cannot tell you where the state might want to subsidise higher education. For example, the earnings of nurses are low but the social value of nurses is high. We would not want to use a low private wage return to conclude that we should not invest in nurses. Quite the contrary, cases where the social return is likely to be far higher than the private return are precisely where we would want to direct our subsidy.
But it’s not the whole picture
Even if we want to use these data in a more narrow way to judge what institutions are doing in terms of adding value with their degree offer, we need to be cautious. The returns to particular subjects reflect relative supply and demand in the labour market. This is precisely the kind of data that students need to consider when making their choices. When considering it from a regulatory perspective however, it is less obvious that one wants to judge degree quality on the basis of what the labour market wants. If the demand in the labour market for creative arts degrees is low, it does not mean that the quality of provision is low. An even more fundamental point is that these data reflect degree courses taken many years ago. Provision may have changed and since it is important to consider earnings some years into a person’s career, one cannot make an automatic link between what the university is doing now and the LEO data on earnings outcomes for previous cohorts of graduates.
In terms of the value added approach, there is a clear analogy here with the ‘value added’ that has been measured in the English school system for many years. In the school context, value added is based on test scores rather than earnings. Data aggregated at school level on pupils’ academic achievement in national examinations taken at age 16 (GCSE scores) has long been published in league tables. Attempts have also been made to account for student intakes which vary by school. So estimates of the value added of secondary schools for example, take account of pupils’ initial achievement level on entry into the school at age 11. This approach has arguably had positive benefits for parents who are better informed about school quality. However, the use of such data in the school system has also raised some real problems that are worth reflecting on before we go down this route in higher education. First, in the school system this has led to schools trying to enrol students who are more likely to achieve higher grades. Second, it has led schools to focus overly on the marginal student i.e. those who are on the cusp of achieving the desired metric (in the past this was 5A*-C grades at GCSE) and providing less support for those who were very high or low achieving and hence either certain to meet the metric or most unlikely to do so. Third, such a high stakes metric ignores the fact that students are not consumers. They jointly produce the human capital developed from their education. If responsibility for achievement is placed too squarely with the teacher, the role of the student and the importance of their motivation and effort may be ignored. This can lead to pressures to ‘dumb down’ or to pass everyone.
In the university context, a high stakes regulatory system based on graduate earnings is only desirable if universities can improve their position in terms of this metric by genuinely improving quality. If on the other hand they can improve their position by changing their student intake (e.g. by admitting fewer students from lower socio economic backgrounds as their earnings tend to be lower) or shifting their focus to different subjects, we would be concerned. On the one hand, we want both students and universities to be aware of the labour market value of different degrees to avoid universities expanding ‘easy’ courses that students like but that have poor labour market outcomes, and indeed avoiding students taking low economic value courses without realising that their labour market outcomes may be relatively poor. Equally we want to maintain subjects that may have lower economic value but are socially vital, for example nursing.
So if we use earnings data to judge whether institutions are doing a good job or not, it needs to be in conjunction with other data. One can make an argument that earnings data should be one of a basket of measures used to assess institutions. If institutions are forced to focus at least somewhat on the earnings of their students, they will be more inclined to embed useful skills in their curricula (across all subjects) that will help students when they enter the world of work. They will also be more likely to support students with better careers advice and work opportunities, particularly those students from poorer backgrounds. These are clearly important ways in which institutions can improve the labour market outcomes of their students and this should be at least one of the aims of higher education.
So in conclusion, LEO provides good evidence on graduates earnings and such data is vitally important to inform students’ choices. LEO can enable us be better informed about where the subsidy for higher education is really being directed. LEO can also help institutions understand the labour market better and arguably adapt their offer with that in mind. However, earnings are not a good guide to course quality and hence need to be used carefully in our higher education accountability system. Most crucially of all, a good job is just one of the many benefits of higher education. Graduates have better health, are more likely to vote, participate more in civic life, contribute more to the arts and tend to have more tolerant political attitudes, to name but a few of their wider benefits. It is vitally important that we have a careful approach to designing a higher education system that can sustain these other benefits whilst not ignoring the fact that for most students, getting a ‘good job’ is a major objective from their higher education.
Note:
1. Anna Vignoles is Professor of Education in the Faculty of Education, University of Cambridge.
References:
Belfield C, Britton J, Buscha F, Dearden L, Dickson M, Van Der Erve L, Sibieta L, Vignoles A, Walker I and Zhu Y, 2018. The relative labour market returns to different degrees: Research report: June 2018. https://dera.ioe.ac.uk/33025/1/The_relative_labour_market-returns_to_different_degrees.pdf
Belfield C, Britton J, Buscha F, Dearden L, Dickson M, van der Erve L, Sibieta L, Vignoles A, Walker I and Zhu Y, 2019. The impact of undergraduate degrees on early-career earnings: Research report: November 2018. https://dera.ioe.ac.uk/33021/1/The_impact_of_undergraduate_degrees_on_early-career_earnings.pdf
Britton J, Dearden L, Shephard N and Vignoles A, (2018). ‘Is improving access to university enough? Socio economic gaps in the earnings of English graduates’, Oxford Bulletin of Economics and Statistics. https://www.repository.cam.ac.uk/handle/1810/279908
Britton J, Shephard N and Vignoles A, 2019, ‘A comparison of sample survey measures of earnings of English graduates with administrative data’ Journal of the Royal Statistical Society: Series A (Statistics in Society), 182(3), pp.719-754.
