In August this year the 31st Olympiad will open in Rio de Janeiro. More than usually, these preparations have been accompanied by doubts about the Rio authorities’ ability to meet the complex requirements and spiralling costs of the event and whether it’s in their interests to do so. In this article Allan Brimicombe¹ decsribes the difficulties involved in evaluating the benefits of hosting major international events of this kind.

The Olympic Games have been dubbed the ‘greatest show on Earth’ and whilst the President of the International Olympic Committee (IOC) stopped short of pronouncing London 2012 as ‘the greatest Games ever’, that they were a successful event is not in doubt. But what of the legacy — has the London 2012 Games been living up to its promises? The IOC’s official impact report — the Olympic Games Impact (OGI) studies — published in December 2015² assesses just that. OGI meet the IOC's desire to develop an objective and scientific analysis of the impact of each edition of the Games and are intended to provide a record of both the individual nature of each Olympiad and its host context. London 2012 were the first Summer Games for which this was mandated through the Host City contract. But there have been quite a number of other enquiries³, reviews⁴, ‘meta-evaluations’⁵ and ‘supra-evaluations’⁶ as well. This article looks at the landscape of assessing the legacy of London 2012.

Defining legacy
So what is Games legacy? It is defined here simply as: any net impact arising from the Games. The term ‘impact’ refers to any change or transformation, for better or for worse, that has taken place and which can be attributed to the Games — in other words the linkage, direct or indirect, to the Games needs to be understood. However, the key term in the definition is ‘net’, that is, the impact that has occurred over and above what would have happened without the Games. Hosting something like the Olympic Games is rarely context-free or designed on a tabula rasa; rather it is superimposed on existing trajectories of historical development. Establishing a plausible counterfactual to measure net impact is therefore critical to knowing what the true legacy of the Games is.

Legacy aspirations are often stated simply for public, non-technical consumption and tend to gloss over their underlying complexity and multi-dimensionality. Thus in the case of London 2012 legacy promises, exactly what is meant by ‘to transform the heart of East London’⁷ in measurable legacy terms? For a start ‘East London’ does not have any standardised (administrative) geographical definition and so where is its ‘heart’ and furthermore what is intended to be transformed over what time horizon? For political reasons these are often left vague so it is easier for politicians and administrators to say this or that aspect was a success (cherry-picking legacy), or if things are not going to plan to say that it is too soon to see an effect. But if we are to measure legacy, then these legacy aspirations need to be translated and decomposed into measurable components of outcomes for defined geographical extents and for defined time horizons.

Data issues
A common misconception is that the measurement of legacy only occurs after an event is over and the ‘show has left town’. Because the very act of preparing for a bid for the Games may have already set in place changes that anticipate some legacy (such as land agglomeration for an Olympic Park), ideally the baseline needs to predate the bid preparation. For the London 2012 OGI study the base year is 2003 and extends twelve years to 2015 and thus includes three years of the legacy period.

Cities rather than countries are contracted to host the Olympic Games and whilst the impact of such an event may be (purposefully) uneven across the city, its impact may extend well beyond the city itself. Location features strongly because legacies happen in specific places — they are geographically rooted — but nevertheless can manifest themselves across a range of scales. The IOC OGI studies stipulate three scales of analysis: city, region, and country; and of course these terms are themselves open to interpretation. Thus in the case of London 2012 the ‘city’ is deemed to be the London Boroughs hosting the Games venues (the Host Boroughs; initially five boroughs but extended to six in 2011⁸), the ‘region’ is London as a whole and the country is preferably the United Kingdom but could be Great Britain, England and Wales or England depending on availability of compatible data amongst the devolved administrations that currently constitute the United Kingdom.

Most approaches to assessing legacy rely on quantitative indicators to monitor change in aspects of the environment, economy or socio-cultural landscape. There are a number of issues that arise with this. The first is defining the indicator(s) that will adequately reflect change in the phenomena to be monitored. This can be quite hard for soft legacy such as ‘being inspired’ by London 2012. Choice of indicator(s) inevitably needs to be mediated against the availability of open data of appropriate temporal and spatial granularity, or, if resources permit, a program of data collection. Inevitably this leads to a compromise in choice of indicator(s) which may not directly reflect the phenomenon to be monitored but act as good proxy or latent variables, or as a group of variables that triangulate the target phenomenon. Then there is maintaining the integrity of a time series over the period necessary to see a legacy effect. As noted, for the IOC OGI this is deemed to be 12 years. Published data sets can be discontinued, the method of compilation changed, definitions of variables and the questions asked in official surveys can be revised, and sample sizes (more often than not) reduced. Then there is the lag in official and administrative statistics. It takes time to collate returns, quality assure, disclosure control and finally be approved for publication. This generally means an 18-24 month lag and so data released in 2015 might only be reflecting the situation in 2013.

Approaches to the London 2012 games
At the time of writing, the author has on his hard disk nearly 200 reports and papers on all aspects of (mostly) anticipated and (some) measured legacy of the London 2012 Games. This volume of reports is symptomatic of the government’s preoccupation with legacy and a deep-felt need to show that the Games were worthwhile and beneficial (profitable) for the whole country, not just London. This is part of a trend where democratically accountable governments increasingly seek to justify their policies, actions and public sector spending by evidencing their merit, quality and efficacy. Thus formal evaluation of policy interventions and their outcomes have been an integral part of achieving transparency in accountability. The UK Government has a number of documents giving guidance for evaluation, impact assessment and cost-benefit analysis. Key amongst these is:

• The Green Book :⁹ This book provides a framework for the appraisal and evaluation of government policy and projects. The difference between ‘appraisal’ and ‘evaluation’ is in the timing. Appraisal is an assessment as to whether a proposed project or policy intervention is worthwhile and most commonly takes the form of a cost-benefit analysis. Evaluation takes place post-implementation to see to what extent the objectives of the project have been achieved and what lessons might be learnt. A supplement to the Green Book has been issued¹⁰ to cover techniques in social cost-benefit analysis. An example of cost-benefit analysis for London 2012 is one carried out at the point of deciding whether or not London should make a bid for the Games¹¹. On an estimated expenditure of £1.9bn the income ranged from £1.65bn to £1.98bn, that is, from an 8 per cent loss to a 4 per cent surplus. Much of this variation arose from uncertainty in the tourism income which in many Olympic cost-benefit analyses is viewed as the key wider economic benefit of staging the Games. The decision, as we know, was to go ahead with a successful bid. But by 2005 the estimate for public sector funding had risen to £2.4bn and by 2007 to £9.3bn¹² primarily because the anticipated private sector involvement in the construction did not materialise. Studies of the Sydney 2000¹³ Games have shown that many such ex ante cost-benefit valuations are over optimistic of the economic stimulus that can be expected from the Olympics.

• The Orange Book:¹⁴ This book provides a model of risk management. Any Games event has substantial risks (financial, reputational and otherwise). If these risks are identified and monetised at an early stage, then sufficient mitigation and financial contingency can be put in place to cover the risks. In 2007 the National Audit Office (NAO) carried out a risk assessment of the preparation phase for the London 2012 Games¹⁵ and identified six areas of risk that needed to be managed. Prime amongst these was delivery against an immovable deadline; also included was planning for a lasting legacy. By 2010 a contingency of £2.2bn was in place to cover risks. These were of three types: programme contingency (£0.97bn) for the construction of the Olympic Park on a constricted site to a fixed deadline; funders’ contingency (£1bn) for changes in scope and wider economic conditions; and a security contingency (£0.24bn). The post-Games audit¹⁶ showed that not all the risks materialised and that an underspend of £0.38bn on the contingency was achieved. The same audit also looked at the immediate legacy benefits which were viewed principally from the point of view of job creation (177,000 job years employment in the construction 2007-2012; 34,500 people in Games-related employment), progress on planned legacy use of the venues, and problems of governance and coordination of legacy delivery. ‘… it remains the case that numerous individual organisations are delivering aspects of the legacy and that coordination of this activity remains a challenge’.

• The Magenta Book:¹⁷ This book provides further guidance on programme evaluation and complements the Green Book. The key focus is to identify ‘what works’, highlight good practice, identify any unintended consequences or unanticipated results, and value for money that can be used to improve future decision-making. ‘Not evaluating, or evaluating poorly, will mean that policy makers will not be able to provide meaningful evidence in support of any claims they might wish to make about a policy’s effectiveness. Any such claims will be effectively unfounded.’ Meta-evaluation, the synthesis of separate smaller evaluations, is also covered in this volume, and is further discussed below in relation to London 2012.

From the preceding discussion it is clear that measuring legacy can take a number of forms — valuation, evaluation, meta-evaluation, audit — and whilst they tend to be quantitative, can nevertheless be qualitative or a mixture of both. There is also a tendency to include historical analogues, that is, to use previous events as a benchmark or as a comparator to gauge progress or likelihood of legacy emerging. One such study was carried out by researchers at the University of Westminster for the Royal Institution of Chartered Surveyors¹⁸. This study used six case studies of previous events including the Barcelona 1992 and Sydney 2000 Olympic Games, the 1998 FIFA World Cup, and the Manchester 2002 and Melbourne 2006 Commonwealth Games, from which to draw conclusions about the likelihood of a regeneration effect from the London 2012 Games. A scorecard approach was used whereby, having identified from the case studies 15 criteria of good practice, London 2012 was rated and scored 165 out of a possible total of 200, or 82.5 per cent. What becomes clear is that where cities plan for legacy from the moment they start planning for an event, the more likely it is that the legacy will come to fruition. Legacy was embedded in the planning for London 2012.

The IOC naturally has a much broader interest in legacy that just regeneration and has mandated, through its Host City Contracts, for Olympic Games Impact (OGI) studies and, as already discussed above, the baseline is two years prior to being awarded the Games. OGI need to be carried out independently of the local organising committee and so for London 2012, the main phases of the OGI (pre-Games, Games time, post-Games) have been carried out by the University of East London (UEL), funded by the Economic and Social Research Council (ESRC)¹⁹. OGI are based on an IOC Technical Manual which provides the specification for the production of standardised data on up to 120 possible indicators in the environmental, social-cultural and economic spheres. Some of these indicators, such as So9 Health, are themselves baskets of indicators capturing many dimensions. Not all 120 indicators are expected to be reported on for all Host Cities, but an appropriate selection is negotiated between the IOC and the local organising committee. Thus for the final post-Games London 2012 report, 67 indicators were deemed necessary (15 environmental, 27 socio-cultural, 25 economic) to adequately assess the legacy.

In relation to OGI, no official counterfactual was established. Instead the trend in the data from 2003 onwards are analysed and interpreted as to whether the changes reflect a net impact that can be attributed to the Games. In some cases, such as the transformation of the Olympic Park site or improvements in the transport infrastructure, this is straightforward. In others it has been difficult to disentangle regional and national trends and government policy changes and interventions from the effect that the Games have had per se. A case in point would be crime rates which have fallen in East London but then been generally falling nationally since 1997 as part of the ‘great crime decline’ affecting many western countries²⁰. Since London 2012 was touted as the ‘sustainable Games’, a method has been devised to calculate a sustainability rating using all the indicators where a minus score reflects outcomes that are unsustainable, zero is where the status quo has been maintined and a score of one achieves full sustainable. London 2012 has achieved an overall score of 0.63 (environmental 0.56, socio-cultural 0.68, economic 0.61) which is up from 0.37 in the 2010 Pre-Games OGI report and is because the nature of the impact of the London 2012 Games have become clearer in the years following the Games.

The OGI, as stated, shows change in relation to a baseline, but does not have a coherent counterfactual. To establish such a counterfactual for each and every indicator would be a daunting task. The counterfactual is more easily achieved where there is a single or small group of indicators such as when valuing the contribution to GDP of the London 2012 Games. The meta-evaluation study²¹ carried out by DCMS22 provides one such example²². The direct spending on the preparation for the Games by the public sector — the Public Sector Funding Package (PSFP) — totalled just over £8.9bn. This included the land purchase, infrastructure and venue construction. The economic calculation of the gross GVA²³ impact of this spending for the period 2007-2012 is £11.5bn. Regarding the counterfactual for calculating the net impact, the report states ‘The modelling compares the impact of the Olympics with the counterfactual assumption that the Olympics weren’t awarded to London and therefore there was no construction or operational spending. There is no counterfactual assumption related to spending the public money on anything else.’ This, in the author’s view, is a politically convenient assumption to make because it means that the gross benefit is also the net benefit. Some displacement was accounted for (movement of production from other parts of the economy to the Games preparation) giving a net GVA impact of about £10bn and therefore the Games make an economic surplus. However, about £3bn of the PSFP came from sources such as the National Lottery and would have circulated in the economy and contributed to GVA even if London had not won the Games — so the counterfactual cannot be zero. The other £6bn of central government spending, a miniscule amount of total government spending, may well have been spent on other projects in the boom years prior to the recession. A more realistic counterfactual would have been to model the value of PSFP as government consumption (spending on goods and services e.g. more doctors and teachers). The net impact would then more properly reflect the difference between spending on consumption versus spending on infrastructure and thus value the true legacy.

Conclusions
There have been a considerable number of different ways to assess the legacy impact of the London 2012 Games. The IOC post-Games OGI report is the latest and probably not the last. As with any long term project that is intended to be a catalyst for long term change and transformation, the analysis of three years into legacy that the OGI report presents is only the beginning. The urban transformation of the Olympic Park is not expected to be complete before 2030. Cultural changes towards, say, more healthy and active lifestyles can take a long time, may even be generational. That London 2012 has been a catalyst for positive change is not in doubt, but when and where the process ends and what will be the full magnitude of the effect is not yet known. The story of London 2012 will continue to unfold for a long time to come.

Notes:

1. Professor Brimicombe, University of East London, has been the project manager for the IOC’s Olympic Games Impact study for the London 2012 Games.

2. Online version available at http://www.uel.ac.uk/geo-information/

3. e.g. House of Lords Select Committee on Olympic and Paralympic Legacy (2013) Keeping the Flame Alive: the Olympic and Paralympic Legacy.

4. e.g. Cabinet Office (2015) Inspired by London 2012: the legacy from the Olympic and Paralympic Games.

5. Department for Communities, Media & Sport (2013) Meta-Evaluation of the Impacts and Legacy of the London 2012 Olympic Games and Paralympic Games – Report 5: Post-Games Evaluation.

6. Centre for Sport, Physical Education & Activity Research (2013) London Legacy Supra-Evaluation.

7. Department for Communities, Media & Sport (2008) Before, During and After.

8. London Boroughs of Newham, Hackney, Tower Hamlets, Waltham Forest and Greenwich, to which was added Barking & Dagenham

9. Her Majesty's Treasury (2003) The Green Book: Appraisal and Evaluation in Central Government.

10. Her Majesty’s Treasury and Department for Work and Pensions (2011) Valuation Techniques for Social Cost-Benefit Analysis: Stated Preference, Revealed Preference and Subjective Well-Being Approaches.

11. ARUP (2002) London Olympic 2012 Costs and Benefits.

12. Berman, G. (2010) Financing the London 2012 Olympic Games. House of Commons Library Standard Note SN3790.

13. Giesecke, J.A. and Madden, J.R. (2011) ‘Modelling the economic impacts of the Sydney Olympics in retrospect – game over for the bonanza story?’ Economic Papers, 30: 218-232

14. Her Majesty’s Treasury (2004) The Orange Book: Management of risk – principles and concepts.

15. National Audit Office (2007) Preparations for the London 2012 Olympic and Paralympic Games – Risk assessment and management.

16. National Audit Office (2012) The London 2012 Olympic Games and Paralympic Games: post-Games review.

17. Her Majesty's Treasury (2011) The Magenta Book: guidance notes for policy evaluation and analysis.

18. Royal Institution of Chartered Surveyors (2011) The 2012 Games: the regeneration legacy.

19. The three reports and all underlying data are available from https://www.uel.ac.uk/geo-information/ .

20. Zimring, F. (2007) The Great American Crime Decline. New York: Oxford University Press.

21. A meta-evaluation is an over-arching synthesis of the findings of individual project-level evaluations in order to provide a comprehensive understanding of outputs and impacts associated with a mega-project.

22. Department for Communities, Media & Sport (2013) Meta-Evaluation of the Impacts and Legacy of the London 2012 Olympic Games and Paralympic Games; Report 5: Post-Games Evaluation, Economic Evidence Base.

23. Gross Value-Added (GVA) measures the economic contribution of each producer and contributes to the calculation of Gross Domestic Product (GDP); GVA + taxes – subsidies = GDP.