Please view a summary of the case study and a list of relevant resources (LSE Research Impacts website).
Summary of the impact
Research by Professor Leonard Smith and the LSE Centre for the Analysis of Time Series (CATS) on forecasting in non-linear and often chaotic systems, with particular attention to weather, has led to advances in three areas: 1) national and international weather industry products and services that are built upon state-of-the-art research and knowledge, 2) dissemination of state-of-the-art practice in forecast production and verification to national, regional and local weather centres around the world, and 3) the introduction of, and new applications in, state-of-the-art forecasting methods in industries facing high uncertainty and risk, e.g. insurance and energy.
Underpinning research
Research Insights and Outputs: From its beginnings in 2000, CATS has advanced the generation and interpretation of state-of-the-art weather forecasts in support of decision-making. Professor Leonard Smith’s research over the past 25 years has focused on forecasting nonlinear, often chaotic, systems, including weather forecasting. In 2003, Professor Smith was awarded two of the ten Department of Trade and Industry Faraday Maths grants (REMIND and DIME). One focused on improving the interpretation and evaluation of ensembles of simulations, the other on their application in a variety of industrial settings. Modern operational weather forecasts take an ensemble (Monte Carlo) approach to account for uncertainty (in the initial condition); scientific, statistical and philosophical questions still remain as to how to interpret an ensemble of model-trajectories as a probability forecast for the future of the system. The research involved translating a sample of about 51 points (the ensemble forecast) in a 10,000,000 dimensional space into information regarding the reliability of a single, high resolution forecast or into a probability forecast for a similar target variable (which was sometimes a nonlinear combination of several meteorological variables). A key aspect is that, since the weather model is nontrivially imperfect, on some days the ensemble is seen, in hindsight, to capture the evolution of the weather, while on other days the model is unable to shadow the events due to model error. The research [5] combined (a) kernel dressing the ensemble members (turning simulations into probabilities), (b) blending with the climatological distribution (a prior based upon historical observations for the relevant phase of the seasonal cycle), (c) an empirically-driven recognition of fundamental limitations for the method itself and (d) input from energy traders in terms of presentation and tolerances, in order to achieve a robust, actionable probabilistic tool.
In 2005 the applied aspects of this research were extended under the NERC grant NAPSTER and both theory and application under the EU FP6 ENSEMBLES project. Following IJ Good’s work on quantifying skill in the 1950s, CATS [1,4] enabled the improved tuning of models and the communication of probabilistic skill [6], alternatively in “bits” for those familiar with information theory and as an “effective interest rate” for those more familiar with financial work (traders in the energy sector; managers in the weather sector). This work was performed jointly with Hagedorn of the European Centre for Medium-range Weather Forecasts, which hired Smith as a consultant.
The key research insight here was that a sample of simulations from one (or more) imperfect physics-based simulation models can provide more information if they are not treated as a sample drawn from some target probability density function. CATS led research development of kernel dressing of individual ensemble members and blending with the background climatological distribution. While this method is superficially similar to Bayesian Model Averaging techniques in weather forecasting, which appeared shortly after (and cite) the CATS work, the epistemological framework is rather different. The novel use of a prior distribution (the “climatology”) to lessen the impact of model error was important. Breadth of applicability was shown in additional research papers on off-shore wave height with Royal Dutch Shell [3] and on wind energy production [2].
Key researchers:
Professor Leonard Smith, Reader/Professor of Statistics, LSE, March 2000-present
Dr Jochen Broecker, Postdoctoral Research Officer, LSE March 2003-April 2007
Dr Liam Clarke, Postdoctoral Research Officer, LSE March 2003-May 2008
Dr Renate Hagedorn, Scientific Researcher, ECMWF, ~2000-2011
Dr Hailiang Du, Graduate Student/Postdoctoral Research Officer, LSE, 2004-present
References to the research:
1. MS Roulston & LA Smith (2002) Evaluating probabilistic forecasts using information theory, Monthly Weather Review 130 6: 1653. DOI: 10.1175/1520
2. MS Roulston, DT Kaplan, J Hardenberg & LA Smith (2003) Using Medium Range Weather Forecasts to Improve the Value of Wind Energy Production, Renewable Energy 29 (4) April 585.DOI: 10.1016/S0960-1481(02)00054-X
3. MS Roulston, J Ellepola & LA Smith (2005) Forecasting Wave Height Probabilities with Numerical Weather Prediction Models, Ocean Engineering 32 (14-15), 1841. DOI:10.1016/j.oceaneng.2004.11.012
4. J Bröcker & LA Smith (2007) Scoring Probabilistic Forecasts: The Importance of Being Proper Weather and Forecasting, 22 (2), 382. DOI: 10.1175/WAF966.1
5. J Bröcker & LA Smith (2008) From Ensemble Forecasts to Predictive Distribution Functions Tellus A 60(4): 663. DOI: 10.1111/j.1600-0870.2008.00333.x
6. R Hagedorn & LA Smith (2009) Communicating the value of probabilistic forecasts with weather roulette. Meteorological Applications 16 (2): 143. DOI: 10.1002/met.92
Evidence of quality: Publications appeared in peer-reviewed journals; Professor Smith received the FitzRoy Prize of the Royal Meteorological Society for distinguished work in applied meteorology in 2002; and the research received grant funding that included:
• Direct & Inverse Modelling in End-to-End Environmental Estimation (DIME) EPSRC, 2003- 2005 (£94,360), GR/R92363/01.
• Real-time Modelling of Nonlinear Datastreams (REMIND). EPSRC, 2003- 2005. (£85,827, plus industrial in-kind support from National Grid Company and Intertec), GR/R92271/01
• Nonlinear Analysis & Prediction Statistics from Timeseries & Ensemble-forecast Realizations (NAPSTER) NERC, 2005 – 2008. (£152,481), NE/D00120X/1.
• ENSEMBLE-based Predictions of Climate Changes and their Impacts (ENSEMBLES), EU FP6, 2004-2009. (£112,926), GOCE-CT-2003-505539.
• Towards Identifying and Increasing the Socio-Economic Value of High-Impact Weather Forecasts, US NOAA, 2003-2004 (£94,538).
• BIOS RPI Grant for evaluation of seasonal forecasts for the insurance industry.($50k) 2012
Details of impact
Weather has an impact on people, businesses and economies every single day. Even fairly common weather, with no extreme meteorological elements whatsoever, can produce costly and stressful disruptions - to power and water supplies, food production and distribution networks, travel systems, communication networks (e.g. GPS) and other infrastructure necessary to the smooth functioning of families, communities and societies. And changes in forecasts themselves, not the weather at all, can have huge impacts on prices and the provision of (cancellation of) services. There is therefore an imperative for tools which enhance the use of modern (probabilistic) weather forecast in addition to the need for weather predictions that are timely, accurate and enable users with varying needs and levels of understanding to plan for and manage their responses to weather conditions and events. There are three ways in which Professor Smith and his colleagues at CATS have responded to this need: 1) advancing the state of the art in probabilistic forecasting in the weather industry; 2) advancing the state of practice in weather forecasting; and 3) advancing understanding and application amongst various types of users in business and industry.
Advancing the state of the art in the weather industry: Professor Smith and CATS have significant and often longstanding relationships focused on transferring knowledge and advancing the state of the art with a number of key institutions involved in weather forecasting. The relationship with the European Centre for Medium Range Weather Forecasts (ECMWF), "considered the worldwide leader in global, medium-range, monthly and seasonal predications" dates back to the 1994 Royal Society of Meteorologists meeting on predictability [A]. Longstanding partners also include: the UK Met Office [B]; the US Naval Research Laboratory (NRL), where one of Professor Smith's former students now leads a group forecasting hurricanes in the Pacific; and the US National Centre for Atmospheric Research (NCAR), where Dr. Du is sharing methods and providing a test bed for data assimilations. More recent partnerships include: the International Research Institute for Climate and Society (IRI) [C], which focuses on climate service development, primarily within developing countries; the Risk Prediction Initiative (RPI) of the Bermuda Institute of Ocean Sciences (BIOS) [D]; and the Industrial Mathematics Knowledge Transfer Network of the Smith Institute. These relationships focus on two critical areas of weather forecasting: predictability and verification.
According to the head of ECMWF's Predictability Division, Roberto Buizza, "During the past 15 years, Prof. Lenny Smith has contributed to a range of key subjects in the area of predictability, which had an impact on the design of the operational systems of ECMWF, and in the use of ensemble-based, probabilistic forecasts...The work we did on the estimation of the impact of non-linearity...provided estimates on the time when non-linearity impacted on forecast accuracy, thus helping us refining our techniques and methodologies. Prof. Smith and his group's contribution in this area continued throughout the years, and the systems operational today are still benefitting from his research done years ago...and supported their gradual extension of the forecast range to the monthly, seasonal and decadal time scale" [A]. IRI's Chief Climate Scientist, Dr. Simon Mason, likewise has been using Professor Smith's work on second-order uncertainty and intractability in "communicating forecast uncertainty honestly at all times scales" and relying on Smith’s advice in contributing to the World Climate Research Programme Working Group on Regional Climate, "where issues of estimating forecast uncertainty are paramount" [C].
ECMWF, IRI and BIOS have all been influenced by the work of Professor Smith and CATS on verification, i.e. metrics to evaluate the accuracy of weather forecasts. For ECMWF, "his work on the assessment of probabilistic forecasts using information theory provided a new and extremely valuable measure of the quality of a probabilistic forecasting scheme. Based on the amount of a data compression it allows, called ignorance (Roulston & Smith 2002), this measure has been used routinely to assess the quality of ECMWF operational products" [A]. IRI's Dr. Mason was also influenced by the underpinning research [e.g. 6] and by extensive discussions with Professor Smith on the properties of verification scores, which is reflected in guidance and training that Dr. Mason has produced and delivered for the World Meteorology Organisation (WMO) [C] (see next section). Professor Smith has worked with BIOS on evaluating the skill of seasonal forecasts of sea surface temperatures in the tropical Atlantic and the equatorial Pacific Ocean where such temperatures play a role in the formation of hurricanes. This knowledge has been transferred through papers and through two workshops with members of BIOS’ Risk Prediction Initiative [D].
Advancing practice in probabilistic weather forecasting: The advance of practice is where the reach of the impact has perhaps been greatest, through tools, guidance, training, software and products based on the underpinning research that have influenced weather forecasters around the world. What has proved to be highly influential is a conceptual framework called Weather Roulette [6] that allows weather predictors to more easily communicate the skill and value of forecasts to customers and users by translating the probabilities into effective daily interest rates, which is particularly useful in situations where small probabilities can lead to large costs or benefits. Weather Roulette has been incorporated into ECMWF's Ensemble Verification Training Course since at least 2010 [E]. It is also an essential component of a verification training course run by the WMO's Commission on Climatology over the past five years to train trainers from regional climate centres, who then assist in training seasonal forecasters in their regions. To date trainings have been held in China (2x), Trinidad, South Africa, Colombia, Argentina, Kenya (2x) and the US, involving over 150 trainers [C].
Weather Roulette has also been incorporated into the Climate Predictability Tool, a software package being used by all meteorological services in South America and extensively in Central America, the Caribbean, Africa, South and East Asia, and elsewhere for the production of seasonal forecasts [C]. Equally significant, effective interest rates (i.e. weather roulette) [6] and ignorance scores [1] have been included as two of the seven recommended verification scores and procedures in the Guidance on Verification of Seasonal Climate Forecasts being officially disseminated to the 191 member countries and territories of the WMO to guide seasonal forecasting by their National Climate Centres and regional and local forecasters [C,F]. This guidance document is also expected to influence the approaches utilised by both governmental and commercial seasonal forecasters around the globe.
In addition, the UK Met Office has used the concepts of kernel dressing and blending climatology from the underpinning research to redesign its 3-month Outlook product, which is constantly available on behalf of the Cabinet Office to assist contingency planners across the public and private sectors to prepare for extreme weather events and potential emergencies [B,G].
Advancing the application of probabilistic weather forecasting (industry): Professor Smith and CATS have a long history of partnership with industry in embedding the research findings in contexts where real-time forecasting is helpful in managing uncertainty and risk. Particular attention has been given to the challenges of constructing actionable information from raw ensemble weather forecasts and framing this information in a format useful for specific users. The wide range of applications has included: road gritting, food sales, horse race-track conditions, crop forecasting, insurance brokerage, energy trading, wind power production, hydrocarbon exploration and production (oil reservoirs and flows), and electricity generation. About one-third of Professor Smith's students and postdocs now continue this applied work in organisations such as the Bank of England, Royal Dutch Shell, the Met Office and Risk Management Solutions. Current CATS partnerships are under way with the UK Department of Energy and Climate Change (DECC) and the Royal National Lifeboat Institute.
The application impact has been most obvious in the energy sector. Altalo and Smith (2002) "promoted the use of probabilistic weather forecasts in business, thus increasing the return on investments in weather forecasting and reducing energy production costs. The benefit of this work is still felt today, which sees the energy sector (producers and energy traders) as one of the main users of weather forecasts" [A]. More specifically, work was done with EDF Energy to develop and implement improved methods for electricity demand forecasting, which helped to reduce risk, avoid u-turn trades, manage supply and improve performance. Metra, the global commercial arm of the New Zealand meteorological service, also partnered with CATS on the design and marketing of a product called Vantage to help energy managers and traders in managing weather-related opportunity and risk and in making operational decisions [H], which has led to a new generation of Metra products and services in use worldwide [I].
Sources to corroborate the impact:
A ECMWF training:
https://apps.lse.ac.uk/impact/download/file/1573
B. IRI Guidance on Verification of Seasonal Climate Forecasts:
https://apps.lse.ac.uk/impact/download/file/1575
C. UK Met Office 3-month outlook (as example of impact):
https://apps.lse.ac.uk/impact/download/file/1576
D. Metra Vantage marketing brochure:
https://apps.lse.ac.uk/impact/download/file/1577