Earth System Model Evaluation Tool (ESMValTool)Axel Lauer1, Yoko Tsushima2, Mattia Righi1, Veronika Eyring1, Alexander Löw3, Ulrika Willén4, and Mark Ringer2

1 Deutsches Zentrum für Luft- und Raumfahrt e.V. (DLR), Oberpfaffenhofen, Germany2 Met Office, Exeter, United Kingdom3 Department of Geography, University of Munich (LMU), Germany4 Swedish Meteorological and Hydrological Institute (SMHI), Norrköping, Sweden

27 May 2015

Motivation – Development of anEarth System Model Evaluation Tool

• Facilitate the evaluation of complex Earth System Models, e.g.

• quick look at standard diagnostic plots & output diagnostic variables

• easy comparison of new simulations (e.g. sensitivity runs or runs with new model versions) with existing runs and with observations

• Raise the standard for model evaluation

• include additional diagnostics of ongoing evaluation activities so that we do not have to start from scratch each time

• implement more observations, account for observational uncertainties

• Quick assessment of where we stand with a new set of model simulations via standard namelists that reproduce specific papers, reports, etc.

• Traceability and reproducibility

• Easy expendability

• synergies with ongoing projects to expand the tool (e.g. NCAR CVDP)

• useful for model groups & those analyzing models

• useful for model development

Technical ImprovementsSchematic Overview of the ESMValTool Structure

Development of an Earth System Model Evaluation Tool Within EMBRACE: DLR, SMHI & EMBRACE partners in collaboration with NCAR, PCMDI, GFDL

• Open Source: Python Script that calls NCL (NCAR Command Language) and other languages (e.g. R, Python)

• Input: CF compliant netCDF model output (CMIP standards)

• Observations: can be easily added

• Extensible: easy to (a) read models (b) process output [diagnostic] with observations and (c) use standard plot types (e.g. lat-lon map)

Current developments include for example• Essential Climate Variables, e.g.

- Sea-Ice- Temperatures & water vapor- Radiation- CO2- Ozone

• Tropical variability (incl. monsoon, ENSO, MJO)• Southern ocean• Continental dry biases and soil-hydrology-climate interactions (e.g. standardized precipitation index) • Atmospheric CO2 and NO2 budget• Aerosol, clouds• Stratospheric ozone

Goal: Standard namelists to reproduce certain reports or papers (e.g., IPCC AR5 Chapter 9, Massonnet et al., 2012; Anav et al., 2012; Wenzel et al. 2014; Eyring et al., 2013)

Release as open-source

software planned for later

this year.

ESMValTool: A community diagnostic and performance metrics tool for routine benchmarking and process evaluation of Earth System Models in CMIP

Similar to Figure 9.24 of AR5

Similar to Figure 9.7 of AR5

Similar to Figure 9.5 of AR5 Similar to Figure 9.24 of AR5

CMIP5 MMM

CMIP5 MMM - OBS

Monsoon Precipitation Intensity and Domain

Similar to Figure 9.7 of AR5

Link to projections

Running along-

side the ESGF

Considering Observational Uncertainty in the ESMValTool

Observational uncertainty needs to be considered in the assessment of model performance and should be an integral part of the evaluation tools

The ESMValTool is quite flexible and can consider observational uncertainty in many ways, e.g.:- using more than one observational dataset to evaluate the models

- showing the differences between the observational datasets in addition to the difference between the model and the reference dataset

- easily usage of an observed uncertainty ensemble spanning the observed uncertainty space, consistent with statistics

Example: HadCRUT4 surface temperatures, 5 (out of 100) realizations

WCRP Grand Challenges: (1) Clouds, circulation and climate sensitivity, (2) Changes in cryosphere, (3) Climate extremes, (4) Regional climate information, (5) Regional sea-level rise, and (6) Water availability, plus an additional theme on “biospheric forcings and feedbacks”

Meehl et al., EOS, 2014

Goal

ESMValTool as one of the CMIP documentation functions to routinely assess the performance of CMIP DECK and CMIP6 simulations running alongside the ESGF

Work as part of ESA CCI CMUG

• Model benchmarking initiatives have become increasingly important to evaluate the quality of coupled Earth System Models (ESMs) and to support the model development process.

• However, ESA data is currently not used in the context of routine model evaluation.

• Model benchmarking is also important for the Climate Research Groups (CRGs) within the CCI and various CMUG activities.

• CMUG will therefore establish a standardized model benchmarking approach for the CCI in close collaboration with the CMIP and WGNE/WGCM Climate Model Metrics Panel, and will contribute to the development of a community-wide Earth System Model Evaluation Tool (ESMValTool) that is currently developed by different partners in different projects – coordinated by DLR.

• The goal is to work towards a standardized community based benchmarking toolkit that includes ESA CCI data and that could be used operationally for CMIP6 analysis in the forthcoming years.

It is expected that the preparation of the individual ESA CCI datasets in CMIP compliant format for obs4MIPs as well as the corresponding technical notes for obs4MIPs are created and submitted by the individual ESA CCI teams.

WP 5.1 – Subtasks

1. Contribute to the community-wide development of an Earth System Model Evaluation Tool (ESMValTool) by developing code and integrating diagnostics and performance metrics into the tool

2. Ensure integration of ESA CCI datasets as well as other ESA datasets (in particular GlobAlbedo or other Glob-products) in the ESMValTool for routine model evaluation and benchmarking.

3. Provide comparison to other observations that are routinely used in model evaluation, in particular those that are contributed to obs4MIPs (http://obs4mips.llnl.gov/).

4. Provide recommendations of standardized benchmarking to the CMIP and WGNE/WGCM climate model metrics panels.

5. Visualize the results through a website to enhance visibility of the ESA CCI data. The ESMValTool will be extended with capabilities to visualize the resulting plots on a website.

6. Support the CCI Climate Research Groups in the use of the ESMValTool by providing a user guide with the goal that the CCI CRGs can use the tool in their own evaluation work.

Goal: Performance Metrics calculated with ESA CCI Data

• Relative error measures of CMIP5 model performance, based on the global seasonal-cycle climatology (1980–2005) computed from the historical CMIP5 experiments. Figure 9.8 of IPCC AR5 (Flato et al., 2013).

• A similar figure will be produced for selected ESA CCI ECVs using ESA CCI as the reference data set and if available an alternate observational data set for comparison.

Baseline Proposal and Batch Options 1 and 2

Baseline Proposala) Atmospheric ECVs (aerosols) [DLR]

b) Terrestrial ECVs (soil moisture, fire, land cover) [MPI-M]

c) Ocean ECVs (SST, sea-ice) [DLR, MPI-M]

Batch Options 1 – O5.1d) Atmosphere: Ozone, GHG (CO2) [DLR], Clouds [Met Office]

e) Ocean: ocean colour [MPI-M]

f) Atmosphere, land and ocean [SMHI]

Batch Options 1 – O5.2g) Evaluation of a selected set of CMIP5 [DLR, MPI, Met Office] and CCI4MIP simulations [MPI-

M]

h) Using the results from the ESMValTool, contribute to a report that includes an assessment of:

i. how well state-of-the-art Earth system models simulate climatological mean, variability and trends of selected ECVs,

ii. how the new observational time series of the ESA CCI change the model evaluation and benchmarking assessment of the models in comparison to observations provided by obs4MIPs,

iii. whether observational uncertainty is still a limiting factor in benchmarking models.

Batch Options 2 – O5.1i) Atmosphere: GHG (CH4) [DLR]

DLR contributions

What has been done so far

1. Maintenance and documentation (Eyring/Lauer/Righi) (WP5.1)

a) A subversion controlled repository has been made available to allow the development by multiple users. In addition, a Mantis bug tracking system and a wiki page is provided to facilitate communication and documentation of the tool.

b) A user guide is in preparation that will support the ESA CCI teams in the use of the toolc) Work on ESMValTool documentation paper startedd) Technical improvements are on-goinge) First release planned for fall 2015

2. Development and integration of ESA CCI data into the ESMValTool (WP3.6, WP3.7, WP5.1, O3.2, O5.1, O5.2)

f) Atmospheric ECVs: aerosols, Righi/Lauer), ozone and GHG/CO2 (Wenzel), clouds (Lauer/Righi)

g) Ocean: sea ice (Senftleben)

• Work with new ESA’s new 17 year ATSR climate data record (AOD) startedWill be continued in year 2, performance metrics plots will be provided for all CMIP5

models, AOD will be added (also WP3.7)• Cloud products (CLT, LWP) from non-ESA sources (also WP3.7)

• O3, CO2, sea ice (also for WP3.6): work with ESA CCI data will be started in year 2

Technical ESMValTool workshop3-4 March 2015, Munich, Germany

• Present status and future perspectives of the ESMValTool

• Specific questions addressed

• What technical improvements can be made that would facilitate broad community development and usage?

• How to simplify ESMValTool integration with existing data repositories and the ESGF infrastructure?

• What are necessary steps towards a roadmap for ESMValTool enhancement for CMIP6 and beyond?

• How to prepare the software release and what is an appropriate license?

→ Technical improvements of the ESMValTool are supported also by ESA CMUG

Technical ESMValTool workshop3-4 March 2015, Munich, Germany

• Timeline for submission of first ESMValTool documentation paper

• The workshop was of particular importance for the technical development of diagnostics of the ESA CCI datasets and their integration into the ESMValTool, and for technical requirements to run the tool on new CMIP data alongside the ESGF.

• Defining the further mid-term roadmap for ESMValTool development on the international level.

• A first release of the ESMValTool as open-source software is planned this year as a deliverable of EMBRACE. This release will form an important basis for the work on the ESMValTool as part of ESA CMUG and will allow to include the ESA CCI data in an efficient manner. From ESA CMUG, we are planning to include some selected diagnostics using the Aerosol CCI 17 year ATSR climate data record already in this first release.

→ Technical improvements of the ESMValTool are supported also by ESA CMUG

Assessment of CMIP5 models with new ESA’s 17 year ATSR climate data record

As in Fig. 9.29 of IPCC Chapter 9.AOD integrated over the oceans for CMIP5 models,

MODIS and ESA-CCI

Clouds20

-yr

aver

age

LWP

(19

86-2

005)

CM

IP5

in

com

par

iso

n w

ith

UW

isc

sate

llit

e cl

imat

olo

gy

LMU contributions (Alexander Löw)

Technical work

• LMU has implemented a flexible python interface into ESMValTool that allows for integration of new diagnostics.

• LMU has developed a prototype for flexible automated testing of diagnostics within ESMValTool. This will maximize the quality of diagnostics across all ECVs.

• LMU has started to implement a soil moisture diagnostic to integrate ESA CCI soil moisture data (Loew et al., 2013) – to be finalized for first CMUG release of ESMValTool.

Administration

• LMU has so far not signed a sub-contract; draft of sub-contract was submitted by Met Office and is currently with LMU administration. Until signature and hiring of staff only minor progress can be expected.

SMHI contributions (Ulrika Willén)

First tests using ESA-CCI SST with the ESMValTool

Example plot

© Crown copyright Met Office

Metrics of Clouds and RadiationYoko Tsushima

ESA CMUG integration meeting May 2015

© Crown copyright Met Office

Auto-assess:

• Aims: Primary aim is to produce a comprehensive set of metrics & diagnostics to inform model development.

• Current status:

• Scientific development currently frozen whilst code is re-written in Python using iris library.

• Aim to produce a first external release to UM user community in 2015.

• Currently assumes native UM file format (pp).

Current & planned assessment areas

• Global tropospheric circulation• Conservation• Clouds & radiation• Hydrological cycle• Asian monsoon systems• MJO• ENSO• Processes over Africa• Tropical cyclones• Mid-latitude storm tracks and blocking• Stratosphere• Land surface• Aerosols• Ocean• Sea Ice

© Crown copyright Met Office

© Crown copyright Met Office

Auto-assess metric plot

© Crown copyright Met Office

RMSE metric based on cloud clustering of daily ISCCP simulator data

The error can be broken down into different cloud regimes, contributions of frequency of occurrence component and in-regime cloud radiative effects

Code provides as one of CFMIP diagnostics

Cloud Regime Error Metric for present day (CREMpd)(Williams and Webb 2009, Tsushima et al.,2013)

Tsushima et al 2013

© Crown copyright Met Office

CF

MIP

1 -> C

MIP

5 Change

in in-cloud Net C

RE

error in regim

e Net C

RE

RM

SE

Improvements are seen in mainly in the radiative properties of optically thicker low and high topped cloud regimes rather than frequency of occurrence.

Change between CFMIP-1 and CMIP5 of in-cloud Net Cloud Radiative Effect (Net CRE) Error in Net CRE RMSE within daily ISCCP simulator cloud regimes in the Tropics

Tsushima et al 2013

Shallow Mid-level Thin Low level Convective Deep Stratocu Cumulus Topped Cirrus Transition Anvil Conv

Cloud Feedback Model Intercomparison Project (CFMIP) CFMIP diagnostics code repository

Klein et al., 2012

Nam et al., 2012Qu et al., 2012 Sherwood et al., 2014

Tsushima et al.,2013, Williams and Webb, 2009Cloud Regime Error Metric for present day (CREMpd)

CFMIP encourages climate modelling and diagnostic groups to both use the existing metrics andprovide additional relevant codes.

Summary and OutlookAn ESM Evaluation Tool is available that facilitates the complex evaluation of ESMs and

their simulations (e.g. submitted to international Model Intercomparison Projects such as CMIP, C4MIP, CCMI)

The tool is developed under a subversion controlled repository

‒ Allows multiple developers from different institutions to join the development team

‒ Broad community-wide and international participation in the development is envisaged

‒ Collaboration with the WGNE/WGCM metrics panel

Current extensions

‒ Atmospheric dynamics, biogeochemical processes, cryosphere and ocean

‒ Need for a wide range of observational data to be used with the tool

‒ Observations should ideally be provided with uncertainty and a technical documentation (e.g. similar to those from obs4MIPs) and in CF compliant netCDF

‒ Improving statistical comparison and visualization

Regular releases to the wider international community

‒ Further develop and share the diagnostic tool and routinely run it on CMIP DECK output and according observations (obs4MIPs) alongside the Earth System Grid Federation (ESGF)

‒ Release of ESMVal tool to the public → will contribute to metrics panel code repository

‒ Work towards a full community-tool developed by multiple institutions / countries

Thank you!