%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Chemistry Climate Model Validation (CCMVal) Diagnostic Package (CCMVal-Diag)
README File
Version 3.0
Release: 19 July 2012
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
A. Gettelman, V. Eyring, C. Fischer, H. Shiona, I. Cionni, M. Neish
Sponsored by: NASA-ACMAP, NCAR, DLR, U. Toronto, NIWA
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Contacts: andrew@ucar.edu, Veronika.Eyring@dlr.de

This document describes the Chemistry Climate Model Validation Diagnostic (CCMVal-Diag) tool. A basic code description is provided, followed by
installation and running instructions. This documents the VERSION 3 release, designed to convert model output to the CCMVal-2 data standard and to
produce standard diagnostics.  This version specifically works with
CCMVal-2 and CMIP5 model output.

Users are encouraged to please contribute code back to the tool code repository for derived variables processed with the tool (e.g: PV, Eq-lat, Tropopause, etc).  See contact information above.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Quick Start
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Yes you should read this document. But to get started with the supplied
namelists, you will need to (note 4 & 5 for modifying the namelists):

1. Follow the installation instructions
2. Pick a sample namelist
3. Download some model output
3.1 Sample CMIP5 data is contained on the website, (renamed files)
3.2 CCMVal1 and CCMVal2 output is contained at the British Atmospheric Data Center
    http://badc.nerc.ac.uk
3.3 Sample reanalysis files are also on the website: 
    ERA40 is set to run with namelist_test_ccmval2
4. Modify the path locations of the model output
5. Run: python main.py <namelist_file>
6. Plots should end up in the work directory. For errors see standard output
Note: The 'work_dir' can be local (as in the namelists), or a full path
  to a different location (since climo files can be large).

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
OUTLINE
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

1. Further Information
2. Introduction
3. Versions
4. Structure
5. Installation
6. Setup to process model output
7. Model specific modifications for conversion of model output only
8. Editing diagnostic sets for plotting
9. NCL routines to reproduce Eyring et al. JGR (2006) plots

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
1. FURTHER INFORMATION
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

More information on the standard and the data request to run this tool
can be found at:

http://www.pa.op.dlr.de/CCMVal/DataRequests/CCMVal-2_Datarequest_FINAL.pdf

For general information on the CCMVal project see:

http://www.pa.op.dlr.de/CCMVal/

This document and the diagnostic code, with latest bug fixes and updates
is available by link from:

http://www.pa.op.dlr.de/CCMVal/CCMVal_DiagnosticTool.html 

A description of the version 3 tool has been published at:

Gettelman, A., Eyring, V., Fischer, C., Shiona, H., Cionni, I., Neish, M., Morgenstern, O., Wood, S. W., and Li, Z.: A community diagnostic tool for Chemistry Climate Model Validation, Geosci. Model Dev. Discuss., 5, 1229-1261, doi:10.5194/gmdd-5-1229-2012, 2012

http://www.geosci-model-dev-discuss.net/5/1229/2012/gmdd-5-1229-2012.html

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
2. INTRODUCTION
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

This Chemistry Climate Model Validation Diagnostic (CCMVal-Diag) Package 
is designed to process model output from coupled Chemistry Climate Models
(CCMs). It is targeted and developed for the CCM Validation Project for SPARC,
and specifically the CCMVal-2, Climate and Forecast (CF) metadata compliant
data request 
http://www.pa.op.dlr.de/CCMVal/DataRequests/CCMVal-2_Datarequest_FINAL.pdf

The code also works for CMIP5 model output, and in principle any output conforming to the Climate and Forecast (CF) Metadata conventions.
 

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
3. VERSIONS
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

REVISION HISTORY:

Version 3.0 (June 2012): the infrastructure has been modified to read across multiple files to reduce memory usage. The tool now works with CMIP5 output. A new facility for modifying model data to adjust for unconformities has been added, and additional testing and corrections to the Eyring et al 2006 plots have been made. 

We have introduced two new environment variables 'MAX_DATA_FILESIZE' and 'MAX_DATA_BLOCKSIZE' to reduce the memory usage at one time. 'MAX_DATA_FILESIZE' determines which files are stored in memory or left in HDD till they are required by the program. 'MAX_DATA_BLOCKSIZE' controls the data sizes to be handled at one time during the processing. They are set in 'main.py' and user can change the values and have some control on memory usage.

The file 'attribute.ncl' is called for reformatting of files. It loads '<ProjectName>_<ModelName>.ncl' (if exists) in the ./reformat/fix_<ProjectName> directory which fixes model depended problems such as dimension name, dimension units and data ranks. These corrections were used to be in few different parts of codes.  Please see examples in ./reformat/fix_CCMVal2 directory.


This version corresponds to the publication of a reference document on the tool.

Gettelman, A., Eyring, V., Fischer, C., Shiona, H., Cionni, I., Neish, M., Morgenstern, O., Wood, S. W., and Li, Z.: A community diagnostic tool for Chemistry Climate Model Validation, Geosci. Model Dev. Discuss., 5, 1229-1261, doi:10.5194/gmdd-5-1229-2012, 2012

http://www.geosci-model-dev-discuss.net/5/1229/2012/gmdd-5-1229-2012.html

Version2.0: Expanded version of 1.0 designed to correct for 'unique'
aspects of data submitted to the CCMVal-2 archive. Code corrects
for various inconistencies in time and coordinates on a model by 
model basis with specific flags. Also assorted bug fixes to total ozone
calculations and some CCSM/WACCM specific corrections in convesion routines
related to CCSM dates and time stamps.

Version1.0:  Designed for SPARC CCMVal Report co-authors 
and others working with CCMVal-2 model output. Features basic functionality 
for reading and working with CCMVal-2 output files from the BADC.
Basic plot types and variables are supplied. Basic code for reproducing 
diagnostics done on CCMVal-1 output (Eyring et al., JGR, 2006; hereafter 
E06) is also supplied.

Eyring, V. et al., Assessment of temperature, trace species and ozone 
in chemistry-climate model simulations of the recent past, J. Geophys. Res.,
111, D22308, doi:10.1029/2006JD007327, 2006.

Not implemented yet: (a) comparisons to observations for diagnostics except
for the E06 diagnostics, (b) web interface for plots.

Beta Release (Beta0-9): The Beta release is targeted specifically at the
CCMVal model groups, to produce output needed for the CCMVal-2 data 
request, and the SPARC CCMVal Report. 

The basic plotting structure is there, but the code is developed to 
convert and check output. The code will compare 2 (or more) model runs.
Plotting routines based on past papers (Eyring et al. 2006) are not
contained in this code.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
4. STRUCTURE
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

The CCM-diag code is based on Python and the NCAR Command Language (NCL) 
scripting language. It requires these two packages to be installed. It
takes as input netCDF files either from (a) model output or (b) in CCMVal-2
data request format. If (a), it will process model output into (b). The
code will further create climatology and timeseries files for the
specified variables.

Customization is required for each model to get output into the right
format. This initial release comes with code for translating NCAR
Community Climate System Model (CCSM) format netCDF files as a template.
Each model will need it's own piece of code to do this.

The code will read MOST files in CCMval-2 or CMIP5 format. In general, most files that are Climate and Forecast (CF) compliant netCDF (with the right meta data) will work. See http://cf-pcmdi.llnl.gov/ for more information on the CF Metadata standard. There are some models
with slightly incompatible formats due to improper specifications or 
use of the specifications. These can usually be read by small modifications either to the files, or ideally to the input meta data by use of the reformat code in ./reformat/fix_* directories that defines a special fix_data function (see below).

The diagnostic code runs by running a main python script. This script calls a 
control namelist which specifies (a) global flags, (b) model output to
process and (c) a file of diagnostic sets to run.  The diagnostic
set file (in the diag_att subdirectory) is another namelist type file 
which lists each variable, and which 'diagnostics' to run on it.
A diagnostic is specified as an ncl program. It can be as simple as a 
standard 'plot type' (zonal mean latitude height, lat-lon at a level, 
etc), or include a complex set of functions. In general complex
diagnostics can be specified as a new variable (see below), using a 
standard plot type. 

This is illustrated schematically in the jpg figure contained in this 
directory.

Variable names are either standard names from the (CCMVal-2 or other) CF
specification, or 'derived' variables. Each variable name must have an 
attribute file. The variable attribute file (in the var_att
subdirectory, called <varname>_att.ncl) is ncl code that sets parameters 
for plotting, as well as the function and other variables needed if it 
is a 'derived' variable.

Observations can be simply treated as (a) another 'model' to include 
in the set, or (b) diagnostics and variables for each observation will have to be specified for individual plot types.

The code is set up to read in netCDF files with either one time sample per
file or multiple time samples per file. WACCM uses both, so examples are
coded into ccsm.ncl

The code can also now read across files with multiple times in them, and is designed to read time sequentially to reduce memory usage.

Further details are provided below.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
5. INSTALLATION
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

1 Python: verify that you have a python distribution (python -V)
    The code has been tested with python 2.3.4 and should work with that
	version or later.

    If not, install Python on your system: http://www.python.org/
    Windows, MacOSX and Linux binaries are available


2. NCL: make sure you have at least NCL version 5.1 installed (ncl -V)

    If not, visit: http://www.ncl.ucar.edu/Download/index.shtml
    Binaries for Supported systems include: 

    * AIX
    * IRIX
    * Linux (i686, x86_64, IA64 systems, GCC v3.x or v4.x) 
	    (see installation notes)

    * MacOSX (PPC and Intel) (see installation notes)
    * Solaris
    * Windows under Cgywin Unix emulator (see installation notes)

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
6. SETUP TO PROCESS MODEL OUTPUT
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

------------------------------------------------------------------------
6.1. Introduction setup to process model output
------------------------------------------------------------------------


The following assumes that the code to convert model output and the name
translation has already been written. If not, see Section 6.2. This
will work for CCSM (WACCM) files in the released beta code. Other models
will be added.

The code generates two types of files: Timeseries files (T3M, T2Ms, etc)
which contain one variable at all times. This type is suitable for uploading
to the BADC. The code also makes 'climatology files', (C3M,etc) that are used
internally for plotting. Timeseries files are in the CCMVal-2 CF compliant
netCDF standard.

Directory structure and location of namelist control files:

The top level directory is: '$home'

$home/
main.py       (main python script used to run the conversion and plotting codes)
namelist_tst  (master namelist)
ccsm.py       (model specific formattting code: see section 7)
ccsm.ncl      (model specific formattting code: see section 7)

$home/reformat

ccsm_convert.txt  : mapping from CF standard names to CCSM model variable names
attribute.ncl: defines fix_data function for adjusting model output read into the tool by project and model (in ./reformat/fix_{PROJECT_NAME} directories)

$home/diag_att

testplot.att	(test plotting script)
E06Diag.att	(sample plots from Eyring 2006 paper)	
	(namelist used to select which diagnostic plots will be created)

$home/var_att

PS_att.ncl              
PSL_att.ncl
T_850_att.ncl
... etc
(variable attribute files used to control plots of the 
 particular variables e.g. PS, PSML, T_850 ...)

$home/plot_type
	(NCL code for standard plot types and Eyring 2006 plots)
	Standard types include:

======================================================================
(AFTER) - beta02_modified
For multiple models,
# table_one         regional table mean,diff,rmse  "T2Ms","T2Ds","C2Ms"
# vertconplot       lat vs height contour plot     "T3M","T2Mz","T2Dz","C3M","C2Mz"
# zonlnplot         lat vs height line plot        "T2Ms","T2Ds","T1Ms","C2Ms","C2Ds","C1Ms"
# plrconplot        polor contour plot             "T2Ms","T2Ds","C2Ms"
# surfconplot       surface contour plot           "T2Ms","T2Ds","C2Ms"
# seacycplot        line plot of seasonal cycle    "T2Ms","T2Ds","C2Ms"
# anncycplot        seasonal cycle                 "T2Ms","T2Ds","T1Ms","C2Ms","C2Ds","C1Ms"
# seadiffplot       seasonal difference plot       "T2Ms","T2Ds","C2Ms"
# profiles          vertical profiles of station   "T3M","C3M"
# tsline            1-D timeseries plots	   "T3M","T2Mz","T2Ms","T0M"
For two models,
# vertconplot_pair  lat vs height contour plot     "T3M","T2Mz","T2Dz","C3M","C2Mz"
# plrconplot_pair   polor contour plot             "T2Ms","T2Ds","C2Ms"
# surfconplot_pair  surface contour plot           "T2Ms","T2Ds","C2Ms"
# anncycplot_pair   seasonal cycle                 "T2Ms","T2Ds","T1Ms","C2Ms","C2Ds","C1Ms"

	see the '$home/diag_att/PLOTTYPE_sample.att' attribute set if you want to play with plots

======================================================================

------------------------------------------------------------------------
6.2. Setup process
------------------------------------------------------------------------

1. Edit a sample namelist ['namelist_test_CCMVal2']

   (rename to your own file)

   1.1 Select options

   1.2 Specify directories
	work directory for plots and converted files

   1.3 Select 'model runs' to process
     	A run can be part of a run (years are specified).
	The code specifies models with 6 arguments separated 
          by spaces on a single line:
	1. Model name
	2. Simulation (REF1)
	3. Ensemble ('1')
	4. Start year
	5. End Year
	6. Directory to output

   1.4 Select diagnostic set.

      This needs to be a file name in the 'diag_att' directory
'convert_M' and convert_I' are basic scripts to just convert variables
E06Diag is a plotting script (see README_E06). EO6 is functional on netCDF
ccmval-2 output (available from the BADC).

2. Edit the diagnostic set if necessary.
	Change variables to process here.

3. Ensure all variable attribute files exist in var_att directory
   Note that for non-derived fields, these files can be copied from 
   existing files. The variable attribute files contain information 
   to control the way the associated plots are generated.
   (see next section)

4. run main.py ( >python main.py namelist_{name}) with the correct master namelist.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
7. MODEL SPECIFIC MODIFICATIONS FOR CONVERSION OF MODEL OUTPUT ONLY
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

For analysis of BADC model output, users can skip this section.

Needed files:  <modelname>.py,
	       ncl_code/<modelname>.ncl,
	       reformat/<modelname>_convert.txt
Modification:  main.py will have to have a conditional call to <modelname>.py

Two routines handle the conversion of model output to CCMVal-2 compliant

netCDF files: ccsm.py and ccsm.ncl main.py calls ccsm.py, and ccsm.py
calls ccsm.ncl. In addition, the file reformat/ccsm_convert.txt maps the
model names to the variable names used in the CCMVal-2 specificaiton and the
rest of the diagnostic package.

     -ccsm.py sets the filenames, and gets the variable names, and then
      calls the ncl file to process the data.

     -ccsm.ncl performs operations on the file list to concatentate files
      together. It assumes netCDF files, but could work on other file types
      as well (anything NCL can read).

1. Make sure <modelname>.py and <modelname>.ncl files are built and exist.
   This means modifying the header and model information, and writing 
   appropriate read code for raw model output files in ncl_code/<modelname>.ncl
   it also means adding a conditional for the unique model name to main.py

NOTE: WACCM output for monthly means comes 1 time sample per file, while for
instantaneous output there are multiple time samples per file. The code expects
a netCDF file. If your model has multiple times per file, use the 'I' 
code as a base, if one per file, use the 'M' code in the ccsm.ncl routine.

2. Remap variable names as necessary in reformat/<modelname>_convert.txt

3. Write appropriate functions into main.py for <modelname>.

4. run


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
8. EDITING DIAGNOSTIC SETS FOR PLOTTING
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

A note on colors: colors are set by model name and project name. Functions
for differnt projects are in the file: 
~/plot_type/CCMVal_FUNCTION/misc_function.ncl
There is no function for CMIP5 model colors, but one could be built
as a copy of (for example): CCMVal2_levelsColor
Colors for observations would be set there as well.

The 'plot_types' directory contains ncl plotting code for a variety of
standard plot types. 

'convert_M' is a basic set that creates climatology files for most
non-derived monthly-mean output fields
	
The version 3.0 release provides a list of basic plot types for comparison.
To control which plots are created, the variable name (CF or derived type) 
is linked to a data type and plot type. These plots should work.

Examples:

Var names Data types   	Plot types
--------- ----------    ----------
ta	  C3M 		none
ta	  T3M		tsline
ua200     C2Ms           vertconplot_plev (lat vs height contour plot)
ua200     T2Ms		tsline (1D line plot with trends)
ua200     T2Ms	   	savetonetcdf (save derived variable timeseries)
ta        C3M           zonlnplot (lat vs height line plot)
PS        C2Ms     	plrconplot (polar contour plot)
O3        C2Mz          seacycplot (line plot of seasonal cycle)
H2O       C2Mz          seadiffplot (seasonal difference plot)
... etc                                                         

Data Types Key 
C3M = climatology, 3d monthly mean
C2Ms= climatology, 2d monthly mean (surface or single level field)
C2Mz= climatology, 2d zonal monthly mean

T3M = timeseries, 3d monthly mean (etc)

NOTE: The data type should correspond to the OUTPUT desired for a 
plot type. Given plot types take specific data types. For example: a 3-D
field or 1-D field cannot be used with a surface contour plot. The specifier 
does not correspond to the INPUT file used to derive a field. For example, 
ua200 uses T3M to derive a T2Ms field. for plotting, use 'ua200 T2Ms' and
'ua200 C2Ms' as noted above. In the variable description for ua200
(./var_att/ua200_att.ncl), the type will be specified as 'T3M'. The code 
will look  first to see if a derived field has been saved (as ..T2Ms_ua200.nc)
in the climo directory, and if not, it will re-derive the variable from the 
specified field(s) in the <var>_att.ncl file.

Plot Types Key:
noplot (no plot, convert only)
anncycplot (annual cycle of monthly values, zonal means of a 2D field v. month)
	plot takes a 'refModel' attribute. If set does differences, if not, does full field
save_to_netcdf (save timeseries of a derived variable)
vertconplot (lat vs height contour plot of a 3D or 2D zonal mean field)
vertconplot_plev (lat vs height contour plot: differences)
plrconplot (polar contour plot of a 2D field)
seacycplot (line plot of seasonal cycle, weighted global mean of a 2D field)
seadiffplot (contour plot of seasonal difference DJF-JJA for all models)
tsline (timeseries plot: seasonal and annual, anomalies or full field)
surfconplot (2D surface contour plot)
	plot takes a 'refModel' attribute. If set does differences, if not, does full field
surfcontrend (surface contour plot of trends)
zonlnplot (lat vs height line plot)
zonlntrend (zonal mean trends and difference from a reference model).
profiles (temperature profiles at select stations)

Note: many plot types have specific required attributes (part of the ncl 
'info' structure that is passed). Each variable needs to have these. These 
are usually contour intervals or other aspects of plotting. If an error
message is generated, check the var_att/<varname>_att.ncl file for missing 
attributes (info@....=). Plot types will usually fail and exit with an 
error message for a missing attribute

Examples:

ua_200 C3M seacycplot,zonlnplot,plrconplot,surfconplot,seacycplot,seadiffplot,anncycplot
TS C2Ms seacycplot,zonlnplot,plrconplot,surfconplot,seacycplot,seadiffplot,anncycplot
PS C2Ms seacycplot,zonlnplot,plrconplot,surfconplot,seacycplot,seadiffplot,anncycplot
ua C3M vertconplot_plev
O3 C3M noplot
tatp T2Ms tsline,save_to_netcdf

If 'noplot' is selected (or a plot name is not recognized), 
then Timeseries and Climatology files will be created, but no plots.

If 'savetonetcdf' is selected, then timeseries is saved to the climo 
directory. The diagnostics checks for climo and timeseries first before
recalcuating, and does not recalculate if the files exist (handy for 
variables that take a long time to process).

Derived fields are being built now, and will be shared. 
For example: tropopause pressure (tatp).

Example:

The TS_att.ncl (surface temperature) file is listed below:

;
; Requires: none
;
info = True
info@derived = False
info@long_name="Surf Temp (radiative)"
info@units="K"
info@pcp_cnlvl = (/210,220,230,240,250,255,260,265,270,275,280,285,290,295,300/)
info@pcp_dcnlvl = (/-15,-12,-9,-7,-5,-3,-1,0,1,3,5,7,9,12,15/)

The "info" variable contains the information required by the plotting 
software to control for example contour intervals, x and y limits on plots etc.

In the above example, the line:

info@pcp_cnlvl = (/210,220,230,240,250,255,260,265,270,275,280,285,290,295,300/)

the pcp_cnlvl array specifies the contour intervals for the polar contour 
plot of TS. The pcp_dcnlvl specifies the contour intervals for the
difference plot of TS (polar contour plot).

All the variable attribute files are set up in standard
configurations in the beta download.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
9. NCL ROUTINES TO REPRODUCE Eyring et al. JGR (2006) PLOTS
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

------------------------------------------------------------------------
9.1. General Information how to run E06 NCL Diagnostics
------------------------------------------------------------------------

Authors and Contact for E06 NCL Diagnostics:
Irene Cionni (irene.cionni@dlr.de) & Veronika Eyring (veronika.eyring@dlr.de)

Item 9 of this document describes the NCL routines that produce the plots 
and plot variables for a given set of models and observations as in
Eyring et al. JGR (2006), hereafter E06. 

Eyring, V. et al., Assessment of temperature, trace species and ozone in
chemistry-climate model simulations of the recent past, J. Geophys. Res., 
111, D22308, doi:10.1029/2006JD007327, 2006.

See these figures for CCMVal-1 and CCMVal-2 models in comparison at 
http://www.pa.op.dlr.de/CCMVal/SPARC_CCMValReport/AuthorTeams/CCMVal2_Analysis.html

VERSION 1: 
The CCMVal-Diag_Version1 tar-file includes the observations (except 
for Figure 15) and examples for two models (WACCM and CMAM) for all
figures of E06 (Figures 1-15). It should fully work on your machine once 
the CCMVal-diag tool is installed. To add the other CCMs that 
participated in CCMVal-1 and CCMVal-2, please download the CF compliant
netCDF data from BADC:

For CCMVal-1:
/project_spaces/ccmval/DIAGNOSTICTOOL/CCMVal-1_convertedto_CCMVal-2CFcompliantnetCDFfiles/REF1_netCDF

For CCMVal-2:
/project_spaces/ccmval/CCMVal-2/REF-B1

Sample CCMVal-2 model output that is displayed in the E06 figures
is available for testing in a separate E06_CCMVal-2 tar file.

Please make sure that the original sources for models and
observations are correctly cited if you show plots, see 
information in E06. Please also follow the CCMVal data policy 
(http://www.pa.op.dlr.de/CCMVal/CCMValDataPolicy_Feb2008.pdf).

------------------------------------------------------------------------
9.2. Description of Content in E06 NCL diagnostics directory
------------------------------------------------------------------------

The following assumes that the code to convert model output and
the name translation has already been written. If not, see Section 6. 
The E06 NCL diagnostics will work for CCMVal files that follow the
standard format as defined in
http://www.pa.op.dlr.de/CCMVal/DataRequests/CCMVal-2_Datarequest_FINAL.pdf

Directory structure and location of namelist files:

#########################################
./ (main directory)
SPECIFY THE INPUT HERE
#########################################

main.py       (main python script used to run the conversion and plotting codes)
namelist_E06  (master namelist: select conversion tool & models)

#########################################
./diag_att
SPECIFY THE DIAGNOSTICS HERE
#########################################

E06_diag      (namelist to select which diagnostic plots will be created)

#########################################
./var_att
SPECIFY OBS, LAT, LON, MONTH etc HERE
#########################################

* CH4_att.ncl   (variable attribute files used to control plots of the particular variables e.g. ta, ua, O3, H2O ...)
* Cly_att.ncl
* H2O_att.ncl
* HCl_att.ncl
* mean_age_att.ncl
* O3_att.ncl
* ta_att.ncl
* toz_att.ncl
* ua_att.ncl

#########################################
Execute main.py with the command:
python main.py namelist_E06_CCMVal2
#########################################

------------------------------------------------------------------------
9.3. Setup to process model output for E06 NCL diagnostics
------------------------------------------------------------------------

#########################################################################
9.3.1. Use correct filenames for the models and observations
#########################################################################

The file names for all models will have to follow the filename
convention specified in the CCMVal-2 data request, i.e.

The output file name is ?CCMVal2_${CCMVal-model-experiment}_${MODELNAME}_${ID-ENSEMBLE}_?
followed by ?${Output field number}_? and '${output variable name}' 

For example:

CCMVal2_REF-B1_CMAM_1_T3M_ta.nc for monthly-mean 3-d temperature data 
or CCMVal2_REF-B1_CMAM_1_T2Mz_ta.nc for monthly-mean zonal mean 2-d T data.

Not working is e.g. test.nc

#########################################################################
9.3.2. Example for ?./diag_E06NCL/namelist_E06?
#########################################################################

9.3.2.1 Select options

(E.g.
  write_plots    yes/no
  write_netcdf   yes/no
  force_processing     true/false
  project      CCMVal1/CCMVal2
  plot_dir     ./plots/ )

9.3.3.2 Specify input files

   Select  'model ' ,'REF', 'run',  'years',   'dir'          to process

(E.g.
          |'model '  |'REF'|  |'run'|  |'years'|    |'dir'| 
 
          AMTRAC     REF-1      1     1980 2000    /data/REF1/AMTRAC/
          CCSRNIES   REF-1      1     1980 2000    /data/REF1/Reform/CCSRNIES/
          CMAM       REF-1      1     1980 2000    /data/REF1/Reform/CMAM/
           etc]

9.3.2.2: select diagnostic set

e.g.:
DIAGNOSTICS
  # specify the namelist of the diagnostics
  # the program assumes this namelist is in ./diag_att and has the extention '.att'
  # e.g. './diag_att/E06Diag.att' if specified as below
  E06Diag

#########################################################################
9.3.3. Example ['diag_att\ E06Diag.att']
#########################################################################

This is a partial list. See the file for a full and current list.

# Figures below that start with # are not produced; all others are.

ta  T2Mz  E06FIG01  #With ERA40
ta  T2Mz  E06FIG04  #With ERA40
ta  T2Mz  E06FIG07  #With ERA40
ua  T2Mz  E06FIG02  #With ERA40
H2O  T2Mz  E06FIG05 
H2O  T2Mz E06FIG08
H2O  T2Mz E06FIG07  
HCl  T2Mz E06FIG05  
CH4 T2Mz E06FIG05
HCl  T2Mz  E06FIG05A 
O3 T2Mz  E06FIG05   
Cly  T2Mz  E06FIG12
Cly  T2Mz  E06FIG12B  
Cly  T2Mz E06FIG05A  
toz  T2Ms E06FIG14


#########################################################################
9.3.4. Select diagnostic set ['diag_att\ E06Diag.att']
#########################################################################

'diag_att\E06Diag.att' selects the E06 plots that are produced, for example:

To control which plots are created, the variable name (CF or derived type)
is linked to a data type (ta, ua, H2O, ect) and plot type 
(E06FIG01, E06FIG02, etc).

  Fig1 of 'Eyring et al 2006' ->E06FIG01
  Fig2 of 'Eyring et al 2006' ->E06FIG02
  Fig4 of 'Eyring et al 2006' ->E06FIG04
  Fig5 of 'Eyring et al 2006' ->E06FIG05
  Fig5 top figures of 'Eyring et al 2006' ->E06FIG05A
  Fig5 bottom figures of 'Eyring et al 2006' ->E06FIG05B
  Fig7 of 'Eyring et al 2006' E06FIG07
  Fig8 of 'Eyring et al 2006' E06FIG08
  Fig9 of 'Eyring et al 2006' E06FIG09
  Fig10 of 'Eyring et al 2006' E06FIG05B
  Fig11 of 'Eyring et al 2006' E06FIG05
  Fig12 of 'Eyring et al 2006' E06FIG12
  Fig13 of 'Eyring et al 2006' E06FIG05
  Fig14 of 'Eyring et al 2006' E06FIG14
  Fig15 of 'Eyring et al 2006' E06FIG15

Examples:

Variable names                Data types           Plot types
--------------                ----------           ------------
ua (u wind )                    T2Mz               E06FIG02
O3 (ozone mixing ratio)         T2Mz               E06FIG05,E06FIG05A,E06FIG05B
ta (atmospheric temperature)    T2MZ               E06FIG01,E06FIG04,E06FIG07
etc
(for more details see the header info in the NCL routines)

In the E06Diag.att namelist file, each primary variable denoted
by its CF variable name or derived type variable is associated 
with a data type (e.g. T2Mz = 2D monthly zonal mean data) and a
series of plot types (comma separated). Please read CCMValDataRequest.


------------------------------------------------------------------------
9.4. Edit the diagnostic set if necessary
------------------------------------------------------------------------

#########################################
/diag_E06NCL/var_att
SPECIFY OBS, LAT, LON, MONTH etc HERE
#########################################

* CH4_att.ncl  (variable attribute files used to control plots of the particular variables e.g. ta, ua, O3, H2O ...)
* Cly_att.ncl
* H2O_att.ncl
* HCl_att.ncl
* mean_age_att.ncl
* O3_att.ncl
* ta_att.ncl
* toz_att.ncl
* ua_att.ncl

#########################################################################
9.4. Ensure all variable attribute files exist in 'var_att' directory
#########################################################################

The "info" variable contains the information required by the plotting 
software to control for example contour intervals, x and y limits
on plots etc.  

The variable attribute files contain information to control
the way the associated plots are generated.

Example:

The ta_att.ncl (surface temperature) file is listed below:
--------------------------------------------------------------------------------
; Requires: none
;
info = True
info@derived = False
info@long_name="Temperature"
info@units="K"

;------------------------------------------------------------------------------- 
;mean reference period between the anomalies are calculated

info@fig04_yr_min = 1980   ;start 
info@fig04_yr_max = 1989   ;end

;reference period betweeen linear trend is calculated as K/decades

info@fig04_trend_year_min = 1980
info@fig04_trend_year_max = 1999

;latitude boundary  (n-dim) minimun values in *lat_min maximum values in *lat_max)

info@fig04_lat_min = (/60.,-90.,-90./)
info@fig04_lat_max = (/90.,-60.,90./)

; selection of seoason (3 consecutive months) referring to latitude boundary
; same dimensions as *lat_min  and  then  *lat_max (n-dim)

info@fig04_season = (/(/"FEB","MAR","APR"/),(/"SEP","OCT","NOV"/),(/"ANN","ANN","ANN"/)/)

;es. [first plot latitutes between 60N 90N, season ("FEB","MAR","APR"),
;     second plot latitutes between 90S 60S, season ("SEP","OCT","NOV"),
;     thirt plot latitutes between GLOBAL, season("ANNUAL")]

;level selected (only one-dim)

info@fig04_lev_sel = 50.

;plot features
;x-axis min and max

info@fig04_xmin = 1960
info@fig04_xmax = 2006

;Observation NAME and file location (dir/name_file) 
;It can be omitted if comparison with Observation is not required

info@fig04_obs_name = (/"ERA40"/)
info@fig04_obs_file = "./plot_type/input_data/OBS/CCMVal1_1980-1999_ERA-40_1_T2Mz_ta.nc"
;-------------------------------------------------------------------------------;reference period of climatological mean for the models

info@fig07_year_min = 1990
info@fig07_year_max = 1999
;reference period of climatological mean for the Observation 

info@fig07_year_min_ERA = 1992
info@fig07_year_max_ERA = 2001

;latitude boundary  (n-dim) (minimun values in *lat_min maximum values in *lat_max)
info@fig07_lat_min = (/0./)
info@fig07_lat_max = (/0./)

;level selected   (one-dim)
info@fig07_lev_sel = (/100./)

;Observation NAME and file location (dir/name_file)
;It can be omitted if comparison with Observation is not required

info@fig07_obs_name = (/"ERA40"/)
info@fig07_obs_file = "./plot_type/input_data/OBS/CCMVal1_1980-1999_ERA-40_1_T2Mz_ta.nc"
;-------------------------------------------------------------------------------
;reference period of climatological mean 

info@fig01_yr_min = 1980
info@fig01_yr_max = 1999

;latitude boundary  (n-dim) (minimun values in *lat_min maximum values in *lat_max)

info@fig01_lat_max = (/90.,90.,-60.,-60./)
info@fig01_lat_min = (/60.,60.,-90.,-90./)

; selection of seoason (3 consecutive months 'DJF'or 'MAM') referring to latitude boundary
; same dimensions as *lat_min  and  then  *lat_max (n-dim)

info@fig01_season = (/"DJF","MAM","JJA","SON"/)

;Observation file location (dir/name_file) vs. bias are calculated. 
;It Cannot be omitted!!!!! This file is need to calculated the bias

info@fig01_obs_file = "./plot_type/input_data/OBS/CCMVal1_1980-1999_ERA-40_1_T2Mz_ta.nc"


------------------------------------------------------------------------
9.5. Run E06 NCL diagnostics
------------------------------------------------------------------------

Execute main.py with the command:

python main.py namelist_E06_CCMVal2

------------------------------------------------------------------------

