Long-term Archiving of ClimateModel Dataat WDC Climate and DKRZ
Michael Lautenschlager
WDC Climate / Max-Planck-Institute for Meteorology, Hamburg
Wolfgang Stahl + Joachim Biercamp
German Climate Computing Centre (DKRZ) Hamburg
Visit at NCAR
October 27th – 29th, 2008 in Boulder, USA
NCAR (Oct. 27-29, 2008)Lautenschlager (WDCC/MPI-M)2
DKRZ:
• Earth system model development
• Simulations of past, present andfuture climate
WDC Climate:
• Long-term data archiving
• Inter-disciplinary datadissemination
NCAR (Oct. 27-29, 2008)Lautenschlager (WDCC/MPI-M)3
Block Diagram HLRE-II System at DKRZ
250 IBM Power6nodes
(240 Compute,10 I/O)
GPFS Filesystem
IBM DS5300
(2-5 PByte)
GPFS Filesystem
IBM DS5300
(2-5 PByte)
StorageTek Silos
Total Capacity:60000 Tapes
Approx. 60 PB
(LTO and Titan)
StorageTek Silos
Total Capacity:60000 Tapes
Approx. 60 PB
(LTO and Titan)
NCAR (Oct. 27-29, 2008)Lautenschlager (WDCC/MPI-M)4
Increase in installed compute power motivates
• finer spatial and temporal model resolution and
• integration of additional physical and chemical processes
into climate models.
NCAR (Oct. 27-29, 2008)Lautenschlager (WDCC/MPI-M)5
Next generation of compute server (HLRE-II) and of climate models atDKRZ implies data production increase with implications for long-term archiving.
HLRE2: Compute power increase by a factor of 60 (sustained)
Experience at DKRZ: Linear increase in data production withinstalled compute power
Previous data storage strategy: all data migrated to the long-termmass storage archive(that means archive increase follows directly the compute powerincrease)
Resulting problem: Since the total amount of money for investmentand for operations is fixed the cost relation between computeservice and data service shifts towards data service while reducingthe compute service fraction
This is not any longer feasable for HLRE-II at DKRZ. Thereforelong-term archive increase has been limited to 10 PB/year which isfive times more than the present data archive increase. Thedatabase increase of WDCC has been limited to 1 PB/year(presently 60 – 100 TB/year).
NCAR (Oct. 27-29, 2008)Lautenschlager (WDCC/MPI-M)6
Compute server architectures: Cray C90 (1996-2003) / HLRE: NEC SX-6(2003-2008) / HLRE-II: IBM Power 6 (2009-2013+)
(HLRE: Höchstleistungsrechnersystem für die Erdsystemforschung)
Data Archive at DKRZ
NCAR (Oct. 27-29, 2008)Lautenschlager (WDCC/MPI-M)7
Increase of WDCC data archive
NCAR (Oct. 27-29, 2008)Lautenschlager (WDCC/MPI-M)8
WDCC data downloads for 2007(catalogue accesses neglected)
NCAR (Oct. 27-29, 2008)Lautenschlager (WDCC/MPI-M)9
Analysis of data classes
Test data from model code development,life cycle: weeks to months
Project data from scientific model evaluation and researchprojects (DKRZ resources at project level),life cycle: 3 – 5 years
Final results as data products for international projects(IPCC) and scientific publications,life cycle: 10 years and longer
Resulting data hierarchy levels
Temp(orary)scratch discs at compute server
Workfixed disc space at project level for evaluation
Arch(ive)tape storage space (single copy) with expiration date forproject data beyond available disc space
Docu(mentation)documented, long-term tape archive (security copy) for dataproducts, focus on interdisciplinary data utilisation,data are fixed and no longer matter of change
NCAR (Oct. 27-29, 2008)Lautenschlager (WDCC/MPI-M)10
Tape space distributon to archive classes at DKRZ beginof 2007:
• part of the “work” space on tape because GFS too small
• “docu” domain consists of WDCC
• no expiration dates in “arch” domain, parts of “arch”domain belongs to “docu” but not yet documented
2007
NCAR (Oct. 27-29, 2008)Lautenschlager (WDCC/MPI-M)11
The new project based data storage strategy andresource assigment at DKRZ contains:
Separation of project data and long-term archive
Expiration date for project data
Aware, scientific decision to move data into the long-term archive within the given archive limits
Data documentation requirements for long-term archive
Long-term data archive (“docu” hierarchy level)accomplishes the rules for good scientific practice
NCAR (Oct. 27-29, 2008)Lautenschlager (WDCC/MPI-M)12
Data documentation requirements are accomplished byusing the WDCC infrastruture
CERA-2 metadata model developed in 1999
Catalogue interface: cera.wdc-climate.de
Input interface: input.wdc-climate.de
CERA-2 metadata content is complete with respect tobrowse, to discover and to use climate data which arestored in the database system or outside in flat files
Missing: structured information on data provenance(topic of EU-project METAFOR)
The WDCC matches international descriptionstandards like ISO 19115, Dublin Core or GCMD and isintegrated in international data federations
Data storage structure assembles storage of climatetime series per variable in BLOB data tables. Thisallows for web-based data catalogue search and dataaccess in small data granules.
NCAR (Oct. 27-29, 2008)Lautenschlager (WDCC/MPI-M)13
WDCC / CERA: General Statistics at 01-10-2008 00:00:10
Database Size (TByte): 370
Number of blobs: 6663287791 (6.6 billion)
Data access by fields and not by files.
Number of experiments: 1146
Number of datasets: 142062
Total size divided by number of BLOBs gives theaverage size of data access granules: 50 - 60 kB/BLOB
NCAR (Oct. 27-29, 2008)Lautenschlager (WDCC/MPI-M)14
WDCC User Categories
Experienced User
From numerical modeldevelopment and analysis
Consolidated knowledge ofmodel data structure andlimitations of model results
Application experience in toolsand infrastructure to processmodel raw data (files)
Familiar with Unixenvironments andprogramming languages
Non-experienced user
From climate mitigation andadaptation
Only little knowledge of modeldata structure and limitationsof model results
Require application adaptedmodel data products and field-based data access
Familiar with MS-Windowsenvironments and OfficeTools
NCAR (Oct. 27-29, 2008)Lautenschlager (WDCC/MPI-M)15
CERA Data Model
Entry
Reference
Status
Distribution
Contact
Coverage
Parameter
Spatial
Reference
Local Adm.
Data Access
Data Org
NCAR (Oct. 27-29, 2008)Lautenschlager (WDCC/MPI-M)16
Coloured columns correspond to BLOB data tables inWDCC.
Collections of matrix rows represents storage in modelraw data files (complete model output storage time stepby storage time step).
NCAR (Oct. 27-29, 2008)Lautenschlager (WDCC/MPI-M)17
Additionally WDCC offers the primary data publication service forfinal data entities which are of general scientific interest
Following the STD-DOI concept (Scientific and TechnicalData – Digital Object Identifier, URL: www.std-doi.de)
Important aspects of the publication process are
The identification of independent data entities which aresuitable for publication at the level of scientific literature,
The execution of an elaborated review process formetadata and climate data,
The assigment of additional metadata for electronicpublication (ISO 690-2) and of persistent identifiers (DOI /URN) and
The integration of publication metadata and persistentidentifiers into the TIB library catalogue (TechnicalInformation Library, Hannover) so that primary data entitiesare searchable and citable together with scientific literature.
Quality characteristic is presently “approved by author”,future development should be “peer reviewed”.
NCAR (Oct. 27-29, 2008)Lautenschlager (WDCC/MPI-M)18
STD-DOI data publication workflow
NCAR (Oct. 27-29, 2008)Lautenschlager (WDCC/MPI-M)19
NCAR (Oct. 27-29, 2008)Lautenschlager (WDCC/MPI-M)20
Data infrastructure integrates data stewardship in thelong-term archive
• Bit-stream preservation
• Quality assurance
• Usability enabling
NCAR (Oct. 27-29, 2008)Lautenschlager (WDCC/MPI-M)21
Long-term archive data stewardship
Bit-stream preservation
Secondary tape copies on different tapes andtechnology at separate location
Copy to new tapes after maximum number of tapeaccesses are reached (Refreshment)
NCAR (Oct. 27-29, 2008)Lautenschlager (WDCC/MPI-M)22
DKRZ archive development
NCAR (Oct. 27-29, 2008)Lautenschlager (WDCC/MPI-M)23
DKRZ archive increase and transition in tape technology
• In 2002 most data on SD3 (helical scan)
• Migration to 9940A and 9940B since they were available
• Migration to T1A (Titanium T10000)
NCAR (Oct. 27-29, 2008)Lautenschlager (WDCC/MPI-M)24
Number of files in DKRZ archive on different tape media
• Small files are stored on 9840C (small capacity but fast access)
• 1. peak: Start with NEC-SX6 yields exponential increase in numberof files and inposes the implementation of file quotas
• 2. peak: Delay in cleaning up number of files
NCAR (Oct. 27-29, 2008)Lautenschlager (WDCC/MPI-M)25
Long-term archive data stewardship (continued)
Quality assurance
Semantic examinations: behavior of a numerical modelcompared to observations and to other models, part ofthe scientific evaluation process
Syntactic examinations: formal aspects of dataarchiving and ensurance that data archiving is free oferrors as far as possible
Consistency between metadata and climate data
Completeness of climate data
Standard range of values (expectation ranges andsimple data statistics)
Spatial and temporal data arrangement
NCAR (Oct. 27-29, 2008)Lautenschlager (WDCC/MPI-M)26
Long-term archive data stewardship (continued)
Usability enabling
Complete and searchable documenation of climatedata entities (database tables and flat files) in thecatalogue system of the WDCC
WDCC offers web-based data access to small datagranules (individual entries in BLOB DB tables)
Archive technology transfer must be downwardcompatible to keep old data technically readable
Data processing tools and data format access librariesmust be migrated to new architectures
NCAR (Oct. 27-29, 2008)Lautenschlager (WDCC/MPI-M)27
WDCC Architecture
NCAR (Oct. 27-29, 2008)Lautenschlager (WDCC/MPI-M)28
Summary
DKRZ long-term data archive will still grow but slower than linearwith the installed compute power
Key increase factors arefor long-term archive: 10 PB/year,for WDCC: 1 PB/year.
Improvement of reliability of long-term archive because of moreemphasis on data stewardship than on technical data serviceoperations
At the end the new data archive concept will result in a completelydocumented and searchable long-term data archive.
In the future more server side data processing is requested for on-site data reduction, on the fly generation of application dataproducts and visualisation at working level and for presentations.
NCAR (Oct. 27-29, 2008)Lautenschlager (WDCC/MPI-M)29
References
2008
Michael Lautenschlager
Preservation of Earth System Model Data
In: Digital Preservation Europe, Briefing Paper 30th June 2008(http://www.digitalpreservationeurope.eu/publications/briefs/)
2007
Lautenschlager, M., Stahl, W.
Long-Term Archiving of Climate Model Data at WDC Climate and DKRZ
In: E Mikusch (Ed.): PV2007 - Ensuring the Long-Term Preservation andValue Adding to Scientific and Technical Data, Conference Proceedings.DLR, German Remote Sensing Data Center, Oberpfaffenhofen, 2007