CMIP5 and C3-INAD

logoWDC-C

Preservation and Long TermAccess of Data at theWorld Data Centre for Climate

Frank Toussaint

N.P. Drakenberg, H. Höck, S. Kindermann, M. Lautenschlager,H. Luthardt, H. Ramthun, M. Stockhause, H. Thiemann

World Data Centre for Climateat the German Climate Computing Centre (DKRZ)

Hamburg, Germany

logoWDC-C

•The World Data Centre for Climate

•Data Storage: Technology – Tapes and Disks

•Data Storage: LObStER – the Tape Storage Tool

•Storage Policy

•Long Term Archiving

•DOI - Digital Object Identifier

logoWDC-C

The German ClimateComputing Centre (DKRZ)is held by…

•Max Planck Society,University of Hamburg,and others.

•Mission:Provide HP computingpower and storage for theGerman Earth Sciencecommunity (mainly)

The World DataCentre for ClimateThe World DataCentre for Climate

$C:\Users\Frank Toussaint\Desktop\DKRZ\PIC\logoDKRZ2.gif$

$C:\Users\Frank Toussaint\Desktop\DKRZ\PIC\logoMPImet2010.gif$

$C:\Users\Frank Toussaint\Desktop\DKRZ\PIC\logoWDCC.gif$

$C:\Users\Frank Toussaint\Desktop\DKRZ\PIC\logoAWI.jpg$

$C:\Users\Frank Toussaint\Desktop\DKRZ\PIC\logoUniHH.gif$

$C:\Users\Frank Toussaint\Desktop\DKRZ\PIC\logoHZG.jpg$

logoWDC-C

$C:\Users\Frank Toussaint\Desktop\DKRZ\1104_EGU_Wien\WISaroundWDCC - Kopie.png$

WMO Information System (WIS)

•National Centres

•Global InformationSystem Centres

•Data Collection andProduction Centres

The WDCC as WISData Collection & Production CentreThe WDCC as WISData Collection & Production Centre

logoWDC-C

$C:\Users\Frank Toussaint\Desktop\DKRZ\PIC\logoDLRwdcRSAt.jpg$

The WDCC in theICSU World Data SystemThe WDCC in theICSU World Data System

•International Council for Science (ICSU)World Data System (WDS)World Data Centres (WDC)

•WDC Cluster Earth System Research:WDC-Mare, WDC-RSAT, WDC-Climate

$C:\Users\Frank Toussaint\Desktop\DKRZ\PIC\logoDLR.gif$

$C:\Users\Frank Toussaint\Desktop\DKRZ\PIC\logoWDCC.gif$

$C:\Users\Frank Toussaint\Desktop\DKRZ\PIC\logoWDCmare.gif$

$C:\Users\Frank Toussaint\Desktop\DKRZ\PIC\logoICSU.png$

$C:\Users\Frank Toussaint\Desktop\DKRZ\PIC\logoICSUwdc.png$

logoWDC-C

The Evolutionof Data QuantitiesThe Evolutionof Data Quantities

Climate Model Data:

Relativehomogeneousbut huge amounts!Needed: Tapeaccess (nearline)

Archive:

files

Container:

Blobs

Appl.

Server

Storage@DKRZ

TDS

LobServer

HPSS9 PB

CERA

DB Layer

• What

• Where

• Who

• When

• How

Midtier

logoWDC-C

Huge amounts of data in each container file

Very different sizes of records: 64b .. 2 Gb

Efficient administration of all records

Irregular access patterns(access latency independent of the record position)

Transactional behaviour for read/write

Fault tolerance for HD, controller, tapes, etc

LObStER:Large Object Storage andEfficient RetrievalLObStER:Large Object Storage andEfficient Retrieval

Lobsterconfigurationmanager

genericJDBC-driver

Application





specific JDBC-drivers loaded

LObStERLObStER

Intranet

Internet

$C:\Program Files (x86)\Microsoft Office\MEDIA\CAGCAT10\j0292020.wmf$

Application

show-container

read-record

fetch-records

Lobsterobjectmanager

Cache

Oracle RDB(or other)‏

LObStERLObStER

Application

logoWDC-C

Container files with blocked format

64-bit files and 64-bit internal position referencing

Max file size: 16384 PBytes

Entries stored in ≥1 blocks

Block sizes 2k, k ∈ { 8, 9, 10, …, 62 }

LObStER:The Data ContainersLObStER:The Data Containers

direct-pointer-blocks

data-blocks

indirect-pointer-block

header-blocks

LObStER:The Data ContainersLObStER:The Data Containers

logoWDC-C

Several steps:

ospecification & concept

ofilling of metadata & data

oquality checks & DOI

•LTA for, e.g., EUCLIPSE, MedCLIVAR, combine

Long Term ArchivingLong Term Archiving

logoWDC-C

LTALTA

$C:\Users\Frank Toussaint\Desktop\DKRZ\1104_EGU_Wien\1104_HLuthardt_Flowchart_v2_en.jpg$

Costs depend oncomplexity andefforts at our site:

•metadata

•reformatting

•etc

logoWDC-C

•Quality Checks on three levels for LTAQC L1: conformity to general standards(format, ...)QC L2: coarse automated content checksQC L3: detailed spot checks:TQA – Technical Quality AssuranceSQA – Scientific Quality Assurance

Long Term ArchivingLong Term Archiving

logoWDC-C

The WDC-Climate asPublishing Agency of the IDFThe WDC-Climate asPublishing Agency of the IDF

International DOIFoundation

International DOIFoundation

International

DOI Foundation

RegistrationAgencies

NationalOrganizations

Publisher

DataCite

DataCite

doi.org

DataCite.org

tib-hannover.de

wdc-climate.de

TIB, BL, …

TIB, BL, …

WDCC, …

WDCC, …

logoWDC-C

The Visibility of LTA Datain Public CataloguesThe Visibility of LTA Datain Public Catalogues

•DOI is given

•Catalogue metadata issent to the RegistrationAgency via thenational organization

logoWDC-C

The Data Life CycleManagementThe Data Life CycleManagement

Virtual ResearchEnvironment

DataProduction

DataEvaluation

DataDissemination

Long TermArchive

logoWDC-C

Thank you,

Questions?