Technische Implementation von CERA

Hannes ThiemannMax-Planck-Institut für Meteorologie

Modelle und Datenhannes.thiemann @ zmaw.de

Jena, 24. Januar 2007

Inhalt

Aufgabe und Motivation

Umsetzung Datenbanken

Anbindung an das HSM

Ausblick

Klimasystem

Klimamodell: Grid

Klimamodell: Auflösung

T42 (300 km)

T106 (120 km)

Datenmengen

Horizontalauflösung des Klimamodells

T42: 128 * 64 = 8192 Punkte pro Globalfeld T106: 160 * 320 = 51200 Punkte pro Globalfeld

Erforderliche Speichereinheiten (GRIB Format)

Horizontalfeld (Zugriffseinheit): 17.1 kB (T42) / 100.1 kB (T106)

Unix Filegröße für monatsweise akkumulierte Ergebnisse mit 6 Std. Speicherintervall und 300 2d Variablen (Physikalische Einheit):

616 MB (T42) / 3500 MB (T106)

240 Jahre Modellintegration (Logische Einheit): 1.7 TB (T42) / 10 TB (T106)

Umsetzung Datenbanken

The Winter TopTen Program identifies the world’s largest and most heavily used databases.

….. Congratulations on achieving Grand Prize award winner status (1) in Database Size, Other, All and TopTen Winner status Database Size, Other, Linux;Workload, Other, Linux in Winter Corp.'s 2005 TopTen Program! .......

(1) Grand prizes are  awarded for first place winners in the All Environments categories only.

WDCC's CERA DB has been identified as the largest Linux DB.

Wintercorp (2005) - DB Size: Scientific, Archive, and other

Company Size (TB)

DBMS Platform System Vendor

Max-Planck 222 Oracle Federated/SMP NEC

USGS/EROS 17 Oracle Centralized/SMP Sun

USGS/EROS 17 Oracle Centralized/SMP Sun

HP 1 NonStop SQL Centralized/MPP HP

T-Systems 1 Oracle RAC Centralized/Cluster Sun

See: www.wintercorp.com

Wintercorp (2005) - DB Size: Data Warehouse

Company Size (TB)

DBMS Platform System Vendor

Yahoo 100 Oracle Centralized/SMP Fujitsu Siemens

AT&T 1) 94 Daytona Federated/SMP HP

KT IT-Group 50 DB2 Centralized/Cluster IBM

LGR 25 Oracle Centralized/SMP HP

Amazon 25 Oracle RAC Centralized/Cluster HP

See: www.wintercorp.com1) 330 GB Norm. Data Volume

Oracle 9.2 single instance running on TX7 Enterprise Edition Partitioning Option Advanced Security

24 Tbyte disk attached to database nodes Database size ~260 Tbyte (logical) Database nodes connected to HSM system Data accessible on the internet 800 named users worldwide Daily access 300 GB/Day (average) New data 250 GB/Day (average)

CERA: Some Facts

BLOB

SX-6

AsAmA 16way AsAmA 4way

DXSM DXDB

Oracle DBDiskXtenderDisk cache

PostprocessingSystemraw

META + DataGFSEnvironment

GFS/Server

AsAmA 4way

DXDB. . .

Oracle DB

AsAmA 16way

GE Network

Users

GFS/Server GFS/Server

DXSN DXSN

BLOB

DXDM DXDM

Climate Model

1.Climate Model writes raw output (GFS I/O)

AP

GFS/Client

Post Process Application

2.PP reads raw data (GFS I/O)

PP writes data (local I/O)AP

OCI Application

3.OCI reads data (Local I/O)

AP

Local disk

Migration &

Staging

Oracle ApplicationServer

5.Data inquiry (OCI)

Oracle AS

© NEC Corporation

4.OCI writes BLOB (via networks)

Oracle InstanceOracle Instance

Level 1 - Interface:Metadata entries(XML, ASCII)+ Data Files

Level 2 – Interf.:Separate filescontaining BLOBtable data in application adapted structure(time series ofsingle variables)

Experiment Description

Pointer toUnix-Files

Dataset 1Description

Dataset nDescription

BLOB DataTable

BLOB DataTable

WDCC Data Topology

BLOB DB Table corresponds to scalable, virtual file at the operating system level.

Datenbanken: Aufteilung

11121311415

16

MetadatenDaten

EnterpriseUserSecurity

OID

Entry

Reference

Status

Distribution

Contact Coverage

Parameter

SpatialReferenceLocal Adm.

Data Access

Data Org

100.000 Tabellen800 GB

Data matrix of model experiment

Model variables

Mod

el R

un T

ime

2 D: small BLOBS (16 KB)

3 D: large BLOBS (3 MB)

Raw data file: direct model output (0.7 – 16.2 GB)

Each columm is one BLOB Table and one META Table in CERA-DB

Raw

data file in D

KR

Z A

rchive

T2M Precip SLP2D variables . . Temp

Water vapour

3D variables . .

T1T2T3.......Tn...................Tend

MetadataTable

Blob_idBlob_sizeStart_dateBlob_minBlob_max

Blob_mean

Structure of metadata tables

Informationen um Einfache Anfragen ohne Zugriff

auf Daten selbst zu beantworten.

Konsistenz zu den Daten selbst überprüfen zu können.

Qualitätskontrollen durchzuführen.

Liegen auf Disk

Metadaten erlauben die Abbildung der blob_id auf die wirkliche Modellzeit

BLOB DataTable

blob_idblob_data

Structure of blob tables

Range Partitioning

Table Partition 1

Table Partition 2

Table Partition n

…

blob_id 1 .. n

blob_id n+1 .. m

blob_id m+1 .. k

…

Time t0 .. tn

Time tn+1 .. tm

Time tm+1 .. tk

…

Datafile 1

Datafile 2

Datafile n

…

Umsetzung: HSM

Anbindung an das HSM

TBS - RW

TblPartition 1

TBS - RW

TblPartition 2

dxdb

TBS - RO

TblPartition 1

All tablespaces are moved

“at once” to dxdb

MigoutMigin

Migout / Migin

Migout takes place after files haven’t been modified for x minutes

Only one migout process per dxdb-filesystem Migin takes place immediately after a file is requested.

Only parts accessed are retrieved from the backend storage.

One migin process per requested file.

dxdb

LWM

HWM

Purging

Criteria for purging

Size of datafiles doesn’t matter Except: “small” datafiles can stay on disk

Time not modified (easy for read only tablespaces) Time not touched

Oracle has the tendency to touch data files quite often

Oracle parameter read_only_open_delayed could be an option

Prerequisite: 2 copies on tape

Inside the datafile

Primary Key

Lob Index

Table

Blob data

Header 128k

Frontend versus Backend

Header 128k

Filesystem Frontend HSM Backend

Header 128k

Part 1 = 512 MB

Part 2 = 512 MB

Retrieving data

4

Header 128k

3 1

2 5

Tape Request

Usage: Downloads

Downloads per year

0100000200000300000400000500000600000700000800000900000

1999 2000 2001 2002 2003 2004 2005 2006

Statistics: Size

Database Size

0

50

100

150

200

250

300

1998 1999 2000 2001 2002 2003 2004 2005 2006

Year

TB

yte

Ausblick: Globalmodell T213 (Atmosphäre)

Horizontalauflösung des Klimamodells

T213: 640 * 320 = 204800 Punkte pro Globalfeld T106: 160 * 320 = 51200 Punkte pro Globalfeld

Erforderliche Speichereinheiten (GRIB Format)

Horizontalfeld (Zugriffseinheit): 400.1 kB (T213) / 100.1 kB (T106)

Unix Filegröße für monatsweise akkumulierte Ergebnisse mit 6 Std. Speicherintervall und 300 2d Variablen (Physikalische Einheit):

14000MB (T213) / 3500 MB (T106)

240 Jahre Modellintegration (Logische Einheit): 40 TB (T213) / 10 TB (T106)

Ausblick: Regionalmodell Auflösung und Datenmengen

REMO-UBA-Modellgebiet

Orography

•Auslösung: 10x10 km

•Datenmenge: 5 TB / 100 Jahre (nur Bodenfelder)

Vielen Dank!