Was ist Object Storage und wer braucht’s? Eine Übersicht und die Lösung von...

YOUR INFORMATION IS OUR BUSINESS

Dr. Wolfgang Obelöer Technischer Leiter itiso GmbH

Was ist Object Storage und wer braucht’s? Eine Übersicht und die Lösung von Huawei

DLR Storagetechnology 2016 15.-17.06.2016

„Radisson Blu Hotel“ in Rostock


Agenda • Über itiso • Was ist Object Storage? • Beispiel: Huawei OceanStor 9000 • Einsatzfelder • Zusammenfassung


Über itiso • Herstellerunabhängiges Systemhaus

• Unser Thema: Datenmanagement

• Gegründet Januar 2005: Hamburg (Zentrale), Hannover, Frankfurt

• Portfolio:


Was ist Object Storage


Object Storage • Objekte

– sind wie Files in einem simplen File-System – werden dynamisch erstellt und gelöscht – haben eine variable Länge – beinhalten erweiterbare Attribute – werden durch eine ID angesprochen

• Der Object Storage managt alle Operationen auf den Objekten – für eine 4TB Platte müssen z.B. 1 Milliarde 4KB-Blöcke

verwaltet werden – dies wird durch den Object Storage komplett verborgen


Object Storage vs. Block vs. File De facto-Standard: S3


Erasure Coding • - Basiert auf Vandermonde-Reed-Solomon error correction • - Anwendbar für Archive, Backup und Multimedia


Erasure Coding (einfaches Beispiel)

Coding

Value Split CodeA1=1A2=2A3=3A4=5

X1=1

X2=2A=12

Example

MirrorA1=1A2=1A3=2A4=2


Vorteile Object Storage • Skalierbarkeit (Verteilter Ansatz möglich) • Schneller Zugriff (Unabhängige Verarbeitung) • Hohe Datensicherheit (Erasure Coding) • S3 als Schnittstelle (Standard in der Cloud) • Niedrige Kosten durch Einsatz von Standard

X86-Hardware • Rechte etc. in Metadaten im Objekt (Selbst-

verwaltung)


Beispiel: Huawei OceanStor 9000


Huawei OceanStor 9000

40 PB

Single file system

740 m2 of equipment room

288 OceanStor 9000 C nodes

Large-capacity cluster verification lab (Chengdu Research Center)


Huawei Storage Portfolio

S2200T/S2600T 18500/18800

Massive Storage

DeviceManager

(Device Management)

eBackup

(Virtualization Backup) (Unified Management)

Stor

age

Solu

tions

So

ftwar

e Pr

oduc

ts

Har

dwar

e Pr

oduc

ts

All-flash array 2 controllers 500 µs latency

Smart Series Hyper Series Info Series

Dorado2100 G2 Dorado5100

(Data Protection Software) (Intelligent Resource Management Software) (Big Data Storage Value-Added Software )

InfoTier InfoEqualizer InfoAllocator InfoExplorer InfoProtector

SmartTier SmartMotion SmartQoS SmartThin SmartPartition SmartVirtualization

HyperCopy HyperSnap HyperClone HyperReplication

2-16 controllers 3 TB cache 3216 disks SPC-1: 1,005,893 IOPS

Unified Storage

Media HPC Analysis Archiving DR & Backup Active-Active File Sharing Virtualization Database Acceleration

Converged Storage Solutions Big Data Storage Solutions Flash Acceleration Solution

2 controllers 16 GB cache 276 disks

eSight

Video Surveillance

5300 V3/5500 V3 6800 V3 5600 V3/5800 V3

2-8 controllers 512 GB cache 750 disks

2-8 controllers 1 TB cache 1250 disks

2-8 controllers 4TB cache 3200 disks

Storage Integration

3-288 nodes 100 PB capacity Mutiple protocol

9000


OceanStor 9000 - Higlights

•Green design hardware

•4.3PB in single rack

Low TCO

•Erasure code distributed data protection

•E2E integration Check

High Reliability

•Convergence of file & object storage

•Multiple Interfaces

Openness & Convergence

OceanStor 9000 Object Storage System •Full distributed architecture

•100 PB single namespace

Flexible Scalability


Original Object Storage

Object Storage

File System

Disk Array

Object Gateway

OceanStor 9000

Original object storage: object does

not need to map to file system,

shorter access delay

Concurrent access, compared to

the object gateway and disk array

architecture, full redundancy, no

single point of failure

Scale-out，capacity and

performance increase linearly with

the increase of nodes

OceanStor 9000： Number of Objects: 10 billion Number of buckets: 100 million

### Number of Objects: 100 million Number of buckets: 4 million

Object Storage


OceanStor 9000: Architektur

Node Node Node Node Node Node Node Node Node Node

Hardware Node Layer

Node Node Node Node Node Node Node Node Node Node

Application layer

Data processing

layer

HPC Video surveillance

Media and entertainment

Object storage

Backup and archiving software

Backup and archiving

Web disk

Data analysis

Precision marketing

Connections to Hadoop

Commercial promotion

FTP NFS CIFS HDFS NDMP

Distributed Storage Layer

Object Storage NAS Storage

Swift/S3

Managem

ent

File storage

Interconnect: 10 GE or Infiniband

https://www.google.com.hk/url?q=http://commons.wikimedia.org/wiki/File:Adobe_PDF_Icon.svg&sa=U&ei=XoAkU9jjN8TwiAeKtoEY&ved=0CCwQ9QEwAA&usg=AFQjCNGIRml6-609bXmaKOJSEQVNmKz6nw

https://www.google.com.hk/url?q=http://www.iconpng.com/tag/%E5%89%AA%E8%BE%91&sa=U&ei=t4EkU6LZI4r7iQfuloCYCA&ved=0CC4Q9QEwAA&usg=AFQjCNFyUY4LL5YAzx8iaS5jHSiWT1kGyQ


Flexible Skalierbarkeit


Zuverlässige Datenabsicherung • N+M redundancy, support N+1 to N+3 redundancy,

maximum 3*36 disks failure - business is not interrupted

• Hot spare space replaces hot spare disks. • Data reconstruction replaces disk reconstruction. • Data recovery speed reaches 1 TB/h, 10 times

higher than traditional mode

Traditional RAID5

Traditional RAID6

Distributed Erasure Code

Max. allowed number of faulty disks 1 2 108

Need hot spare disk or not yes yes no

1 TB data recovery time 9.5 hours 10 hours 1 hour

+1 +2 +3


Data „Aerobics“

Add Checksum

Write diagram Additional Process

System

OBS

local

Data EC

Write to Disk

Rebuild data

Adding checksum to enable detection of corrupted data

Detect errors based on health state of HDD, when reading data or during silent check and process automated error recovery


Object Deduplizierung

Object level (Object with the same fingerprint would be deduplicated )

Post proceed, calculate fingerprint when upload, backend task compare the fingerprint of all object and delete duplicate data

Global deduplication, Duplicate data comparison is based on all the objects of the entire system, not limited to a single user or a single barrel

Flexible settings, deduplication configuration base on user or bucket, bucket will inherit the policy of it’s user

OceanStor 9000

Account (Dedup enable） Account (Dedup disable）

Bucket


Performance-Optimierung Cache-Aware Algorithm

Optimized Protocol

File File

Location of current read

Location of next read

Cache miss causes dropped frames and erratic display

Cache hit, smooth video editing and streaming, rapid response

Legend:

Read data Prefetch data

……

One client, one link, one server

One client, multi-link, multiple-server layout breaks through the bottlenecks in the network links and issues with single-server deployments

Mac 10.10 or Redhat 6.3 private client

Location of current read

Location of next read

Single client performance up to 2.5GB/s

Global Cache

SSD-Layer with for caching and autotiering


Geringe TCO

High-density and energy-saving hardware

Intelligent software and multi-level power conservation

Energy-Saving ATOM CPUs

ATOM Others

50%

Power consumption/processing unit

• 4 U 72-slot enclosure for 3.5-inch hot swappable disks •Single cabinet capable of providing a maximum of 4.3 PB capacity

High-Density nodes

Node control

Normal

Intelligent CPU frequency control

Intelligent fan speed control

Disk start delay

100%

70%

35%

24% 27%

Flow

Hea

vy

Low

Power consumption/node

Intelligent Energy-Saving Design High Efficiency PSU/FAN

•Gold Power PSU: The power

conversion efficiency over 90% •16 level intelligent self-adjusted fan

• Node expansion in 1 min


Huawei OceanStor 9000

OceanStor 9000

Performance

Flexible Scalability

Open Convergence

• 400 GB/s system throughput, exclusive InfoTurbo acceleration technology achieves up to 2.5 GB/s in bandwidth over a single client.

• Fully symmetric and distributed architecture, accommodation of up to 288 nodes, 100 PB storage space

• All-IP network topology to reduce complexity, and supporting various data types, such as file and object


Einsatzfelder


Anwendungs-Szenarien

Data Components

Video

Music

Picture

Data file

Email

Seldom updated

Undefined value

Large capacity and high growth speed

Long storage time

25%

75% (unstructured data)


Object Storage und HSM (Versity) Right (High)

Performance

High Data Protection

Continuous Data Health Check

Flexible Data Placement

HSM (Versity VSM)


Zusammenfassung • Object Storage ist sinnvoll einsetzbar für viele

Anwendungsgebiete • Sichere und kostengünstige

Datenspeicherung • Huawei OceanStor 9000 bietet

– Flexible Skalierung – Sehr umfangreiche Softwarefeatures – Alle wichtigen Protokolle



itiso GmbH Flughafenstraße 52a 22335 Hamburg http://www.itiso.de

Ihr Ansprechpartner Dr. Wolfgang Obelöer

[email protected]

Was ist Object Storage und wer braucht’s? Eine Übersicht und die Lösung von...

Documents

Transcript of Was ist Object Storage und wer braucht’s? Eine Übersicht und die Lösung von...