SDIC'16 - Betrieb des Smart Data Innovation Labs - Vorstellung der Plattform

KIT – Universität des Landes Baden-Württem ber g undnationales Forschungszentrum in der Helmholtz-Gemeinschaft

Steinbuch Centre for Computing

www.kit.edu

Betrieb des Smart Data Innovation LabsVorstellung der Plattform

Nico Schlitter, 13. Oktober 2016

2 13.10.2016 Betrieb des Smart Data Innovation Labs (SDIL) Steinbuch Centre for ComputingSteinbuch Centre for Computing

Partners


SDI-X funded by BMBF 09/2015 – 08/2018to strengthen the cooperation between industry and science

self organizing data innovation communities industry research partner meet at community meetings and define projectsSDIL strategy board approves project proposals within two weeksproject members get access to the SDIL platform hosted by KIT

Smart Data Innovation Lab

MedicineSmart Cities

EnergyIndustry 4.0


Running the SDIL platform at KITDevelopment of tools, best practices und operating conceptsDeveloping a legal frameworkResearch in the field of

data processing infrastructuredata integrationdata analyticsdata curationdata anonymization and pseudonymization

Tasks within SDIL


funded by the state of Baden-Württemberg 10/2014 – 09/2017educating SME in regard to data analytics (~30 contacts to far)free of charge

consulting for local SME in the field of data analyticsevaluating the potential of existing dataperforming data analysisrecommending additional data collectioninvestigating further analysis steps

Smart Data Solution Center BW

• First contact• Checking requirements• Defining Objectives• Legal Issues

• Data hand over• Data analysis• Evaluation

• Presenting results• Recommendations• Next steps

Preparation Realization Finalization


SDIL Platform Overview


#Cores RAM [GB]

Disk Space [TB]

Network Software

IBM Watson FoundationPower 8

7 x 20= 140

4096 300 40Gbit/sEthernet

IBM Open Platform with Hadoop/SparkSPSS Modeler

SPSS Analytic ServerDB2 with BLU Acceleration

SAP HANA 4 x 80= 320

4096 80 10Gbit/s Ethernet

SAP HANAPredictive Analysis LibraryBusiness Function Library

Software AGTerracotta

( * on request * ) BigMemory Max

HTCondor 32 x 4= 128

1024 1Gbit/s Ethernet

RapidMiner, Python, R, Matlab,

Virtualization 3 x 12= 36

576 6 10Gbit/sEthernet

Red Hat Enterprise Virtualizationon GlusterFS

SDIL Platform Resources


security is important due to storing of sensitive datastrict access control of the server roommultiple state-full firewalls are in placeaccess via dedicated login serverfine granular permission management for all SDIL components

unified storage solution enables users to use several different technologies w/o the need to migrate large volumes of datahome and project directories with snapshotsencrypted backup on tape for disaster recoveryticket system (xGUS) for user support

SDIL Operations – user perspective


legal issues are much bigger than expected

heterogeneous infrastructuredifferent architectures and operating systemsspecial software stacks like HANA and Terracottanot everything is nicely packaged

data privacy protection

SDIL Platform Challenges


3 networks: BMC, deployment/configuration, servicesTwo firewalls, strict access control, encryption IBM Spectrum Scale as storage solutionForeman and Puppet for deployment and configuration of SDIL machines (continuous integration)GitLab as code repository

Icinga for monitoring (integrated with puppet)Redhat Enterprise Virtualization (RHEV) on glusterFS

Ticket system for all operation tasksWeekly shifts for incident response

SDIL Operations – nerd perspective


IBM Watson Foundations on Power 8


SAP HANA & RHEV


Continuously upgrading …

SDIC'16 - Betrieb des Smart Data Innovation Labs - Vorstellung der Plattform

Data & Analytics

Transcript of SDIC'16 - Betrieb des Smart Data Innovation Labs - Vorstellung der Plattform