Der Funke springt über: Apache Spark in einem Raspberry ... · Apache Spark top-level Projekt...

Der Funke springt über:

Apache Spark in einem Raspberry Cluster

Dozenten

Burkhard Hoppenstedtatr Software GmbH & Universität Ulmburkhard.hoppenstedt@uni-ulm.de

Nicolas KuhauptUniversität Ulmnicolas.kuhaupt@uni-ulm.de

Gliederung

Architektur

Geschichte

Code Snippets

Cluster

Aufbau

Konfigurierung

Ergebnisse

Algorithmen-SkalierungClusterleistung

Spark Geschichte

MapReduce Forschung (Google)

MapReduce Paper

Hadoop sortiert 1.8TB in 47.9 h

Konferenz: Hadoop Summit

2010 Spark Paper

Apache Hadoop 1.0

Apache Spark top-level Projekt

Spark sortiert 100TB in 23 min, Hadoop in 72 min

Powered By

eBay Inc.

Amazon

Yahoo!

Dateiformate

Format Strukturiert

Textdateien Nein

JSON Halb

CSV Ja

SequenceFile Ja

Protocol buffers Ja

Object files Ja

Dateiformate

Format Strukturiert

Textdateien Nein

JSON Halb

CSV Ja

SequenceFile Ja

Protocol buffers Ja

Object files Ja

Sequence File Header

3 Byte (SEQ) + 1 Byte Version

Key Class Name

Value Class Name

isCompressed

IsBlockCompressed

Class Name

Meta Data

SyncMarker

Dateiformate

Format Strukturiert

Textdateien Nein

JSON Halb

CSV Ja

SequenceFile Ja

Protocol buffers Ja

Object files Ja

message Person {

required string name = 1;

required int32 id = 2;

optional string email = 3;

Person john = Person.newBuilder()

.setId(1234)

.setName("John Doe")

.setEmail("jdoe@example.com")

.build();

output = new FileOutputStream(args[0]);

john.writeTo(output);

Person john;

fstream input(argv[1],

ios::in | ios::binary);

john.ParseFromIstream(&input);

id = john.id();

name = john.name();

email = john.email();

Dateiformate

Format Strukturiert

Textdateien Nein

JSON Halb

CSV Ja

SequenceFile Ja

Protocol buffers Ja

Object files Ja

Java Serialisierung

• Nicht standardisiert mit Hadoop-Output

• Langsam• Schnelle Verarbeitung

beliebiger Objekte

Word count

//Python Example

rdd = textFile(“…path…“)words = rdd.flatMap(lambda x: x.split(“ “))result = words.map(lambda x: (x,1)).reduceByKey(lambda x, y: x + y)

Amazon JSON

{'asin': '0078764343', 'description': 'Brand new sealed!', 'price': 37.98, 'imUrl': 'http://ecx.images-amazon.com/images/I/513h6dPbwLL._SY300_.jpg', 'related': {'also_bought': ['B000TI836G', 'B003Q53VZC', 'B00EFFW0HC', 'B003VWGBC0', 'B003O6G5TW', 'B0037LTTRO', 'B002I098JE', 'B008OQTS0U', 'B005EVEODY', 'B008B3AVNE', 'B000PE0HBS', 'B00354NAYG', 'B0050SYPV2', 'B00503E8S2', 'B0050SY77E', 'B0022TNO7S', 'B0056WJA30', 'B0023CBY4E', 'B002SRSQ72', 'B005EZ5GQY', 'B004XACA60', 'B00273Z9WM', 'B004HX1QFY', 'B002I0K50U'], 'bought_together': ['B002I098JE'], 'buy_after_viewing': ['B0050SY5BM', 'B000TI836G', 'B0037LTTRO', 'B002I098JE']}, 'salesRank': {'Video Games': 28655}, 'categories': [['Video Games', 'Xbox 360', 'Games']]}

FP GrowthCustomer Also bought

John Doe Banana, Unicorn, Sword

Mickey Mouse Unicorn, Steam Boat

Donald Trump Unicorn, Sword

John Cena Banana, Unicorn, Steam Boat

Stephen Hawking Banana, Sword

Nyan Cat Unicorn, Sword

Cpt. Jack Sparrow Banana, Sword

Steve Jobs Banana, Unicorn, Sword, Razor

Mr. Nobody Banana, Unicorn, Sword

Banana: 6 | Unicorn: 7 | Sword : 7 | Steam Boat: 2 | Razor: 1

FP Growth - resortCustomer Also bought

John Doe Unicorn, Sword, Banana

John Cena Unicorn, Banana, Steam Boat

Stephen Hawking Sword, Banana,

Cpt. Jack Sparrow Sword, Banana

Steve Jobs Unicorn, Sword, Banana, Razor

Mr. Nobody Unicorn, Sword, Banana

Banana: 6 | Unicorn: 7 | Sword : 7 | Steam Boat: 2 | Razor: 1

Customer Also bought

StephenHawking

Sword, Banana,

Cpt. JackSparrow

Sword, Banana

StephenHawking

Sword, Banana,

Cpt. JackSparrow

Sword, Banana

Unicorn:1

Banana:1

Sword:1

StephenHawking

Sword, Banana,

Cpt. JackSparrow

Sword, Banana

Unicorn:2

SteamBoat:1

Banana:1

Sword:1

StephenHawking

Sword, Banana,

Cpt. JackSparrow

Sword, Banana

Unicorn:3

SteamBoat:1

Banana:1

Sword:2

StephenHawking

Sword, Banana,

Cpt. JackSparrow

Sword, Banana

Unicorn:4

SteamBoat:1

Banana:1

Sword:2

Banana:1

SteamBoat:1

StephenHawking

Sword, Banana,

Cpt. JackSparrow

Sword, Banana

Unicorn:4

SteamBoat:1

Banana:1

Sword:2

Banana:1

Sword:1

SteamBoat:1

StephenHawking

Sword, Banana,

Cpt. JackSparrow

Sword, Banana

Unicorn:5

SteamBoat:1

Banana:1

Sword:3

Banana:1

Sword:1

SteamBoat:1

StephenHawking

Sword, Banana,

Cpt. JackSparrow

Sword, Banana

Unicorn:5

SteamBoat:1

Banana:1

Sword:3

Banana:1

Banana:2

Sword:2

SteamBoat:1

StephenHawking

Sword, Banana,

Cpt. JackSparrow

Sword, Banana

Unicorn:6

SteamBoat:1

Razor:1

Sword:4

Banana:1

Banana:2

Sword:2

Banana:2

SteamBoat:1

StephenHawking

Sword, Banana,

Cpt. JackSparrow

Sword, Banana

Unicorn:7

SteamBoat:1

Razor:1

Sword:5

Banana:1

Banana:2

Sword:2

Banana:3

SteamBoat:1

StephenHawking

Sword, Banana,

Cpt. JackSparrow

Sword, Banana

Unicorn:7

SteamBoat:1

Razor:1

Sword:5

Banana:1

Banana:2

Sword:2

Banana:3

SteamBoat:1

StephenHawking

Sword, Banana,

Cpt. JackSparrow

Sword, Banana

Frequent Patterns

Banana

Unicorn:7

SteamBoat:1

Razor:1

Sword:5

Banana:1

Banana:2

Sword:2

Banana:3

SteamBoat:1

StephenHawking

Sword, Banana,

Cpt. JackSparrow

Sword, Banana

Frequent Patterns

Banana {Unicorn, Sword, Banana: 3}

Unicorn:7

SteamBoat:1

Razor:1

Sword:5

Banana:1

Banana:2

Sword:2

Banana:3

SteamBoat:1

StephenHawking

Sword, Banana,

Cpt. JackSparrow

Sword, Banana

Frequent Patterns

Banana {Unicorn, Sword, Banana: 3} , {Unicorn, Banana:1}

Unicorn:7

SteamBoat:1

Razor:1

Sword:5

Banana:1

Banana:2

Sword:2

Banana:3

SteamBoat:1

StephenHawking

Sword, Banana,

Cpt. JackSparrow

Sword, Banana

Frequent Patterns

Banana {Unicorn, Sword, Banana: 3} , {Unicorn, Banana:1} , {Sword, Banana: 2}

Unicorn:7

SteamBoat:1

Razor:1

Sword:5

Banana:1

Banana:2

Sword:2

Banana:3

SteamBoat:1

StephenHawking

Sword, Banana,

Cpt. JackSparrow

Sword, Banana

Unicorn:7

SteamBoat:1

Razor:1

Sword:5

Banana:1

Banana:2

Sword:2

Banana:3

SteamBoat:1

Frequent Patterns

Banana {Unicorn, Sword, Banana: 3} , {Unicorn, Banana:1} , {Sword, Banana: 2}

StephenHawking

Sword, Banana,

Cpt. JackSparrow

Sword, Banana

Unicorn:7

SteamBoat:1

Razor:1

Sword:5

Banana:1

Banana:2

Sword:2

Banana:3

SteamBoat:1

Frequent Patterns

Banana {Unicorn, Sword, Banana: 3} , {Unicorn, Banana:1} , {Sword, Banana: 2}, {Unicorn, Banana: 4}

StephenHawking

Sword, Banana,

Cpt. JackSparrow

Sword, Banana

Unicorn:7

SteamBoat:1

Razor:1

Sword:5

Banana:1

Banana:2

Sword:2

Banana:3

SteamBoat:1

Frequent Patterns

Sword {Unicorn, Sword: 5}

Steam boat {Unicorn, Steam Boat: 1}, {Unicorn, Banana, Steam Boat: 1}, {Unicorn, Steam Boat : 2}

Razor {Unicorn, Sword, Banana, Razor:1}

Frequent Patterns

Banana {Unicorn, Sword, Banana: 3} {Sword, Banana: 2}, {Unicorn, Banana: 4}

Steam boat {Unicorn, Steam Boat : 2}

StephenHawking

Sword, Banana,

Cpt. JackSparrow

Sword, Banana

Frequent Patterns

Steam boat {Unicorn, Steam Boat: 1}, {Unicorn, Banana, Steam Boat: 1}, {Unicorn, Steam Boat : 2}

Razor {Unicorn, Sword, Banana, Razor:1}

min support:2

Unicorn:7

SteamBoat:1

Razor:1

Sword:5

Banana:1

Banana:2

Sword:2

Banana:3

SteamBoat:1

100000

120000

140000

160000

180000

200000

Scaling of FP Growth

Scaling of KMeans

Customer Money Spend(€)

Time (h)

John Doe 2 4

Mickey Mouse 4 5

Donald Trump 10 10

John Cena 1 10

Stephen Hawking 0 2

Nyan Cat 2 3

Cpt. Jack Sparrow 2 6

Steve Jobs 5 8

Mr. Nobody 7 2

„John Cena and Donald Trump shop similar to Steve Jobs“

Quellen

• Sequence Files Header: http://hadooptutorial.info/hadoop-sequence-files-example/ [30.01.2017]

• Protocol Buffers: https://developers.google.com/protocol-buffers/ [30.01.2017]

• Raspberry Pi 3: https://www.raspberrypi.org/products/raspberry-pi-3-model-b/ [31.01.2017]

Der Funke springt über: Apache Spark in einem Raspberry ... · Apache Spark top-level Projekt...

Documents

Transcript of Der Funke springt über: Apache Spark in einem Raspberry ... · Apache Spark top-level Projekt...

Projekt Big Data: Semantische Suche mit Apache Solr...Projekt Big Data: Semantische Suche mit Apache Solr Eike Nils Knopp , Minh Hieu Nguyen 29. März 2018 Universität Hamburg - MIN

140801 thumbs one spark 2014

Apache camel

Kreativität macht den Unterschied. - Adobe Inc....Adobe Spark Einstieg mit Spark. Adobe Spark bietet einfache, aber leistungsfähige Tools für Präsentation und Storytelling. Damit

OpenSocial und Apache Shindig

SPARK – DIE KLASSISCHE BAND FRANZISKA DANNHEIM …

Apache Solr Revisited 2015

Apache solr

Karlsruhe, 29.09.2015 Sébastien Jelsch - inovex GmbH · Apache Kylin + Mondrian: Die Idee 26 Saiku Apache Kylin HBase, ... OLAP Cubes on Hadoop Sébastien Jelsch Apache Kylin: Roadmap

BMW Spark Ingition Efficiency Technology

FID Germanistik – Bedarfsermittlung...2019/05/17 · 20. Mai 2019 20 Portal & Admin Interface Apache Solr für TYPO3 TYPO3 Core Apache HTTP Server MySQL Apache Solr OAI Schnittstellen

Spark Architecture · Spark Architecture Spark Shuffle ... Spark Shuffle Spark DataFrames . ... – Entry point of the Spark Shell (Scala, Python, R) – The place where SparkContext

Apache Spark: Untersuchung der Möglichkeiten zur ...Spark+Masterthesis... · Apache Spark: Untersuchung der Möglichkeiten zur verteilten Datenverarbeitung und Analyse von Streaming

Apache Trinidad

Apache ServiceMix 4.2.0

Apache Spark und HBase zur Speicherung und Analyse von ... · TDWI ROUNDTABLE RALTUNGEN tdwi.eu Freiburg. anuar Im Vortrag wurde neben der allgemeinen Arbeitsweise von HBase auch

Masterarbeit - uni-hamburg.de · Masterarbeit Ein Verfahren zur Erzeugung großer Test-Datasets für die Duplikaterkennung unter Verwendung von Apache Spark Kai Hildebrandt 6hildebr@informatik.uni-hamburg.de

AH-64 D Longbow Apache AH-64 D Longbow Apache · 2020. 8. 31. · AH-64D Longbow Apache 04046-0389 2005 BY REVELL GmbH & CO. KG PRINTED IN GERMANY AH-64 D Longbow Apache AH-64 D Longbow

Auf der Big Data - download.e-bookshelf.de · NoSQL-Datenbanken: Apache Cassandra, Apache CouchDB, MongoDB, Neo4J, Redis, Riak NewSQL-Datenbanken: Apache Phoenix, Apache Tajo, Kylin,

MapReduce & Apache Hadoop