Der Funke springt über: Apache Spark in einem Raspberry ... · Apache Spark top-level Projekt...

Der Funke springt über:

Apache Spark in einem Raspberry Cluster

Dozenten

Burkhard Hoppenstedtatr Software GmbH & Universität [email protected]

Nicolas KuhauptUniversität [email protected]

Gliederung

Spark

Architektur

Geschichte

Code Snippets

Cluster

Aufbau

Konfigurierung

Ergebnisse

Algorithmen-SkalierungClusterleistung

Spark Geschichte

MapReduce Forschung (Google)

MapReduce Paper

Hadoop sortiert 1.8TB in 47.9 h

Konferenz: Hadoop Summit

2010 Spark Paper

Apache Hadoop 1.0

Apache Spark top-level Projekt

Spark sortiert 100TB in 23 min, Hadoop in 72 min

2002

2004

2006

2008

2010

2012

2014

2014

Powered By

eBay Inc.

Spark

Amazon

IBM

Yahoo!

Dateiformate

Format Strukturiert

Textdateien Nein

JSON Halb

CSV Ja

SequenceFile Ja

Protocol buffers Ja

Object files Ja

Dateiformate

Format Strukturiert

Textdateien Nein

JSON Halb

CSV Ja

SequenceFile Ja

Protocol buffers Ja

Object files Ja

Sequence File Header

3 Byte (SEQ) + 1 Byte Version

Key Class Name

Value Class Name

isCompressed

IsBlockCompressed

Class Name

Meta Data

SyncMarker

Dateiformate

Format Strukturiert

Textdateien Nein

JSON Halb

CSV Ja

SequenceFile Ja

Protocol buffers Ja

Object files Ja

message Person {

required string name = 1;

required int32 id = 2;

optional string email = 3;

}

Person john = Person.newBuilder()

.setId(1234)

.setName("John Doe")

.setEmail("[email protected]")

.build();

output = new FileOutputStream(args[0]);

john.writeTo(output);

Person john;

fstream input(argv[1],

ios::in | ios::binary);

john.ParseFromIstream(&input);

id = john.id();

name = john.name();

email = john.email();

Dateiformate

Format Strukturiert

Textdateien Nein

JSON Halb

CSV Ja

SequenceFile Ja

Protocol buffers Ja

Object files Ja

Java Serialisierung

• Nicht standardisiert mit Hadoop-Output

• Langsam• Schnelle Verarbeitung

beliebiger Objekte

Word count

//Python Example

rdd = textFile(“…path…“)words = rdd.flatMap(lambda x: x.split(“ “))result = words.map(lambda x: (x,1)).reduceByKey(lambda x, y: x + y)

Amazon JSON

{'asin': '0078764343', 'description': 'Brand new sealed!', 'price': 37.98, 'imUrl': 'http://ecx.images-amazon.com/images/I/513h6dPbwLL._SY300_.jpg', 'related': {'also_bought': ['B000TI836G', 'B003Q53VZC', 'B00EFFW0HC', 'B003VWGBC0', 'B003O6G5TW', 'B0037LTTRO', 'B002I098JE', 'B008OQTS0U', 'B005EVEODY', 'B008B3AVNE', 'B000PE0HBS', 'B00354NAYG', 'B0050SYPV2', 'B00503E8S2', 'B0050SY77E', 'B0022TNO7S', 'B0056WJA30', 'B0023CBY4E', 'B002SRSQ72', 'B005EZ5GQY', 'B004XACA60', 'B00273Z9WM', 'B004HX1QFY', 'B002I0K50U'], 'bought_together': ['B002I098JE'], 'buy_after_viewing': ['B0050SY5BM', 'B000TI836G', 'B0037LTTRO', 'B002I098JE']}, 'salesRank': {'Video Games': 28655}, 'categories': [['Video Games', 'Xbox 360', 'Games']]}

FP GrowthCustomer Also bought

John Doe Banana, Unicorn, Sword

Mickey Mouse Unicorn, Steam Boat

Donald Trump Unicorn, Sword

John Cena Banana, Unicorn, Steam Boat

Stephen Hawking Banana, Sword

Nyan Cat Unicorn, Sword

Cpt. Jack Sparrow Banana, Sword

Steve Jobs Banana, Unicorn, Sword, Razor

Mr. Nobody Banana, Unicorn, Sword

Banana: 6 | Unicorn: 7 | Sword : 7 | Steam Boat: 2 | Razor: 1

FP Growth - resortCustomer Also bought

John Doe Unicorn, Sword, Banana



John Cena Unicorn, Banana, Steam Boat

Stephen Hawking Sword, Banana,


Cpt. Jack Sparrow Sword, Banana

Steve Jobs Unicorn, Sword, Banana, Razor

Mr. Nobody Unicorn, Sword, Banana

Banana: 6 | Unicorn: 7 | Sword : 7 | Steam Boat: 2 | Razor: 1

Customer Also bought





StephenHawking

Sword, Banana,


Cpt. JackSparrow

Sword, Banana



{ }






StephenHawking

Sword, Banana,


Cpt. JackSparrow

Sword, Banana



Unicorn:1

Banana:1

{ }

Sword:1






StephenHawking

Sword, Banana,


Cpt. JackSparrow

Sword, Banana



Unicorn:2

SteamBoat:1

Banana:1

{ }

Sword:1






StephenHawking

Sword, Banana,


Cpt. JackSparrow

Sword, Banana



Unicorn:3

SteamBoat:1

Banana:1

{ }

Sword:2






StephenHawking

Sword, Banana,


Cpt. JackSparrow

Sword, Banana



Unicorn:4

SteamBoat:1

Banana:1

{ }

Sword:2

Banana:1

SteamBoat:1






StephenHawking

Sword, Banana,


Cpt. JackSparrow

Sword, Banana



Unicorn:4

SteamBoat:1

Banana:1

{ }

Sword:2

Banana:1

Banana:1

Sword:1

SteamBoat:1






StephenHawking

Sword, Banana,


Cpt. JackSparrow

Sword, Banana



Unicorn:5

SteamBoat:1

Banana:1

{ }

Sword:3

Banana:1

Banana:1

Sword:1

SteamBoat:1






StephenHawking

Sword, Banana,


Cpt. JackSparrow

Sword, Banana



Unicorn:5

SteamBoat:1

Banana:1

{ }

Sword:3

Banana:1

Banana:2

Sword:2

SteamBoat:1






StephenHawking

Sword, Banana,


Cpt. JackSparrow

Sword, Banana



Unicorn:6

SteamBoat:1

Razor:1

{ }

Sword:4

Banana:1

Banana:2

Sword:2

Banana:2

SteamBoat:1






StephenHawking

Sword, Banana,


Cpt. JackSparrow

Sword, Banana



Unicorn:7

SteamBoat:1

Razor:1

{ }

Sword:5

Banana:1

Banana:2

Sword:2

Banana:3

SteamBoat:1






StephenHawking

Sword, Banana,


Cpt. JackSparrow

Sword, Banana



Frequent Patterns

Banana

Unicorn:7

SteamBoat:1

Razor:1

{ }

Sword:5

Banana:1

Banana:2

Sword:2

Banana:3

SteamBoat:1






StephenHawking

Sword, Banana,


Cpt. JackSparrow

Sword, Banana



Frequent Patterns

Banana {Unicorn, Sword, Banana: 3}

Unicorn:7

SteamBoat:1

Razor:1

{ }

Sword:5

Banana:1

Banana:2

Sword:2

Banana:3

SteamBoat:1






StephenHawking

Sword, Banana,


Cpt. JackSparrow

Sword, Banana



Frequent Patterns

Banana {Unicorn, Sword, Banana: 3} , {Unicorn, Banana:1}

Unicorn:7

SteamBoat:1

Razor:1

{ }

Sword:5

Banana:1

Banana:2

Sword:2

Banana:3

SteamBoat:1






StephenHawking

Sword, Banana,


Cpt. JackSparrow

Sword, Banana



Frequent Patterns

Banana {Unicorn, Sword, Banana: 3} , {Unicorn, Banana:1} , {Sword, Banana: 2}

Unicorn:7

SteamBoat:1

Razor:1

{ }

Sword:5

Banana:1

Banana:2

Sword:2

Banana:3

SteamBoat:1






StephenHawking

Sword, Banana,


Cpt. JackSparrow

Sword, Banana



Unicorn:7

SteamBoat:1

Razor:1

{ }

Sword:5

Banana:1

Banana:2

Sword:2

Banana:3

SteamBoat:1

Frequent Patterns

Banana {Unicorn, Sword, Banana: 3} , {Unicorn, Banana:1} , {Sword, Banana: 2}






StephenHawking

Sword, Banana,


Cpt. JackSparrow

Sword, Banana



Unicorn:7

SteamBoat:1

Razor:1

{ }

Sword:5

Banana:1

Banana:2

Sword:2

Banana:3

SteamBoat:1

Frequent Patterns

Banana {Unicorn, Sword, Banana: 3} , {Unicorn, Banana:1} , {Sword, Banana: 2}, {Unicorn, Banana: 4}






StephenHawking

Sword, Banana,


Cpt. JackSparrow

Sword, Banana



Unicorn:7

SteamBoat:1

Razor:1

{ }

Sword:5

Banana:1

Banana:2

Sword:2

Banana:3

SteamBoat:1

Frequent Patterns


Sword {Unicorn, Sword: 5}

Steam boat {Unicorn, Steam Boat: 1}, {Unicorn, Banana, Steam Boat: 1}, {Unicorn, Steam Boat : 2}

Razor {Unicorn, Sword, Banana, Razor:1}

Frequent Patterns

Banana {Unicorn, Sword, Banana: 3} {Sword, Banana: 2}, {Unicorn, Banana: 4}


Steam boat {Unicorn, Steam Boat : 2}

Razor






StephenHawking

Sword, Banana,


Cpt. JackSparrow

Sword, Banana



Frequent Patterns



Steam boat {Unicorn, Steam Boat: 1}, {Unicorn, Banana, Steam Boat: 1}, {Unicorn, Steam Boat : 2}

Razor {Unicorn, Sword, Banana, Razor:1}

min support:2

Unicorn:7

SteamBoat:1

Razor:1

{ }

Sword:5

Banana:1

Banana:2

Sword:2

Banana:3

SteamBoat:1

0

20000

40000

60000

80000

100000

120000

140000

160000

180000

200000

10

%30

%50

%70

%90

%

Scaling of FP Growth

0

10000

20000

30000

40000

50000

60000

70000

80000

90000

10%

40%

70%

100%

Scaling of KMeans

Customer Money Spend(€)

Time (h)

John Doe 2 4

Mickey Mouse 4 5

Donald Trump 10 10

John Cena 1 10

Stephen Hawking 0 2

Nyan Cat 2 3

Cpt. Jack Sparrow 2 6

Steve Jobs 5 8

Mr. Nobody 7 2

„John Cena and Donald Trump shop similar to Steve Jobs“

Quellen

• Sequence Files Header: http://hadooptutorial.info/hadoop-sequence-files-example/ [30.01.2017]

• Protocol Buffers: https://developers.google.com/protocol-buffers/ [30.01.2017]

• Raspberry Pi 3: https://www.raspberrypi.org/products/raspberry-pi-3-model-b/ [31.01.2017]

Der Funke springt über: Apache Spark in einem Raspberry ... · Apache Spark top-level Projekt...

Documents

Transcript of Der Funke springt über: Apache Spark in einem Raspberry ... · Apache Spark top-level Projekt...