Server Hardware monitoring done right - NETWAYS · 4 Nach diesem Vortrag überwachen Sie sicherer...

Server Hardware Monitoringdone right!Werner Fischer, Thomas-Krenn.AG

2

Status quo

_ Überwachen Sie Ihre Server Hardware?

Ja Nein

3

Nach diesem Vortragüberwachen Sie sicherer

und umfangreicher

4

Nach diesem Vortragüberwachen Sie sicherer

und umfangreicher(hoffe ich zumindest... ;-)

5

Status quo

_ Welche Technologien nutzen Sie?

IPMI / SNMP NRPE CAM CAT

6

CAMera

7

CATinspection → satification?

8

Agenda

_ IPMI (20')

_ RAM (5')

_ RAID (10')

_ SMART (5')

_ GPU (5')

9

monitor yourIPMI-Sensors!

12

Intelligent Platform Management Interface

13

Monitoring(temp, fans, ...)

Recovery Control(on/off/reset)

Logging(System Event Log)

Inventory(FRU information)

Funktionen

14

Chassis board

Motherboard

Processorboard

Memoryboard

BaseboardManagement

Controller(BMC)

System bus

NVS StorageSDRSELFRU

Chassismgmt.

(SatelliteController)

Sensors & ControlsFan sensor

Temp. sensorPower controlReset control

…

FRU

Temp. s.

FRU

private mgmt. busses

IPMB

M/BSerial

Controller

SerialPort

Sharing

BMCSerial

Controller

Serial/Modeminterface

LANinterface

SerialConnector

LANConnector

PCI mgmt. bus

Network(LAN)

Controller

Remote Mmgt. Card(KVM over IP, ...)

AuxillaryIPMB Connector

ICMB

ICMBbridge

System interface

Redundant Powerboard

FRU Temp.sensor

…

FRU

Zugriff mitroot Rechten

Zugriff mitBenutzername

& Passwort

Aufbau

15

Discrete (True/False) Threshold (Schwellwerte)

Mehrere Zustände möglich:● bis zu 15 Status möglich● jeder Status = 1 Bit● mehrere aktive Statusbits möglich

Zustand abhängig von:● Vergleich analoger Messert mit dem

Schwellwerten (Thresholds)

Liefert:● allgemeine Zustände● Sensor-spezifische Zustände

Liefert:● analogen Messwert● diskreten Status

Ähnliche Klasse OEM● Bedeutung der Zustände werden

vom OEM definiert

IPMI Sensor Klassen

16

IPMI Sensor KlassenDiscrete Threshold

[root@test ~]# ipmitool sdr get "PS2 Status"Sensor ID : PS2 Status (0x71) Entity ID : 10.2 (Power Supply) Sensor Type (Discrete): Power Supply States Asserted : Power Supply [Presence detected] [Power Supply AC lost] Assertion Events : Power Supply [Presence detected] [Power Supply AC lost] Assertions Enabled : Power Supply [Presence detected] [Failure detected] [Predictive failure] [Power Supply AC lost][...] Deassertions Enabled : Power Supply[...]

[root@test ~]# ipmitool sdr get "Fan 1"Sensor ID : Fan 1 (0x50) Entity ID : 29.1 (Fan Device) Sensor Type (Analog) : Fan Sensor Reading : 5719 (+/ 0) RPM Status : ok Nominal Reading : 6708.000 Normal Minimum : 2451.000 Normal Maximum : 10965.000 Lower critical : 1720.000 Lower noncritical : 1978.000 Positive Hysteresis : 86.000 Negative Hysteresis : 86.000 Minimum sensor range : Unspecified Maximum sensor range : Unspecified Event Message Control : Perthreshold Readable Thresholds : lcr lnc Settable Thresholds : lcr lnc Threshold Read Mask : lcr lnc Assertion Events : Assertions Enabled : lnc lcr Deassertions Enabled : lnc lcr

17

$ sudo ipmisensors outputsensorstate interpretoemdataPassword: ID | Name | Type | State | Reading | Units | Event4 | System Temp | Temperature | Nominal | 27.00 | C | 'OK'71 | Peripheral Temp | Temperature | Nominal | 35.00 | C | 'OK'138 | CPU Temp | OEM Reserved | Nominal | N/A | N/A | 'Low'205 | FAN 1 | Fan | Nominal | 1800.00 | RPM | 'OK'…942 | VBAT | Voltage | Nominal | 3.15 | V | 'OK'1009 | VSB | Voltage | Nominal | 3.34 | V | 'OK'1076 | AVCC | Voltage | Nominal | 3.38 | V | 'OK'1143 | Chassis Intru | Physical Security | Critical | N/A | N/A | 'Gen...'

IPMI SensorenOK

Critical

18

$ cat /etc/freeipmi/freeipmi_interpret_sensor.conf[…]## IPMI_Physical_Security ## IPMI_Physical_Security_No_Event Nominal# IPMI_Physical_Security_General_Chassis_Intrusion Critical# IPMI_Physical_Security_Drive_Bay_Intrusion Critical[…]# IPMI_Power_Supply_No_Event Nominal# IPMI_Power_Supply_Presence_Detected Nominal# IPMI_Power_Supply_Power_Supply_Failure_Detected Critical# IPMI_Power_Supply_Predictive_Failure Critical# IPMI_Power_Supply_Power_Supply_Input_Lost_AC_DC Critical[…]

IPMI Sensoren (Discrete)

19

$ ./check_ipmi_sensor H 192.168.255.5 f ipmi.cfg vvIPMI Status: OK | 'System Temp'=27.00 'Peripheral Temp'=35.00 'FAN 1'=1800.00 'Vcore'=0.98 '3.3VCC'=3.36 '12V'=11.93 'VDIMM'=1.53 '5VCC'=5.09 '12V'=12.09 'VBAT'=3.15 'VSB'=3.34 'AVCC'=3.38System Temp = 27.00 (Status: Nominal)Peripheral Temp = 35.00 (Status: Nominal)CPU Temp = 'Low' (Status: Nominal)FAN 1 = 1800.00 (Status: Nominal)Vcore = 0.98 (Status: Nominal)3.3VCC = 3.36 (Status: Nominal)12V = 11.93 (Status: Nominal)VDIMM = 1.53 (Status: Nominal)5VCC = 5.09 (Status: Nominal)12V = 12.09 (Status: Nominal)VBAT = 3.15 (Status: Nominal)VSB = 3.34 (Status: Nominal)AVCC = 3.38 (Status: Nominal)Chassis Intru = 'OK' (Status: Nominal)

IPMI Plugin

20

#!/usr/bin/perl# check_ipmi_sensor: Nagios/Icinga plugin to check IPMI sensors## Copyright (C) 20092014 ThomasKrenn.AG,# additional contributors see changelog.txt## This program is free software; you can redistribute it and/or modify it under[…]Version 3.5 20141031 * Fix LAN Driver if called on localhost

Version 3.4 20140929 * Fix implicit array warning with split * Add option to disable LAN protocol version 2.0

Version 3.3 20140606 * Print a warning if ipmisensors only returned a single output row * Ignore sudo errors and warnings in IPMI command output (Thanks to Robert Heinzmann for contributing) * Use LAN protocol version 2.0 per default * Print empty output error only if return code was 0 * Exit the plugin with return code 3 if fru command fails * Added an include list option to only include specific sensors

Version 3.2 20131028 * Added FRU serial number to output

IPMI Plugin

21

so weit so gut?

22

Intelligent? Platform Management Interface

24

Das Abhörsystem in ihrem Computer

The Eavesdropping System in Your Computer(Bruce Schneier, Schneier on Security Blog 31.01.2013)

https://www.schneier.com/blog/archives/2013/01/the_eavesdroppi.html

27

230.000 1HE Server→ 10.223,5 m Höhe

(Mount Everest 8.848 m)

29

IPMI Firmware by ATEN / AMI

_ Mainboard-Herstellerpassen Firmware an

_ OS = Embedded Linux

_ IPMI Firmware Teile Closed-Source

30

Wir empfehlen administrative Zugänge wie IPMI- aber auch etwa SSH-Dienste nicht offen im Internet zu betreiben,

sondern mittels Firewall/VPN den Zugriff auf solche Dienste

ausschließlich berechtigten Personen zu ermöglichen.

31

Was wenn doch?

Enable

&DROP

32

IPMI Top 3 Sicherheitstipps

33

#1 - Netzwerk

34

#1 - Netzwerk

35

#2 – User Management

sjfaiklaz afjhuijoh

Administrator

User

36


msf > use auxiliary/scanner/ipmi/ipmi_dumphashes msf auxiliary(ipmi_dumphashes) > set RHOSTS 10.1.102.141RHOSTS => 10.1.102.141msf auxiliary(ipmi_dumphashes) > set THREADS 128THREADS => 128msf auxiliary(ipmi_dumphashes) > run

[+] 10.1.102.141:623 - IPMI - Hash found: admin:14667523250000004ec525d3852f4fa73c93b674788217fe0000000000000000000000000000000000000000000000000000000000000000140561646d696e:2c76e372d89ac7cd4e3bfecb423962f708d0741c

In short, the authentication process for IPMI 2.0 mandates that the server send a salted SHA1 or MD5 hash of the requested user's password to the client, prior to the client authenticating.

A Penetration Tester's Guide to IPMI and BMCs (rapid7.com)

https://community.rapid7.com/community/metasploit/blog/2013/07/02/a-penetration-testers-guide-to-ipmi

37


$ ./cudaHashcat64.bin --outfile=ipmi.out -m 7300 hash.txt -a 3 ?lu?lu?lu?lu?lu?lu[...]Session.Name...: cudaHashcatStatus.........: ExhaustedInput.Mode.....: Mask (?lu?lu?lu?lu?lu?lu) [12]Hash.Target....: 54414378fb2db5ff365e4bc5856adaf4c1b8a2f2153efd1b81fb54dfe1bf56478788ea7ba154375b40167e34f026e1020010d21d1ea31625040561646d696e:0a0b160231e204a6d0bd086e26718002409b35b7Hash.Type......: IPMI2 RAKP HMAC-SHA1Time.Started...: Thu Sep 18 10:11:17 2014 (6 secs)Time.Estimated.: 0 secsSpeed.GPU.#1...: 52732.3 kH/sRecovered......: 0/1 (0.00%) Digests, 0/1 (0.00%) SaltsProgress.......: 308915776/308915776 (100.00%)Skipped........: 0/308915776 (0.00%)Rejected.......: 0/308915776 (0.00%)HWMon.GPU.#1...: -1% Util, 41c Temp, 31% Fan

38


20Komplexe

& langePasswörter

39

#3 – Dienste limitieren

42

monitor your RAM!(it's ECC, isn't it?)

44

3%min 1 CE/Jahr (DDR2)Google 2009, Jaguar-Cluster 2012

45

70%CE's vor UE's

Google 2009

46

1,3%Server mit UE's/Jahr

Google 2009

47

root@debiantest:/sys/devices/system/edac/mc/mc0/csrow0# ls ltotal 0rrr 1 root root 4096 Nov 12 09:02 ce_countrrr 1 root root 4096 Nov 12 09:02 ch0_ce_countrwrr 1 root root 4096 Nov 12 09:02 ch0_dimm_labelrrr 1 root root 4096 Nov 12 09:02 ch1_ce_countrwrr 1 root root 4096 Nov 12 09:02 ch1_dimm_labelrrr 1 root root 4096 Nov 12 09:02 dev_typerrr 1 root root 4096 Nov 12 09:02 edac_moderrr 1 root root 4096 Nov 12 09:02 mem_typedrwxrxrx 2 root root 0 Nov 12 09:02 powerrrr 1 root root 4096 Nov 12 09:02 size_mblrwxrwxrwx 1 root root 0 Nov 12 09:02 subsystem > ../../../../../../bus/mc0rrr 1 root root 4096 Nov 12 09:02 ue_countrwrr 1 root root 4096 Nov 12 09:02 ueventroot@debiantest:/sys/devices/system/edac/mc/mc0/csrow0# cat ce_count 0root@debiantest:/sys/devices/system/edac/mc/mc0/csrow0# cat ue_count 0

Linux EDAC

48

Linux EDAC SupportmatrixTreibermodul CPUs Kernel Unterstützte Architekturen

amd64_edac.c AMD 2.6.312.6.393.103.133.15

K8 und F10F15F16F15_M30HF16_M30H

i7core_edac.c Intel Single/Dual 2.6.35 Nehalem/Westmere

ie31200_edac.c Intel Single-CPU 3.17 Sandy & Ivy BridgeHaswell

sb_edac.c Intel Dual-CPU 3.23.133.17

Sandy BridgeIvy BridgeHaswell

51

monitor your RAID!

53

LinuxSoftware

RAID

LSI / AdaptecHardware

RAID

54

Avago MegaRAID (LSI)

55

root@debiantest:~# storcli64 Storage Command Line Tool Ver 1.13.06 Sep 03, 2014

(c)Copyright 2014, LSI Corporation, All Rights Reserved.

help lists all the commands with their usage. E.g. storcli help<command> help gives details about a particular command. E.g. storcli add help

List of commands:

Commands Descriptionadd Adds/creates a new element to controller like VD,Spare..etcdelete Deletes an element like VD,Spareshow Displays information about an elementset Set a particular value to a property get Get a particular value to a property compare Compares particular value to a propertystart Start background operationstop Stop background operationpause Pause background operationresume Resume background operationdownload Downloads file to given deviceexpand expands size of given driveinsert inserts new drive for missingtransform downgrades the controller/cx Controller specific commands/ex Enclosure specific commands/sx Slot/PD specific commands/vx Virtual drive specific commands/dx Disk group specific commands/fall Foreign configuration specific commands/px Phy specific commands/[bbu|cv] Battery Backup Unit, Cachevault commands

56

$ /usr/lib/nagios/plugins/check_lsi_raid vvWarning (LD Warn) [c0/v0_Consist = Warning (No)]|CV_Temperature=22;70;85 ROC_Temperature=57;80;90 c0/e252/s0_Drive_Temperature=21;40;45 c0/e252/s1_Drive_Temperature=21;40;45Used storcli commands: /usr/bin/sudo /usr/sbin/storcli64 /c0 /cv show status /usr/bin/sudo /usr/sbin/storcli64 adpallinfo a0 /usr/bin/sudo /usr/sbin/storcli64 /c0/vall show all /usr/bin/sudo /usr/sbin/storcli64 /c0/vall show init /usr/bin/sudo /usr/sbin/storcli64 /c0/eall/sall show all /usr/bin/sudo /usr/sbin/storcli64 /c0/eall/sall show initialization /usr/bin/sudo /usr/sbin/storcli64 /c0/eall/sall show rebuildWarning sensors: c0/v0_Consist (No)

check_lsi_raid

57

Warum adpallinfo a0?

„storcli /0 show all … blocks the whole raid card i/o for … upto ~4 seconds“

58

Warum adpallinfo a0?

„storcli /0 show all … blocks the whole raid card i/o for … upto ~4 seconds“

59

$ /usr/lib/nagios/plugins/check_lsi_raid hcheck_lsi_raid: Nagios/Icinga plugin to check LSI Raid Controller statusPulgin version: 2.0Copyright (C) 20132014 ThomasKrenn.AGCurrent updates available at http://git.thomaskrenn.com/check_lsi_raid.gitThis Nagios/Icinga Plugin checks LSI RAID controllers for controller,physical device, logical device, BBU and CV warnings and errors.In order for this plugin to work properly you need to add the nagiosuser to your sudoers file (or create a new one in /etc/sudoers.d/).Usage: [ h | help ] Display this help page [ v | vv | vvv | verbose ] Sets the verbosity level. No v is the normal single line output for Nagios/Icinga, v is a more detailed version but still usable in Nagios. vv is a multiline output for debugging configuration errors or more detailed information. vvv is for plugin problem diagnosis. For further information please visit: http://nagiosplug.sourceforge.net/developerguidelines.html#AEN39 [ V version ] Displays the plugin and, if available, the version if StorCLI. [ C <num> | controller <num> ] Specifies a controller number, defaults to 0....

check_lsi_raid

60

VMware? → CIM Provider

61

VMware? → Plugin

check_esxi_hardware.py check_vmware_esx.pl

Hardware VMware allgemein

python-pywbem VMware Perl SDK

Claudio Kuenzler et.al.

Infos:

Martin Fürstenau

https://www.youtube.com/watch?v=3StGgZmIOKQ

http://www.thomas-krenn.com/de/wiki/VMware_ESXi_Hardware_mit_Nagios_oder_Icinga_%C3%BCberwachen

62

#!/usr/bin/python# * coding: UTF8 *## Script for checking global health of host running VMware ESX/ESXi## Licence : GNU General Public Licence (GPL) http://www.gnu.org/# This program is free software; you can redistribute it and/or...# Copyright (c) 2008 David Ligeret# Copyright (c) 2009 Joshua Daniel Franklin# Copyright (c) 2010 Branden Schneider# Copyright (c) 20102014 Claudio Kuenzler# Copyright (c) 2010 Samir Ibradzic# Copyright (c) 2010 Aaron Rogers# Copyright (c) 2011 Ludovic Hutin# Copyright (c) 2011 Carsten Schoene# Copyright (c) 20112012 Phil Randal# Copyright (c) 2011 Fredrik Aslund# Copyright (c) 2011 Bertrand Jomin# Copyright (c) 2011 Ian Chard# Copyright (c) 2012 Craig Hart# Copyright (c) 2013 Carl R. Friend

VMware? check_esxi_hardware.py

63

Adaptec by PMC

65

$ ./check_adaptec_raid p /usr/sbin/arcconf AACRAID CRITICAL (Ctrl #1): [ZMM critical]

$ ./check_adaptec_raid hThomasKrenn Adaptec Raid Controller Nagios/Icinga Plugin Version: 1.0Copyright (C) 20092013 ThomasKrenn.AGCurrent updates available via git at: http://git.thomaskrenn.com/check_adaptec_raid.gitThis Nagios/Icinga Plugin checks ADAPTEC RAIDControllers for Controller, PhysicalDevice and Logical Device warnings and errors. In order for this plugin to work properly you need to add the nagiosuser to your sudoers file (or create a new one in /etc/sudoers.d/).This is required as arcconf must be called with sudo permissions.Usage: [ C <Controller number> ] [ LD <Logical device number> ] [ PD <Physical device number> ] [ T <Warning Temp., Crit. Temp.> ] [ h | help ] Display this help page [ v | vv | vvv | verbose ] Sets the verbosity level no v single line output for Nagios/Icinga v single line with more details...

check_adaptec_raid Updategeplant(2015)

66

VMware? → CIM Provider erwartet

_ aktuell:_ „CIM Provider“ für remote arcconf

_ Adaptec MSM in einer VM

_ künftig:_ „echter“ CIM Provider

67

be smart,use SMART ;-)

68

Self-Monitoring,Analysis &ReportingTechnology

69

Standardisiert NICHT standadisiert

DatenformatKommandos

ErrorlogsTests

Attribute

Dokumentationvom Hersteller

erforderlich(oft nicht

öffentlich, außer Intel/Samsung)

70

check_smart_attributes

$ /usr/lib/nagios/plugins/check_smart_attributes \> d /dev/sda \> dbj /etc/nagiosplugins/config/check_smartdb.jsonOK (sda) |sda_Media_Wearout_Indicator=098;16;6 sda_Host_Writes_32MiB=575272 sda_Host_Reads_32MiB=723527

71

/etc/nagiosplugins/config/check_smartdb.json..."Intel DC S3700" : { "Device" : ["Intel DC S3700 Series SSDs","INTEL SSDSC2BA100G3", "ID#" : { "5" : "RAW_VALUE", # Reallocated Sector Count ... "194" : "RAW_VALUE", # Temperature Device Internal Te ... "232" : "VALUE", # Available Reserved Space "233" : "VALUE", # Media Wearout Indicator "234" : "VALUE", # Thermal Throttle Status "241" : "RAW_VALUE", # Total LBAs Written (32MiB) "242" : "RAW_VALUE", # Total LBAs Read (32MiB) "1024" : "VALUE" # ATA error count (custom) }, "Threshs" : { "5" : ["20","40"], ... "232" : ["16:","11:"], "233" : ["16:","6:"], "1024" : ["0","10"] }, "Perfs" : ["194","233","241","242"]},...

72

/etc/nagiosplugins/config/check_smartdb.json

...

73


Ständig neue SSDs&HDDs

74


Ständig neue SSDs&HDDsAktualisierungen?

75


Git(t) seiDank ;-)

76

ja cool, aber was ist mit RAID Controllern?...[d|device <path to device being checked>] Specify the device being monitored. If multiple devices should be checked provide the 'd' option multiple times. E.g. 'd /dev/sda d /dev/sdb' For devices behind LSI RAID controllers specify 'megaraid' and then the device number, e.g. 'd megaraid6'. Use storcli to find out the corresponding device numbers. For devices behind Adaptec RAID controllers specify '/dev/sg<X>' where <X> is the number for your device. Use e.g. sg_scan to find the device. You must also use 'O sat' or 'O scsi' according to the device interface. This are extra options only necessary for '/dev/sg<X>' devices....

77

ja cool, aber was ist mit RAID Controllern?

$ /usr/lib/nagios/plugins/check_smart_attributes \> d megaraid6> dbj /etc/nagiosplugins/config/check_smartdb.jsonOK (megaraid6) |megaraid6_Temperature_Internal=26 megaraid6_Media_Wearout_Indicator=100;16;6 megaraid6_Host_Writes_32MiB=70283 megaraid6_Host_Reads_32MiB=1650800

$ /usr/lib/nagios/plugins/check_smart_attributes \> d megaraid7> dbj /etc/nagiosplugins/config/check_smartdb.jsonWarning (megaraid7) [megaraid7_CRC_Error_Count = Warning]|megaraid7_Temperature_Internal=34 megaraid7_Media_Wearout_Indicator=098;16;6 megaraid7_Host_Writes_32MiB=189904 megaraid7_Host_Reads_32MiB=29658

78

monitor your GPU!

79

$ /usr/lib/nagios/plugins/check_gpu_sensor db 0000:83:00.0OK Tesla K20 |ECCL2AggSgl=0;1;2; ECCTexAggSgl=0;1;2; memUtilRate=0 PWRUsage=49.81;150;200; ECCRegAggSgl=0;1;2; SMClock=705 ECCL1AggSgl=0;1;2; GPUTemperature=38;85;100; memClock=2600 usedMemory=0.24;95;99; fanSpeed=30;80;95; graphicsClock=705 GPUUtilRate=0 ECCMemAggSgl=0;1;2;

check_gpu_sensor

80

NVIDIA: „angezeigte Lüfterdrehzahl lässt nicht

darauf schließen, ob sich der Lüfter tatsächlich dreht.“

„es ist jene Drehzahl, mit der der Lüfter-Algorithmus versucht den Lüfter zu betreiben.“

wir empfehlen: „Temperatursensor“

81

Plugins - Future

_ Überwachung vonFW-Versionen

_ RAID ConsistencyChecks

_ Temperatur von10GBit NICs(siehe Intel X540 FAQs)

82

so, was nun?

83

Relax ...

_ alle Plugins unter git.thomas-krenn.com

_ alle Plugins erfüllenPlugin Developer Guidelines (-h für Hilfe)

_ „Plugin Entwicklung für Einsteiger“von Alexander Wirt heute um 14:15h

84

Relax, start ...

Serverlisteerstellen

IPMIsicher

konfigurieren

relevantePlugins

einrichten

85

Relax, start and have fun at

Server Hardware monitoring done right - NETWAYS · 4 Nach diesem Vortrag überwachen Sie sicherer...

Documents

Transcript of Server Hardware monitoring done right - NETWAYS · 4 Nach diesem Vortrag überwachen Sie sicherer...