Abhinav Singh Software Technical Consulting EngineerVTune+-+Perfor… · Abhinav Singh Software...

Abhinav Singh

Software Technical Consulting Engineer

SPDK, PMDK & VTune™ Amplifier Summit

Agenda

• Introduction to SPDK Architecture

• Data Telemetry

• Data Collection and Metrics

• Conclusion


Drivers

StorageServices

StorageProtocols

iSCSI Target

NVMe-oFTarget

vhost-scsiTarget

NVMe Devices

Blobstore

NVMe-oFInitiator

Intel® QuickDataTechnology

Driver

Block Device Abstraction (BDEV)

Ceph* RBD

Linux* Async IO

Logical Volumes

3rd Party

NVMe

NVMe PCIeDriver

vhost-blkTarget

BlobFS

Integration

RocksDB*

Ceph*

Core

ApplicationFramework

GPT

PMEMvirtio-

scsi

QEMU

SPDK Architecture

Intel®VTune™


Supported SPDK usage models

Application services:

SPDK vhost-scsi for serving optimized block storage to VMs

SPDK NVMe for optimized access to locally attached storage

Disaggregated storage:

NVM Express* over Fabrics

iSCSI target


SPDK data processing using Intel® Data Direct I/O Technology (Intel® DDIO)

CPU Socket

CPU Cores ME

MO

RY

CO

NT

RO

LL

ER

RA

M

1

LLCI/O Data

1

5

2

RXd

Disk

4 3

6

1. Core writes I/O command descriptor

2. Disk reads descriptor to get buffer address

3. Disk writes I/O data

4. Disk writes I/O command descriptor

5. Core reads I/O command descriptor (polling)

6. Core reads IOPs and performs some action


Telemetry points

CPU SocketCPU Socket

CPU Cores ME

MO

RY

CO

NT

RO

LL

ER

1

LLCPacketRXd

Disk

15

23

6 RA

M

4

Core communicates with storage through PCIe MMIO transactions Intel DDIO makes LLC the

primary target ofDMA operations

Core I/O data processing

Memorybandwidth

Writing back descriptors may result in partial PCIe transaction

Intel® VTune™ integrationwith SPDK


Analyzing results – Summary

Get an overall I/O performance data Notice I/O

performance imbalance among

SSDs

We can also seehow each device

performs individually


Analyzing results – Summary

See throughput distributionper each device

Set reasonable performance targets and get throughput utilization per each range

All metrics can be seen on timeline within thread quanta…see next slides


Analyzing results – Bottom-up

in-thread activitycolored according

to Throughput utilization levels SPDK Effective time: CPU usage is accounted separately for spin-wait loops

I/O Statistic: IOPs and Throughput

PCIe bandwidth with traffic breakdown per physical device


Analyzing results – Bottom-up

Locate recessions and filter in to see I/O performance changes

Check which code was executed and caused I/O communication

being dropped for ~160ms!


Analyzing results – Platform view

PCIe bandwidth with traffic breakdown per physical device

Intel DDIO missesresulted in write back

to RAM

MMIO traffic. AvoidReads and control

Writes

DRAM bandwidth

Socket interconnect traffic



huge writeback traffic to host memory due to misses in IO-ways



Ʃ= 93,362,354

Almost 1:1 between doorbell signaling

and total number of IO operations. IO* 46,681,177


Easy

Advance

Calibrate I/O bandwidth with expected performance

Check for unexpected cross-NUMA traffic

Check for unexpected PCIe MMIO traffic

Optimize workload to make use of Intel DDIO efficiently

Analyzing performance with Intel® VTune™


Conclusion and Call to ActionSPDK proves to be useful to accelerate storage applications equipped with NVMe based devices

Run Disk I/O analysis in Intel® VTune™ Amplifier 2019 to ensure that I/O solution can deliver the performance required for the intended usage model

Call to action:

Welcome to use SPDK and contribute into SPDK community

Check out SPDK at http://spdk.io Download

Intel® VTune Amplifier Performance Profilersoftware.intel.com/vtune

Documentation:

User guide: SPDK IO Data Viewsoftware.intel.com/vtune-amplifier-help-spdk-code-analysis

IO Issues: Remote socket accesssoftware.intel.com//vtune-amplifier-cookbook-i/o-issues-remote-socket-accesses

http://spdk.io/

https://software.intel.com/en-us/vtune/choose-download

https://software.intel.com/en-us/vtune-amplifier-help-spdk-code-analysis

https://software.intel.com/en-us/vtune-amplifier-cookbook-i/o-issues-remote-socket-accesses


Legal Disclaimer & Optimization Notice

INFORMATION IN THIS DOCUMENT IS PROVIDED “AS IS”. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO THIS INFORMATION INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT.

Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more complete information visit www.intel.com/benchmarks.

Copyright © 2018, Intel Corporation. All rights reserved. Intel, Pentium, Xeon, Xeon Phi, Core, VTune, Cilk, and the Intel logo are trademarks of Intel Corporation in the U.S. and other countries.

Optimization Notice

Intel’s compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice.

Notice revision #20110804

http://www.intel.com/benchmarks

https://software.intel.com/en-us/articles/optimization-notice


BACKUP


cd <path_to_SPDK>

scripts/gen_nvme.sh

[Nvme]

TransportID "trtype:PCIe traddr:0000:00:0e.0" Nvme0

paste the ouput to test/lib/bdev/bdev.conf.in

cd test/lib/bdev/bdevperf

run SPDK sample application

sudo ./bdevperf -c ../bdev.conf.in -q 32 -s 4096 -w rw -M 50 -t 10

cd <path_to_Vtune/bin64>

run VTune

sudo ./amplxe-cl -collect hotspots -knob enable-user-tasks=true -- <app>

Abhinav Singh Software Technical Consulting EngineerVTune+-+Perfor… · Abhinav Singh Software...

Documents

Transcript of Abhinav Singh Software Technical Consulting EngineerVTune+-+Perfor… · Abhinav Singh Software...