Performance Measurement and Analysis of H.323 Traffic
Transcript of Performance Measurement and Analysis of H.323 Traffic
17 March 2005
Performance Measurement and Analysis of H.323 Traffic
Performance Measurement and Analysis of H.323 Traffic
Prasad Calyam ([email protected])
Mukundan Sridharan ([email protected])
Weiping Mandrawa ([email protected])
Paul Schopis ([email protected])
19th April 2004, PAM 2004, Juan-Les-Pins, France
Prasad Calyam ([email protected])
Mukundan Sridharan ([email protected])
Weiping Mandrawa ([email protected])
Paul Schopis ([email protected])
19th April 2004, PAM 2004, Juan-Les-Pins, France
2“ PAM 2004, April 19th 2004” www.oar.net
Background
Today H.323 Videoconferencing is a popular collaboration technology in both industry and academia
Demand for high-quality Video and Audio in H.323 applications is always on the rise even given the rapid advancements in the recent past
The “Network” is the critical variable that needs to be better understood in assuring satisfactory end-user experience while using H.323 applications
Our fundamental question:“What can any given network health diagnostic actually tell us about what the end-user experience might be when using an H.323 application such as Videoconferencing on that network?”
3“ PAM 2004, April 19th 2004” www.oar.net
Topics of Discussion
Goals of our studyTerminologyTest Setup and MethodologyResults of our studyConclusions
4“ PAM 2004, April 19th 2004” www.oar.net
Goals of our Study
To obtain “Good”, “Acceptable” and “Poor” performance bounds for network metrics such as delay, jitter and loss for H.323 applications, based on simultaneous subjective and objective quality assessment of H.323 audio and video streams
To identify the most dominating factor amongst delay, jitter and loss that affects end-user perception of audiovisual quality
5“ PAM 2004, April 19th 2004” www.oar.net
H.323 Overview
H.323 is an umbrella standard that defines how real-time multimedia communications such as videoconferencing can be supported on packet switched networks (Internet)
Devices: Terminals, Gateways, Gatekeepers and MCUs
Codecs: H.261, H.263, G.711, G.723.1Signaling: H.225, H.245Transport Mechanisms: TCP, UDP, RTP and RTCPData collaboration: T.120Many others…
6“ PAM 2004, April 19th 2004” www.oar.net
H.323 Protocol Stack
NETWORK
DATA LINK
PHYSICAL
TRANSPORT
SESSION
PRESENTATION
APPLICATION
Supplementary Services
Audio Signal
Video Signal Data
Control
G.711 G.728H.261 H.263 T.127
T.126
T.124
T.125/T.122
G.722 G.729
G.723.1
RTCP RAS RTP
H.450.3 H.450.2
H.450.1H.235
H.245 H.225UDP TCP
X.224.0
7“ PAM 2004, April 19th 2004” www.oar.net
Understanding the factors that affect H.323 Traffic performance assessment…
Human FactorsIndividual perception of audiovisual quality, Lack of training to use the system effectively, …
Device FactorsVoIP endpoints, Gateways, MCUs, Routers, Firewalls, NATs, Modems, Operating System, Processor, memory, …
Network FactorsDelay, Jitter, Loss, throughput, BER, …
8“ PAM 2004, April 19th 2004” www.oar.net
E2E Measurements of Audio and Video Applications
Two approaches to evaluating the performance of audiovisual quality
Subjective MeasurementsInvolve human participants to rate audiovisual quality Can you hear me now?We used the Mean Opinion Score (MOS) Ranking
technique (ITU-T P.800) Not just “Good”!
Objective MeasurementsAutomated techniques to rate audiovisual quality We used the “E-Model” (ITU-T G.107)
9“ PAM 2004, April 19th 2004” www.oar.net
MOS Rankings
10“ PAM 2004, April 19th 2004” www.oar.net
What constitutes a “Task”?
Any activity that takes place in a routine Videoconference
Casual conversation, intense discussion, a class lecture, movements involving minimal physical activity, …
Recommendations insist realistic scenarios, not just passive viewing
Guidelines can be found for task selection, participant training for scoring the audiovisual quality, task ordering, overall environment setup for quality assessments, …
H.323 Beacon loopback feature provided additional tasks for the various assessments scenarios
11“ PAM 2004, April 19th 2004” www.oar.net
Understanding Delay…
Compression Delay
Transmission Delay
Electronic Delay
Propagation Delay
Processing Delay
Queuing Delay
Resynchronization Delay
Decompression Delay
Presentation Delay
SENDER SIDE NETWORK RECEIVER SIDE
Delay is the amount of time that a packet takes to travel from the sender’s application to reach the receiver’s destination application
Caused by codecs, router queuing delays, …One-way delay requirement is stringent for H.323
Videoconferencing to maintain good interaction between both endsGood (0ms-150ms), Acceptable (150ms-300ms), Poor
(> 300ms) Our Results!
12“ PAM 2004, April 19th 2004” www.oar.net
Understanding Jitter…
Jitter is the variation in delay of the packets arriving at the receiving end
Caused by congestion, insufficient bandwidth, varying packet sizes in the network, out of order packets, …
Excessive jitter may cause packet loss in the receiver jitter buffers thus affecting the playback of the audio and video streamsGood (0-20ms), Acceptable (20ms-50ms), Poor (>50ms)
Our Results!
13“ PAM 2004, April 19th 2004” www.oar.net
Understanding Loss…
Packet Loss is the packets discarded deliberately (RED, TTL=0) or non-deliberately by intermediate links, nodes and end-systems along a given transmission path
Caused by line properties (Layer 1), full buffers (Layer 3) or late arrivals (at the application)
Good(0%-0.5%), Acceptable (0.5%-1.5%), Poor (>1.5%) Our Results!
14“ PAM 2004, April 19th 2004” www.oar.net
Overall Test Setup
15“ PAM 2004, April 19th 2004” www.oar.net
World Sites involved in Testing…
More than 500 one-on-one subjective quality assessments from Videoconferencing end-users and corresponding traffic traces were obtained from the Testing!!!
16“ PAM 2004, April 19th 2004” www.oar.net
Various Tools Used in the Testing…
OARnet H.323 Beacon – a tool to troubleshoot H.323 performance problems
Network path characteristics determination, Audio loopback, customized tests, …Ethereal – a network traffic sniffer tool
To capture traffic traces for Objective Ratings analysisTelchemy VQMon – a network traffic trace analyzer for VoIP
Used for obtaining Objective Ratings for various network settingsNISTnet – a network emulator tool
To create realistic network scenarios by introducing Delay, Jitter and LossSpirent SmartBits – a commercial network measurement suite
Used to qualify NISTnetNLANR Iperf – a TCP/UDP bandwidth measurement tool
Network path characteristics determinationjaalaM appareNet – a commercial network measurement suite
Network path characteristics determinationPolycom FX Videoconferencing Station – Audio/Video clips source
17“ PAM 2004, April 19th 2004” www.oar.net
H.323 Beacon Screenshots…
18“ PAM 2004, April 19th 2004” www.oar.net
H.323 Beacon Screenshots…(Contd.)
19“ PAM 2004, April 19th 2004” www.oar.net
H.323 Beacon Screenshots… (Contd.)
http://www.itecohio.org/beacon
20“ PAM 2004, April 19th 2004” www.oar.net
Internet Tests Methodology…
Design of Experiments → 2 Phase ApproachPhase I: LAN with WAN Emulation Tests
Full factorial (n**3 =27; n=3) Tasks with various network health scenario configurations on NISTnet
Phase II: Internet Tests 9 equivalent tasks out of possible 27 tasks + 3 H.323 Beacon tasks using the loopback featureEnd-to-End delay, jitter and loss values configured on NISTnet for each Internet test site = (LAN NISTnet Settings) – (Inherent Path Characteristics for corresponding Internet test site)
VII. G G PVIII. G P GIX. P G G I. BG BG BGII. BA BA BAIII. BP BP BP
I. G G GII. A A AIII. P P PIV. G G AV. G A GVI. A G G
G → Good; A → Acceptable; P → Poor; B(G/A/P) → Beacon Task
21“ PAM 2004, April 19th 2004” www.oar.net
Results: Subjective and ObjectiveMOS Vs Delay
Subjective and Objective MOS Vs Delay
Pearson correlation co-efficient for Subjective and Objective Scores for Delay = 0.827
22“ PAM 2004, April 19th 2004” www.oar.net
Results: Subjective and ObjectiveMOS Vs Jitter
Subjective and Objective MOS Vs Jitter
Pearson correlation co-efficient for Subjective and Objective Scores for Jitter = 0.737
23“ PAM 2004, April 19th 2004” www.oar.net
Results: Subjective and ObjectiveMOS Vs Loss
Subjective and Objective MOS Vs Loss
Pearson correlation co-efficient for Subjective and Objective Scores for Loss = 0.712
24“ PAM 2004, April 19th 2004” www.oar.net
Overall Results of Quality Grade Assessments for LAN/Internet Tests
GA*/PAG*/AA/PA/PPA/PGResultGGAGGPPAGLossGAGGPGPAGJitterAGGPGGPAGDelayS9S8S7S6S5S4S3S2S1
G → Good; A → Acceptable; P → Poor; S1 - S9 → Scenario 1-9
* next to a grade → end-user will more often perceive that Grade of quality in that particular scenario
25“ PAM 2004, April 19th 2004” www.oar.net
Results: Effects of Normalized Delay, Jitter and Loss variations on Subjective MOS
0 1 2 3 4 5 61
1.5
2
2.5
3
3.5
4
4.5
5
5.5
Su
bje
ctiv
e M
OS
Normalized Scale
DelayJitterLoss
Normalized Scale: 1 Unit → 150ms Delay; 20ms Jitter; 0.5% Loss
26“ PAM 2004, April 19th 2004” www.oar.net
Results: Effects of Normalized Delay, Jitter and Loss variations on Objective MOS
0 1 2 3 4 5 61
1.5
2
2.5
3
3.5
4
4.5
Ob
ject
ive
MO
S
Normalized Scale
DelayJitterLoss
End-user perception of audiovisual quality is more sensitive tochanges in jitter than to changes in delay and loss
27“ PAM 2004, April 19th 2004” www.oar.net
Conclusions
We have determined performance bounds for latency, jitter and loss by conducting exhaustive testing in a LAN environment with WAN Emulation
By using many incremental values of each of the 3 metrics meticulously, we determined what ranges of these values in isolation and in combination affect the user perception of audiovisual quality as "good", "acceptable" and "poor" grades
We have demonstrated our theory of bounds developed in a LAN with WAN emulation, actually holds good in the “Internet”
By using literally every type of last mile connection and by testing with sites all over the world, we proved our LAN results scale consistently to the Internet
28“ PAM 2004, April 19th 2004” www.oar.net
Conclusions (Contd.)
We have observed the H.323 Beacon loopback assessments (again both subjective and objective) to be in accord with the audiovisual tasks test results
This demonstrates the handy utility of the H.323 Beacon in determining user perceived audiovisual quality in H.323 production systems without remote end-user intervention
Our normalized results indicate that end-user perception of audiovisual quality is more sensitive to changes in jitter than to changes in delay and loss
29“ PAM 2004, April 19th 2004” www.oar.net
Questions