Wanna test NFV ... where do I start ?? what do I need ? is there a cookbook somewhere ??

1. NFV Testing
Landscape
MP Odini/HPE – ETSI NFV Vice Chair
Feb, 2016

2. Overview
– ETSI has worked on 2 aspects: Performance characteristics & metrics + testing methodology
– GS NFV-PER 001: NFV Performance & Portability Best Practises - published in ph1 (attached)
– GS NFV-INF 010: Quality metrics - published in ph1
– GS NFV-REL 005 - Report on Quality Accountability Framework – Published in ph2
– This is also referring to QuestForum: QuestForum (Release 5.0, July 2012): "TL 9000 Measurements Handbook".
– Available at http://www.tl9000.org/handbooks/measurements_handbook.html.
– NFV Testing methodology: TST001 on pre-validation test and TST002 on Interop test
– TST001 and TST002 are not published yet but are publicly available – as any ETSI NFV Draft (all publicly available) https://docbox.etsi.org/ISG/NFV/Open/Drafts/
– => we have a new proposal on the table to define Interop test suites , and ETSI is planning an NFV PlugTest later this year
– IETF has couple papers on benchmarking NFV:
– IETF draft-ietf-bmwg-virtual-net-01: “Considerations for Benchmarking Virtual Network Functions and Their Infrastructure”
– draft-huang-bmwg-virtual-network-performance-01: “Benchmarking methodology for Virtualization Network Performance” (but very ‘network / vswitch focused)
– OPNFV (Apache licence) has built a complete opensource Continuous Integration platform (Octopus) with a set of opensource tests environments
(pdf on FuncTest)
– Base system functionality testing (FuncTest) => tempest, rally etc
– Platform Performance Benchmarking (Qtip) => Compute, Storage, Network test suite >> leveraging ETSI PER001
– VSwitch Performance characteristics (VsPerf)
– Infrastructure verification (Yardstick) =>
– Storage Performance Benchmarking for NFVI (StorPerf)
– Policy Test (policytest)
– Other opensource projects examples:
– Clearwater (GNU licence) – opensource IMS project – have developed some opensource test :
– clearwater-live-test - Live test for Clearwater deployments

3. ETSI NFV & Test
3

4. ETSI GS NFV-PER 001: NFV Performance & Portability Best
Practises
Captures the HW and SW needs for VNFs/VMs dealing with network workloads, including data plane at
middleboxes.
Table of Content:
– Introduction
– Methodology and relation to NFV use cases
– Bottleneck analysis based on tests for each kind of workload
– One sub-clause per kind of workload.
– To be extended with future tests
– Best practises and recommendations
– HW & Hypervisor capabilities
– Templates for portability
– List of parameters in Compute Host Descriptors and VM Descriptor templates
– Annex A: Gap analysis of hypervisor and cloud OS
– Annex B: Relevant technologies (Direct I/O, SRIOV, RDMA, Infiniband)
– Annex C: NFV Test Methodologies (Control plane testing, Data plane testing, benchmarking hypervisors, benchmark
performance metrics,
– Annex D: Performance evaluation of an IP edge data plane

5. Benchmark Performance Metrics
QoS Metrics :
– Throughput
– Latency
– Frame Loss Rate
– Back-to-Back
– Packet delay variation
– Service Disruption Time for Fail-over Convergence
QoE Metrics:
QoE metrics are specific to the type of service or application, ex:
– HTTP: page load time, load time variance.
– Video: MOS-AV score, range = 2-5 with 5 being the best.
– HTML5 video - AS score, 100 % score as the maximum.
5

6. ETSI GS NFV-REL 005 : Quality Accountability Framework
– Goal – Develop an Informative Technical Report that:
– Promotes the development of capabilities by which VNFs, NFV infrastructure and MANO can eventually enable rapid
and reliable root cause analysis of service quality impairments, corrective action, and SLA management.
– Scope:
– Define high level roles, service boundaries and identify associated NFV reference points; and their quality-related
responsibilities to enable rapid root cause analysis and corrective actions
– Align high level roles with TMForum’s SLA management service delivery relationships
– Frame how traditional PNF quality measurements (e.g., TL 9000 Service Outage measurements) will be applied to
VNFs to permit side-by-side quality comparisons

7. ETSI GS NFV-INF 010 : Service Quality Metrics
The Service Quality Metrics document provides a general taxonomy for NFV service quality metrics. It
identifies metrics relevant to the service quality of virtual machines (VM), virtual network interfaces,
technology components, and orchestration. It also describes use cases for service quality metrics and
provides additional recommendations on aspects of measurements and service level agreements.
Table of Content:
• Taxonomy
• VM Quality Metrics (VM stall, Premature VM Release Ratio, VM Scheduling Latency, VM Clock Error)
• Virtual Network Interface Service Quality Metrics
• Technology Component Service Quality Metrics
• Orchestration Service Quality Metrics
• Service Quality Metrics Use Case
• Recommendations
• Annex: ex NFVI-VIM Interactions related to Service Quality Metrics
7

8. Service Quality Metrics & NFV Reference Architecture
8

9. Service Quality Metrics on NFVI
9

10. Summary of NFV Quality Metrics
10

11. NFVI-VIM Interactions related to Service Quality Metrics
11
across the [Nf-Vi]/<C,N,H> reference points
Resource Establishment:
VM Provisioning latency:
VM Provisioning reliability:
VM Dead on Arrival (DOA) ratio
This metric intends to track the
liveliness of the requested VM.
(Example)

12. TST001: PRE-DEPLOYMENT TESTING
Recommendations for lab validation of VNFs, their interaction with the NFV functional
blocks and the NFV blocks themselves
TST002: INTEROPERABILITY TESTING METHODOLOGY
Study how interoperability test methodology can be applied to NFV by analysing the
functional blocks and interfaces defined within the NFV architecture
TST001 & TST002

13. ETSI NFV GS TST001: Pre-deployment Testing
Table of content
• Definition of SUTs
• Test methods for pre-deployment validation of SUTs
• Pre-deployment validation of NFV Infrastructure
• Pre-deployment validation of VNFs
• Pre-deployment validation of Network Services
• Reliability & Resiliency Requirements
• Security Considerations
• ( Annex A: Open source communities in NFV )
13
The functions under test (FUT) are entities which are also commonly
known as Devices Under Test (DUT) in the testing community

14. SUT (System Under Test)
14

15. Pre-deployment validation of VNFs
15
VNF Lifecycle testing
 VNF on-boarding
 VNF instantiation =>
 VNF scaling/updating
 VNF termination
Specs incl the same for scaling, termination

16. ETSI NFV GS TST002
Table of content
Interoperability Testing Methodology Guidelines for NFV
Basic concepts for interoperability testing (SUT, FUT, etc)
Interoperability Test Specifications
NFV SUT Architecture
NFV Interoperability Features
16

17. TST002 basic concepts
17

18. NFV SUT architecture
18

19. VNF Package
19
Similar table for:

20. OPNFV & Test

21. OPNFV Projects
45 projects approved
As of Feb 11th , 2016
https://wiki.opnfv.org/approved_projects
testing
performance
- Pharos
- FuncTest
- Yardstick
- Policy test
- Vswitch performance
- Platform performance Qtip
- StorPerf
- Software Fastpath service quality metrics

22. OPNFV Continuous Integration (Octopus)

23. OPNFV Pharos - Testbed
23
12 Labs
(as of Feb 11th
2016)
Linux Foundation, Oregon, US
Spirent, California, US
China Mobile, China
Dell, California, US
Enea, Sweeden
Ericsson, Canada
Huawei, China
Huawei, California
Intel, Oregon
Orange, France (Lannion, Paris)
Zte, China
Soon: HPE, Israel
Each pharos lab has a number of OPNFV POD
Some POD are dedicated to certain activities -
https://wiki.opnfv.org/pharos_rls_b_labs

24. FuncTest: OPNFV functional testing
https://wiki.opnfv.org/opnfv_functional_testing
Scope: functional testing framework along with a set of test suites and test cases to test and verify the
functionality OPNFV platform (VIM +NFVI) – there are 2 functest : Arno and Bramahputra
– Tooling: Rally (benchmark, Tempest), Robot framework, Teston framework
Ex
24
Ex:
https://wiki.opnfv.org/list_of_t
empest_smoke_tests
Functest integrated with Octopus CI
4 pharos labs connected incl Ericsson, Orange

25. Yardstick : Infrastructure Verification
https://wiki.opnfv.org/yardstick
The goal of the Yardstick Project is to verify the infrastructure compliance when running VNF applications.
NFV Use Cases described in ETSI GS NFV 001 show a large variety of applications, each defining specific
requirements and complex configuration on the underlying infrastructure and test tools.The Yardstick
concept decomposes typical VNF work-load performance metrics into a number of
characteristics/performance vectors, which each of them can be represented by distinct test-cases.
The project's scope is to develop a test framework, test cases and test stimuli. The methodology used by the
Project, to verify the infrastructure form the perspective of a VNF, shall be aligned with ETSI TST001.
25

26. Yardstick test cases
26
TST001, Clause 6

27. Yardstick test metrics
27Leverages ETSI NFV PER001

28. Brahmaputra testing
28

29. Test coverage
29

30. OPNFV Test Coverage & ETSI NFV
30

31. Brahmaputra release
31

32. Qtip: Platform performance benchmarking
https://wiki.opnfv.org/platform_performance_benchmarking
Qtip is a performance benchmark suite for OPNFV platform
QTIP aims to benchmark OPNFV platforms through a "Bottom up" approach, testing baremetal components
first and then VM, and comparing baremetal results with VM results.
32

33. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Thankyou