The document discusses model-driven telemetry as an approach to network visibility and monitoring. It describes some of the challenges with traditional monitoring approaches like SNMP polling. Model-driven telemetry uses data models to push analytics-ready data from network devices to collectors. Key aspects covered include using YANG models to map native device data, encoding the data using protocols like gRPC and Google Protocol Buffers, and configuring subscriptions to stream telemetry data from sensors to destinations.

1. © 2016 Cisco and/or its affiliates. All rights reserved. 1
Model-Driven Telemetry
#ConnectCA
Jimmy Fanizzi
Systems Engineer – Global Service Provider
November 28, 2017
Connect
Cisco

3. © 2016 Cisco and/or its affiliates. All rights reserved. 3
Look
Twice
Before
You
Leap.
-- Charlotte Bronte
What Automation Without Visibility Looks Like

4. © 2016 Cisco and/or its affiliates. All rights reserved. 4
Source: Google @ Bay Area OpenDaylight Meetup 06/16

5. 5© 2016 Cisco and/or its affiliates. All rights reserved.
Current
challenges

6. © 2016 Cisco and/or its affiliates. All rights reserved. 6
syslog
SNMP
CLI
Too Slow
Incomplete
Network-Specific
Hard to Operationalize
Why Network Visibility Is Hard

7. © 2016 Cisco and/or its affiliates. All rights reserved. 7
SNMP Polling Is Hard on Everybody
Request-ID 1:
Sent, No Response
Request-ID 2:
Sent, No Response
ManagersNetworkRouters

8. © 2016 Cisco and/or its affiliates. All rights reserved. 8
What Happens When You Push SNMP Too Hard
• 10 second poll / push
• 3 pollers / telemetry receivers
• 30 minute measurement intervals
• 288 100Gig E Interfaces (Line Rate)
• SNMP: IF-MIB (query by row)

9. 9© 2016 Cisco and/or its affiliates. All rights reserved.
Telemetry
fundamentals

10. © 2016 Cisco and/or its affiliates. All rights reserved. 10
Three Enablers for Telemetry
Push Not Pull
Analytics-Ready Data
Data-Model Driven

11. © 2016 Cisco and/or its affiliates. All rights reserved. 11
Push Means Don’t Wait To Be Asked
T1, interface stats
T2, interface stats
T1, interface stats
T2, interface stats
• Collect Once, Send Many
• Wait for a Period of Time
• Repeat

12. © 2016 Cisco and/or its affiliates. All rights reserved. 12
Push Beats SNMP Pull for Speed & Scale
• 10 second poll / push
• 3 pollers / telemetry receivers
• 30 minute measurement intervals
• 288 100Gig E Interfaces (Line Rate)
• SNMP: IF-MIB (query by row)

13. © 2016 Cisco and/or its affiliates. All rights reserved. 13
“OSI model” of Telemetry
Data store
Data model
Producer
Exporter
Collector
Native (raw) data inside a router’s
database
Raw data mapped to a model
(YANG native, OpenConfig, etc)
Sending requested data in model format
to the “Exporter” at defined intervals
Encoding and delivering data to the
collector(s) destination(s)
Information collection for processing
(e.g., data monitoring, automation, analytics)
“Data”“Layer”
Telemetryend-to-end

14. © 2016 Cisco and/or its affiliates. All rights reserved. 14
“OSI model” of Telemetry
Data store
Data model
Producer
Exporter
Collector
Native (raw) data inside a router’s
database
Raw data mapped to a model
(YANG native, OpenConfig, etc)
Sending requested data in model format
to the “Exporter” at defined intervals
Encoding and delivering data to the
collector(s) destination(s)
Information collection for processing
(e.d., data monitoring, automation, analytics)
“Data”“Layer”
Telemetryend-to-end

15. © 2016 Cisco and/or its affiliates. All rights reserved. 15
“OSI model” of Telemetry
Data store
Data model
Producer
Exporter
Collector
Native (raw) data inside a router’s
database
Raw data mapped to a model
(YANG native, OpenConfig, etc)
Sending requested data in model format
to the “Exporter” at defined intervals
Encoding and delivering data to the
collector(s) destination(s)
Information collection for processing
(e.d., data monitoring, automation, analytics)
“Data”“Layer”
Telemetryend-to-end

16. © 2016 Cisco and/or its affiliates. All rights reserved. 16
YANG Is A Modeling Language
module ietf-interfaces {
import ietf-yang-types {
prefix yang;
}
container interfaces {
list interface {
key "name";
leaf name {
type string;
}
leaf enabled {
type boolean;
default "true";
}
…
Edited
for
Brevity
Self-contained top-level hierarchy of nodes
Import or define data types
Leaf nodes for simple data
Lists for sequence of entries
Containers group related nodes
Other YANG Features
• RO or RW
• Optional nodes
• Choice
• Augment
• When
• Arbitrary XML
• RPC
• etc

17. © 2016 Cisco and/or its affiliates. All rights reserved. 17
Models are Available in Github
You
Should
Do
This
• Telemetry only cares about operational (*-oper.yang)
models.
• 143 oper YANG models published for XR 6.1.1
• 151 oper YANG are for XR 6.1.2
• 177 oper YANG are for XR 6.2.1
• 180 oper YANG for XR 6.2.2
• 198 oper YANG for XR 6.3.1https://github.com/YangModels/yang/tree/master/vendor/cisco/xr

18. © 2016 Cisco and/or its affiliates. All rights reserved. 18
Finding the Data You Want To Stream
$ pyang -f tree Cisco-IOS-XR-infra-statsd-oper.yang
--tree-path infra-statistics/interfaces/interface/latest/generic-counters
telemetry model-driven
sensor-group SGROUP1
sensor-path Cisco-IOS-XR-infra-statsd-oper:infra-
statistics/interfaces/interface/latest/generic-counters

19. © 2016 Cisco and/or its affiliates. All rights reserved. 19
What Will Be Pushed With That Config
{
"Timestamp": 1480547974706,
"Keys": {
"interface-name": "MgmtEth0/RP0/CPU0/0"
},
"Content": {
"applique": 0,
"availability-flag": 0,
"broadcast-packets-received": 25035,
"broadcast-packets-sent": 0,
"bytes-received": 165321050,
"bytes-sent": 233917498,
"carrier-transitions": 3,
"crc-errors": 0,
"framing-errors-received": 0,
"giant-packets-received": 0,
"input-aborts": 0,
"input-drops": 62,
"input-errors": 0,
"input-ignored-packets": 0,
"input-overruns": 0,
"input-queue-drops": 0,
"last-data-time": 1480547974,
"last-discontinuity-time": 1479244159,
"multicast-packets-received": 457,
"multicast-packets-sent": 0,
"output-buffer-failures": 0,
"output-buffers-swapped-out": 0,
"output-drops": 0,
"output-errors": 104,
"output-queue-drops": 0,
"output-underruns": 0,
"packets-received": 373156,
"packets-sent": 311583,
"parity-packets-received": 0,
"resets": 0,
"runt-packets-received": 0,
"seconds-since-last-clear-counters": 0,
"seconds-since-packet-received": 0,
"seconds-since-packet-sent": 0,
"throttled-packets-received": 0,
"unknown-protocol-packets-received": 0
}
Repeated for all interfaces

20. © 2016 Cisco and/or its affiliates. All rights reserved. 20
“OSI model” of Telemetry
Data store
Data model
Producer
Exporter
Collector
Native (raw) data inside a router’s
database
Raw data mapped to a model
(YANG native, OpenConfig, etc)
Sending requested data in model format
to the “Exporter” at defined intervals
Encoding and delivering data to the
collector(s) destination(s)
Information collection for processing
(e.d., data monitoring, automation, analytics)
“Data”“Layer”
Telemetryend-to-end

21. © 2016 Cisco and/or its affiliates. All rights reserved. 21
Configuring Destination
telemetry model-driven
destination-group DGROUP
address family ipv4 192.168.1.1 port 2104
---- and/or ----
address family ipv6 2001:db8::1 port 2104
encoding self-describing-gpb
protocol tcp
Specify where you want to send your data
Specify how you want your data to look like
Specify how you want your data to be delivered

22. © 2016 Cisco and/or its affiliates. All rights reserved. 22
Basic Concept: Encoding
Encoding (or “serialization”) translates data (objects, state) into a format that
can be transmitted across the network. When the receiver decodes (“de-
serializes”) the data, it has an semantically identical copy of the original data.
DATA
DATA
“Decode”
“Encode”
IOS XR platforms
Encodings
• Compact GPB
• Key-Value GPB
• JSON (6.3.1)

23. © 2016 Cisco and/or its affiliates. All rights reserved. 23
GPB Encoding
Design Goals
• Simplicity
• Performance
• Forward/Backward
Compatibility
Non-Goals
• Human-Readable
• Self-Describing
• Text-based
Google Protocol Buffers (GPB)
Call them
“protobufs”
for short
“Protocol buffers are Google's language-neutral, platform-
neutral, extensible mechanism for serializing structured data
– think XML, but smaller, faster, and simpler.”

24. © 2016 Cisco and/or its affiliates. All rights reserved. 24
Telemetry Has Two GPB Encoding Options
data_gpb {
row {
timestamp: 1485794640469
keys: "n026GigabitEthernet0/0/0/0"
content:
"220003010230003001240003002250003000260003
000270003000300003000310003000320003300204=3
30003000340003000350003000360003377001"
}
2X faster
Operationally more complex (but not
relative to SNMP!)
data_gpbkv {
timestamp: 1485793813389
fields {
name: "keys"
fields { name: "interface-name" string_value:
"GigabitEthernet0/0/0/0" }
}
fields {
name: "content"
fields { name: "input-data-rate" uint64_value: 8 }
fields { name: "input-packet-rate" uint64_value: 1 }
<<< 9 lines are skipped >>>
fields { name: "input-load" uint32_value: 0 }
fields { name: "reliability" uint32_value: 255 }
}
}
...
3X larger
Native models: still need heuristics for key
names
GPB – “compact” GPB – “self-describing”

25. © 2016 Cisco and/or its affiliates. All rights reserved. 25
Dial-Out
• TCP & gRPC (from 6.1.1)
• UDP (from 6.2.1)
Dial-In
• gRPC only (from 6.1.1)
Transport Options
Collector
Data
SYN
SYN-ACK
ACK
Collector
Data
SYN
SYN-ACK
ACK

26. © 2016 Cisco and/or its affiliates. All rights reserved. 26
gRPC: Like REST But Different
Runs over HTTP/2
Optimize for page load time
Server push, header compression, multiplexing, TLS
RFC 7540 (May 2015)
Preserves most HTTP1.1 syntax
Defines Services (“RPCs”)
Encodes Using Google Protocol
Buffers (“GPB” or “protobufs”)
Services and Messages
Auto-generate code in many languages
http://www.grpc.io/docs/#hello-grpc

27. © 2016 Cisco and/or its affiliates. All rights reserved. 27
“OSI model” of Telemetry
Data store
Data model
Producer
Exporter
Collector
Native (raw) data inside a router’s
database
Raw data mapped to a model
(YANG native, OpenConfig, etc)
Sending requested data in model format
to the “Exporter” at defined intervals
Encoding and delivering data to the
collector(s) destination(s)
Information collection for processing
(e.d., data monitoring, automation, analytics)
“Data”“Layer”
Telemetryend-to-end

28. © 2016 Cisco and/or its affiliates. All rights reserved. 28
A Telemetry Subscription
telemetry model-driven
subscription Sub1
sensor-group-id SGROUP1 sample-interval 30000
destination-id DGROUP1
*Omit Destination Group For gRPC Dial-In

29. 29© 2016 Cisco and/or its affiliates. All rights reserved.
Telemetry on
Cisco products

30. © 2016 Cisco and/or its affiliates. All rights reserved. 30
Cisco XR Telemetry overview
Classic XR ASR9k Evolved XR ASR9k NCS5500 NCS6k
MDT support 6.1.1 6.1.1 6.1.1 6.1.3
Data models
YANG
(native, OC, IETF)
Link for models
YANG
(native, OC, IETF)
Link for models
YANG
(native, OC, IETF)
Link for models
YANG
(native, OC, IETF)
Link for models
Transport
(Control
protocols)
TCP (dial-out),
UDP (dial-out)*
gRPC (dial-in, dial-out),
TCP (dial-out),
UDP (dial-out)*
gRPC (dial-in, dial-out),
TCP (dial-out), UDP
(dial-out)*
TCP (dial-out), UDP
(dial-out)*
Encoding
GPB /
GPB-KV / JSON**
GPB / GPB-KV / JSON**
GPB /
GPB-KV / JSON**
GPB /
GPB-KV / JSON**
Collectors Pipeline*** Pipeline*** Pipeline*** Pipeline***
* UDP support from 6.2.1
** JSON support from 6.3.1
*** Open-sourced and ready to use: https://github.com/cisco/bigmuddy-network-telemetry-pipeline

31. © 2016 Cisco and/or its affiliates. All rights reserved. 31
NX OS IOS-XE
MDT support 7.0(3)I6(1) 16.6.1*
Data models
Data Management Engine, NX-API, YANG
(native, OC, IETF)
YANG (native**, IETF)
Link for models
Transport
(Control protocols)
gRPC* (dial-out), UDP** (dial-out), HTTP***
(dial-out)
Netconf (for YANG), GNMI
(16.8.1), gRPC (16.9.1)
Formats GPB/JSON XML, GPB (16.9.1)
Collectors Pipeline TBD
Min sample interval 5 sec 1 sec
Max # of dial-out destinations 5 TBD
Cisco NXOS/XE Telemetry high-level overview
* gRPC supports GPB only * supported on Catalyst 3650/3850/9300/9500, ASR1000, ISR4000
** UDP from 7.0(3)I7(1), supports both, GPB and JSON ** Native models are different from YANG models in XR
*** HTTP suppors JSON only
https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/7-x/programmability/guide/b_Cisco_Nexus_9000_Series_NX-
OS_Programmability_Guide_7x/b_Cisco_Nexus_9000_Series_NX-OS_Programmability_Guide_7x_chapter_011000.html
https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/prog/configuration/166/b_166_programmability_cg/model_driven_telemetry.html

32. © 2016 Cisco and/or its affiliates. All rights reserved. 32
Some Commonly Used Models and Subtrees
32BRKSPG-2333

33. © 2016 Cisco and/or its affiliates. All rights reserved. 33
Your infrastructure should be ready
Bytes are per a single collection* Go with EDT starting from IOS XR 6.3.x
Data
NCS5516
numbers
gRPC/KVGPB
BW (mbps)
gRPC/GPB
BW (mbps)
UDP/JSON
BW (mbps)
Interface Oper State* 2160 0.6806876 0.145277467 0.769932267
Interface Data Rate 2160 0.6365896 0.151012 0.802756
Interfaces Stats 2160 1.7155632 0.3023296 2.0160128
Optics Ports Info 576 0.477302222 0.065839378 0.483407289
Uptime Info 1 0.0002816 0.0002288 0.0005968
CPU State 18 0.335916 0.053052 0.3043256
Memory Info 18 0.004036 0.0011584 0.0055872
Processes Memory 538 0.061503111 0.019817778 0.0832968
LLDP Info 574 0.185312 0.0703288 0.2401824
IPv4 RIB Info * 650,022 47.37393978 10.95847427 56.86328129
IPv6 RIB Info * 14876 3.6415044 0.856933333 4.371237733
BGP IPv4 Routes Info 650,000 0.0004744 0.0002424 0.000632
BGP IPv6 Routes Info 12800 0.000472 0.00024 0.0006272
BGP ipv4 Neighbor Info 2 0.0390272 0.0054728 0.0391232
MPLS-TE Tunnels
Summary Info 1003 0.004604 0.0005632 0.004692
RSVP Interface Info 5 0.0015352 0.0004256 0.001972
NCS5500 NPU Stats 96 1.59755976 0.255440227 1.726576827
NCS5500 NPU Resources
Info 96 0.2052016 0.0378816 0.2130816
~ 57 mbps ~ 13 mbps ~ 68 mbps

34. 34© 2016 Cisco and/or its affiliates. All rights reserved.
Collectors
and analytics
platforms

35. © 2016 Cisco and/or its affiliates. All rights reserved. 35
Kafka
Different Collection Models
Logstash
ElasticSearch
Kibana
Panda
BYO
Custom Open Source, Customizable
Proprietary
or OS-
based
Commercial Stack
Prometheus /
InfluxDB
Grafana
Applications
Storage
Collection

36. © 2016 Cisco and/or its affiliates. All rights reserved. 36
Pipeline: An Open Source Collector
Pipeline
Kapacitor
Output to file, TSBD, Kafka…
Ingest, transform, filter
Self-monitoring, horizontally scalable

37. © 2016 Cisco and/or its affiliates. All rights reserved. 37
MDT in real time
Demo

38. © 2016 Cisco and/or its affiliates. All rights reserved. 38
Topology
tgn-01
Pagent Traffic Generator
xrv-01
XRv9k
rcv-01
IOS
Gi0/0/0/0 Gi 0/0/0/1
Traffic
Pipeline
Telemetry
QoS Data Policy-map output

39. © 2016 Cisco and/or its affiliates. All rights reserved. 39
How does the stack work?
Pipeline
Transforms telemetry data
for DB storage
InfluxDB
Stores telemetry data
Grafana
Data visualization and alerting
Python
Flask App to trigger
Ansible playbooks
Ansible
Automation Engine
Cisco Network Service
Orchestrator
IOS XR - core data source
NSO

40. © 2016 Cisco and/or its affiliates. All rights reserved. 40
The power of a modular tool chain
Data transformation
Data storage
Data visualization
and alerting tool
Webhook end point
Automation engine
Service orchestrator
IOS XR - core data source
NSO
= New Service Offering

41. © 2016 Cisco and/or its affiliates. All rights reserved. 41
NSO
telemetry model-driven
destination-group dev
address family ipv4 10.85.204.18 port 5432
encoding self-describing-gpb
protocol tcp
!
!
sensor-group QoS
sensor-path Cisco-IOS-XR-qos-ma-oper:qos/nodes/node/policy-
map/interface-table/interface/output
!
!
subscription Sub1
sensor-group-id QoS sample-interval 30000
destination-id dev
!
How does the stack work? – IOS XR
Pipeline
address
Sensors
Subscription

42. © 2016 Cisco and/or its affiliates. All rights reserved. 42
Key Takeaways
• Speed & Scale Require Visibility
• It’s Not Hard to Beat SNMP
• Data Models Are Your Friends
• A Big Data Platform Is In Your Future

43. © 2016 Cisco and/or its affiliates. All rights reserved. 43
Resources
Tutorials, Blogs, VoDs
https://xrdocs.github.io/telemetry/
http://blogs.cisco.com/sp/the-limits-of-snmp
http://blogs.cisco.com/sp/boring-is-the-new-awesome
http://blogs.cisco.com/sp/why-you-should-care-about-model-driven-telemetry
https://youtu.be/tIN8BjHwpNs (NANOG 67: 10 Lessons from Telemetry)
YANG
https://github.com/YangModels/yang/tree/master/vendor/cisco (Cisco YANG models)
http://blogs.cisco.com/getyourbuildon/yang-opensource-tools-for-data-modeling-driven-management
(YANG open source tools)
https://developer.cisco.com/site/ydk/ (YDK intro)
Telemetry Tools :
https://github.com/cisco/bigmuddy-network-telemetry-pipeline
https://github.com/cisco/bigmuddy-network-telemetry-stacks
Demos and Lab
https://dcloud-cms.cisco.com/demo/mdt-ios-xr-611-v1 (dCloud)
https://youtu.be/F_S9-ctNFe0 (demo on NCS 5508)

44. Thank you.