Dealing with real time events is hard, but required to gain visibility in the clouds; perform low-latency monitoring (seconds) or troubleshoot large number of machines at the same time. LINQ for Traces (http://tx.codeplex.com/) provides a powerful way to deal with standing queries over real-time data feeds, such as Event Tracing for Windows (ETW) sessions or performance counters, as well as ad-hoc queries over past history in the form of trace and log files. This session will take you through the discovery and exploration of TX by building a real-time event analysis solution for web services. The presentation is based on building the actual monitoring event pipeline for Xbox Live Services.

1. Real-time Event
Processing using
LINQ for Logs &
Traces (TX)
Gert Drapers (#DataDude)
Principle Software Design Engineer

2. Agenda
•Rx (Reactive Extensions)
•Tx (LINQ to Logs and Traces)
•Case Study
Subtitel

3. TX (LINQ to Logs and Traces)
•http://tx.codeplex.com/
•TX allows you use LINQ
directly on raw event
sources:
• Ad-hoc query on past history in
trace and log files
• Standing queries on real-time
feeds, such as Event Tracing for
Windows (ETW) sessions

4. Builds on Reactive Extensions (RX.Net)
http://rx.codeplex.com/

5. Reactive Essential Interfaces
namespace System {
// Producers (.NET events, WinRT Events, Sensors, APM method etc.)
public interface IObservable<out T> {
IDisposable Subscribe(IObserver<T> observer);
}
// Events stream consumers
public interface IObserver<in T> {
void OnNext(T value);
void OnError(Exception error);
void OnCompleted();
}
}

6. Event processing at all
scales with Reactive
ExtensionsBart de Smet
Thrusday 17 april 13:15 - 14:30

7. Demo
Using TX with LINQPad

8. Event Sources
• Tx.Windows
• Event Tracing for Windows (ETW)
• Performance Counters - interop with the "Performance Data
Helper" (PDH) native API
• Event Logs (.evtx)
• IIS W3C text log files
• Tx.SqlServer
• SQL Server Extended Events (XEvent) is the tracing technology
used by the SQL-Server engine.
• Extensibility

9. Type Generation

10. Playback
•Structured Mode
• Lose analogy is that events of given type are like a "Table“
• playback.GetObservable< T> is similar to dataContext.GetTable<T>
in LINQ to SQL
• More precisely, IObservable is a sequence (i.e. order matters)
• This is the same distinction as IEnumerable vs.. SQL Table
•Timeline Mode
• This mode is useful to ask questions that are about the timeline
of all events, regardless of type

11. Playback Features
•Multiplexed sequences
•Hiding the heterogeneity of the event sources
•Same API for real-time and past history
•Single-Read to answer multiple queries from
file(s)
•Occurrence Time Scheduler

12. Demo
HttpSpew

13. Case Study XblWatch
•Objective of XblWatch:
• Replace existing rule based SCOM pack monitoring of web
service
• Provide near-realtime monitoring
• Create Synthetic Performance Counters for success and failure
rates per API and service
• Publish Synthetic Performance Counters to integrate with
existing monitoring and alerting system

14. XblWatch/er
xblwatcher.prod.live
XblWatcher.exe XblWatchCmd.exe
http://*:80
tcp://*:9000
Perf Counters
Site/Service/API
Machine 1
XblWatch.exe
http.sys ETW trace
Perf Counters
Machine/Service/API
Machine 2
XblWatch.exe
http.sys ETW trace
Perf Counters
Machine/Service/API
Machine 3
XblWatch.exe
http.sys ETW trace
Perf Counter
Machine/Service/API
Machine N
XblWatch.exe
http.sys ETW trace
Perf Counters
Machine/Service/API

15. Demo
XblWatch

16. Resources
•Tx (LINQ to Logs and Traces)
•(Rx) Reactive Extensions
•Event processing at all scales with Reactive
Extensions
Bart de Smet
Thrusday 17 april 13:15 - 14:30

17. Laat ons weten wat u vindt van deze sessie! Vul de evaluatie
in via www.techdaysapp.nl en maak kans op een van de 20
prijzen*. Prijswinnaars worden bekend gemaakt via Twitter
(#TechDaysNL). Gebruik hiervoor de code op uw badge.
Let us know how you feel about this session! Give your
feedback via www.techdaysapp.nl and possibly win one of
the 20 prices*. Winners will be announced via Twitter
(#TechDaysNL). Use your personal code on your badge.
* Over de uitslag kan niet worden gecorrespondeerd, prijzen zijn voorbeelden – All results are final, prices are
examples