Bart De Smet's ECOOP 2011 Keynote talk.

1. Category theory, Monads, and Duality in the world of (BIG) Data Bart J.F. De Smet bartde@microsoft.com Cloud Programmability Team

2. What’s in a name? Cloud Programmability Team Logical role: Research-oriented team Collaboration with MSR Physical placement: Oasis within the product team Close to the SQL Server business (Dual) 80/20 rule of success Portfolio Language Integrated Query (LINQ) XML literals in Visual Basic 9 Reactive Extensions (Rx) Various undisclosed projects Democratizing the cloud

3. Take One Democratizing data access with LINQ

4. A quick reminder on LINQ Solving the impedance mismatch between objects and data through querying.

5. Back to the future 5+ years ago Censored

6. Democratizing data access varres = from p inctx.Products wherep.UnitPrice > 100 group p byp.Categoryinto g selectnew { Category = g.Key, Avg = g.Average() }; “Lost in translation” varres = ctx.Products .Where(p => p.UnitPrice > 100) .GroupBy(p => p.Category) .Select(g => new { Category = g.Key, Avg = g.Average() }); (In-memory) iterators Query providers

7. C# 3.0 compilation to C# 2.0

8. Language Integrated Monads IEnumerable<T> IQueryable<T> IEnumerable<R> SelectMany<T, R>(this IEnumerable<T> source,Func<T, IEnumerable<R>> selector) SelectMany

9. Maybe baby! Billion Null-propagating dot string s = name?.ToUpper(); Syntactic sugar name.SelectMany( _ => _.ToUpper(), s => s) from _ in name from s in _.ToUpper() select s Compiler Can useextension method

10. Closing the loop LINQ to Haskell 

11. Take two Democratizing event processing with Rx

12. Once upon a time… Tier splitting “LINQ to Events”

13. Reactive Extensions (Rx) GPS RSS feeds Stock tickers Social media UI events Server management

14. Pull-based data access interfaceIEnumerable<out T> { IEnumerator<T> GetEnumerator();} interfaceIEnumerator<out T> : IDisposable { boolMoveNext(); T Current { get; } void Reset(); } You could get stuck

15. Duality in the world around us(Or… the Dutch are cheap) Electricity: inductor and capacitor Logic: De Morgan’s Law Programming? ¬𝐴∨𝐵≡¬𝐴∧¬𝐵 ¬𝐴∧𝐵≡¬𝐴∨¬𝐵

16. Duality as the secret sauce?Give me a recipe http://en.wikipedia.org/wiki/Dual_(category_theory) Reversing arrows…Input becomes output and vice versa Making a U-turnin synchrony

17. Distilling the essenceProperties and unchecked exceptions interfaceIEnumerable<out T> { IEnumerator<T> GetEnumerator();} interfaceIEnumerator<out T> : IDisposable { boolMoveNext(); T Current { get; } }

18. Distilling the essenceProperties and unchecked exceptions interfaceIEnumerable<out T> { IEnumerator<T> GetEnumerator();} interfaceIEnumerator<out T> : IDisposable { boolMoveNext() throwsException; T GetCurrent(); }

19. Distilling the essenceEmbracing a (more) functional style interfaceIEnumerable<out T> { IEnumerator<T> GetEnumerator();} interfaceIEnumerator<out T> : IDisposable { boolMoveNext() throwsException; T GetCurrent(); }

20. Distilling the essenceEmbracing a (more) functional style interfaceIEnumerable<out T> { IEnumerator<T> GetEnumerator();} interfaceIEnumerator<out T> : IDisposable { (void | T | Exception) MoveNext(); } () -> (() -> (void | T | Exception))

21. Flipping the arrowsPurely mechanical transformation () -> (() -> (void | T | Exception)) ((void | T | Exception) -> ()) -> ()

22. Harvesting the resultSo far for abstract nonsense interfaceIBar<out T> { voidQux(IFoo<T> foo);} interfaceIFoo<in T> { voidWibble(); void Wobble(T value); voidWubble(Exception error); }

23. Harvesting the resultThe observer pattern in disguise interfaceIObservable<out T> { void Subscribe(IObserver<T> observer);} interfaceIObserver<in T> { voidOnCompleted(); voidOnNext(T value); voidOnError(Exception error); }

24. The observer pattern revisited Stateful!

25. Interface hierarchy interfaceIObservable<out T> { IDisposableSubscribe(IObserver<T> observer);}

26. Message grammar OnNext(42) OnNext(43) OnCompleted source1 OnNext(“Hello”) OnError(error) source2 OnNext* [OnError | OnCompleted]

27. Observable.Create<T> operator IObservable<int> o = Observable.Create<int>(observer => { // Assume we introduce concurrency (see later)… observer.OnNext(42); observer.OnCompleted(); return () => { /* unsubscribe action */ }; }); IDisposable subscription = o.Subscribe( onNext: x => { Console.WriteLine("Next: " + x); }, onError: ex => { Console.WriteLine("Oops: " + ex); }, onCompleted: () => { Console.WriteLine("Done"); } ); C# doesn’t have anonymous interface implementation, so we provide various extension methods that take lambdas. C# 4.0 named parameter syntax

28. Observable.Create<T> operator IObservable<int> o = Observable.Create<int>(observer => { // Assume we introduce concurrency (see later)… observer.OnNext(42); observer.OnCompleted(); return () => { /* unsubscribe action */ }; }); IDisposable subscription = o.Subscribe( onNext: x => { Console.WriteLine("Next: " + x); }, onError: ex => { Console.WriteLine("Oops: " + ex); }, onCompleted: () => { Console.WriteLine("Done"); } ); Thread.Sleep(30000); // Main thread is blocked… F10

29. Observable.Create<T> operator IObservable<int> o = Observable.Create<int>(observer => { // Assume we introduce concurrency (see later)… observer.OnNext(42); observer.OnCompleted(); return () => { /* unsubscribe action */ }; }); IDisposable subscription = o.Subscribe( onNext: x => { Console.WriteLine("Next: " + x); }, onError: ex => { Console.WriteLine("Oops: " + ex); }, onCompleted: () => { Console.WriteLine("Done"); } ); Thread.Sleep(30000); // Main thread is blocked… F10

30. Observable.Create<T> operator IObservable<int> o = Observable.Create<int>(observer => { // Assume we introduce concurrency (see later)… observer.OnNext(42); observer.OnCompleted(); return () => { /* unsubscribe action */ }; }); IDisposable subscription = o.Subscribe( onNext: x => { Console.WriteLine("Next: " + x); }, onError: ex => { Console.WriteLine("Oops: " + ex); }, onCompleted: () => { Console.WriteLine("Done"); } ); Thread.Sleep(30000); // Main thread is blocked… F5

31. Observable.Create<T> operator IObservable<int> o = Observable.Create<int>(observer => { // Assume we introduce concurrency (see later)… observer.OnNext(42); observer.OnCompleted(); return () => { /* unsubscribe action */ }; }); IDisposable subscription = o.Subscribe( onNext: x => { Console.WriteLine("Next: " + x); }, onError: ex => { Console.WriteLine("Oops: " + ex); }, onCompleted: () => { Console.WriteLine("Done"); } ); Thread.Sleep(30000); // Main thread is blocked… Breakpoint got hit

32. Iterators dualized IObservable<int> GetXs() { returnObservable.Create(o => for(int i = 0; i < 10; i++) o.OnNext(i * i); o.OnCompleted(); ); } GetXs().Subscribe(x => { Console.WriteLine(x); }); IEnumerable<int> GetXs() { for (int i = 0; i < 10; i++) yieldreturni * i; yield break; } foreach(var x inGetXs()) { Console.WriteLine(x); } Synchronous Asynchronous

33. Compositionality matters IObservable<T>Merge<T>(thisIObservable<T> left, IObservable<T> right) { return Create<T>(observer => { // Ignoring a few details for OnCompleted var d1 = left.Subscribe(observer); var d2 = right.Subscribe(observer); returnnewCompositeDisposable(d1, d2); }); } Lazy evaluation

34. Bridging Rx with the WorldWhy .NET events aren’t first-class… Hidden data source How to pass around? form1.MouseMove+= (sender, args) => { if(args.Location.X==args.Location.Y) // I’d like to raise another event }; form1.MouseMove -=/* what goes here? */ Lack of composition Resource maintenance?

36. Bridging Rx with the World…but observable sequences are first-class Source of Point values Objects can be passed IObservable<Point>mouseMoves= Observable.FromEvent(frm, "MouseMove"); varfiltered = mouseMoves .Where(pos => pos.X == pos.Y); varsubscription = filtered.Subscribe(…); subscription.Dispose(); Can define operators Resource maintenance!

37. Composition and QueryingIt’s the continuation monad! // IObservable<string> from TextChanged events varchanged = Observable.FromEvent(txt, "TextChanged"); var input = (from e in changed let text = ((TextBox)e.Sender).Text wheretext.Length >= 3 select text) .DistinctUntilChanged() .Throttle(TimeSpan.FromSeconds(1)); // Bridge with the dictionary web service var svc = newDictServiceSoapClient();var lookup = Observable.FromAsyncPattern<string, DictionaryWord[]> (svc.BeginLookup, svc.EndLookup); // Compose both sources using SelectMany var res = from term in input from words in lookup(term) select words; input.SelectMany(term => lookup(term))

38. Introducing schedulers How to be asynchronous? Different ways to Introduce of concurrency Parameterization by schedulers interfaceIScheduler { DateTimeOffset Now { get; } IDisposableSchedule<T>( T state, Func<IScheduler, T, IDisposable> f); // Overloads for time-based scheduling }

39. Example: creation operators static classObservable { static IObservable<T> Return<T>(T value, IScheduler scheduler) { return Create<T>(observer => { var state = new { value, observer }; returnscheduler.Schedule(state, (self, s) => { s.observer.OnNext(s.value); s.observer.OnCompleted(); }); }); } } Resource mgmt Avoiding closures (serialization)

40. Operational layering

41. IQbservable<T> LINQ to Twitter How? ToQbservable Translatable (Expression trees) IQueryable<T> LINQ to SQL ToQueryable LINQ to *.* AsObservable Homo-iconic AsEnumerable AsQbservable AsQueryable ToObservable IEnumerable<T> LINQ to Objects IObservable<T> LINQ to Events Fixed (MSIL) ToEnumerable Pull(interactive) Push (reactive) What? Duality Concurrency(IScheduler) Where? Message loops Distributed Worker pools Threads

42. Take THREE Democratizing cloud data processing with CoSQL

43. NoSQL is CoSQL!

44. The NoSQL trend

45. Object graphs var_1579124585 = newProduct{ Title = “The Right Stuff”, Author = “Tom Wolfe”, Year = 1979, Pages = 304, Keywords = new[] { “Book”, “Hardcover”, “American” }, Ratings = new[] { “****”, “4 stars” }, }; var Products = new[] { _1579124585 };

46. Queries over object graphs varq = from product in Products whereproduct.Ratings.Any(rating => rating == “****”) selectnew { product.Title, product.Keywords };

47. The O/R paradox Objects Fully compositional value ::= scalar new {…, name = value, … } Tables Non compositional value ::= new {…, name = scalar, … }

48. Relational (de)normalization

49. Queries over tables var q = from product in Products fromrating in Ratings whereproduct.ID == rating.ProductId && rating == “****” fromkeyword in Keywords whereproduct.ID == keyword.ProductID selectnew { product.Title, keyword.Keyword }; varq = from product inProducts joinrating in Ratings onproduct.ID equalsrating.ProductId whererating == “****” selectproduct intoFourStarProducts fromfourstarproductinFourStarProducts joinkeyword in Keywords onproduct.ID equalskeyword.ProductID selectnew { product.Title, keyword.Keyword };

50. Welcome to O/R voodoo varq = from product in Products whereproduct.Ratings.Any(rating => rating == “****”) selectnew { product.Title, product.Keywords };

51. What did we gain? Ad-hoc queries? But what about scale… The relational Gods invented indexes Going against the PK-FK flow… from p1 in WWW from p2 in WWW where p2.Contains(p1.URL) selectnew { p1, p2 };

52. Job security?

53. Spot the difference

54. Duality to the rescue again?

55. Consequences of duality

56. More work in the area

57. Thank you! Bart J.F. De Smet bartde@microsoft.com Cloud Programmability Team