.NET Fest 2018. Антон Молдован. One year of using F# in production at SBTech

About me:
@AntyaDev
like types*

- Biggest provider in sport offering
- Supports a lot of regulated markets
- About 1000 microservices (200 distinct types)
- 5 datacenters maintained fully by SBTech
- About 500 concurrent live events at pick time
- On average we handle about 100K+ RPS

If we have sexy C#
class Person
{
public string FirstName; // Not null
public string? MiddleName; // May be null
public string LastName; // Not null
}

If we have sexy C#
switch (shape)
{
case Circle c:
WriteLine($"circle with radius {c.Radius}");
break;
case Square s when (s.Length == s.Height):
WriteLine($"{s.Length} x {s.Height} square");
break;
case Rectangle r:
WriteLine($"{r.Length} x {r.Height} rectangle");
break;
}

If we have sexy C#
public (string, string) LookupName(long id)
{
// some code..
return (first, middle);
}
var names = LookupName(id);
WriteLine($"found {names.first} {names.last}.");

Story of a new C# project
Get("/api/bets", async request =>
{
var result = Validator.Validate(request); // unit testable
if (result.IsOk)
{
var response = await _httpClient.Call(request);
return BetsMapper.MapResponse(response); // unit testable
}
else return Errors.MapError(result); // unit testable
});

public class BetsProviderService : IBetsProviderService
{
readonly IOpenBetsRepository _openBetsRepository;
readonly ISettledBetsRepository _settledBetsRepository;
public BetsProviderService(IOpenBetsRepository openBetsRepository,
ISettledBetsRepository settledBetsRepository)
{
_openBetsRepository = openBetsRepository;
_settledBetsRepository = settledBetsRepository;
}
}
Story of a new C# project

Different Culture
OOP FP
abstraction
extensibility
purity
composability
correctness

Triggers:
Much better tooling support
Community growth
Recursive modules

Tooling support
Visual Studio
RiderVisual Studio Code

Recursive modules
type Plane() = member x.Start() = PlaneLogic.start(x)
// types and modules can't currently be mutually
// referential at all
module private PlaneLogic =
let start (x: Plane) = ...

Recursive modules
module M
type First =
| T of Second
and Second =
| T of First

Recursive modules
module M
type First =
| T of Second
type Second =
| T of First

Recursive modules
module rec M
type First =
| T of Second
type Second =
| T of First

Recursive modules
module M
let private helper () = ...
let publicApiFunc () = helper()

Recursive modules
module rec M
let publicApiFunc () = helper()
let private helper () = ...

Community growth
• To begin, F# has grown to be bigger than ever, at least as far as
we can measure, through product telemetry, twitter activity,
GitHub activity, and F# Software Foundation activity.
• Active unique users of F# we can measure are in the tens of
thousands.
• Measured unique users of Visual Studio Code
with Ionide increased by over 50% this year, to become far larger
than ever.
• Measured unique users of Visual Studio who use F# increased by
over 20% since last year to be larger than ever, despite quality
issues earlier in the year that we believe have inhibited growth.
• Much of the measured growth coincides with the release of .NET
Core 2.0, which has shown significant interest in the F#
community.

namespace SportData.ETL.Core
type ItemRawUpdate<'Item> = {
Id: string
Data: 'Item option
}
module TransformationFlow =
let getUpdates (feed: ChangeFeed, lastOffset: uint32) =
Code Style

Code sample
let createGame (globalCache: GlobalItemsCache,
masterEvent: MasterEventItem, currentTime: DateTime) = trial {
let! league = Cache.get(globalCache.Leagues, masterEvent.LeagueID)
let! country = Cache.get(globalCache.Countries, league.CountryID)
let! sport = Cache.get(globalCache.Branches, masterEvent.BranchID)
...
return { Id = masterEvent.ID.ToString()
Type = getEventType(masterEvent) }
}

I need website
Website
SBTech story

I need an API
to build
unique UI
The Problem
Oops?!

The Problem
Oops?!
1. well defined contracts (Event, Market, Selection)
2. flexible way to query data and subscribe on changes
3. push updates notifications about changes
4. near real time change delivery (1 sec delay)
We need to provide:

PUSH based
Queryable API
(Change Feed)

select * from Games where isLive = true
order by totalBets desc
limit 10
PUSH changes

Change Feed
(Actor)
Change Stream
Feed View
2:2
1:0 Chelsea - Milan
Milan - Liverpool

Subscribers
Feed View
2:2
1:0 Chelsea - Milan
Milan - Liverpool
Change Log
Query
Change Feed
(Actor)

Subscribers
type NodeName = string
type LastHeartBeat = DateTime
type NodeSubscribers = Map<NodeName, LastHeartBeat>
type FeedStatus =
| ReadyForDeactivation
| HasSubscribers
| NoSubscribers
type ChangeFeed = {
Id: FeedId
View: FeedView
ChangeLog: ChangeLog
Subscribers: NodeSubscribers
Query: Query
Status: FeedStatus
}
Query
Feed View
2:2
1:0 Chelsea - Milan
Milan - Liverpool
Change Feed
Change Log

module ChangeFeed
let getSnapshot (feed: ChangeFeed, fastPreloadAmount: uint32) = ...
let getUpdates (feed: ChangeFeed, lastOffset: uint32) = ...
let getLatestUpdate (feed: ChangeFeed) = ...
let subscribe (feed: ChangeFeed, name: NodeName, currentTime: DateTime) = ...
let subscribeByOffset (feed: ChangeFeed, name: NodeName, lastOffset: uint32) = ...
let unsubscribe (feed: ChangeFeed, name: NodeName) = ...
let getInactiveNodes (feed: ChangeFeed, currentTime: DateTime, inactivePeriod: TimeSpan) = ...
let reloadView (feed: ChangeFeed, queryResults: QueryResult seq) = ...

Feed View
2:2
1:0 Chelsea - Milan
Milan - Liverpool
type PartialView = {
EntityType: EntityType
Entities: IEntity seq
}
type FeedView = {
OrderType: EntityType
OrderIds: string seq
OrderFilter: FilterFn option
Views: Map<EntityType, PartialView>
}
let tryCreate (queryResults: QueryResult seq, orderType: EntityType,
orderFilter: FilterFn option) =
match queryResults with
| NotMatchFor orderType -> fail <| Errors.OrderTypeIsNotMatch
| Empty -> ok <| { OrderType = orderType; OrderIds = Seq.empty
OrderFilter = orderFilter; Views = views }
| _ -> ok <| ...

[<Property>]
let ``empty ChangeLog should start with offset 0 and requested maxSize`` (maxSize: uint32) =
let changeLog = ChangeLog.create(maxSize)
Assert.Equal(0u, changeLog.Offset)
Assert.Equal(maxSize, changeLog.MaxSize)
[<Property(Arbitrary=[|typeof<Generators.Generator>|])>]
let ``all changes in changeLog.stream should have UpdateType.Update`` (payloads: Payload array) =
let mutable changeLog = ChangeLog.create(5u)
for p in payloads do
changeLog <- ChangeLog.append(changeLog, p)
let result = changeLog.Stream.All(fun change -> change.Type = UpdateType.Update)
Assert.True(result)
TDD without Mocks

We have our own frontend
but
we need a Data
The Next Challenge
Oops?!

RDBMS
All changes
from
40 tables
ETL
Operator
Well defined entities
compose entities
react on changes
recalculate odds
(data locality)

C#
F#Asp NET Core
Orleans
DB Drivers
[ DDD; Tests]
Infrastructure
business logic

public Task<FullFeedUpdate> GetSnapshot(GetSnapshot msg)
{
// invoke F# code from C#
var updates = ChangeFeedModule.getSnapshot(FeedState, msg.FastPreloadAmount);
return updates;
}

Applied at the heart of the system
Zero Null Reference exceptions
Domain errors instead of exceptions
DDD with types (strong determinism)
Dependency Rejection
TDD without mocks (property based testing)

- Technology agnostic
(no dependency on HTTP, WebSockets, SSE)
- Very simple API
(to test Pull and Push scenarios)
- CI/CD integration

type Step =
| Request of RequestStep // to model Request-response pattern
| Listener of ListenerStep // to model Pub/Sub pattern
| Pause of TimeSpan // to model pause in your test flow
type TestFlow = {
FlowName: string
Steps: Step[]
ConcurrentCopies: int
}
type Scenario = {
ScenarioName: string
TestInit: Step option
TestFlows: TestFlow[]
Duration: TimeSpan
}

let execStep (step: Step, req: Request, timer: Stopwatch) = task {
timer.Restart()
match step with
| Request r -> let! resp = r.Execute(req)
timer.Stop()
let latency = Convert.ToInt64(timer.Elapsed.TotalMilliseconds)
return (resp, latency)
| Listener l -> let listener = l.Listeners.Get(req.CorrelationId)
let! resp = listener.GetResponse()
timer.Stop()
let latency = Convert.ToInt64(timer.Elapsed.TotalMilliseconds)
return (resp, latency)
| Pause time -> do! Task.Delay(time)
return (Response.Ok(req), 0L)
}

type TestFlowRunner(flow: TestFlow) =
let createActors (flow: TestFlow) =
flow.CorrelationIds
|> Set.toArray
|> Array.map(fun id -> TestFlowActor(id, flow))
let actors = createActors(flow)
member x.Run() = actors |> Array.iter(fun x -> x.Run())
member x.Stop() = actors |> Array.iter(fun x -> x.Stop())
member x.GetResult() = actors
|> Array.collect(fun actor -> actor.GetResults())
|> TestFlowStats.create(flow)

// C# example
var flow = new TestFlow(flowName: "READ Users",
steps: new[] {step1, step2, step3},
concurrentCopies: 10);
new ScenarioBuilder(scenarioName: "Test MongoDb")
.AddTestFlow(flow)
.Build(duration: TimeSpan.FromSeconds(10))
.RunInConsole();

// F# example
let flow = { FlowName = "GET flow"
Steps = [step1; step2; step3]
ConcurrentCopies = 100 }
Scenario.create("Test HTTP https://github.com")
|> Scenario.addTestFlow(flow)
|> Scenario.withDuration(TimeSpan.FromSeconds(10.0))
|> Scenario.runInConsole

var httpClient = new HttpClient();
var httpStep = Step.CreateRequest("GET request", execute: async _ => {
var response = await httpClient.SendAsync(request);
return response.IsSuccessStatusCode
? Response.Ok()
: Response.Fail();
});
var pause_1s = Step.CreatePause(duration: TimeSpan.FromSeconds(1));
var scenario = new ScenarioBuilder("Test HTTP")
.AddTestFlow("PULL", new[] { httpStep, pause_1s }, concurrentCopies: 10)
.Build(duration: TimeSpan.FromSeconds(10));
scenario.RunInConsole();

RDBMS
Storage
Kafka ETL
Kafka
NoSQL
Storage
Push
API

- 1470 LOC
- (4) OOP objects
- (~100) functions
- try/catch used 2 times

async { let! x = myFunction()
match x with
| Some v -> ...
| None -> ... }
F# vNext
match!

async { match! myFunction() with
| Some v -> ...
| None -> ... }
F# vNext
match!

F# vNext
Struct Records
type Vector3 = { X: float; Y: float; Z: float }
[<Struct>]
type Vector3 = { X: float; Y: float; Z: float }

F# vNext
Struct Records
[<Struct>]
type Result<'T,'TError> =
| Ok of 'T
| Error of 'Terror
[<Struct>]
type ValueOption<'T> =
| Some of ‘T
| None

F# vNext
Anonymous Records
type Data = {
X: int
Y: string
}
let data = { X = 1; Y = "abc" }

F# vNext
Anonymous Records
let data = {| X = 1; Y = "abc" |}

F# vNext
Structural Type System
type User() =
member this.GetName() = "test user"
type Car() =
member this.GetName() = "test car"
let inline printName (x: ^T when ^T : (member GetName: unit -> string)) =
x.GetName()
|> Console.WriteLine

F# vNext
Applicative functors
validate {
let! name = validateName(name)
let! age = validateAge(age)
let! email = validateEmail(email)
return User(name, age, email)
}

F# vNext
validate {
let! name = validateName(name)
and! age = validateAge(age)
and! email = validateEmail(email)
return User(name, age, email)
}

F# vNext
parallel {
let! x = slowRequestX()
and! y = slowRequestY()
and! z = slowRequestZ()
return f(x, y, z)
}

THANKS
@antyadev
antyadev@gmail.com
@nbombertest
http://nbomber.com

.NET Fest 2018. Антон Молдован. One year of using F# in production at SBTech

Recomendados

Recomendados

Mais conteúdo relacionado

Semelhante a .NET Fest 2018. Антон Молдован. One year of using F# in production at SBTech

Semelhante a .NET Fest 2018. Антон Молдован. One year of using F# in production at SBTech (20)

Mais de NETFest

Mais de NETFest (20)

Último

Último (20)

.NET Fest 2018. Антон Молдован. One year of using F# in production at SBTech