Introduction to Flux and Functional Data Scripting

Scott Anderson
Sr. Technical Writer & Tech Lead
September 2021
Introduction to Flux &
Functional Data Scripting

© 2021 InﬂuxData. All rights reserved.
2
What is Flux?
A functional data scripting
language designed to query,
process, and act on data.

3
Design Goals
– Turing-complete
– Useable
– Readable
– Flexible
– Extensible
– Testable
– Contributable
– Shareable

4
How does
it work?

5 © 2021 InﬂuxData. All rights reserved.
5

6

7

8

9

10

11
from(bucket: "system-data")
|> range(start: -1h)
|> filter(fn: (r) =>
r._measurement == "mem" and
r._field == "used_percent"
)
|> aggregateWindow(every: 5m, fn: mean)

12
Data
Structure

13

14

15
Stream of tables

16
Table

17
version userID sessionID _ﬁeld _value
Table

18
v1 ex123 ex456 example 1.2
Table

19
Table
Group key: [ ]

20
Tables
Group key: ["version", "userID" ]

21
Tables
Group key: ["version", "userID" ]

22
sql.from(
driverName: "mysql",
dataSourceName: "${username}:${password}@tcp(localhost:3306)/db",
query:"SELECT CustomerName,City FROM Customers;"
)
CustomerName City
John Doe New York
Jane Seymour London
Rufus Peck San Francisco
Group key: [ ]

23
from(bucket: "example-bucket")
|> filter(fn: (r) => r._measurement == "m1" and r.field == "f1")
_start _stop _time _measurement host _ﬁeld _value
2021-01-01T00:00:00Z 2021-01-01T01:00:00Z 2021-01-01T00:09:00Z m1 host1 f1 1.2
2021-01-01T00:00:00Z 2021-01-01T01:00:00Z 2021-01-01T00:28:00Z m1 host1 f1 2.5
2021-01-01T00:00:00Z 2021-01-01T01:00:00Z 2021-01-01T00:39:00Z m1 host1 f1 2.2
2021-01-01T00:00:00Z 2021-01-01T01:00:00Z 2021-01-01T00:18:00Z m1 host2 f1 1.2
2021-01-01T00:00:00Z 2021-01-01T01:00:00Z 2021-01-01T00:24:00Z m1 host2 f1 2.5
2021-01-01T00:00:00Z 2021-01-01T01:00:00Z 2021-01-01T00:42:00Z m1 host2 f1 2.2
Group key: ["_start", "_stop", "_measurement", "host", "_ﬁeld"]

24
Flux
Syntax

25
Pipe-forward
|>

26
Basic data types
string "example string"
bool true
float 1.20
int 123456789
uint 123456789
duration 1h
time 2019-12-01T00:00:00Z
bytes [104 101 108 108 111]

27
Composite data types
record {key1: "value1", key2: 2.0}
array [12, 34, 56]
dict [1: "foo", 2: "bar"]
function f = (x) => x

28
Variable declarations
varName = varValue
foo = "bar"
bazRecord = {this: 1, that: 8}
data = from(...)

29
Reference composite types
r = {key1: "value1", key2: 2.0}
Dot notation
r.key1
Returns "value1"
Bracket notation
r["key1"]
Returns "value1"

30
arr = [12, 34, 56]
Bracket notation
arr[0]
Returns 12

31
dictEx = [1: "foo", 2: "bar"]
Function reference
dict.get(dict: dictEx, key: 1, default: "baz")
Returns "foo"

32
dictEx = [1: "foo", 2: "bar"]
Function reference
dict.get(dict: dictEx, key: 3, default: "baz")
Returns "baz"

33
r = {key1: "value1", key2: 2.0}
key = "key1"
r.key Returns an error
r["key"] Returns an error
r["${key}"] Returns an error
r = ["key1": "value1", "key2": "2.0"]
key = "key1"
dict.get(
dict: r,
key: key,
default: ""
) Returns "value1"

34
Functions
f = (x, y) => x / y
f(x: 12, y: 2)
Returns 6

35
Function types
Static
f = (x, y=2) => x / y
Transformation
f = (t=<-) => t
Predicate
(r) => r.key1 == "foo"

36
Predicate functions
(r) => 1.0 == 1.0
Parameter(s) Predicate Expression

37
Predicate functions
(r) => true

38
Predicate functions
(r) => 1.0 != 1.0

39
Predicate functions
(r) => false

40
{
Predicate functions
(r) => 1.0 > 1.0
== != < > <= >= =~ !~

41
Predicate functions
(r) => 1.0 == 1.0 and "foo" != "bar"

42
Predicate functions
(r) => true and true

43
Predicate functions
(r) => true

44
Predicate functions
(r) => 1.0 > 1.0 and "foo" != "bar"

45
Predicate functions
(r) => false and true

46
Predicate functions
(r) => false

47
Predicate functions
(r) => 1.0 > 1.0 or "foo" != "bar"

48
Predicate functions
(r) => true

49
Predicate functions
(r) =>
1.0 == 1.0 and
("foo" == "bar" or "baz" != "quz")

50
Predicate functions
r = {fname: "John", lname: "Doe", age: 42}
(r) => r.fname == "John" and r.age > 40

51
Predicate functions
r = {fname: "Mike", lname: "Smith", age: 29}
(r) => r.fname == "John" and r.age > 40

52
Basic
Query

53
Basic Query
Source

54
Basic Query
sql.from(
driverName: "postgres",
dataSourceName: "...",
query: "..."
)
Source

55
Basic Query
csv.from(csv: "...")
Source

56
Basic Query
Source

57
Basic Query
r._field == "used_percent")
Source
Filter

58
Basic Query
|> group(columns: ["host"])
Source
Filter
Shape

59
Basic Query
|> mean()
Source
Filter
Shape
Process

60
Basic Query
|> mean()
Source
Filter
Shape
Process

61
Process Data
data = from(bucket: "system-data")
r._field == "used_percent"
)

62
2021-01-01T00:00:00Z 2021-01-01T01:00:00Z 2021-01-01T00:09:00Z mem host1 used_percent 62.1
Process Data

63
Process Data
data
|> map(fn: (r) => ({_time: r._time, _value: r._value}))

64
_time _value
2021-01-01T00:09:00Z 62.1
2021-01-01T00:18:00Z 44.0
2021-01-01T00:24:00Z 52.4
2021-01-01T00:28:00Z 63.8
2021-01-01T00:39:00Z 65.4
2021-01-01T00:42:00Z 50.7
Process Data

65
Process Data
data
|> map(fn: (r) => ({r with _value: r._value * 0.01}))

66
Process Data

67
Process Data
data
|> map(fn: (r) => ({
r with
state: if r._value > 65.0 then "high" else "ok"
}))

68
_start _stop _time _measurement host _ﬁeld _value state
2021-01-01T00:00:00Z 2021-01-01T01:00:00Z 2021-01-01T00:09:00Z mem host1 used_percent 62.1 ok
2021-01-01T00:00:00Z 2021-01-01T01:00:00Z 2021-01-01T00:39:00Z mem host1 used_percent 65.4 high
_start _stop _time _measurement host _ﬁeld _value state
Process Data

69
Process Data
data |> mean()

70
|> mean()

71
Process Data

72
_start _stop _measurement host _ﬁeld _value
2021-01-01T00:00:00Z 2021-01-01T01:00:00Z mem host1 used_percent 63.8
Process Data

73
Process Data
data |> sum()

74
Process Data

75
Process Data
data |> max()

76
Process Data

77
Process Data
data |> last()

78
Process Data

79
Flux in
Practice

80
from(bucket: "market-summary")
|> range(start: -6mo)
r._measurement == "stockPrices" and
r.symbol == "GME"
)
|> holtWinters(n: 24, seasonality: 12, interval: 1w)

81
heatIndex = (t, h) => //...
from(bucket: "sensor-data")
|> range(start: -1d)
|> filter(fn: (r) => r._measurement == "sensors")
|> filter(fn: (r) => r._field == "temp" or r._field == "hum")
|> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value")
|> map(fn: (r) => ({ r with heatIndex: heatIndex(t: r.temp, h: r.hum) }))

82
import "sql"
sensorInfo = sql.from(
driverName: "postgres",
dataSourceName: "postgresql://localhost?sslmode=disable",
query: "SELECT * FROM sensors"
)
sensorMetrics = from(bucket: "example-bucket")
|> filter(fn: (r) => r._measurement == "airSensors")
join(tables: {metric: sensorMetrics, info: sensorInfo}, on: ["sensor_id"])
|> group(columns: ["sensor_id", "lastInspected"]),
|> aggregateWindow(
every: 5m,
fn: (tables=<-, column="error") => tables |> count(column: column)
)

October 11-12, 2021
Hands-On Flux Training
October 26-27, 2021
Virtual Experience
We look forward to bringing together our
community of developers to learn, interact and
share tips and use cases.
https://www.inﬂuxdays.com/inﬂuxdays-north-america-2021-virtual-experience/

Introduction to Flux and Functional Data Scripting

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a Introduction to Flux and Functional Data Scripting

Semelhante a Introduction to Flux and Functional Data Scripting (20)

Mais de InfluxData

Mais de InfluxData (20)

Último

Último (20)

Introduction to Flux and Functional Data Scripting