This document discusses architectures for AI systems and big data engineering. It addresses topics like what constitutes data engineering; how to store and process event data using pipelines; when to use batch vs stream processing; how AI models can be integrated into data pipelines; and considerations for engineering systems involving big data and AI like non-functional requirements, reprocessing data as models improve, and storing model outputs. The overall aim is to provide an overview of key concepts in engineering systems that handle large volumes of data and incorporate artificial intelligence.

1. Architectures of AI systems
Engineering for Big Data & AI
HCMC, Sep 6th 2019 herve@quod.aiHerve Roussel

2. What is
Data Engineering ?

3. Is this data engineering?
UploadData.java
upload_data.py

4. cat console.log
| grep “ERROR”
> errors.log
Is this data engineering?

5. Data engineering?
Transformed dataEvent data
Program

6. Backend vs Data?

7. cat console.log
| grep “ERROR”
> errors.log
Is this data engineering?
Event data
Transform
Transformed data

8. What is
Big Data Engineering ?

9. Where is Big Data?

10. How to query news feed?
SELECT
*
FROM posts
INNER JOIN friends
WHERE ...
ORDER BY
posts.timestamp DESC

11. Notify? Web,
mobile?
Who can
see this?
Racist? Vulgar?
Is this a face? Who’s
this? Friend? Celebrity?
Courtney likes. Is that
good or bad?
Paddy commented. Is
that good or bad?
Chris posted. Is that
good or bad?
Anybody tagged?
What rank
in feed?

12. Copyright violation?

13. Is Big Data just for big companies?
300K QPS [R]
6K QPS [W]
As of JULY 8, 2013
1B+ QPM [P]
250M+ QPM [R]
400M LOC [P]
1.8 TB per year [P]

14. Data Engineering
Augmented dataEvent data
Program

15. Event data
Transform
Augmented data
Big Data Engineering + AI

16. Pipeline (Transform)
Source (Event data)
Sink (Augmented data)

17. What is a
source ?

18. Synchronous_
( 10-100 ms )_
Where is data coming from?
Main data
Event source
Why split?
Asynchronous_
( 3-5 s )_

19. What’s in an event data?
Post
{
id: 12345,
content: “hello world”,
created_at: …
updated_at: …
author_id: 67890,
…
}
PostCreatedEvent
{
story_id: 12345,
type: “story_posted”
…
}

20. Job 1
Job 2
Scheduler
What’s batch processing?

21. Which DB for event source?

22. ● Volume?
● Velocity? QPS reads? QPS writes?
● Latency?
● Cost? Storage & R/W
● How to write?
○ Integrity?
○ Consistency?
○ Durability?
○ Version?
● How to read?
○ Random access or sequential?
○ Full text search?
○ Geo distance?
How to store events?

23. MySQL MongoDB JSON on S3 (or
GCS)
30 GB OK Good Very good
10K WPS OK Good Very good
1K RPS OK Good Very good
Range read OK Good Very good
Cost $$ $$$ $
MySQL MongoDB
30 GB OK Good
10K WPS OK Good
1K RPS OK Good
Sequential read OK Good
Cost $$ $$$
How to store events?

24. Who wants to become architect?

25. Job 1
Job 2
Scheduler
What’s the problem with batch?
LATENCY

26. How to process real-time?
Stream processing

27. How can 2 processes talk?

28. QUEUE
Why not use database?

29. Importance MySQL Kafka Redis
10K WPS 1.0 5 10 10
1K RPS 1.0 5 10 10
Sequential
read
1.0 10
(with B-TREE)
10 10
(using Lists)
Order
guarantee
0.2 10 0 10
Durability 0.1 10 5 (but perf. hit) 0
Deployability 0.5 10 5 7.5
Score 5.6 / 10 6.6 / 10 7.15 / 10
Why not database?

30. What is a
transform ?

31. Transforms
Source
Sink

32. Functional vs OOP
Librarian
.startShift()
Catalog.open() Library.close()
Books.create()
Operations on things
Add more things
find(book)
assign(book)
Things with operations
Add more operations
remove(book)
load_cover(book)

33. Functional vs OOP
find_similar(vid_uploaded)
transcribe_captions(vid_uploaded
)
Things with operations
Add more operations
alert_subscribers(vid_uploaded)
generate_thumbnails(vid_uploaded
)

34. What’s supporting data?
Transform
Supporting data
event
{
id: 12345,
type: “story_posted”
user_id: 67890
coordinates: [ 10.76, 106.66
]
}
Friends or city DB

35. Who uses ext. supporting data?

36. API vs Pipeline: availability?
Requests in thread Long running

37. API vs Pipeline: performance?
100ms
⇓
10ms
100ms * 300,000/60/60 = 9H
⇓
10ms * 300,000/60/60 = 55 min

38. Where is the data coming from?
Is this a face? Who’s
this? Friend? Celebrity?

39. Data pipelines & AI
TransformAI model

40. How can 2 processes talk?
Transform
AI model

41. What is a
sink ?

42. Which DB to sink to?

43. What to do with the sink?
Write Read
Data scientist
Sales

44. What are the read use cases?
Give me summary
report of last
month’s activity
Give me posts that
contain the words
Donald Trump,
Trump or President
Give me all posts by
female, age 18-35
Aggregation Full text search Bulk data, filtered

45. ACID

46. Denormalization: good or bad?

47. What is BCNF?

48. What’s distributed data systems?

49. Why re-run the pipeline?
TransformAI model Transform v2

50. Idempotency & backfill
f(f(x)) = f(x)
POST “/BankAccount/AddFunds”
{ value: 1000, token: TX123 }

51. Another reason for backfill?

52. What if the AI model improves?
TransformAI model v2

53. AI systems ≠ traditional systems?
93.2%
ProbabilisticDeterministic

54. Store output of model v1 or v2?
AI Model v1
( accuracy: 83.1% )
AI Model v2
( accuracy: ?? )

55. What have we
learned ?

56. Source: Uber Engineering
[DE] Collect data
[DE] Process data
[DS] Build DL model
[BE/FE] Use DL model in app
[DA] Validate DL model

57. Which NFR for Big Data?
• Scalability
• Availability
• Interoperability
• Portability
• Modifiability
• Maintainability
• Testability
• Usability
• Buildability
• Deployability
• Ease of Development
• Performance
• Security
• Localization
• Legal
• Reusability
• Supportability
• Monitorability

58. • Deployability
• Ease of Development
• Performance
• Security
• Localization
• Legal
• Reusability
• Supportability
• Monitorability
Which NFR for Big Data?
• Scalability
• Availability
• Interoperability
• Portability
• Modifiability
• Maintainability
• Testability
• Usability
• Buildability

59. Main data
+
Materialized view
Event data
⇓
Pipeline
⇓
Augmented data
What have we learned?

60. Want to learn more about
AI & Big Data?
We’re hiring:
● Big Data Engineer, in training (Java)
● Big Data Engineer (Java)
● Data Scientist (Python)
http://bit.ly/quod-ai-join
herve@quod.aiHerve Roussel