인공지능, 모바일, 클라우드 등 현대적인 온라인 서비스를 만드는데 다양한 Challenge가 있습니다. 본 발표에서는 이런 문제를 해결하는데 있어서 Data Science가 어떤 역할을 할 수 있는지를

1. Data Science for Online Services:
Problems & Frontiers
Jinyoung Kim
Data & Analytics
Naver Search US
2021 Changbal Conference

2. About Me
Head of Data Science @ Naver Search
Executive Director of Naver Search US
Ex-MSFT / Ex-Snap (Search & RecSys)
Co-founder / 1st President of Changbal
* 한국/미국에서 Data Scientist & Engineer
채용중입니다! (jin.y.kim@navercorp.com)
https://medium.com/naver-dna-tech-blog

3. Mission: Making Data Science as Cool as AI

4. Mission: Making Data Science as Cool as AI

5. Challenges in Building
Modern Online Services

6. What constitutes a modern online service?
AI-
enabled
Mobile-
first
Cloud-
backend
Powerful
algorithms
Limitless
Computing
Infrastructure
Contextual
awareness,
Omnipresence

7. Challenge#1: Dynamic Environment
Societal and
environment
changes
New apps
and services
launch daily
Underpinning
technologies
evolve
COVID-19
Climate
Change
GPT-3 and large-
scale language
models Yelp
Pinterest
AirBnB
Election
Breakthrough in
Computer Vision
TicTok
DoorDash

8. Solution: Metric Regression Detection
Source: KDD’20 Paper from Microsoft

9. Challenge#2: Ecosystem / Social Impact
• Online services need to look
beyond user satisfaction
• Search / RecSys results can instill
and reinforce social bias
• Algorithmic ranking means
life/death for content providers
• Manipulating algorithmic results
became an industry

10. Solutions to Bias / Fairness in Ranking
Source: http://naversearchconf.naver.com/

11. Challenge#3: Internal Alignment
• Major online service
companies have dozens
of teams with different
focus
• Blind optimization in one
area can lead to negative
user experience

12. Modern AB Test for Naver Search (DeView’21)

13. Data Science Problems
for Online Services

14. Problems & Solutions by Service Lifecycle
• Each stage present different analysis and decision problems,
with corresponding data science solutions
PLAN DEVELOP LAUNCH MONITOR
Defect
Monitoring
Competitive
Analysis
Offline
Experiment
A/B
Experiment
KPI
Monitoring
Opportunity
Analysis
Parameter
Optimization
Crowdsourced
Evaluation

15. DS Problems during Planning Stage
Questions
• How can we acquire &
onboard new users?
• What are the pain points
with existing customers?
• Where are we lagging
behind competing
services?
Solutions
• User funnel / journey
analysis
• Side-by-side comparison
• In-depth user study

16. Where do users come & go? Funnel Analysis
• New Users for E-Commerce • Existing Users for Contents
Discovery (Search / App Store)
Connect / Install
Create Account
Checkout
Revisit
App Open
Page View (browsing)
Page View
Consume
Revisit
Where do most users drop off the funnel?
Which part of the funnel has the biggest leverage?
How is the funnel shifting over time?

17. Beyond Funnel: User Journey Analysis
• Sanky chart can visualize diverging / converging user journey
Source: Medium

18. Example: Analyzing Snapchat User Journey
• Use app events to build & predict user journey graph
Source: Characterizing and Forecasting User Engagement
with In-app Action Graph: A Case Study of Snapchat

19. Example: Analyzing Snapchat User Journey
• Use these to predict and optimize user satisfaction / retention

20. Applications for
Naver Mobile App
• What does overall user
journey look like?
• Can we understand
which factors improve /
hinder user satisfaction?
• Can we optimize user
experience, thereby
improving KPI (DAU)?

21. DS Problems during Dev. / Launch Stage
Questions
• Is the new
design/ranking better
than the old one?
• How can we choose the
best design/ranking
parameters?
Solutions
• Online (A/B)
experiment
• Multi-armed Bandit

22. Before/After Comparision vs. AB Test
• Hard to measure true impact of given feature
• High user impact in case of full launch & roll-back
Control Treatment
External Factors
(day-of-week / seasonality / ...)
Full Launch Roll-back
Control
100%
Traffic

23. Before/After Comparison vs. AB Test
• AB testing allows measurement w/o impact of external factors
• Multiple treatments with versioning & roll-out support
Control
Treatment1 (V1)
AB Test V1
External Factors (*)
(* no impact under randomized controlled experiment)
Roll-out &
Monitoring
Control
Treatment1 (V2)
AB Test V2
Treatment2
(gradual roll-out)
Problem
Solving
10~20%
Traffic
Treatment2 (V1) Treatment2 (V2)

24. Pitfalls in Real-world AB Testing (DeView’21)

25. Modern AB Test for Naver Search (DeView’21)

26. Can we use the user
bucketing idea for
dynamically optimize AI
model / UI parameters?
But I have hundreds of
parameters to choose
from. I’d also like to
minimize user impact

27. Contextual Bandits for Parameter Optimization
• Converge to the best action (max. reward) given a context

28. DS Problems during Monitoring Stage
Questions
•Has there been any
shift in user behavior /
metrics?
•Has the algo results
shown any new
defects?
Solutions
•Metric monitoring /
detection framework
•Results defect
detection framework

29. Metric Regression
Detection Framework
• Generating alerts for
metric regression is easy
• The hard part is
minimizing false positives
• Solutions to control false
discovery rate is available
Source: KDD’20 Paper from Microsoft

30. Defect Monitoring for Naver Search Results
Input Query
Set
Scraping &
Annotation
Automated
Quality Estimate
Collect Human
Quality Rating
Defect
Reporting
Continuous
Defect Monitoring
and Improvement
Manual reporting
Sample from traffic
SRP side-by-side
rating
Snippet rating
SRP scraping & parsing
Query taxonomy
SRP layout signals
Engagement signals
Query defect alerts
Defect weekly discussion

31. Closing Thoughts

32. Closing Thoughts
• Modern / AI-powered online services are
more powerful, but along came greater
challenges and responsibilities.
• Data Science provides various solutions
throughout the lifecycle of online
services
• The choice of techniques depends on
specifics of the service (Contents /
Commerce / Social network / …)
• Learning and starting career in data
science is also more accessible than ever
(bootcamps, better tools and guides)
AI-
enable
d
Mobile
-first
Cloud-
backend

33. We’re hiring @ Naver Search US!
• Exciting Data Science &
Artificial Intelligence problems
across Naver & Line
• Best of both Korean and US
tech working culture
• Huge growth opportunities
(150+ people in 3-5 years)
• Locations in Seattle & Bay
Area (+remote options)
https://naver-career.gitbook.io/en/