Strata hadoop world 2015 context computing - jonas keynote - final

© 2015 IBM Corporation
Context Computing
Strata + Hadoop World 2015
Jeff Jonas, IBM Fellow
Chief Scientist, Context Computing
http://www.twitter.com/jeffjonas
www.jeffjonas.typepad.com

Jeff Jonas
IBM Fellow
Chief Scientist, Context Computing
 Founded Systems Research & Development (SRD) in 1985
 Architected, designed, developed roughly 100 systems over
the last three decades
– Financial services
– Defense, intelligence
– Manufacturing
– Humanitarian efforts
 Acquired by IBM in 2005
 Currently focused on Context Computing, Sensemaking and
Privacy by Design

No Context
Newsletter
Subscriber

Context
“Better understanding something by taking into
account the things around it.”

I ducked as the bat flew my way.
Another exciting baseball game.

In Context
Social
Media
Influencer
Newsletter
Subscriber
Loyalty
Club Member
High Value
Customer
Job
Applicant Watch
Listed Party

Context Accumulating
Context
Accumulation
Contextualized
Observations
Observation
(Any kind of data from
any kind of sensor)

Context Informs Decisioning
Context
Accumulation
Contextualized
Observations
Observation
In Context
Decisioning
Act
Data Finds Data Relevance Finds You
Observation
(Any kind of data from
any kind of sensor)

The Puzzle Metaphor
 Imagine an ever-growing pile of puzzle pieces of varying sizes, shapes, colors
 What it represents is unknown – there is no picture on hand
 Is it one puzzle, 15 puzzles, or 1,500 different puzzles?
 Some pieces are duplicates, missing, incomplete or have errors
 Some pieces may even be professionally fabricated lies
 Until you take the pieces to the table, it is nearly impossible to assess the scene

Puzzling Images: Courtesy Ravensburger © 2011
270 pieces
90%
200 pieces
66%
150 pieces
50%
6 pieces
2%
30 pieces
10%
(duplicates)

First Discovery

More Data Finds Data

Duplicates in Front Of Your Eyes

First Duplicate Found Here

Incremental Context – Incremental Discovery
6:40pm START
22min “Hey, this one is a duplicate!”
35min “I think some pieces are missing.”
37min “Looks like a bunch of hillbillies on a porch.”
44min “Hillbillies, playing guitars, sitting on a porch, near a barber sign
and a banjo!”

150 pieces
50%

Incremental Context – Incremental Discovery
47min “We should take the sky and grass off the table.”
2hr “Let’s switch sides, and see if we can make sense of this from
different perspectives.”
2hr10m “Wait, there are three … no, four puzzles.”
2hr18m “I think you threw in a few random pieces.”

How Context Accumulates
 With each new observation one asserts: 1) unrelated; 2) related; or 3) connected
 Must favor the false negative
 New observations sometimes reverse earlier assertions
 Some observations produce novel discovery
 The emerging picture helps focus collection interests

Big Data [in context]. New Physics.
More data: better the predictions
– Lower false positives
– Lower false negatives
More data: bad data good
– Suddenly glad your data is not perfect
More data: less compute

Big Data
Pile of ______ Information In Context

One Essential Form of Context: “Entity Resolution”
 Is it 5 people each with 1 account or is it 1 person with 5 accounts?
 Is it 20 cases of SARS in 20 cities or one case reported 20 times?
 If one cannot count, one cannot estimate vector or velocity (direction, speed).
 Without vector and velocity prediction is nearly impossible.

Who is Fang Wong?
Fang Wong
Top 100 Customer
F A Wong
Seattle, DOB: 6/12/82
Former Customer
@FangWong
2.5M Followers
FangWong@Email.com
Newsletter Subscriber
Fang Wong
FangWong@Email.com
Marketing Department’s
Prospect List

Resolving the Fang Wong
Fang Wong
Top 100 Customer
F A Wong
Seattle, DOB: 6/12/82
Former Customer
@FangWong
2.5M Followers
FangWong@Email.com
Fang Wong
FangWong@Email.com
Marketing Department’s
Prospect List

Resolving the Fang Wong
Fang Wong
Top 100 Customer
2.5M Followers

Graphing the (resolved) Fang Wong
Bill Smith
Member of the Board
Employee
Customer
Customer
Fraudster
Fang Wong
Top 100 Customer
2.5M Followers

Contextualizing Sandy Maden
Bill Smith
Member of the Board
Sandy Maden
New Account
Employee
Lives With
Co-signer
Former
Customer
Customer Customer
Customer
Fraudster
Fang Wong
Top 100 Customer
2.5M Followers

“Entities”
Bill Smith
Member of the Board
Lives With
Co-signer
Sandy Maden
New AccountFormer
Customer
Employee
Customer Customer
Customer
Fraudster
Fang Wong
Top 100 Customer
2.5M Followers
Company
Boat
Plane
Asteroid
Car

Asteroid Hunting

Single Detection
Image courtesy of: Eva Lilly, Institute of Astronomy, University of Hawaii

From Orphans to Orbits
Single
Detections
(trash)
Tracklette
Track
Orbit
Forecasting
Named entity: S100ZUtza
Single Detection
(orphan)
Anticipation

http://www.space.com/7854-slam-asteroids-suspected-space-collision.html

"We have directly
observed a collision
between asteroids for
the first time, instead of
having to infer that
they happened from
million-year-old
remains."
Colin Snodgrass
Planetary Scientist
Max Planck Institute for Solar System
Research

Geospatial Context via “Space Time Boxes”

Detecting Colocation
TIME
1 day
1 hour
Determine
encounter
distance
and time
Space Time Boxes

Computing 600k Asteroid Interactions over 25 Years
4-5 orders of magnitude improvement
Initial Analysis
Adding 1 New Trajectory
Space-Time Box Method
2,880 CPU hours
15 CPU minutes
N-body Simulation Method
10,000,000 CPU hours
4,000 CPU hours

Asteroid vs. Asteroid Encounters
Encounter Distance Asteroid 1 Size Asteroid 2 Size
May 1, 2032 299km 00A9170 2-4km 0008758 4-9km
Nov 24, 2016 449km 00P5634 1-2km 0055711 2-5km
Jan 11, 2018 449km K08E88J 530-1200m 00N0062 2-4km

June 12th, 2015
Hi Jeff & the gang,
I have great news! On Tuesday I happened to observe a close encounter
you guys predicted - one 1 km and the other one 2 km in diameter!
To my knowledge this is the first case ever of direct observation of a close
encounter in the small main belt asteroids. The closest point of encounter
unfortunately happened during bright daylight in Hawaii, so I missed it …
Cheers!
Eva -

Image courtesy of: Eva Lilly, Institute of Astronomy, University of Hawaii

[Theatrical Pause]

Action
Red Analytics
Green Analytics
Blue Analytics
Observation
Space
Old School: Isolated Analytics

Observation
Space
ActionInformation
In Context
Next: General Purpose Context Computing
Context Computing

Observation
Space
ActionInformation
In Context
Context Computing
Helping Focusing Human Attention
General Purpose
• Marketing
• Customer service
• Fraud detection
• Asteroid hunting
Simultaneously!

Making Data Work: Recommendations
 Widen the observation space
 Accumulate context to improve understanding
 Deliver significantly higher quality outcomes … everywhere
– Life sciences
– Financial services
– Public safety
 Leverage Hadoop/Spark to accelerate innovation

More
 Blog: www.jeffjonas.typepad.com
 Email: JeffJonas@us.ibm.com
 Next: San Francisco, Nov 10-12, @Datapalooza

Strata hadoop world 2015 context computing - jonas keynote - final

Recomendados

Recomendados

Mais conteúdo relacionado

Destaque

Destaque (14)

Semelhante a Strata hadoop world 2015 context computing - jonas keynote - final

Semelhante a Strata hadoop world 2015 context computing - jonas keynote - final (20)

Último

Último (20)

Strata hadoop world 2015 context computing - jonas keynote - final