Investigative journalists use graphs and graph databases like Neo4j to connect disparate pieces of data and uncover hidden relationships. The Panama Papers investigation involved loading over 2.6 TB of leaked data into Neo4j to allow over 370 journalists from 80 countries to collaborate and find connections between entities, addresses, intermediaries and officers. Visualizing the data in Neo4j helped journalists tell the full story and have a global impact, exposing offshore dealings of world leaders and others.
6. Our world today
• Billions of exchanges
• Messages, data, transactions
• Sometimes hidden in plain sight
• Between people, companies,
organizations, governments
• Which includes criminals
18. • Individual Reporters
• Working the Paper trail
• Much like an detective story
• Long turnaround times
• Local impact
• Large amounts of data were not easily shareable
Investigative Journalism
19. Investigative Journalism Today
• Benefit from whistleblowers & leaks
• Sharing a large amount of data
• Use technology and data engineers
• Collaborate globally (Trust!)
• Corrobate suspicions with
other sources locally
• Affects the world at large
• „Golden Age of IJ, never been as important“
20. Organization of ca. 200 journalists
Based in 65 countries
“Our aim is to bring journalists from different countries
together in teams - eliminating rivalry and promoting
collaboration. Together, we aim to be the
world’s best cross-border investigative team.”
icij.org/about
24. Panama Papers Timeline
• early 2015
First contact John Doe with SZ
• Spring 2015
Involving ICIJ
• Summer 2015
Start of investigations
• April 4. 2016
Public Launch
26. Exposed the offshore holdings
of 12 current and former
world leaders.
Dealings of 128 more
politicians and public officials
around the world.
Exposure of hidden secrets
32. Nuix Investigator
• OCR
• Entity Extraction
• Analytical tools
• Philantrophic Donation
to ICIJ
Nuix is an Australian (founded 2000 Sydney) company focused on data
extraction from unstructured sources.
33. 3 million files
x
10 seconds per
file
=
1 yr / 35 servers
= 1.5 weeks
Nuix
Investigator
41. Need to store and query
our connections!
Real, inferred and integrated
42. Neo4j
A native graph database
• Manage and store your
connected data as a graph
• Query relationships
easily and quickly
• Evolve model and applications
to support new requirements and
insights
• Built to solve relational pains
44. NODE
key: “value”
properties
Property Graph Model
Nodes
• The entities in the graph
• Can have name-value properties
• Can be labeled
Relationships
• Relate nodes by type and direction
• Can have name-value properties
RELATIONSHIP
NODE NODE
key: “value”
properties
key: “value”
properties
key: “value”
properties$
45. Neo4j: All about Patterns
(:Person { name:"Dan"} ) -[:KNOWS]-> (:Person {name:"Ann"})
KNOWS
Dan Ann
NODE NODE
LABEL PROPERTY
neo4j.com/developer/cypher
LABEL PROPERTY
48. Cypher: Find Patterns
MATCH (:Person { name:"Dan"} ) -[:KNOWS]-> (who:Person) RETURN who
KNOWS
Dan ???
LABEL
NODE NODE
LABEL PROPERTY ALIAS ALIAS
neo4j.com/developer/cypher
50. Getting Data into Neo4j
• Bulk Load from CSV Files
• Update Graph from
• Web APIs (JSON,XML)
• Other Databases
• CSV Files
• User Activity (Logs, Callbacks)
,
,,
51. Import Demo – CSV dump
==> /Users/mh/Downloads/panama/import/Addresses.csv <==
address,icij_id,valid_until,country_codes,countries,node_id:ID,sourceID
27 ROSEWOOD DRIVE #16-19 SINGAPORE 737920,6991059DFFB057DF310B9BF31CC4A0E6,The Panama Papers data is current through
2015,SGP,Singapore,14000001,Panama Papers
==> /Users/mh/Downloads/panama/import/Entities.csv <==
name,original_name,former_name,jurisdiction,jurisdiction_description,company_type,address,internal_id,incorporation_date,inactivation_date,
struck_off_date,dorm_date,status,service_provider,ibcRUC,country_codes,countries,note,valid_until,node_id:ID,sourceID
"TIANSHENG INDUSTRY AND TRADING CO., LTD.","TIANSHENG INDUSTRY AND TRADING CO., LTD.",,SAM,Samoa,,ORION HOUSE SERVICES (HK) LIMITED ROOM 1401;
14/F.; WORLD COMMERCE CENTRE; HARBOUR CITY; 7-11 CANTON ROAD; TSIM SHA TSUI; KOWLOON; HONG KONG,1001256,23-MAR-2006,
18-FEB-2013,15-FEB-2013,,Defaulted,Mossack Fonseca,25221,HKG,Hong Kong,,The Panama Papers data is current through 2015,10000001,Panama Papers
==> /Users/mh/Downloads/panama/import/Intermediaries.csv <==
name,internal_id,address,valid_until,country_codes,countries,status,node_id:ID,sourceID
"MICHAEL PAPAGEORGE, MR.",10001,MICHAEL PAPAGEORGE; MR. 106 NICHOLSON STREET BROOKLYN PRETORIA 0002; GAUTENG (PWV) SOUTH AFRICA,
The Panama Papers data is current through 2015,ZAF,South Africa,ACTIVE,11000001,Panama Papers
==> /Users/mh/Downloads/panama/import/Officers.csv <==
name,icij_id,valid_until,country_codes,countries,node_id:ID,sourceID
KIM SOO IN,E72326DEA50F1A9C2876E112AAEB42BC,The Panama Papers data is current through 2015,KOR,"Korea, Republic of",12000001,Panama Papers
==> /Users/mh/Downloads/panama/import/all_edges.csv <==
node_id:START_ID,rel_type:TYPE,node_id:END_ID
11000001,intermediary of,10208879
57. Data is available
• 785,000 Offshore Leaks Companies from several investigations
• For online browsing and visualization
• offshoreleaks.icij.org
• As CSV dump download
• As Neo4j Database download
• offshoreleaks.icij.org/pages/database
• As Neo4j sandboxes sandbox.neo4j.com
58.
59. Data exposed as interactive Visualization
• Public figures and leaders
• Different shell companies & involvements
63. More steps for the ICIJ and all of us
• Data integration with other sources
• Entity extraction
• Email pattern analysis
• Content & Data mining
• Machine learning
• Alerts with real time news / social media
• Investigative recommendations
• Active search for new sources ...
64. Current Investigation of Mossack Fonseca
Jürgen Mossack – Ramon Fonseca
Arrested, free on bail, ongoing investigations
65. • Flow to US tax havens
• Transparancy laws UK, EU
• $1.3bn taxes recouped
publicly
• Investigations into banks, public figures, companies
• Criminal cases solved & ongoing
World Wide Results of the Offshore leaks investigations
icij.org/investigations/panama-papers/panama-papers-helps-recover-more-than-1-2-billion-around-the-world/
67. Most recent SEB Bank Investigation in Sweden
• Operations of
Nordic banks
In Baltic states
• Danske Bank,
Swedbank, SEB
• Resignations,
Investigations
• Straw-men mentioned in reporting can be found
in offshore leaks db
• SEB bank in Nov 2019 !
68. Murder Daphne Caruana Galizia
• Maltese Investigative
Reporter
• Reported on Panama Papers
appearances of influential
Maltese Politicians
• Murdered Oct 16 2017
with car bomb
• „The Daphne Report“
• PM finally resigning
en.wikipedia.org/wiki/Daphne_Caruana_Galizia
69. Murder Jan Kuciak
• Slovak Investigative
Reporter
• Reported on criminal
behavior of businessmen
• Jan & fiancé shot Feb 2018
• Massive reactions +
political crisis
• PM and several ministers
resigned
en.wikipedia.org/wiki/Murder_of_J%C3%A1n_Kuciak
87. Graph & ML Algorithms in Neo4j+35
neo4j.com/
graph-algorithms-
book/
Pathfinding
& Search
Centrality /
Importance
Community
Detection
Link
Prediction
Finds optimal paths
or evaluates route
availability and quality
Determines the
importance of distinct
nodes in the network
Detects group
clustering or partition
options
Evaluates how
alike nodes are
Estimates the likelihood
of nodes forming a
future relationship
Similarity