Intuidex - To be or not to be iid by William M. Pottenger (NYC Machine Learning Group)

 William M. Pottenger, Ph.D.
All Rights Reserved
To be or not to be IID:
That is the Question
Higher Order Learning
William M. Pottenger, Ph.D.
Rutgers University and Intuidex, Inc.
DrWMP@rci.rutgers.edu; www.dimacs.rutgers.edu/~billp
DrWMPottenger@Intuidex.com; www.intuidex.com

All Rights Reserved
Dr. William M. Pottenger
www.dimacs.rutgers.edu/~billp
www.intuidex.com
• Example Application Areas
– Homeland Security/Law
Enforcement/Criminal Justice
Information Systems
– Decision Support Systems
– Information Retrieval Systems
– High Performance Computing
• Research Funded by
– National Science Foundation
– National Institute of Justice
– Department of Homeland Security
– Army Research Lab
– Commonwealth of Pennsylvania
– Corporate Partners
– E.g., Lockheed-Martin, Kodak,
PNNL, Boeing, etc.
• Associate Research Professor
@ Rutgers University
– DIMACS & Computer Science
• CEO of Intuidex, Inc.
• Director of Transition for
DHS S&T CCI Center
• Research Scientist @ NCSA
• M.S., Ph.D. in CS at UIUC
• Research Interests
– Statistical Relational Learning
– Leveraging higher-order
relations in graphs of data
– Parallel and Distributed Visual
& Data Analytics
– Analytics in a parallel and/or
distributed environment
– Information Extraction
– Automatic extraction of
keywords/features from text
2

All Rights Reserved
What is Higher Order Information?
• Swanson (‘91) posed problem: Migraine headaches (M)
– stress associated with M
– stress leads to loss of magnesium
– calcium channel blockers prevent some M
– magnesium is a natural calcium channel blocker
– spreading cortical depression (SCD) implicated in M
– high levels of magnesium inhibit SCD
– M patients have high platelet aggregability
– magnesium can suppress platelet aggregability
• All extracted from medical journal titles
Slide reused with permission of Marti Hearst @ UCB
3

All Rights Reserved
Gathering Evidence
stress
migraine
CCB
magnesium
PA
magnesium
SCD
magnesiummagnesium
4

All Rights Reserved
Higher Order Paths!
migraine magnesium
stress
CCB
PA
SCD
5

All Rights Reserved
Related Work:
Link Mining and Collective Classification
 Link-based approaches (Taskar et al., 2001; Getoor and
Diehl, 2005; Lu and Getoor, 2003; Neville and Jensen
2004) to collective classification use explicit link
information within networked data
 Studies (Chakrabarti et al., 1998; Neville and Jensen,
2000; Taskar et al., 2001) have shown that collective
classifiers can achieve significant reductions in
classification errors by performing inference about
multiple data instances simultaneously
 Collective classifiers are context-dependent and are not
designed to classify stand-alone data instances
 We propose classification methods that leverage implicit
links between features in small training sets, and that
maintain the ability for “context-free” classification of
individual data instances
6

All Rights Reserved
Is there a theoretical basis for the use
of higher order co-occurrence relations?
• Research agenda: study machine learning
algorithms in search of a theoretical foundation
for the use of higher order relations
• First algorithm: Latent Semantic Indexing (LSI)
– Widely used technique in text mining and IR based on
the Singular Value Decomposition (SVD) matrix
factoring algorithm
– Research question: Does LSI use higher order term
co-occurrence?
– First step: study SVD
7
April Kontostathis
Associate Professor
@ Ursinus College

All Rights Reserved
Is there a theoretical basis for the use of
higher order co-occurrence relations in LSI?
s1
s2
s3
sr
A (m x n)

T (m x r) S (r x r) DT (r x n)
Term by Doc Term by
Dimension
Singular
Values
Dimension by Document
s1 <= s2 <= s3 <= . . . <=sr
r = rank of A, m = num terms, n = number docs
Singular Value Decomposition
8

All Rights Reserved
s1
s2
s3
sr
A (m x n)

T (m x k) S (k x k) DT (k x n)
Reduced
Term by Doc
Term by
Dimension
Singular
Values
Dimension by Document
s1 <= s2 <= s3 <= . . . <=sr
r = rank of A, m = num terms, n = number docs
LSI: Truncation of Singular Values
9

All Rights Reserved
human
interface
computer
user
system
response
time
EPS
Survey
trees
graph
minors
human x 1 1 0 2 0 0 1 0 0 0 0
interface 1 x 1 1 1 0 0 1 0 0 0 0
computer 1 1 x 1 1 1 1 0 1 0 0 0
user 0 1 1 x 2 2 2 1 1 0 0 0
system 2 1 1 2 x 1 1 3 1 0 0 0
response 0 0 1 2 1 x 2 0 1 0 0 0
time 0 0 1 2 1 2 x 0 1 0 0 0
EPS 1 1 0 1 3 0 0 x 0 0 0 0
Survey 0 0 1 1 1 1 1 0 x 0 1 1
trees 0 0 0 0 0 0 0 0 0 x 2 1
graph 0 0 0 0 0 0 0 0 1 2 x 2
minors 0 0 0 0 0 0 0 0 1 1 2 x
Deerwester Term by Term Matrix
human
interface
computer
user
system
response
time
EPS
Survey
trees
graph
minors
human x 0.54 0.56 0.94 1.69 0.58 0.58 0.84 0.32 -0.32 -0.34 -0.25
interface 0.54 x 0.52 0.87 1.50 0.55 0.55 0.73 0.35 -0.20 -0.19 -0.14
computer 0.56 0.52 x 1.09 1.67 0.75 0.75 0.77 0.63 0.15 0.27 0.20
user 0.94 0.87 1.09 x 2.79 1.25 1.25 1.28 1.04 0.23 0.42 0.31
system 1.69 1.50 1.67 2.79 x 1.81 1.81 2.30 1.20 -0.47 -0.39 -0.28
response 0.58 0.55 0.75 1.25 1.81 x 0.89 0.80 0.82 0.38 0.56 0.41
time 0.58 0.55 0.75 1.25 1.81 0.89 x 0.80 0.82 0.38 0.56 0.41
EPS 0.84 0.73 0.77 1.28 2.30 0.80 0.80 x 0.46 -0.41 -0.43 -0.31
Survey 0.32 0.35 0.63 1.04 1.20 0.82 0.82 0.46 x 0.88 1.17 0.85
trees -0.32 -0.20 0.15 0.23 -0.47 0.38 0.38 -0.41 0.88 x 1.96 1.43
graph -0.34 -0.19 0.27 0.42 -0.39 0.56 0.56 -0.43 1.17 1.96 x 1.81
minors -0.25 -0.14 0.20 0.31 -0.28 0.41 0.41 -0.31 0.85 1.43 1.81 x
Deerwester Term by Term Matrix, truncated to two dimensions
10

All Rights Reserved
• Answer is in the following theorem we proved:
If the ijth element of the truncated term by
term matrix, Y, is non-zero, then there exists a
co-occurrence path of order  1 between terms
i and j.
– Kontostathis, A. and Pottenger, W. M. (2006) A
Framework for Understanding LSI Performance.
Information Processing & Management, volume 42,
issue 1, pages 56-73.
• We have both proven mathematically and
demonstrated empirically that LSI is based on
the use of higher order co-occurrence relations.
• Next step?
11

All Rights Reserved
Using Higher Order Information in both
Generative and Discriminative Learning
• Extend the theoretical foundation that April and
I developed by studying characteristics of
higher-order information in other machine
learning approaches including both generative and
discriminative supervised learning as well as
unsupervised approaches
– Ganiz, M. C., Lytkin, N. I. and Pottenger, W. M.
(2009) Leveraging Higher Order Dependencies Between
Features for Text Classification. In the Proceedings
of the European Conference on Machine Learning and
Principles and Practice of Knowledge Discovery in
Databases (ECML PKDD). Bled, Slovenia, September.
Nikita Lytkin
Research Scientist @
NYU Medical Center
Murat Ganiz
Assistant Professor
@ Dogus University

All Rights Reserved
Representation of Boolean
Data by a Bipartite Graph
13

All Rights Reserved
Multinomial vs. Multivariate Event Model
McCallum & Nigam (1998)
14

All Rights Reserved
First Order Paths in a Data Graph
15

All Rights Reserved
Second Order Paths in a Data Graph
16

All Rights Reserved
Patterns of Connectivity between Features
17

All Rights Reserved
Probabilistic Characterization of Features by
Second Order Paths
18

All Rights Reserved
Higher Order Naïve Bayes:
A Generative Learner
Murat Ganiz
Assistant Professor
@ Dogus University
19

All Rights Reserved
20
Slonim & Tishby (2001) vs. HONB
Ganiz, M. C., Pottenger, W. M. and George, C. (2010) Higher Order
Naïve Bayes: A Novel Non-IID Approach to Text Classification. IEEE
Transactions of Knowledge and Data Engineering (TKDE).
multinomial features binary features
Dataset NB NB_wc improvement % NB HONB improvement %
COMP (5) 0.473 0.508 7.4 0.51 0.65 26.5
SCIENCE (4) 0.65 0.725 11.5 0.6 0.84 41.6
POLITICS (3) 0.62 0.67 8.1 0.68 0.83 22.8
RELIGION (3) 0.525 0.553 5.3 0.64 0.74 15.7
8.075 26.65
 HONB achieves statistically significantly better performance
than NB for four datasets based on t-test results
 (Slonim & Tishby, 2001) did not report std dev or t-test results

All Rights Reserved
Supervised Second Order Transformation
for Discriminative Learning
21
Nikita Lytkin Research
Scientist @ NYU Medical
Center

All Rights Reserved
Influence of Higher-Order Paths
22

All Rights Reserved
Experimental Setup
 Support Vector Machine (Vapnik 1998) was
used to evaluate the Supervised Second Order
Transformation
 Multi-class classification by SVM was
performed using the “one-against-one”
scheme
 Used RBF and linear kernels in SVM and
varied soft margin cost from 10-4 to 104
 Training set size varied from 5% to 60%
 Eight experiments performed at each sample
size
25

All Rights Reserved
 Six benchmark text corpora were selected
 Stop words were removed, others were stemmed
 For the RELIGION, POLITICS, SCIENCE and COMP
subsets of the 20 Newsgroups dataset, the top 2000
terms ranked by Information Gain were selected;
500 documents per class were sampled at random for
comparison with Slonim and Tishby (2001)
Experimental Setup (continued)
Dataset # classes total # docs # terms
RELIGION 3 1500 2000
POLITICS 3 1500 2000
SCIENCE 4 2000 2000
COMP 5 2500 2000
Citeseer 6 3312 3703
Cora 6 2708 1433
26

All Rights Reserved
Scalability Across Training Set Sizes
27

All Rights Reserved
Results for Naïve Bayes, SVM, HONB and
HOSVM on 20NG REL & SCI Datasets
28

All Rights Reserved
Results for Naïve Bayes, SVM, HONB and
HOSVM on Citeseer & Cora Datasets
29

All Rights Reserved
Significance of Results for Naïve Bayes,
SVM, HONB and HOSVM on All Datasets
30
 HONB consistently and statistically significantly outperformed NB
on all datasets (significant at <= 5% p-value)
 HOSVM outperformed SVM on the RELIGION, POLITICS and
SCIENCE datasets (significant at <= 5% p-value)
 Although, the difference between HOSVM and SVM on the COMP
dataset was significant at the level 0.158, HOSVM outperformed
SVM on seven out of eight trials by an average of 3%

All Rights Reserved
What role do higher-order relations play in
supervised machine learning?
• Higher-Order Collective Classification (HOCC)
– Classifies a set of instances simultaneously and thus exploits the
relationships between them; Based on a record-relation graph
– Capable of both supervised event detection as well as
unsupervised anomaly detection
• Application: Classification and Anomaly Detection of
Interdomain Routing Events
– Goal: detect and categorize such events
– Menon, V. and Pottenger, W. M. (2009) A Higher Order
Collective Classifier for Detecting and Classifying Network
Events. In the Proceedings of the IEEE International Conference
on Intelligence and Security Informatics 2009 (ISI 2009)
31
Vikas Menon
Software Developer @
Bridgewater Associates

All Rights Reserved
HOCC Results
• Detection of Interdomain Routing Events and Anomalies Based on
Higher-Order Path Analysis
– Slammer worm attack, Witty worm attack, 2003 East Coast Blackout
• Real Time Classification of Abnormal Events
– Sliding window samples of 120 three-second instances
– 180th window = start of event
– HOCC detects events and distinguishes anomalies
Witty (Supervised) Witty (Unsupervised)
32

All Rights Reserved
What role do higher-order relations play in
unsupervised machine learning?
• Next step? Consider unsupervised learning…
– Association Rule Mining (ARM)
• ARM is one of the most widely used algorithms in
data mining
– Extend ARM to higher order… Higher Order
Apriori
• LHOIM (Latent Higher-Order Information Mining)
• Experiments confirm the value of Higher Order
Apriori on real world e-marketplace data
33
Shenzhi Li
Senior Software Engineer
@ Ask (Ask.com)

All Rights Reserved
LHOIM Results on 20NG Computer Dataset
• Average error rate for 1st-order (top left) 2nd-order (top right)
• Average stdev for 1st-order (bottom left) 2nd-order (bottom right)
34
Li, S. Z., Wu, T., and Pottenger, W. M. (2005) Distributed Higher
Order Association Rule Mining Using Information Extracted from
Textual Data. SIGKDD Explorations, volume 7, issue 1, pages 26-35.

Higher Order Graph Sampling on Reuters
Naï…
0
10
20
30
40
50
60
70
1 2 3 4 5 6 7 8 9 10
Naïve Bayes Random Sampling
Higher Order Naïve Bayes Random Sampling
Higher Order Naïve Bayes Higher Order Sampling
Naï…
0
10
20
30
40
50
60
70
1 2 3 4 5 6 7 8 9 10
Naï…
0
10
20
30
40
50
60
70
1 2 3 4 5 6 7 8 9 10
Naï…
0
10
20
30
40
50
60
70
1 2 3 4 5 6 7 8 9 10
Higher Order Naïve
Bayes with Higher
Order Sampling gives
even better results
Higher Order Naïve
Bayes improves the
accuracy by at least 10%
Accuracy
in %
Patterns can be
discovered using a
much smaller sample –
important for online
learning
Training
Sample %
Cibin
George
M.S. in
CS @
Rutgers

All Rights Reserved
Higher Order (Online)
Latent Dirichlet Allocation
Intuitively, this formula can be interpreted as
a word being assigned to a topic proportional
to its frequency of occurrence in that topic.
This is in fact, our guiding intuition and we
simply replace these term frequencies with
higher order frequencies.
36
Nir Grinberg
Ph.D. in CS
@ Rutgers
Kashyap Kolipaka
Ph.D. in CS @
Rutgers
Christie Nelson
Ph.D. at RUTCOR
@ Rutgers

All Rights Reserved
Modeling Social Media for Emergency
Response in Port-au-Prince, Haiti
Cluster Geolocation

All Rights Reserved
Modeling Social Media for Emergency
Response in Port-au-Prince, Haiti
Cluster Geolocation with predicted resource

All Rights Reserved
Research Futures: Privacy-Enhanced
Higher Order Community Partitioning
),()(
11
=
,
1
1=
jiIPA
nl
Q k
ij
k
ij
ji
k
l
k
l  
),()(=),()
2
(=
,,
jiIPAjiI
m
dd
AQ ijij
ji
ji
ij
ji
 
Let I(I,j) be 1 if vertices i and j are in the same
community (social network), and 0 otherwise, then
Newman’s Q-Modularity is defined as:
Generalization
Q-Modularity counts edges inside each community and
subtracts the expected number of edges inside the same
community. Higher-order Ql counts number of paths inside
each community and subtracts the expected number of
paths. We propose Ql as a measure of a community split
and consider a combinatorial optimization approach.
39
Alex Nikolov, Ph.D.
in CS @ Rutgers

All Rights Reserved
Results on Ground Truth Data
• We optimized Ql using an LP rounding based
approximation algorithm for correlation
clustering.
• We ran our experiments on networks with
known communities, and compared the known
communities to our clustering using the
Adjusted Rand Index.
Datasetl 1 2 3 4
Karate 0.5414 0.5669 0.5669 0.5669
Political
Books
0.6250 0.6463 0.6463 0.6463
40

All Rights Reserved
Is Ql easier to approximate?
• We approximated Ql on random Gn,p graphs for
different values of l and p.
• We used the ratio of the value of the found
solution to the value of an LP relaxation as an
estimate of the approximation factor.
• It seems that Ql is harder for denser graphs (p
high) but easier for higher l.
l = 1 2 3 4 5
p = 0.03 0.9678 0.9840 1.0000 1.0000 0.9986
p = 0.12 0.1828 0.4542 -0.1179 0.8447 1.0000
p = 0.60 -0.1130 0.3975 1.0000 1.0000 1.0000
41

All Rights Reserved
Differential Privacy
• Differential Privacy [DMNS]: A randomized
function K gives ε-differential privacy if for all
graphs G1,G2 differing in a single edge and all
subsets S of Range(K):
• The global sensitivity of a real valued function f
is:
where G1,G2 differ in a single edge.
S])G([KPrS])G(K[Pr 21 
GSf  maxG1,G2
| f (G1) f (G2) |
42

All Rights Reserved
Sensitivity of Ql
The global sensitivity of Ql is at most 5(2l – 1)/l
for any fixed clustering.
By [DMNS], given a community split, outputting Ql
+ Lap(5(2l – 1)/lε) satisfies ε-differential privacy.
43

All Rights Reserved
Differentially Private Community Discovery
• The measure of community split Ql is insensitive.
– We can output the value of a community split
differentially privately
• But we would like a to design an algorithm Alg,
such that:
– Alg outputs a community partition with high Ql ;
– Alg satisfies ε-differential privacy
• Considered in Differentially Private Combinatorial
Optimization (Gupta et al. 2009), but there is
no general method.
44

All Rights Reserved
 In HOQL, we classify states as being in a high reward class or a low reward class. States are
added to a class based on a threshold. We use HONB classification for action selection. We
combine our method with greedy action selection based on the formula:
ε = 1- εstart* (1-episodecurrent / episodetotal)
 Q-values are updated based on the traditional formula:
Q(st
, at
) ← Q(st
, at
) + α[rt+1
+ γmaxa
Q(st+1
, a) – Q(st
, at
)
 Where α is the learning rate and γ is the discount factor. In these results, α = .91, γ =
1, and εstart = 0.8
REU Ashley Edwards
Higher Order Q-Learning (HOQL)
Ashley
Edwards,
Applicant for
Ph.D. in CS
@ Rutgers
Edwards, A. and
Pottenger, W. M. 2011.
Higher Order Q-
Learning. IEEE
Symposium on Adaptive
Dynamic Programming
and Reinforcement
Learning. Paris, France.
45

All Rights Reserved
Anomaly detection through
machine-learning exposed that the
Chinese government is capable of
“line rate” MITM attacks.
Due to pipelining in modern
browser implementations,
“censorware” is forced to remember
a 5-tuple for every attempt a user
makes to view censored content.
<ipSrc, ipDst, srcPort, dstPort,
proto>
Chinese government routers use
fiber-optics to do censorship at
“line rate.”
They lose the ability to drop
packets, so every censorware
router in the path must store a 5-
tuple and block responses.
This begs the question: “What
kinds of computational complexity
bottlenecks in ‘censorware’ can we
exploit?”
For example, how large of a
“botnet” would be required to cause
Chinese censorware routers to run
out of memory?
A BMITM
User attempts to restart the
connection.
Government servers useSEQ-1460
attack on TCP.
Government servers get user to
establish new, fake connection
User accepts new, fake connection
and retransmits.
Government rejects data
transmission with RST packet.
Server doesn’t understand new,
fake connection. Sends RSTs.
User rejects attempt to restart
the connection.
Server assumes user is adversarial.
Sends RSTs and kills connection.
REU Becker Polverini
Using Clustering to Detect Censorware
46
Polverini, A. B. and Pottenger, W. M. 2011. Using Clustering to Detect
Chinese Censorware. CSIIRW ’11 Oak Ridge National Labs, TN USA

All Rights Reserved
CCICADA technology transfer efforts
• Goal: Technology transfer to DHS users and
customers
• Several Tech Transfer programs @ DHS S&T:
– E2E – Engage to Excel
– Tech Solutions
– SECURE
• CCICADA is committed to support these existing
programs and to innovate new approaches – what can
you do?
– Publish your open-source software!
– Commercialize your software!
– Start your own company… and sell to DHS!
4747

www.intuidex.com ©Intuidex 2013 48
Intuidex, Inc.
Presenter: William M. Pottenger, Ph.D.
DrWMPottenger@intuidex.com

About Intuidex
Data Analytics and Data Model provider
Focused on helping Organizations discover
actionable intelligence from large, varied, and
complex data sources
Provides an open, extensible analytics platform,
Watchman AnalyticsTM
Platform and components that facilitate enhanced
real-time information extraction, consolidation,
fusion and discovery from disparate structured
and unstructured data streams

The problem we solve: “Big Data”
 Data volume and complexity has increased exponentially
 The number of data sources has exploded as well as
data formats, schemas and types
 The most valuable data is often unstructured and
fragmented
 The necessary data to drive better decisions is often
scattered across multiple data silos
 Data that is useful and valuable is often incomplete and
requires other data sources to validate
 Data storage systems are often proprietary with limited
interoperability
 Data from different sources regarding the same entities
sometimes conflicts.

www.intuidex.com ©Intuidex 2013 51www.intuidex.com ©Intuidex 2013 51
Differentiation
• Academic: Commercial Technology
Development
o Lab @ Rutgers University
o Director of Tech Transition for DHS S&T CCI Center
o Close cooperation with Rutgers Office of
Commercialization
o Three patents allowed, fourth pending
• Strategic Partnerships
o Rutgers University and DHS S&T Center of Excellence
o PNNL-DHS S&T National Visual Analytics Center
o Law Enforcement Partners: 3M (PIPS Technology)
o Customers in Intel / Defense sectors

Analyst Information Overload
FMV
COMINT
SIGINT
HUMINT
SIGACTS
OTHER
Analyst
Applications
and
Visualization
Platforms
e.g., TIGR

Data
Source
Data
Source
Data
Source
Data
Source
HighPerformanceIndex(IxHPI™)
Indexing
Routine
Indexing
Routine
Indexing
Routine
Indexing
Routine
Watchman Analytics™
Entity Extraction (IxExtract™)
Feature Selection (IxFeatures™
Topic Modeling (IxTopics™)
Rule Learning (IxRules™)
Recommender (IxRecommend™)
Alerting (IxAlert™)
Clustering (IxCluster™)
Data Validation (IxValidate™)
Trending (IxEntityTrend™)
Link Analysis (IxLinks™)
Data Fusion (IxRelClu™)
Entity Resolution (IxResolve™)
U
S
E
R
Watchman
Analytics™
Visualization
Customer
Visualization

• Web-based advanced data analytics and visualization solution
• Adobe Flex RIA framework
• Component Modules
• Synchronized
Watchman Analytics™ for BOSS

Intuidex and 3M Partnership
Intuidex, Inc., a leader and innovator in data analytics (machine learning), is the
pioneer of Higher Order Learning™ technologies that deliver unprecedented accuracy and
efficiency in identifying linkages, trends and patterns across disparate information
systems, in real time or near real time. Intuidex analytics have been licensed by
customers in the US Defense and Intelligence Agencies, US Law Enforcement Agencies
and the Fortune 500 to extract latent intelligence and insights from both structured and
unstructured data sources.
3M (formerly PIPS Technology) is the worldwide leader in Automated License
Plate Recognition (ALPR) technology. PIPS designs, manufactures, and supports its
complete line of ALPR products and services for use in law enforcement, parking, tolling,
and intelligent transportation systems. With over 20,000 cameras deployed around the
globe and a wide range of patents covering their technology and its application, PIPS
Technology is easily recognized as the leading provider of traffic related video imaging
and license plate capture technology for public safety agencies everywhere.

APPLICATIONS OF
HIGHER ORDER LEARNING™
FROM

• Objective: determine which COMINT is likely important and
require further analysis
• Data: plain text representation of comm-hits
• 400 samples drawn from Afghanistan theater
• Classification: two classes
• Class A, Class B
• Evaluation
• Compared IxHONB™ to Naïve Bayes (NB)
• Train on 5% to 90%, test on rest
• Averages (accuracy, precision, recall, ...) across 10-folds
Military Threat Detection Applications of
Intuidex’s Higher Order Learning™

Weighted F-measure performance of NB vs. IxHONB™

MIRC (Chat) Entity Extraction
 Data from MIRC chat Comm Hits (COMINT) has
been helpful to GMTI analysts in
 Determining the nature of movements detected by radar (e.g.,
wild animals don't radio their friends for help)
 Whether ground targets may represent a threat
 Validating known movements by corroborating with statements
of locals (if they see a vehicle WE see, then we KNOW what
the “dots” are)
 Some “dots” can talk!
 Tactical Ground Reporting System (TIGR)
 A TIGR user on the battlefield has limited ability to refine a
search the way an analyst can
 Only has temporal and spatial filters, and relies on pre-
packaged intel from various sources input to TIGR (HUMINT,
SIGACT, HUMINT)

Example Actionable Information
• IxRules™ aids a user in discovering rules for multiple entity types
• IED Trigger
“On 23 February 2006, at 12:30 PM, in Ba'qubah, Diyala, Iraq,
assailants detonated a probable command-initiated improvised
explosive device (IED) hidden in a soup vendor's handcart near an
Iraqi Army patrol in the central market, killing eight Iraqi soldiers and
eight civilians, wounding four Iraqi soldiers and 11 civilians, and causing
unspecified damage to the public market. The Mujahidin Shura
Council in Iraq (MSC) claimed responsibility.”
• Height
“… The suspect is described as black, medium complexion, 28-30 years
old, clean-shaven, approximately 6 feet 8 inches tall, weighing 180-
200 pounds, with a muscular build. He was last seen wearing a black
sweatshirt, black pants, and a dark blue or black knit hat. …”

Tactical Ground Reporting System: TIGR

Benefits to the Warfighter
1. Fusion of high-value COMINT intel provides
significantly improved situational awareness for
warfighters with ‘boots on the ground’
2. Extraction / summarization of high-value COMINT,
SIGACT, HUMINT from unstructured, unleveraged
text sources
3. Fusion of high-value COMINT and other text-
based intel with GMTI and other intel sources
• Transitioned to: ESC/CIEF, used in DARPA
Tactical Ground Reporting System (TIGR)
Technology Transition Description
• Fielded operationally at: Afghanistan and other
theaters
• Customer(s): TIGR and users, e.g., GEOINT, FSR,
S2, ISR, MAI, CPTI, JIEDDO MID, CIED, RFI, NASIC,
Centcom TFs
Information extraction, summarization and fusion
technologies to provide warfighter with
situational awareness
From theater: “These are exactly the sort of quick and
dirty SIGINT summaries I am trying to get. … Just
wanted to make sure you know how happy our ground
units are to get this information in a wrap up. This daily
tipper has made our supported units very happy. Thanks
for the consistent help.”

• Objective: Classify confidence in perpetrator identification for
incidents in NCTC Worldwide Incident Tracking System (WITS)
• Data: relational tables from WITS
• Sampled ~1,000 incidents from 80,000 record corpus
• Included some free text
• Classification: five confidence classes
• Plausible, Likely, Unknown, Unlikely, Inferred (analyst)
• Evaluation
• Compared IxHONB™ to NB and LSI-kNN
• Train on 5% to 90% of sample, test on rest
• Averages (accuracy, precision, recall, ...) across 10-folds
Counterterrorism Applications of Intuidex’s
Higher Order Learning™

0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
5 10 20 30 40 50 60 70 80 90
F-measure
Percentage of Training Set Available for Training
HONB
LSI-kNN
NB
Non-weighted F-measure performance of NB, LSI-kNN and IxHONB™

Nuclear Detection
•Data was taken from a Thermo Scientific
handheld Spectroscopic Personal Radiation
Detector called the InterceptorTM
• 302 gamma-ray spectrum files
•20 from Tc99m, the rest from other
isotopes or background
•Small positive class size
• 1024 numeric channels per spectrum
•High dimensional space
• 14 labeled, high confidence isotopes
•Potassium (40K; 1.3 billion years)

Sample of Results - Accuracy
Accuracy
65% 60% 55% 50% 45% 40% 35% 30% 25% 20%
Ga67 – D-B 0.0002 0 0 0 0 0 0 0 0 0
Ga67 – N-D-B 1 1 1 0.343 0.778 0.697 0.39 0.57 0.26 0.06
I131 – D-B 0 0 0 0 0 0 0 0.01 0.251 0.16
I131 – N-D-B 0 0 0.002 0.008 0 0 0 0.01 0.002 0
In111 – D-B 0.136 0.017 0.01 0.001 0 0 0 0 0 0
In111 – N-D-B 1 0.08 0.389 0.005 0.001 0.037 0 0.18 0 0.45
Tc99m – D-B 0.049 0.095 0.001 0 0 0 0 0 0 0
Tc99m - N-D-B 0 0 0 0 0 0 0 0 0 0
Key
Statistically Significant difference: NB < HONB
Not Statistically Significant

Typical Intuidex Engagement
• Client environment analysis
Infrastructure (hardware, software)
Data sources
Operations (relevant and related policies)
• Requirements Specification with SMEs
Iterate until approved
• Deploy high-performance index engine
Install, configure, test
• Deploy indexing routines
Develop, configure, optimize
• Deploy analytics services
(Optional) Develop custom services to spec
Install, configure, test

Typical Intuidex Engagement
• (Optional) Existing visualization interface
Design interface specification for existing framework
• Ground-truth development with SMEs
• System documentation
Usage documentation
Administration and Configuration documentation
Visualization interface documentation (optional)
• Deployment validation
Quality assurance
Load testing
• Customer acceptance

Watchman Analytics™ Functionality
Entity Resolution Online Monitoring
Data Deconfliction Automated Alerting
Interactive
Analysis
Entity Extraction
Ad-hoc Reporting Entity Classification
Privacy
Protection*
Quality Assurance
Link-based
Analysis
Embedded Analytics
* Privacy protection is a major Intuidex research area and development thrust

• Intuidex, Inc. is a hi-tech start-up incorporated by
William. M. Pottenger, Ph.D.
• Thought Leadership in Data Analytics
• Key Partnerships

All Rights Reserved
Acknowledgements
• I am very grateful to my hardworking, intelligent and creative
(current and former) students and postdocs without whom none of
this would have been possible: Kunikazu Yoda, Christie Nelson,
Aleksandar Nikolov, Nir Grinberg, Cibin George, Christopher
Janneck, Nikita Lytkin, Shenzhi Li, Murat Ganiz, Chirag Pandya,
Kashyap Kolipaka, Vikas Menon, April Kontostathis, Tianhao Wu,
Jirada Kuntraruk, Jason Perry, Mark Dilsizian (and >> others).
• I also thank Rutgers University, the National Science Foundation,
the Department of Homeland Security and the National Institute of
Justice. This material is based upon work partially supported by the
National Science Foundation under Grant Numbers 0703698 and
0712139. Any opinions, findings, and conclusions or recommendations
expressed in this material are those of the authors and do not
necessarily reflect the views of the National Science Foundation or
Rutgers University.
• I also gratefully acknowledge the continuing help of my Lord and
Savior, Yeshua the Messiah (Jesus the Christ) in my life and work.
72

All Rights Reserved
References
 Soumen Chakrabarti, Byron Dom, and Piotr Indyk. Enhanced hypertext
categorization using hyperlinks. SIGMOD Rec., 27(2):307–318, 1998.
 Scott Deerwester, Susan T. Dumais, George W. Furnas,Thomas K.
Landauer, and Richard Harshman.
 Indexing by latent semantic analysis. Journal of the American Society for
Information Science, 41:391–407, 1990.
 Lise Getoor and Christopher P. Diehl. Link mining: a survey. SIGKDD Explor.
Newsl., 7(2):3–12, 2005.
 Murat Can Ganiz, Sudhan Kanitkar, Mooi Choo Chuah, and William M.
Pottenger. Detection of interdomain routing anomalies based on higher-order
path analysis. In ICDM ’06: Proceedings of the Sixth International
Conference on Data Mining, pages 874–879, Washington, DC, USA, 2006.
IEEE Computer Society.
 Leo Katz. A new status index derived from sociometric analysis.
Psychometrika, 18(1):39–43, March 1953.
 April Kontostathis and William M. Pottenger. A framework for understanding
latent semantic indexing (LSI) Performance. Inf. Process. Manage.,
42(1):56–73, 2006.
74

All Rights Reserved
 Qing Lu and Lise Getoor. Link-based classification. In Tom Fawcett and
Nina Mishra, editors, ICML, pages 496–503. AAAI Press, 2003.
 Shenzhi Li, Tianhao Wu, and William M. Pottenger. Distributed higher order
association rule mining using information extracted from textual data.
SIGKDD Explorations Newsl., 7(1):26–35, 2005.
 J. Neville and D. Jensen. Iterative classification in relational data. In Proc.
AAAI, pages 13–20. AAAI Press, 2000.
 J. Neville and D. Jensen. Dependency networks for relational data. Data
Mining, 2004. ICDM ’04. Fourth IEEE International Conference, pages 170–
177, Nov. 2004.
 Noam Slonim and Naftali Tishby. The power of word clusters for text
classification. In In 23rd European Colloquium on Information Retrieval
Research, 2001.
 Ben Taskar, Eran Segal, and Daphne Koller. Probabilistic classification and
clustering in relational data. In Proceedings of the Seventeenth
International Joint Conference on Artificial Intelligence, pages 870–878,
2001.
 Vladimir Vapnik. Statistical Learning Theory. John Wiley, 1998.
References
75

Intuidex - To be or not to be iid by William M. Pottenger (NYC Machine Learning Group)

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Destaque

Destaque (19)

Semelhante a Intuidex - To be or not to be iid by William M. Pottenger (NYC Machine Learning Group)

Semelhante a Intuidex - To be or not to be iid by William M. Pottenger (NYC Machine Learning Group) (20)

Mais de Hakka Labs

Mais de Hakka Labs (20)

Último

Último (20)

Intuidex - To be or not to be iid by William M. Pottenger (NYC Machine Learning Group)