Legal Analytics Course - Class #2 - Introduction to Machine Learning for Lawyers - Professors Daniel Martin Katz + Michael J Bommarito

Legal Analytics
Professor Daniel Martin Katz
Professor Michael J Bommarito II
legalanalyticscourse.com
Class 2
Machine Learning for Lawyers

< What is Machine ‘Learning’?>
access more at legalanalyticscourse.com

we are trying to learn from existing
data to infer future data
rough and ready goal for ML:

“A computer program is said to learn from
experience E with respect to some task T
and some performance measure P, if its
performance on T, as measured by P,
improves with experience E.”
Tom Mitchell
Carnegie Mellon University

inference is tricky for
a variety of reasons

data might be noisy
thus hard to infer true signal
(1)

the dynamics change such that
past data is a bad guide for
future data
(2)

researcher chooses wrong
method to do forecasting
(3)

remember to take stock
of relevant baselines

For example, how accurate
is the existing method
of forecasting?

sometimes it is good but far too
often existing method(s) is not as
good as many would think

There are 3 Known Ways
to Predict Something

Experts, Crowds, Algorithms

historically decision making
in law is heavily skewed
toward the use of
Human Experts

rarely is the historic
performance of
Human Experts
extensively benchmarked
or validated using
statistical methods

in related ﬁelds
this is becoming
increasingly common

it is even becoming
common among
those who use
statistical and other
methods to forecast

Poll Aggregation is one form of ensemble where
the learning question is to determine how much
weight (if any) to assign to each individual poll

selecting the
ulitmate ensemble
of polls requires
grading the
historical
performance
of each pollster

The goal of this course is
to expose you to some of
the methods used in
predictive analytics

< Supervised Learning
versus
Unsupervised Learning>
*note reinforcement learning will not be covered herein

Supervised Learning

typcially supervision is
undertaken through the
provision/development
of “gold standard” data

Classic Example from Law
is so called ‘predictive coding’

imagine your client is served
with a request for production

in random
order
assume
this is the
size
of the
hypothetical
document
set
(emails,
memos,
etc.)

we can
sample
a subset
of the
documents

classification
clustering
regression
dimension reduction

classification

predictive coding =
~ binary classification

LearningTask = Determine Whether a Given
Document is Relevant?
Relevant
Not Relevant
f( )
relevance?
Binary Classiﬁcation (Supervised Learning)
and/or
010
101
001

take the sample set as
a training set and
use human experts

the use of the human
experts is called
“supervised learning”

in the simple binary case,
ask humans to assign
objects to two piles

Apply Human Coders

yellow = relevant
white = non-relevant
and return this

Non RelevantRelevant

Non RelevantRelevant
gold standard data

Key Insight ...

What Allows A
Human To Separate
These Two Classes of
Documents?

that precise human
process is what
“predictive coding”
is trying to mimic

most vendors are selling a
largely undifferentiated product

Humans are selecting
upon some “features”
of the documents

to place those
documents in their
respective bins
(i.e. relevant, non-relevant)

features =?
text,
author,
date,
other metadata

machine learning task is
trying to recover (learn)
what separates the
relevant from the
non-relevant documents

once we learn the
rule / boundary
we can apply it to separate
the remain documents into
the two classes

we want to take what we learn here

we want to take what we learn here
and apply it here

By Contrast
Unsupervised Learning

Pre-Clustering Documents
based on some sort of criterion

Must determine
the
distance metric
(similarity index)

features =?
text,
author,
date,
other metadata
distance
metric
/
similarity
index

< Bias vsVariance Tradeoff >

The Bias vsVariance Tradeoff
“The problem of simultaneously minimizing the
bias (how accurate a model is across different
training sets) and variance of the model error
(how sensitive the model is to small changes in
training set).
Intuitively, it means that a model must be
chosen that at the same time captures the
regularities in its training data, but also
generalizes well to unseen data.”

The Bias vsVariance Tradeoff
http://scott.fortmann-roe.com/docs/BiasVariance.html
Worst Case Scenario
Best Case Scenario

Michael Clark citing Hastie, et al (2009)

< Precision vs Recall >

http://en.wikipedia.org/wiki/Precision_and_recall

Legal Analytics Course - Class #2 - Introduction to Machine Learning for Lawyers - Professors Daniel Martin Katz + Michael J Bommarito

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (17)

Semelhante a Legal Analytics Course - Class #2 - Introduction to Machine Learning for Lawyers - Professors Daniel Martin Katz + Michael J Bommarito

Semelhante a Legal Analytics Course - Class #2 - Introduction to Machine Learning for Lawyers - Professors Daniel Martin Katz + Michael J Bommarito (20)

Mais de Daniel Katz

Mais de Daniel Katz (20)

Legal Analytics Course - Class #2 - Introduction to Machine Learning for Lawyers - Professors Daniel Martin Katz + Michael J Bommarito