This document summarizes a presentation on simultaneous deep transfer across domains and tasks. It discusses training deep neural networks on labeled source domain data and adapting them to an unlabeled target domain. The key idea is to minimize the discrepancy between source and target domains by using a domain confusion loss and soft labels on target data during training. Experimental results on the Office dataset show that combining domain confusion with soft labels leads to improved accuracy compared to other domain adaptation methods or training on source or target data alone.

1. Simultaneous,Deep,Transfer,Across,
Domains,and,Tasks,
!
Presenta)on!by!Alejandro,Cartas!
Eric!Tzeng,!Judy!Hoffman,!Trevor!Darrell,!Kate!Saenko!

2. Domain Adaptation: Train on source adapt to target
backpack chair bike
Source Domain
lots of labeled data
⇠ PS(X, Y )
DS = {(xi, yi), 8i 2 {1, . . . , N}}
Original slide: J. Hoffman, Adapting Deep Networks Across Domains, Modalities, and Tasks, ICCV 2015

3. Domain Adaptation: Train on source adapt to target
backpack chair bike
Source Domain
lots of labeled data
⇠ PS(X, Y )
DS = {(xi, yi), 8i 2 {1, . . . , N}}
bike??
Target Domain
unlabeled or limited labels
⇠ PT (Z, H)
?DT = {(zj, ), 8j 2 {1, . . . , M}}
Original slide: J. Hoffman, Adapting Deep Networks Across Domains, Modalities, and Tasks, ICCV 2015

4. Domain Adaptation: Train on source adapt to target
backpack chair bike
Adapt
Source Domain
lots of labeled data
⇠ PS(X, Y )
DS = {(xi, yi), 8i 2 {1, . . . , N}}
bike??
Target Domain
unlabeled or limited labels
⇠ PT (Z, H)
?DT = {(zj, ), 8j 2 {1, . . . , M}}
Original slide: J. Hoffman, Adapting Deep Networks Across Domains, Modalities, and Tasks, ICCV 2015

5. Source Data
backpack chair bike
fc8conv1 conv5
source data
fc6 fc7 classif cation
loss
i
Adap)ng!across!domains!minimize!discrepancy!!
L(xS, yS, xT , yT , ✓D; ✓repr, ✓C) = LC(xS, yS, xT , yT ; ✓repr, ✓C)
+ Lconf (xS, xT , ✓D; ✓repr)
+⌫Lsoft(xT , yT ; ✓repr, ✓C)
Eric!Tzeng,!et.!al.!Simultaneous!Deep!Transfer!Across!Domains!and!Tasks,!2015!

6. Adap)ng!across!domains!minimize!discrepancy!!
L(xS, yS, xT , yT , ✓D; ✓repr, ✓C) = LC(xS, yS, xT , yT ; ✓repr, ✓C)
+ Lconf (xS, xT , ✓D; ✓repr)
+⌫Lsoft(xT , yT ; ✓repr, ✓C)
Source Data
backpack chair bike
Target Databackpack
?
fc8conv1 conv5 fc6 fc7
labeled target data
fc8conv1 conv5
source data
fc6 fc7
classif cation
loss
shared
shared
shared
shared
i
Eric!Tzeng,!et.!al.!Simultaneous!Deep!Transfer!Across!Domains!and!Tasks,!2015!

7. Adap)ng!across!domains!minimize!discrepancy!!
LD(xS, xT , ✓repr; ✓D) =
X
d
[yD = d] log qd Lconf (xS, xT , ✓D; ✓repr) =
X
d
1
D
log qd
Adapted!from!J.!Hoffman,!Adap)ng!Deep!Networks!Across!Domains,!Modali)es,!and!Tasks,!2015!
q = softmax(✓T
Df(x; ✓repr))

8. Adap)ng!across!domains!minimize!discrepancy!!
✓C
object
classifier
LD(xS, xT , ✓repr; ✓D) =
X
d
[yD = d] log qd Lconf (xS, xT , ✓D; ✓repr) =
X
d
1
D
log qd
q = softmax(✓T
Df(x; ✓repr))
Adapted!from!J.!Hoffman,!Adap)ng!Deep!Networks!Across!Domains,!Modali)es,!and!Tasks,!2015!

9. Adap)ng!across!domains!minimize!discrepancy!!
✓C
object
classifier
LD(xS, xT , ✓repr; ✓D) =
X
d
[yD = d] log qd Lconf (xS, xT , ✓D; ✓repr) =
X
d
1
D
log qd
q = softmax(✓T
Df(x; ✓repr))
Adapted!from!J.!Hoffman,!Adap)ng!Deep!Networks!Across!Domains,!Modali)es,!and!Tasks,!2015!

10. Adap)ng!across!domains!minimize!discrepancy!!
✓C
object
classifier
Discrepancy
LD(xS, xT , ✓repr; ✓D) =
X
d
[yD = d] log qd Lconf (xS, xT , ✓D; ✓repr) =
X
d
1
D
log qd
q = softmax(✓T
Df(x; ✓repr))
Adapted!from!J.!Hoffman,!Adap)ng!Deep!Networks!Across!Domains,!Modali)es,!and!Tasks,!2015!

11. Adap)ng!across!domains!minimize!discrepancy!!
✓C
object
classifier domain
classifier
✓D
LD(xS, xT , ✓repr; ✓D) =
X
d
[yD = d] log qd Lconf (xS, xT , ✓D; ✓repr) =
X
d
1
D
log qd
q = softmax(✓T
Df(x; ✓repr))
Adapted!from!J.!Hoffman,!Adap)ng!Deep!Networks!Across!Domains,!Modali)es,!and!Tasks,!2015!

12. Adap)ng!across!domains!minimize!discrepancy!!
LD(xS, xT , ✓repr; ✓D) =
X
d
[yD = d] log qd Lconf (xS, xT , ✓D; ✓repr) =
X
d
1
D
log qd
domain
classifier
✓D
q = softmax(✓T
Df(x; ✓repr))
Adapted!from!J.!Hoffman,!Adap)ng!Deep!Networks!Across!Domains,!Modali)es,!and!Tasks,!2015!

13. Adap)ng!across!domains!minimize!discrepancy!!
LD(xS, xT , ✓repr; ✓D) =
X
d
[yD = d] log qd Lconf (xS, xT , ✓D; ✓repr) =
X
d
1
D
log qd
q = softmax(✓T
Df(x; ✓repr))
✓C
object
classifier
Adapted!from!J.!Hoffman,!Adap)ng!Deep!Networks!Across!Domains,!Modali)es,!and!Tasks,!2015!

14. Adap)ng!across!domains!minimize!discrepancy!!
L(xS, yS, xT , yT , ✓D; ✓repr, ✓C) = LC(xS, yS, xT , yT ; ✓repr, ✓C)
+ Lconf (xS, xT , ✓D; ✓repr)
+⌫Lsoft(xT , yT ; ✓repr, ✓C)
Source Data
backpack chair bike
Target Databackpack
?
fc8conv1 conv5 fc6 fc7
alltargetdata
sourcedata
labeled target data
fc8conv1 conv5
source data
fcD
fc6 fc7
classif cation
loss
domain
confusion
loss
domain
classif er
loss
shared
shared
shared
shared
shared
i
i
Eric!Tzeng,!et.!al.!Simultaneous!Deep!Transfer!Across!Domains!and!Tasks,!2015!

15. Domain Adaptation: Train on source adapt to target
backpack chair bike
Adapt
Source Domain
lots of labeled data
⇠ PS(X, Y )
DS = {(xi, yi), 8i 2 {1, . . . , N}}
bike??
Target Domain
unlabeled or limited labels
⇠ PT (Z, H)
?DT = {(zj, ), 8j 2 {1, . . . , M}}
Original slide: J. Hoffman, Adapting Deep Networks Across Domains, Modalities, and Tasks, ICCV 2015

16. Domain Adaptation: Train on source adapt to target
backpack chair bike
Adapt
Source Domain
lots of labeled data
⇠ PS(X, Y )
DS = {(xi, yi), 8i 2 {1, . . . , N}}
bike??
Target Domain
unlabeled or limited labels
⇠ PT (Z, H)
?DT = {(zj, ), 8j 2 {1, . . . , M}}
Original slide: J. Hoffman, Adapting Deep Networks Across Domains, Modalities, and Tasks, ICCV 2015

17. Source!soKlabels!
Source
CNN
Source
CNN
Source
CNN
Bottle
Mug
Chair
Laptop
Keyboard
Bottle
Mug
Chair
Laptop
Keyboard
Bottle
Mug
Chair
Laptop
Keyboard
Bottle
Mug
Chair
Laptop
Keyboard
+
softmax
high
temp
softmax
high
temp
softmax
high
temp
Eric!Tzeng,!et.!al.!Simultaneous!Deep!Transfer!Across!Domains!and!Tasks,!2015!
l(bottle)

18. Source!soKlabels!
Bottle
Mug
Chair
Laptop
Keyboard
Bottle
Mug
Chair
Laptop
Keyboard
Adapt CNN
“Bottle”
Source Activations
Per Class
backprop
Cross Entropy Loss
softmax
high
temp
Eric!Tzeng,!et.!al.!Simultaneous!Deep!Transfer!Across!Domains!and!Tasks,!2015!
Lsoft(xT , yT ; ✓repr, ✓C) =
X
i
lyT
i log pi
p = softmax(✓T
Cf(xT ; ✓repr)/⌧)

19. Class!correla)on!transfer!loss!
L(xS, yS, xT , yT , ✓D; ✓repr, ✓C) = LC(xS, yS, xT , yT ; ✓repr, ✓C)
+ Lconf (xS, xT , ✓D; ✓repr)
+⌫Lsoft(xT , yT ; ✓repr, ✓C)
Source Data
backpack chair bike
Target Databackpack
?
fc8conv1 conv5 fc6 fc7
Source softlabels
alltargetdata
sourcedata
labeled target data
fc8conv1 conv5
source data
softmax
high temp
softlabel
loss
fcD
fc6 fc7
classif cation
loss
domain
confusion
loss
domain
classif er
loss
shared
shared
shared
shared
shared
i
i
Eric!Tzeng,!et.!al.!Simultaneous!Deep!Transfer!Across!Domains!and!Tasks,!2015!

20. Office dataset ExperimentAdapting Visual Category Models to New Domains 9
31 categories
keyboardheadphonesfile cabinet... laptop letter tray ...
amazon dSLR webcam
...
instance1instance2
...
...
...
instance5
...
instance1instance2
...
...
...
instance5
...
3 domains
Fig. 4. New dataset for investigating domain shifts in visual category recognition tasks.
Images of objects from 31 categories are downloaded from the web as well as captured
by a high definition and a low definition camera.
popular way to acquire data, as it allows for easy access to large amounts of
• all classes have
source labeled
examples
• 15 classes have
target labeled
examples
• evaluate on remaining
16 classes
[saenko`10]Original slide: J. Hoffman, Adapting Deep Networks Across Domains, Modalities, and Tasks, ICCV 2015

21. DeCAF6 S+T [9] 80.7 ± 2.3 – – 94.8 ± 1.2 – – –
DaNN [13] 53.6 ± 0.2 – – 71.2 ± 0.0 – 83.5 ± 0.0 –
Source CNN 56.5 ± 0.3 64.6 ± 0.4 47.6 ± 0.1 92.4 ± 0.3 42.7 ± 0.1 93.6 ± 0.2 66.22
Target CNN 80.5 ± 0.5 81.8 ± 1.0 59.9 ± 0.3 80.5 ± 0.5 59.9 ± 0.3 81.8 ± 1.0 74.05
Source+Target CNN 82.5 ± 0.9 85.2 ± 1.1 65.8 ± 0.5 93.9 ± 0.5 65.2 ± 0.7 96.3 ± 0.5 81.50
Ours: dom confusion only 82.8 ± 0.9 85.9 ± 1.1 66.2 ± 0.4 95.6 ± 0.4 64.9 ± 0.5 97.5 ± 0.2 82.13
Ours: soft labels only 82.7 ± 0.7 84.9 ± 1.2 66.0 ± 0.5 95.9 ± 0.6 65.2 ± 0.6 98.3 ± 0.3 82.17
Ours: dom confusion+soft labels 82.7 ± 0.8 86.1 ± 1.2 66.2 ± 0.3 95.7 ± 0.5 65.0 ± 0.5 97.6 ± 0.2 82.22
Table 1. Multi-class accuracy evaluation on the standard supervised adaptation setting with the Office dataset. We evaluate on all 31 categories
using the standard experimental protocol from [28]. Here, we compare against three state-of-the-art domain adaptation methods as well as a
CNN trained using only source data, only target data, or both source and target data together.
A ! W A ! D D ! A D ! W W ! A W ! D Average
MMDT [18] – 44.6 ± 0.3 – – – 58.3 ± 0.5 –
Source CNN 54.2 ± 0.6 63.2 ± 0.4 36.4 ± 0.1 89.3 ± 0.5 34.7 ± 0.1 94.5 ± 0.2 62.0
Ours: dom confusion only 55.2 ± 0.6 63.7 ± 0.9 41.2 ± 0.1 91.3 ± 0.4 41.1 ± 0.0 96.5 ± 0.1 64.8
Ours: soft labels only 56.8 ± 0.4 65.2 ± 0.9 41.7 ± 0.3 89.6 ± 0.1 38.8 ± 0.4 96.5 ± 0.2 64.8
Ours: dom confusion+soft labels 59.3 ±0.6 68.0±0.5 43.1± 0.2 90.0± 0.2 40.5±0.2 97.5± 0.1 66.4
Table 2. Multi-class accuracy evaluation on the standard semi-supervised adaptation setting with the Office dataset. We evaluate on 16
held-out categories for which we have no access to target labeled data. We show results on these unsupervised categories for the source only
model, our model trained using only soft labels for the 15 auxiliary categories, and finally using domain confusion together with soft labels
on the 15 auxiliary categories.
target domain. We report accuracies on the remaining un-
labeled images, following the standard protocol introduced
with the dataset [28]. In addition to a variety of baselines, we
report numbers for both soft label fine-tuning alone as well
as soft labels with domain confusion in Table 1. Because the
Office dataset is imbalanced, we report multi-class accura-
following the standard protocol introduced with the Office
dataset [28]. We then evaluate our classification performance
on the remaining 16 categories for which no data was avail-
able at training time.
In Table 2 we present multi-class accuracies over the 16
held-out categories and compare our method to a previous
Office dataset Experiment
Multiclass accuracy over 16 classes which lack target labels
Original slide: J. Hoffman, Adapting Deep Networks Across Domains, Modalities, and Tasks, ICCV 2015

22. back pack
bike
bike helmet
bookcase
bottle
calculator
desk chair
desk lamp
desktop computer
file cabinet
headphones
keyboard
laptop computer
letter tray
mobile phone
monitor
mouse
mug
paper notebook
pen
phone
printer
projector
punchers
ring binder
ruler
scissors
speaker
stapler
tape dispenser
trash can
0
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
0.09
0.1
Ours soft label
back pack
bike
bike helmet
bookcase
bottle
calculator
desk chair
desk lamp
desktop computer
file cabinet
headphones
keyboard
laptop computer
letter tray
mobile phone
monitor
mouse
mug
paper notebook
pen
phone
printer
projector
punchers
ring binder
ruler
scissors
speaker
stapler
tape dispenser
trash can
0
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
0.09
0.1
Baseline soft label
ring binder
monitor
Baseline soft activation
Our soft activation
Target test image
Originalslide:J.Hoffman,AdaptingDeepNetworksAcrossDomains,Modalities,andTasks,ICCV2015

23. back pack bike bike helmet
bookcase bottle calculator
desk chair desk lamp desktop computer
file cabinet headphones keyboard
laptop computer letter tray mobile phone
Source soft labels
Originalslide:J.Hoffman,AdaptingDeepNetworksAcrossDomains,Modalities,andTasks,ICCV2015

24. Cross-dataset Experiment Setup
Source: ImageNet
!
Target: Caltech256
!
40 categories
!
Evaluate adaptation performance with 0,1,3,5
target labeled examples per class
[tommasi`14]
Originalslide:J.Hoffman,AdaptingDeepNetworksAcrossDomains,Modalities,andTasks,ICCV2015

25. ImageNet adapted to Caltech
Number Labeled Target Examples per Category
0 1 3 5
Multi-classAccuracy
72
73
74
75
76
77
78
Source+Target CNN
Ours: softlabels only
Ours: dom confusion+softlabels
[ICCV 2015]
400 120 200
Number of labeled target examples
MulticlassAccuracy
Originalslide:J.Hoffman,AdaptingDeepNetworksAcrossDomains,Modalities,andTasks,ICCV2015