Reversible visual privacy protection

1
Multimedia Signal Processing Group
Swiss Federal Institute of Technology, Lausanne
Reversible visual
privacy protection
Touradj Ebrahimi
touradj.ebrahimi@epfl.ch
COST IC1206 Training School
De-identification for privacy protection in multimedia content
7-11 October 2015, Limassol, Cyprus

2
Outline
• Part I:
– Motivations and context
– Conventional privacy protection
filters
– Advanced privacy protection
filters
• Part II:
– Visual privacy evaluation
framework
– Impact of new imaging
modalities on privacy
– Illustrative example

3
Motivation
• People are increasingly exposed:
– video surveillance
– social media

4
Proliferation of video surveillance applications
• Surveillance of sensitive locations
– Embassies, airports, nuclear plants, military zone, border
control, …
• Intrusion detection
– Residential surveillance, retail surveillance, …
• Traffic control
– Speed control
• Access to places
– Car license plate recognition in cities
• Event detection
– Child/Elderly care
• Marketing/statistics
– Customers habits
– Number of visitors
• …

5
Proliferation of social media applications

6
Main security solutions for visual content protection
• Encryption
– Secure communication
– Conditional access
• Integrity verification
– Digital signature
– Proof for lack of manipulation after capture

7
Alternatives to implement video surveillance with privacy
• Fully automatic surveillance without intervention of
human operators
– False positives and false negatives
• Encrypting the whole video
– Not good for monitoring
• Distorting/blocking sensitive regions
– Impact on intelligibility
• Reversible encryption/scrambling of sensitive regions
with a key
– Identification can take place when crime happens
• Legal and best practices in video surveillance
– Recorded materials locked in secure locations
• Only extract/record needed information from the scene
– MPEG-7 visual descriptors

8
Privacy-sensitive visual information
• Predefined zones
– Windows, doors
– Bank teller
– Casino playing tables
– …
• Automatic identification of Regions of Interest (ROI)
– People in the scene
– Human faces
– Cars license plates
– Moving objects
– …
• Advanced image/video analytics
– Deep learning
– Big data analytics

9
Social media/networks business model
• User profiling
• Targeted advertisement/marketing

10
Requirements for visual privacy protection
• Maximize intelligibility
• Minimize invasion of privacy
• Visually pleasing
• Reversible
• Reasonable computational resources
• Format preserving/independent
• Reliable
• Secure
• …

11
Legacy solutions to visual privacy protection
• Masking
• Blur
• Pixelization

12
Masking

13
Blur

14
Pixelization

15
More recent solutions for privacy protection
• (ROI) Encryption
• (ROI) Scrambling

16
ROI selective encryption

17
ROI selective decryption

18
ROI selective scrambling

19
Bitstream encryption
• Selective encryption of the bitstream at packet level
• One or more secret keys
• Symmetric encryption
– Packet body
– Block cipher: e.g. AES
packet
encrypted
packet
private key

20
Scrambling approaches
• Image-domain
– Randomly flip bits in one or more bit planes
• Pros
– Very simple
– Independent from the subsequent encoding scheme
– Does not affect the bitstream syntax → standard compliance
• Cons
– Significantly alter statistics of video signal
– Ensuing compression less efficient
bitstreamimage Scrambling
Encoder
Transform Entropy Coding

21
• Transform-domain
– Randomly flip sign of transform coefficients
• Pros
– Does not adversely affect subsequent entropy coding
– Strength of scrambling can be controlled
– Does not affect the bitstream syntax → standard compliance
• Cons
– Must be integrated inside the encoder
bitstreamimage Transform
Encoder
Scrambling Entropy Coding

22
• Bitstream-domain
– Randomly flip bits in bitstream
• Pros
– Applied on bitstream after encoding
• Cons
– Require parsing of bitstream
– Difficult to guarantee syntax remains compliant and will not crash a
decoder
bitstreamimage
Encoder
Transform Entropy Coding Scrambling

23
Scrambling in JPEG
(a) (b) DC
PRNG
pseudo-randomly
inverse sign
scrambled
codestream
public key
assymetric
encryption
seed
(c) DC
PRNG
pseudo-randomly
inverse sign
scrambled
codestream
public key
assymetric
encryption
seed
(d) DC
PRNG
pseudo-randomly
inverse sign
scrambled
codestream
public key
assymetric
encryption
seed
Figure 4 – AC coefficients scrambling:
(a) 63 AC coefficients, (b) 60 AC coefficients, (c) 55 AC coefficients, (d) 48 AC coefficients.
Straightforwardly, as the scrambling is merely flipping signs of selected coefficients, the technique requires negligible
computational complexity. Clearly, the shape of the scrambled region is restricted to match the 8x8 DCT blocks
boundaries.

24
Scrambling in JPEG
(c) (d)

25
Scrambling in JPEG 2000 (JPSEC)
• Codeblock-based bitstream domain scrambling



≥
<+=
→
900
900900mod)('
xxifx
xxifxmxx
x
Preserve the markers in the bitstream; do not introduce erroneous markers
x=current byte, y=preceding byte
1. If x=0xFF, no modification
2. If y=0xFF
3. Otherwise
where m is an 8-bit pseudo-
random number in [0x00,0x8F]
where n is an 8-bit pseudo-
random number in [0x00,0xFE]
xFFnxxx 0mod)(' +=→
selective
scrambling
PRNG
seed encryption
encrypted
seed
scrambled
codestream
JPSEC
codestream
JPSEC
syntax
codestream
quantizer selective
scrambling
PRNG
seed
wavelet
transform
arithmetic
coder
encryption
encrypted
seed
scrambled
codestream
JPSEC
codestream
JPSEC
syntax
image
Encoder Decoder

26
Scrambling in JPEG 2000 (JPSEC)
• ROI-based wavelet domain scrambling
– Arbitrary-shape regions
• Exploit ROI mechanisms in JPEG 2000
Encoder
Decoder
quantizerwavelet
transform
arithmetic
coder
segmentation
mask ?
image
down-shift
wavelet
coefficient
PRNG
seeds
encrypt
seeds
ROI-based scrambled
JPSEC code-stream
scramble
wavelet
coefficient
resolution
level l
< TI ?
up-scale
code-block
distortion
foreground
objects background
keys
resolution
level l
≥ TS ?
yes
no
yesno
inverse
quantizer
inv. wavelet
transform
arithmetic
decoder
coefficient
< 2
s
?
image
up-shift
wavelet
coefficient
PRNG
seeds
decrypt
seeds
ROI-based scrambled
JPSEC code-stream
unscramble
wavelet
coefficient
foreground
objects background
keys
resolution
level l
≥ TS ?
yes
no
Decoder

27
Scrambling in JPEG 2000

28
Scrambling in MPEG-4

29

30

31
Scrambling in H.264/AVC

32
Scrambling in H.264/AVC

33
An existing product
Scrambler Unscrambler

34
Scrambling in DVCScrambling in DVC
• Key frame privacy (JPEG)
– Scrambling in the transform domain on the DCT coefficients.
– Driven by a Pseudo-Random Number Generator (PRNG) to pseudo-
randomly invert the sign of the DCT Coefficients.
• WZ frames
DCT scrambler
DVC scheme with privacy protection

35
Scrambling in DVCScrambling in DVC
a) Key frame (JPEG). b) Wyner-Ziv transform domain scrambling.

36
MPEG-7 camera
XML scene description
The MPEG-7 camera describes a scene in terms of
semantic objects and of their properties

37
MPEG-7 camera
– Image analysis: segmentation, change detection, and tracking
(implemented on the camera DSP).
– MPEG-7 coder: scene description represented using MPEG-7 (XML).
– MPEG-7 decoder: MPEG-7 description is parsed. Extraction of the
information related to the specific applications.

38
MPEG-7 camera
<!-- ################################################## --!>
<!-- DDL output for object 4 --!>
<!-- ################################################## --!>
<Object id="4">
<RegionLocator>
<BoxPoly> Poly </BoxPoly>
<Coords1> 237 222 </Coords1>
</RegionLocator>
<DominantColor>
<ColorSpace> YUV </ColorSpace>
<ColorValue1> 143.4 </ColorValue1>
</DominantColor>
<HomogeneousTexture>
<TextureValue> 9.02 </TextureValue>
</HomogeneousTexture>
<MotionTrajectory>
<TemporalInterpolation>
<KeyFrame> 100 </KeyFrame>
<KeyPos> 268.6 251.7 </KeyPos>
...
</TemporalInterpolation>
</MotionTrajectory>
</Object>
XML scene
description

39
MPEG-7 camera for video surveillance
original frame segmentation mask bounding box

40
Existing product

41
Warping-based privacy filter
• Compute map between original and shifted points
• Interpolate in-between pixels with ‘cubic’
original points shifted points
transformation

42
Recovery from warping
• Know original points and seed for random
algorithm to compute shifted points
• Perform reverse mapping and interpolation
shifted points original points
transformation

43
mild medium strong
Facial features-based warping

44
mild medium strong
Unwarping

45
Morphing

46
Morphing

47
Privacy filter with false color

48
JPEG Transmorphing
JPEG
Transcoder
Mask matrix Sub-image Morphed JPEG image
Original image
Processed image
−
Sub-image embedded
in APPn Markers
T
Reconstructed image
0 ! 0 ! 0 ! 0 ! 0 ! 0 ! 0 ! 0 ! 0 ! 0 !
0 ! 0 ! 0 ! 0 ! 0 ! 0 ! 1 ! 1 ! 0 ! 0 !
0 ! 0 ! 1 ! 1 ! 0 ! 1 ! 1 ! 1 ! 1 ! 0 !
0 ! 1 ! 1 ! 1 ! 1 ! 1 ! 1 ! 1 ! 1 ! 0 !
0 ! 1 ! 1 ! 1 ! 1 ! 0 ! 1 ! 1 ! 0 ! 0 !
0 ! 0 ! 1 ! 1 ! 0 ! 0 ! 0 ! 0 ! 0 ! 0 !
0 ! 0 ! 0 ! 0 ! 0 ! 0 ! 0 ! 0 ! 0 ! 0 !
JPEG
Transcoder
Threshold t
JPEG
Transcoder

49
JPEG Security and Privacy
SOI
APP1 (Exif)
EOI
SOI
APP1 (Exif)
EOI
APP11
(protected
metadata)
JPEG-1 decoder
JPEG Privacy &
Security
decoder
APP1 (Exif)
APP1 (Exif)
original JPEG
codestream
JPEG compatible
codestream with
data protection
Image Data
Image data
APP11
(protected
image data)
Image Data
APP11
(protected
metadata)
Image data
APP11
(protected
image data)
APP3 (JPSearch)
APP3 (JPSearch)
APP3 (JPSearch)

50
Homework!
• Any non-reversible privacy protection filter
can be converted into a reversible
version!
– Propose how this can be done!

51
Thanks for your attention!
End of Part I

52
Outline
• Part I:
– Motivations and context
– Conventional privacy protection
filters
– Advanced privacy protection
filters
• Part II:
– Visual privacy evaluation
framework
– Impact of new imaging
modalities on privacy
– Illustrative example

53
Visual privacy evaluation framework

54
Privacy-intelligibility tradeoff
• Protect privacy – obfuscate or remove
personal information from the video
• Perform surveillance – determine
suspicious event/person, apprehend
and prosecute criminals
• Where is the balance?

55
Evaluation tools
• Subjective evaluation
• Crowdsourcing
• Objective metrics

56
Subjective evaluation
• Three naïve filters: blurring, pixelization,
and masking
• Dataset of 8 annotated videos
– People acting normally or abnormally
– Wearing glasses, scarf, hats, etc.
– Blinking into the camera

57
Example

58
Questions asked to subjects
race white asian I don’t know
gender female male I don’t know
glasses yes no I don’t know
sunglasses yes no I don’t know
scarf yes no I don’t know
blinking yes no I don’t know
Privacy
Intelligibility

59
Subjective results
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
blurring filter
pixelization filter
masking filter
Privacy

60
Facebook-based evaluations
• The same experiment as for subjective
evaluations
• Facebook-based system
– Shows videos
– Collects answers
– Trusted workers

61
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Offline results
Onlineresults
blurring
masking
pixelization
Crowdsourcing: privacy

62
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Offline results
Onlineresults
blurring
masking
pixelization
Crowdsourcing: intelligibility

63
• The cheapest and the most scalable option
• People counting in public transport
– Face recognition is the metric of privacy
– Face detection is the metric of intelligibility
• An ideal privacy protection filter
– Degrades face recognition
– Does not affect face detection
Objective evaluations

64
• Increase strength of privacy filter
• Note relative decrease in accuracy of face
detection and recognition
Evaluation examples

65
Detection Recognition
Gaussian kernel size
Accuracy
Blurring, FERET dataset

66
Morphing, FERET dataset
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Accuracy
Intensity strength
interpolation_0
interpolation_0.2
interpolation_0.4
interpolation_0.6
interpolation_0.8
interpolation_1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Accuracy
Intensity strength
interpolation_0
interpolation_0.2
interpolation_0.4
interpolation_0.6
interpolation_0.8
interpolation_1
Recognition morphed Recognition recovered

67
Impact of new imaging modalities on privacy

68
New imaging modalities
• Ultra high definition (UHD)

69
• High dynamic range (HDR)

70
• Video from mini-drones

71
Ultra High Definition (UHD)

72
Walking example

73
Exchanging bag example

74
Fighting example

75
Subjective evaluations of privacy
• 4K UHD Sony reference monitor
• Evaluation questions about
– People’s accessories
– Main action
– Visible items
– Gender
– Race
• Accompanied questions on certainty

76
Measuring privacy
%ofcorrectanswers

77
High Dynamic Range (HDR)

78
Woman
Elderly
Wears sunglasses
Woman??
Woman
Young??
Woman
Young??
Man?? Middle Age??Man?? Middle Age??
Man
Middle Aged
Man
Middle Aged
Woman
Young
Woman
Young
• HDR surveillance cameras
• More details in the scene
• More privacy intrusive?
• HDR monitor is needed
• Tone-mapped images
Implications of HDR

79

80

81

82

83
Crowdsourcing evaluations of images
• About 400 people participated
• Evaluation questions about
– Gender
– Race
– Age
– Color of clothes
– How many people

84
Crowdsourcing results

85
(Mini-)drone surveillance

86
Drones & surveillance
• Can collect sensitive data
– Different heights
– Different angles
– Fly over fences
• Bird-fly view
• Harass and follow targets
• Privacy protection?

87
Privacy protection filters
• Blurring
• Pixelization
• Masking
• Warping
• Morphing

88
Privacy protection filters
• Different strength levels
– Mild
– Noticeable
– Obfuscating
– Completely obfuscating

89
Crowdsourcing evaluations
• Seven selected video contents
– 5 privacy filters at 4 levels of strength +
original = 21 versions of each content
• 850 online workers from around the world
(Microworkers platform)
– Worker accesses one version of the content
• Six questions about personal privacy and
intelligibility of surveillance
– Also, asked how certain is the answer

90
Evaluation framework
• Questions
– Activity in the scene
– Number of people
– Items (camera, wallet, etc.)
– Accessories people wear (jacket, hat, helmet,
sunglasses, etc.)
– Gender
– Ethnicity

91
Results – correctness
Pixelization

92
Results – correctness
Morphing

93
Results – certainty
Pixelization

94
Results – certainty
Morphing

95
Privacy-intelligibility tradeoff

96
ProShare: A privacy-aware photo sharing platform

97
Illustrative example
User 1
Client-side
Notify User 2
Server-side
Protect User 2
User 2
User 3
User 1User 2
Protect User 1
Friend relationship:
User 1 & 2: ✔ User 1 & 3: ✔ User 2 & 3: ✖
Social Networking Services
URL
• Sender-side operations
– Protection and upload
• Server-side operations
– Hosting and Access control
• Recipient-side operations
– Download and Reconstruction

98
JPEG Privacy & Security

99
JPEG Security and Privacy
SOI
APP1 (Exif)
EOI
SOI
APP1 (Exif)
EOI
APP11
(protected
metadata)
JPEG-1 decoder
JPEG Privacy &
Security
decoder
APP1 (Exif)
APP1 (Exif)
original JPEG
codestream
JPEG compatible
codestream with
data protection
Image Data
Image data
APP11
(protected
image data)
Image Data
APP11
(protected
metadata)
Image data
APP11
(protected
image data)
APP3 (JPSearch)
APP3 (JPSearch)
APP3 (JPSearch)

100
Thanks for your attention!
End of Part II

Reversible visual privacy protection

Recomendados

Recomendados

Mais conteúdo relacionado

Semelhante a Reversible visual privacy protection

Semelhante a Reversible visual privacy protection (20)

Mais de Touradj Ebrahimi

Mais de Touradj Ebrahimi (20)

Último

Último (20)

Reversible visual privacy protection

Notas do Editor