Compressed sensing techniques for sensor data using unsupervised learning

Compressed Sensing Techniques for Sensor Data
using Unsupervised Learning
SONG CUI

COPYRIGHT©2013 SONG CUI. ALL RIGHTS RESERVED

Outline
• Concept of compressed sensing
• Basic theories with interpretations
• Case studies:
Medical imaging (Considered as a sensor network problem)
Wearable electroencephalography (EEG)
Recommendation system


Concept of compressed sensing
Data compression

Data acquisition

Sensor (limited computational capability, transmission power et al.)

• Feature extraction
• Prediction
• Decision making

Data analytics

Learning algorithms

CPU or GPU (excellent computational capability)

Compression techniques
• Traditional lossy compression techniques: JPEG, wavelet et al.
• Traditional compression techniques are not efficient for sparse
data.

Non-sparse data

Sparse data

Why JPEG is ineffective for sparse data
• Discrete cosine transform converts the image (2-D matrix) from

spatial domain to frequency domain in JPEG.
• Insignificant coefficients in frequency domain are discarded in JPEG.
• Sparse data have comparable coefficients in all frequencies.

Keep the first 5 coefficients in
frequency domain and
recover the signal


Data compression


Data compression
• Data compression is through underdetermined linear system.

• It is a dimension reduction process in machine learning.
y = Cx , Where n<m

• Compression matrix C is pre-determined where the compression ratio
is: m/n.

Comparison with PCA
• The compression matrix C in compressed sensing is pre-determined
with some restrictions.
• PCA is more computational demanding:
Normalizing means and variances in training samples x1 , x2 , x3 , x4 ..., xm .
m
Finding the eigenvalues and eigenvectors for (∑ ( xi xi T )) / m.
i =1
Picking up the first n principle eigenvectors to form C.

• Data compression is efficient and requires limited or no
computational power which favors applications (e. g. mobile sensors)
that have limited power, data storage, transmission, and
computational capabilities.

Data reconstruction


Data reconstruction
Can we fully recover the original signal x from its compressed version
y?
y = Cx, Where n<m

The answer is no in general:

y = x1 + x2


Data reconstruction
• However, the signal can be fully recovered if:
The sparsity of the original signal x is s (the maximum number of
non-zero entry is s).
C must satisfy restricted isometry property (RIP) which means any
s columns in matrix C are independent.

Sparsity is the key!


Data reconstruction method
• Ill imposed inverse Problem: find the x from y:
y = Cx

ɶ
ɶ
min || x ||0 s.t . y = Cx

If RIP holds and x is sparse
• L1 norm minimization is used in practice:

ɶ
ɶ
min || x ||1 s.t . || y − Cx ||2 ≤ δ

Minimizing

ɶ
ɶ
|| y − Cx ||2 +λ || x ||1

E. Candès and T. Tao, IEEE Trans. Inform. Theory, vol. 51, no. 12, pp. 4203-4215 (2005).


Why L1 regularization?
Analogy to help
understand L1 and L2
regularizations:
L1 norm minimization

Personal income tax
rate

Original sparse signal

ɶ
ɶ
|| y − Cx ||2 + λ || x || p (p = 1or 2)

Numerical examples are from Prof. W.
K. Ma’s lecture notes
(http://www.ee.cuhk.edu.hk/~wkma).

L2 norm minimization

Parallel computing for large-scale datasets
Data compression:
Data recovery:
1 CPU

y = Cx

( x1 , y1 ), ( x2 , y2 ), ( x3 , y3 ),...

Multi-thread computing in GPU or CPU clusters
ɶ
ɶ
ɶ
|| y1 − Cx ||2 +λ || x ||1 ⇒ x1

ɶ
ɶ
ɶ
|| y1 − Cx ||2 +λ || x ||1 ⇒ x1
ɶ
ɶ
ɶ
|| y2 − Cx ||2 +λ || x ||1 ⇒ x2

ɶ
ɶ
ɶ
|| y2 − Cx ||2 +λ || x ||1 ⇒ x2

ɶ
ɶ
ɶ
|| y3 − Cx ||2 +λ || x ||1 ⇒ x3
......

ɶ
ɶ
ɶ
|| y3 − Cx ||2 +λ || x ||1 ⇒ x3
….

….

Distributed methods are available for single L1 regularization problem.

Case study 1: Medical imaging
• Positron

emission tomography
(PET) is capable of measuring
positron-emitting radionuclides.

• It is a medical diagnostic
instrument for oncology,
neuroimaging, and cardiology
applications.


Background


Compressed sensing multiplexing circuits
• Data is sparse in spatial domain.
• Data compression is
implemented on PCB circuit
boards with pre-amplifiers,
resistors and capacitors.

P. D. Olcott et al., IEEE NSS-MIC Conference Record
p. 3224 (2011).


General applications to wireless sensor network

The figure is from J. Haupt et al., IEEE Signal Processing Mag.
pp. 92 Mar. 2008.


Case study 2: Wearable electroencephalography (EEG)

Figures are from: AJ. Casson et al., IEEE Eng. Med. Biol. Mag 29:44–56 (2010)


Trend of wearable EEG
• Long-term monitoring capability is needed and large-scale data can
be generated.
• Data are processed with machine learning algorithms in the
remote end with strong computational capabilities and large
database.
• Wearable data transmission enables device miniaturization and
body area network applications.
• Applications include sleep disorders and augmented cognition.


Challenges in wearable EEG
• Electrode Technology
• Battery power consumption
Data acquisition
Data transmission

One possible solution: Compressed sensing!


Compressed sensing for scalp EEG
The data is not sparse in time
domain.
The data has sparse
representations in terms of
other basis functions.

A. M. Abdulghani et al., Med. Biol. Eng. Comput. 50:1137–1145 (2012).


Case study 3: Recommendation system
Think about how you build up
a recommendation system!
• Matrix factorization methods
rui = u + qiT pu

• Baseline methods
rui = u + bi + bu

User – item rating matrix: M
Item 1
User 1

Item 2

User 2

Item 4

?

5

Item 3

3

2

User 3

4

3

2

User 4

5

4

?

…………


Interpretation of matrix completion
• We assume that there are
correlations among some users
rating on the same items
(dependency in rows).
• We assume there are
correlations among ratings on
some items from the same user.

Item 1
User 1

Item 2

User 2

Item 4

?

5

Item 3

3

2

User 3

4

3

2

User 4

5

4

?

We hypothesis that rank (M) is small!

Problem formulation
Find a matrix X which fills
up the unknown useritem ratings and satisfy:

NP hard
problem

Minimize rank(X), s.t .X iu = M iu (M iu ∈ Ω)
where rank(X) =|| Σ ||0 , X = U ΣV *

Item 1
User 1

Item 2

User 2

Item 4

?

5

Item 3

3

2

User 3

where

r

|| X ||* = ∑ σ i , σ i

is the singular value

3

2

User 4
Minimize || X ||* ,s.t .X iu = M iu (M iu ∈ Ω)

4
5

4

?

E. Candès, et al., Foundations of Computational Mathematics,
vol. 9, pp. 717 (2009).

i =1


Connection with compressed sensing
• A more robust and computational efficient approach:
Minimize || X − M ||F +λ || X ||*

∑∑ (X
i

u

2

iu

r

− M iu ) + λ ∑ σ i

Previous results:
Minimize || y − Cx ||2 + λ || x ||1

i =1

Interpretation: M is a compressed version
of X!

• The method has been tested on Netflix dataset.
N. Srebro and R. Salakhutdinov., Advances in Neural Information Processing Systems, vol. 23, pp. 2056
(2010).


Challenges
• Does the hypothesis min rank (X) always match with the truth?
• How to deal with non-uniform sampling (e. g. Some user have much
more ratings than others)?
Weighted regularization

• How to deal with cold start problems in the model?
• How to incorporate additional information such as sex, time
drifting, and geography location in the model?

Personal opinions!

Summary
• Compressed sensing is a new data compression and recovery method.
• It is effective for sparse data.
Sparse in time domain.
Sparse in frequency domain.
Sparse in other representations.

•It is useful for mobile sensors which has large-scale data transmission,
limited battery power, computational capabilities and requires device
miniaturization.
• It has seen applications in machine learning such as recommendation
systems.


Extra materials and links
• Stanford course website: http://statweb.stanford.edu/~candes/stats330/index.shtml
• A very resourceful website: Nult Blanche’s blog: http://nuit-blanche.blogspot.com/p/teachingcompressed-sensing.html covers updates from theories and applications such as MRI and
machine learning.
• You can also add me on Linkedin: http://www.linkedin.com/profile/view?id=71703589 or
contact andycui1@hotmail.com if you want to have further discussions.


Compressed sensing techniques for sensor data using unsupervised learning

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (19)

Semelhante a Compressed sensing techniques for sensor data using unsupervised learning

Semelhante a Compressed sensing techniques for sensor data using unsupervised learning (20)

Último

Último (20)

Compressed sensing techniques for sensor data using unsupervised learning