O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.
イベントカメラの研究動向と
ニューラルネットワークによる処理
2
関川雄介
(デンソーアイティーラボラトリ)
Features of
Event-Based
Camera
3
High
speed
1µs
No
motion
blur
Low Low
Power
Sparse
High
dynamic
range
130dB
4
5
240fps⾼速カメラ
6
d-itlab.co.jp
Nagata et.al, SSII2019
6
7
Video from Falanga et.al, How Fast is Too Fast? The Role of Perception Latency in High-Speed Sense and Avoid, RAL2019
7
88
Video from Event-based, 6-DOF Camera Tracking from Photometric Depth Maps, Gallego et.al., PAMI2017
99
Video from Event-based, 6-DOF Camera Tracking from Photometric Depth Maps, Gallego et.al., PAMI2017
Event-Based
Vision
10
• Fast&Sparce,
No-Blur,
HDR,...
Event-Based
Camera
• Process only
for changes
Event-Based
Processing
ニューラルネットワークによる処理, SSII2020
お伝えしたい内容
11
どうやって
処理するの︖
フレームベースカメラ:
輝度の画像を取得するカメラ
イベントベースカメラ:
輝度の差分を取得するカメラ
なにが嬉しいの︖
Video fro...
ニューラルネットワークによる処理, SSII2020
⾃⼰紹介
§2004- 経済産業省特許庁
➢特許審査 (移動体通信)
§2008- オリンパスイメージング
➢無線ファームウェア開発
➢カメラ商品企画
§2012- デンソーアイティーラボラ...
ニューラルネットワークによる処理, SSII2020
もくじ
§フレームベースビジョンとその課題 (5min)
§イベントベースカメラ (10min)
§特徴/原理/難しさ
§イベントカメラの研究動向(アルゴリズム+嬉しさ) (50min)
§...
Frame-based Vision
概要と課題
14
ニューラルネットワークによる処理, SSII2020
Frame-based Vision: Sensing & Processing
15
Image from Wikipedia / Video from Inivation
Y.Sekik...
16
Problem?
16
ニューラルネットワークによる処理, SSII2020
Motion blur
17Y.Sekikawa, イベントカメラの研究動向と,
ニューラルネットワークによる処理, SSII2020
Limited Dynamic Range
18
Image from expertphotography.com
Y.Sekikawa, イベントカメラの研究動向と,
ニューラルネットワークによる処理, SSII2020
Speed vs Power/Data Rate/Price tradeoff
19
Fram
e-based
cam
eraSpeed[fps]
Datarate[bps]
Image f...
ニューラルネットワークによる処理, SSII2020
Frame-based Processing
20
106FPS
Tracking, Recognition, …
Y.Sekikawa, イベントカメラの研究動向と,
Event-based Vision
21
ニューラルネットワークによる処理, SSII2020
Biological Vision
“Retina is sensitive to temporal brightness gradients”
“Retina is blind to st...
ニューラルネットワークによる処理, SSII2020
Event-based Vision
23
Sensing
Retina (Event Camera)
Processing
Brain (CPU, GPU, SNN-Proc.)
Y.Se...
ニューラルネットワークによる処理, SSII2020
History
24
2010
・ 1991 Mahowald et.al
1990 20202000
・2020 Gen.4
1280 x 720 w/ SONY
・2018 Celexl...
ニューラルネットワークによる処理, SSII2020
Comparison between different event cameras
25
Prophesee(Chronocam) iniVation(iniLabs) Samsung C...
ニューラルネットワークによる処理, SSII2020
Consumer/Industrial products
26
Images from Samsung (left)/ Prophesee.ai (right)
• Low-power
• ...
ニューラルネットワークによる処理, SSII2020
Bio-inspired Retina: Event-based Cameras driven by intensity changes
Event-Based: Asynchronous
...
28Video from, Hanme Kim, Event-Based Camera vs Standard Camera)
29Video from, Hanme Kim, Event-Based Camera vs Standard Camera)
30Video from, Hanme Kim, Event-Based Camera vs Standard Camera)
31
Video from, Inivation
Video from, Inivation
Time
LogIntensity
31
ニューラルネットワークによる処理, SSII2020
Event generation Model
§Each pixel asynchronously report intensity changes
Image from Kim et.al...
33
Time
LogIntensity
!
""
§ Intensity difference Δ" > C ⟶ trigger event
Δ" #$, %$ ≐ " #$, %$ − " #$, %$ − Δ%$
Δ" #$, %$ = ...
34
Δ" = $!% ≈ −("/(* ⋅ ,Δ-
v Temporal relation
v Spatial relation (optical flow constraint)
) #$, %$ − ) #$, %$%& ≈ ,
$∈(
...
ニューラルネットワークによる処理, SSII2020
Event-Based Camera
35
• High speed (1µs)
• Low data rate/Sparse (0-30Mbps)
• No motion blur
• H...
ニューラルネットワークによる処理, SSII2020
Difficulties when dealing with event data
§Sparse data representation: Frame-based alg. cannot ...
ニューラルネットワークによる処理, SSII2020
Wide Range of Usage
Algorithm
§Tracking
§Optical Flow
§Visual odometry
§SLAM
§Image Reconstruct...
イベントカメラの研究動向
第1部 Model-Based
うれしさ
技術ポイント
38
ニューラルネットワークによる処理, SSII2020
ML-Based (End-to-end neural-network for event processing)
39Y.Sekikawa, イベントカメラの研究動向と,
Algorith...
ニューラルネットワークによる処理, SSII2020
Speed Invariant Time Surface
for Learning to Detect Corner Points with Event-Based Cameras
Mand...
ニューラルネットワークによる処理, SSII2020
EKLT: Asynchronous Photometric Feature Tracking Using Events and Frames
Gehrig et.al., Depts. I...
4242
ニューラルネットワークによる処理, SSII2020
Simultaneous Optical Flow and Intensity Estimation from an Event Camera
Bardow et.al., Imperial...
ニューラルネットワークによる処理, SSII2020
Continuous-time Intensity Estimation Using Event Cameras
Scheerlinck et.al., ACCV2018
37
Real-t...
4545
ニューラルネットワークによる処理, SSII2020
Simultaneous Mosaicing and Tracking with an Event Camera
§Kim et.al, Imperial College London, B...
4747
ニューラルネットワークによる処理, SSII2020
Real-Time 3D Reconstruction and 6-DoF Tracking with an Event Camera
Kim et.al, Imperial College...
ニューラルネットワークによる処理, SSII2020
Focus Is All You Need: Loss Functions For Event-based Vision
Guillermo et.al., UZH@ETH, CVPR201...
5050
51
Motion segmentation
Video from: Event-Based Motion Segmentation by Motion Compensation (ICCV'19) 51
ニューラルネットワークによる処理, SSII2020
EMVS: Event-based Multi-View Stereo
Henri et.al, RPG@ETH, BMVC 2016
52
195
Simple/Fast/Easy 3D ...
53
ニューラルネットワークによる処理, SSII2020
EVO: A Geometric Approach to Event-Based 6-DOF Parallel Tracking
and Mapping in Real-time
Rebec...
5555
ニューラルネットワークによる処理, SSII2020
Ultimate SLAM? combining events, images, and IMU
for robust visual SLAM in HDR and high speed s...
5757
ニューラルネットワークによる処理, SSII2020
CameraProjector
MC3D: Motion Contrast 3D Scanning Nathan
Nathan et.al., Evanston, ICCP2015
58
2...
5959
6060
ニューラルネットワークによる処理, SSII2020
Event-Based Structured Light for Depth Reconstruction using Frequency Tagged
Light Patterns
Ler...
イベントカメラの研究動向
第2部 ML-Based
うれしさ
技術ポイント
62
ニューラルネットワークによる処理, SSII2020
ML-Based (End-to-end neural-network for event processing)
63
Event camera
Pros: Can be utilize ...
ニューラルネットワークによる処理, SSII2020
ML-Based (End-to-end neural-network for event processing)
64
Event camera
Pros: Can be utilize ...
ニューラルネットワークによる処理, SSII2020
Event-based Vision meets Deep Learning
on Steering Prediction for Self-driving Cars
Maqueda et....
ニューラルネットワークによる処理, SSII2020
Industrial DVS Design: Key Features and Applications
Ryu et.al., Samsung, CVPR2019WS
66
ニューラルネットワークによる処理, SSII2020
Learning an event sequence embedding for dense event-based deep stereo
Tulyakov et.al., EPLF, I...
ニューラルネットワークによる処理, SSII2020
End-to-End Learning of Representations for Asynchronous Event-Based Data
Gehrig et.al., RPG@ETH...
ニューラルネットワークによる処理, SSII2020
Matrix-LSTM: a Differentiable Recurrent Surface for Asynchronous Event-Based Data
Cannici et.al...
ニューラルネットワークによる処理, SSII2020
Constant Velocity 3D Convolution
Sekikawa et.al., 3DV IEEE Access 2018
70
[Background] 3D Convo...
71
constant velocity 3d kernel
3dconv
>1,000x less MAP※
※multiply–accumulate operation
decompose cv3dconv(ours)
=∑*
71
#
7272
ニューラルネットワークによる処理, SSII2020
High Speed and High Dynamic Range Video with an Event Camera
Rebecq et.al., UZH@ETH, CVPR 2019 ...
7474
ニューラルネットワークによる処理, SSII2020
ML-Based (End-to-end neural-network for event processing)
75
Event camera
Pros: Can be utilize ...
ニューラルネットワークによる処理, SSII2020
ML-Based (End-to-end neural-network for event processing)
76
Event camera
Pros: Can be utilize ...
ニューラルネットワークによる処理, SSII2020
SNN (Spikingk Neural Network):
3rd generation of neural network: Spiking Neural Network
77
Leak...
ニューラルネットワークによる処理, SSII2020
SNN Hardware
TrueNorth DYNAP Loihi Braindrop
Manufacture IBM aiCTX Intel Stanford
Type of neuro...
ニューラルネットワークによる処理, SSII2020
Categorization of training SNN
Supervised
Rewarded-STDP ANN (Back-propagation) to SNN
Unsupervi...
ニューラルネットワークによる処理, SSII2020
Synaptic Modifications in Cultured Hippocampal Neurons:
Dependence on Spike Timing, Synaptic St...
81
Image Diehl et.al, l. Unsupervised learning of digit recognition using spike-timing-dependent plasticity
Simple mapping...
ニューラルネットワークによる処理, SSII2020
A Low Power, Fully Event-Based Gesture Recognition System
Amir et.al., IBM Research+UZH-ETH, CV...
8383
ニューラルネットワークによる処理, SSII2020
Training Deep Spiking Neural Networks Using Backpropagation
Lee et.al, Institute of Neuroinform...
85
"
# 0
9
Spike rate
Approximate with differentiable continuous function
Spike rate GradientLow pass
ニューラルネットワークによる処理, SSII2020
Random synaptic feedback weights support error backpropagation
for deep learning
Lillicrap et.a...
ニューラルネットワークによる処理, SSII2020
Event-Driven Random Back-Propagation:
Enabling Neuromorphic Deep Learning Machines
Neftci et.al...
ニューラルネットワークによる処理, SSII2020
ML-Based (End-to-end neural-network for event processing)
Event camera
Pros: Can be utilize exi...
ニューラルネットワークによる処理, SSII2020
EventNet: Asynchronous recursive event processing
Sekikawa et.al, CVPR 2019
89
Event camera
Pro...
Problem Statement: Asynchronously Model Event Stream
90
Requirements
§Sparse Event-based Processing (No densification)
§Re...
ニューラルネットワークによる処理, SSII2020
[Related work] PointNet: Deep Learning on Point Sets for 3D Classification and
Segmentation
§Qi...
92
6(7(#))
Pentagon
Star
8,9({. . . })
{&!, . . . , &"
(,)
}
{&!, . . . , &"
(!)
}
ℎ*+,(⋅)
). = ℎ*+,(+.)
6(7(*))
7(#) = 8,...
ニューラルネットワークによる処理, SSII2020
Idea 1: Recursive computation by temporal coding (t-code)
MLP ℎ
nx1024
shared global
feature
ML...
ニューラルネットワークによる処理, SSII2020
mlp (64,64,64,128,1024)
Idea 2: LUT※ Realization of MLP ℎ
※Look-up table
>
?
> ?
,(-@) = '(./0(...
95
Triangle other
Setup Output
95
ニューラルネットワークによる処理, SSII2020
Event-based Asynchronous Sparse Convolutional Networks
Messikommer et.al, arXiv2020
Synchronous...
9797
まとめ
98
ニューラルネットワークによる処理, SSII2020
まとめ イベントカメラって︖
§明るさの変化を観測するカメラ
§センサーとして良い特徴
§ HDR・⾼速・ブラーレス・コンパクトデータ
§ うまく使えば難しい環境で動作する低計算量で⾼速レス...
ニューラルネットワークによる処理, SSII2020
まとめ どうやって使うの︖
§データの形式や特性がフレーム画像と違うので,“フレーム画像処理“がそのまま使えない
§⾮同期&スパース
§動きで⾒えが変わる
§イベントの特性を⽣かした処理で ...
ニューラルネットワークによる処理, SSII2020
Reference
• Gallego et.al., Event-based Vision: A Survey
• Gallego et.al., Event-based Vision R...
102102
Próximos SlideShares
Carregando em…5
×

SSII2020TS: Event-Based Camera の基礎と ニューラルネットワークによる信号処理 〜 生き物のように「変化」を捉えるビジョンセンサーと 「変化」を処理するニューラルネットワーク 〜​

2.339 visualizações

Publicada em



SSII2020 チュートリアルセッション TS2
6/11 (木) 9:30~10:40 メイン会場 (vimeo + sli.do)
イベントカメラは、全ピクセルが同期して明るさを捉える従来のフレームカメラと異なり、生物の目のように各ピクセルが非同期に明るさの「変化」を捉える新しいタイプのビジョンセンサーである。 HDR、疎、高速といった特徴から近年着目されている。しかしながら、カメラから得られる信号は「画像」ではなく、疎な「変化」であるため従来の画像処理を用いることが困難である。この「変化」を扱う種々の手法について、特に疎なイベント信号をそのまま扱うためのニューラルネットワークにフォーカスして説明する。

Publicada em: Tecnologia
  • Seja o primeiro a comentar

SSII2020TS: Event-Based Camera の基礎と ニューラルネットワークによる信号処理 〜 生き物のように「変化」を捉えるビジョンセンサーと 「変化」を処理するニューラルネットワーク 〜​

  1. 1. イベントカメラの研究動向と ニューラルネットワークによる処理 2 関川雄介 (デンソーアイティーラボラトリ)
  2. 2. Features of Event-Based Camera 3 High speed 1µs No motion blur Low Low Power Sparse High dynamic range 130dB
  3. 3. 4
  4. 4. 5 240fps⾼速カメラ
  5. 5. 6 d-itlab.co.jp Nagata et.al, SSII2019 6
  6. 6. 7 Video from Falanga et.al, How Fast is Too Fast? The Role of Perception Latency in High-Speed Sense and Avoid, RAL2019 7
  7. 7. 88 Video from Event-based, 6-DOF Camera Tracking from Photometric Depth Maps, Gallego et.al., PAMI2017
  8. 8. 99 Video from Event-based, 6-DOF Camera Tracking from Photometric Depth Maps, Gallego et.al., PAMI2017
  9. 9. Event-Based Vision 10 • Fast&Sparce, No-Blur, HDR,... Event-Based Camera • Process only for changes Event-Based Processing
  10. 10. ニューラルネットワークによる処理, SSII2020 お伝えしたい内容 11 どうやって 処理するの︖ フレームベースカメラ: 輝度の画像を取得するカメラ イベントベースカメラ: 輝度の差分を取得するカメラ なにが嬉しいの︖ Video from, Inivation Y.Sekikawa, イベントカメラの研究動向と, イベントデータ︓スパースで⾮同期な時系列情報
  11. 11. ニューラルネットワークによる処理, SSII2020 ⾃⼰紹介 §2004- 経済産業省特許庁 ➢特許審査 (移動体通信) §2008- オリンパスイメージング ➢無線ファームウェア開発 ➢カメラ商品企画 §2012- デンソーアイティーラボラトリ ➢MIT Media Lab Tangible Media ➢Computational Photography ➢画像テンプレートマッチング ➢⽣成モデル学習(GAN) ➢Event-Basedカメラの信号処理 §2020 PhD@慶應⼤ 斎藤研 Event Cameraのテーマ 12Y.Sekikawa, イベントカメラの研究動向と,
  12. 12. ニューラルネットワークによる処理, SSII2020 もくじ §フレームベースビジョンとその課題 (5min) §イベントベースカメラ (10min) §特徴/原理/難しさ §イベントカメラの研究動向(アルゴリズム+嬉しさ) (50min) §モデルベース (20min) §トラッキング,輝度復元,VO/SLAM,3次元復元 §機械学習ベース (30min) §フレームに変換 §そのまま処理(SNN※,我々のアプローチ) § まとめ 13Y.Sekikawa, イベントカメラの研究動向と, ※Spiking Neural Network
  13. 13. Frame-based Vision 概要と課題 14
  14. 14. ニューラルネットワークによる処理, SSII2020 Frame-based Vision: Sensing & Processing 15 Image from Wikipedia / Video from Inivation Y.Sekikawa, イベントカメラの研究動向と, Frame-based Sensing Frame-based Processing frame CMOS sensor
  15. 15. 16 Problem? 16
  16. 16. ニューラルネットワークによる処理, SSII2020 Motion blur 17Y.Sekikawa, イベントカメラの研究動向と,
  17. 17. ニューラルネットワークによる処理, SSII2020 Limited Dynamic Range 18 Image from expertphotography.com Y.Sekikawa, イベントカメラの研究動向と,
  18. 18. ニューラルネットワークによる処理, SSII2020 Speed vs Power/Data Rate/Price tradeoff 19 Fram e-based cam eraSpeed[fps] Datarate[bps] Image from ix-camera EnergyComsumption[W] Price[$] Y.Sekikawa, イベントカメラの研究動向と,
  19. 19. ニューラルネットワークによる処理, SSII2020 Frame-based Processing 20 106FPS Tracking, Recognition, … Y.Sekikawa, イベントカメラの研究動向と,
  20. 20. Event-based Vision 21
  21. 21. ニューラルネットワークによる処理, SSII2020 Biological Vision “Retina is sensitive to temporal brightness gradients” “Retina is blind to static scenes in absence of eye movements ” 22 Receptive fields of single neurons in the cat’s striate cortex David H Hubel et.al.,1959, Nobel prize 1981
  22. 22. ニューラルネットワークによる処理, SSII2020 Event-based Vision 23 Sensing Retina (Event Camera) Processing Brain (CPU, GPU, SNN-Proc.) Y.Sekikawa, イベントカメラの研究動向と,
  23. 23. ニューラルネットワークによる処理, SSII2020 History 24 2010 ・ 1991 Mahowald et.al 1990 20202000 ・2020 Gen.4 1280 x 720 w/ SONY ・2018 Celexl V 1280 x 960 Y.Sekikawa, イベントカメラの研究動向と, 2015- 2014- ・2020 DVXplorer 640 x 480 2012- ・2018 Samsung Gen.3 Event-Based Sencing Device 2017- ・2020 GaAI One ・2018 DyNap CNN2017- ・2014 IBM TrueNorth ・2018 Intel Loihi ・2018 Stanford Braindrop Event-Based Processing Device ・2009 Lichtsteiner et.al 128x128
  24. 24. ニューラルネットワークによる処理, SSII2020 Comparison between different event cameras 25 Prophesee(Chronocam) iniVation(iniLabs) Samsung Celepixel(Hillhouse) Latest version ATIS-Gen4 DAVIS346 DVS Gen.4 CeleX-V Resolution CD : 1280 x 720 CD+EM : ? 346x260 1280 x 960 1280 x 800 Pixel pitch CD : 4.86μm CD+EM : ? 18.5μm 4.95μm 9.8μm Intensity information EM: Exposure Measurement 130dB Event resets a capacitor to a high voltage. Brighter →faster discharges APS: Active pixel sensor 56.7dB Similar to standard frame N/A Intensity at event-rate Other feature / Info IMARGO industrial Camera Joint dev. with SONY Sony acquires Insightness DVXplorer (640 x 480 no intensity) In-home monitoring camera Event-wise optical flow Commercial product for DSM in China Y.Sekikawa, イベントカメラの研究動向と,
  25. 25. ニューラルネットワークによる処理, SSII2020 Consumer/Industrial products 26 Images from Samsung (left)/ Prophesee.ai (right) • Low-power • HDR • High-speed • HDR Y.Sekikawa, イベントカメラの研究動向と,
  26. 26. ニューラルネットワークによる処理, SSII2020 Bio-inspired Retina: Event-based Cameras driven by intensity changes Event-Based: Asynchronous Time LogIntensity Sensor array Frame-Based: Synchronous Exposure time Intensity Time 27Y.Sekikawa, イベントカメラの研究動向と, frame
  27. 27. 28Video from, Hanme Kim, Event-Based Camera vs Standard Camera)
  28. 28. 29Video from, Hanme Kim, Event-Based Camera vs Standard Camera)
  29. 29. 30Video from, Hanme Kim, Event-Based Camera vs Standard Camera)
  30. 30. 31 Video from, Inivation Video from, Inivation Time LogIntensity 31
  31. 31. ニューラルネットワークによる処理, SSII2020 Event generation Model §Each pixel asynchronously report intensity changes Image from Kim et.al., et.al, Simultaneous mosaicing and tracking with an event camera 32Y.Sekikawa, イベントカメラの研究動向と,
  32. 32. 33 Time LogIntensity ! "" § Intensity difference Δ" > C ⟶ trigger event Δ" #$, %$ ≐ " #$, %$ − " #$, %$ − Δ%$ Δ" #$, %$ = '$(, Frame Δ"" $"
  33. 33. 34 Δ" = $!% ≈ −("/(* ⋅ ,Δ- v Temporal relation v Spatial relation (optical flow constraint) ) #$, %$ − ) #$, %$%& ≈ , $∈( ('$ Note: No event when image gradient is perpecdicular to motion −("/(* , Time LogIntensity ! """"#$ 34
  34. 34. ニューラルネットワークによる処理, SSII2020 Event-Based Camera 35 • High speed (1µs) • Low data rate/Sparse (0-30Mbps) • No motion blur • High dynamic range (130dB) ✔ ✔ Fram e-based cam era Speed[fps] EnergyComsumption[W] Event-based camera Datarate[bps] Price[$] ✔ Y.Sekikawa, イベントカメラの研究動向と,
  35. 35. ニューラルネットワークによる処理, SSII2020 Difficulties when dealing with event data §Sparse data representation: Frame-based alg. cannot be applied Image from Gehrig et.al, Asynchronous, Photometric Feature Tracking using Events and Frames 36Y.Sekikawa, イベントカメラの研究動向と, §Motion dependent data: Association in SLAM / Generalization in ML Frame: Motion Independent Event (Histogram): Motion Dependent
  36. 36. ニューラルネットワークによる処理, SSII2020 Wide Range of Usage Algorithm §Tracking §Optical Flow §Visual odometry §SLAM §Image Reconstruction §Stereo depth estimation §3D measurement with SL §Object Recognition §Etc.. 37 Applications §Surveillance at Home §Obstacle avoidance §UAV, automotive §Bin-picking §Gesture recognition §Etc.. Y.Sekikawa, イベントカメラの研究動向と,
  37. 37. イベントカメラの研究動向 第1部 Model-Based うれしさ 技術ポイント 38
  38. 38. ニューラルネットワークによる処理, SSII2020 ML-Based (End-to-end neural-network for event processing) 39Y.Sekikawa, イベントカメラの研究動向と, Algorithm • Feature Tracking • Optical Flow (OF) • Visual Odometry (VO) • Simultaneous Localization and Mapping (SLAM) • 3D Reconstruction • Intensity Reconstruction (IR) Model-based Processing Setup Image from Gallego et.al., Event-based, 6-DOF Camera Tracking from Photometric Depth Maps Geometry Planer/Known/3D? Texture Known/Estimate? Known/Estimate? Rotation/SE(2)/SE(3)? Environment Static/Dynamic Use image as Proxy/Direct? Ext. sensor /Reconstruct ? Algorithm Input
  39. 39. ニューラルネットワークによる処理, SSII2020 Speed Invariant Time Surface for Learning to Detect Corner Points with Event-Based Cameras Manderscheid et.al., CVPR2019 Very fast/robust corner tracking in challenging illumination conditions Corner detetion using SI time surface Event to Time Surface SI:Speed Invariant 40 Corner tracking Event only Y.Sekikawa, イベントカメラの研究動向と, Image (on the bottom) from Alzugaray, et.al., Asynchronous Corner Detection and Tracking for Event Cameras in Real Time Video from Alzugaray et.al., Asynchronous Corner Detection and Tracking for Event Cameras Tracking by simple Nearest Neighbor association
  40. 40. ニューラルネットワークによる処理, SSII2020 EKLT: Asynchronous Photometric Feature Tracking Using Events and Frames Gehrig et.al., Depts. Informatics and Neuro informatics@ETH, ECCV2018 IJCV2019 159 ! = arg min ! ( − ) Very fast/rubust feature tracking in challenging illumination conditions Compare intensity increment from event with prediction from frame 41 Feature tracking Event + Frame Y.Sekikawa, イベントカメラの研究動向と,
  41. 41. 4242
  42. 42. ニューラルネットワークによる処理, SSII2020 Simultaneous Optical Flow and Intensity Estimation from an Event Camera Bardow et.al., Imperial College Dyson Lab., CVPR2016 43 32 Data term "# Optical flow Intensity & OF estimation at high rate in challenging illumination conditions Joint optimization using optical flow constrains IR+OF Event Y.Sekikawa, イベントカメラの研究動向と,
  43. 43. ニューラルネットワークによる処理, SSII2020 Continuous-time Intensity Estimation Using Event Cameras Scheerlinck et.al., ACCV2018 37 Real-time & high rate intensity estimation in challenging inllumination conditions Complementary fusion of frame and event 44 IR Event+Frame Y.Sekikawa, イベントカメラの研究動向と,
  44. 44. 4545
  45. 45. ニューラルネットワークによる処理, SSII2020 Simultaneous Mosaicing and Tracking with an Event Camera §Kim et.al, Imperial College London, BMVC2014 46 Localization (PF) 39 Mapping (EKF based IR) SLAM (rotation only) in challenging illumination conditions Mapping by intensity reconstructing (Pioneering work for event-based SLAM) Intencity MAP SLAM(SO(3)) Event Y.Sekikawa, イベントカメラの研究動向と,
  46. 46. 4747
  47. 47. ニューラルネットワークによる処理, SSII2020 Real-Time 3D Reconstruction and 6-DoF Tracking with an Event Camera Kim et.al, Imperial College London, ECCV2016 (Best Paper) 48 47 Full 6DOF SLAM in challenging illumination conditions Extension of Kim2016 to SE(3) by incorporation depth estimation SLAM(SO(3)) Event Y.Sekikawa, イベントカメラの研究動向と,
  48. 48. ニューラルネットワークによる処理, SSII2020 Focus Is All You Need: Loss Functions For Event-based Vision Guillermo et.al., UZH@ETH, CVPR2018, CVPP2019 49 120 Efficient motion (OF) estimation in challenging illumination conditions OF estimation w/o intensity. Novel focus-based loss OF Event Y.Sekikawa, イベントカメラの研究動向と,
  49. 49. 5050
  50. 50. 51 Motion segmentation Video from: Event-Based Motion Segmentation by Motion Compensation (ICCV'19) 51
  51. 51. ニューラルネットワークによる処理, SSII2020 EMVS: Event-based Multi-View Stereo Henri et.al, RPG@ETH, BMVC 2016 52 195 Simple/Fast/Easy 3D reconstruction in challenging illumination conditions Vote events into DSI using know trajectory DSI: Disparity Space Image (DSI) Mapping Event Y.Sekikawa, イベントカメラの研究動向と, max
  52. 52. 53
  53. 53. ニューラルネットワークによる処理, SSII2020 EVO: A Geometric Approach to Event-Based 6-DOF Parallel Tracking and Mapping in Real-time Rebecq et.al., UZH@ETH, RAL2016 54 48 Very fast SLAM (500Hz on CPU) in challenging illumination conditions Utilize DSI (No IR, edge-map alignment suffice) SLAM(SE(3)) Event Y.Sekikawa, イベントカメラの研究動向と,
  54. 54. 5555
  55. 55. ニューラルネットワークによる処理, SSII2020 Ultimate SLAM? combining events, images, and IMU for robust visual SLAM in HDR and high speed scenarios Vidal et.al., RPG@ETH, ROBOTICS AND AUTOMATION LETTERS 2017 56 50 Efficient SLAM in challenging illumination conditions Utilize all available sensors for computational efficiently and robustness SLAM(SE(3)) Event+Frame+Gyro Y.Sekikawa, イベントカメラの研究動向と,
  56. 56. 5757
  57. 57. ニューラルネットワークによる処理, SSII2020 CameraProjector MC3D: Motion Contrast 3D Scanning Nathan Nathan et.al., Evanston, ICCP2015 58 28 Real-time & precise 3D reconstruction Utilize precise event-time stamp for easy & robust correspondence 3D Rec Event+Projector Y.Sekikawa, イベントカメラの研究動向と, (%!, '!, (!, )!) (%"!, '", (", )")
  58. 58. 5959
  59. 59. 6060
  60. 60. ニューラルネットワークによる処理, SSII2020 Event-Based Structured Light for Depth Reconstruction using Frequency Tagged Light Patterns Leroux, et.al., University of Pittburgh&CMU&Sorbonne&Universitasx&Prpphesee, arXiv 2018 61 Light weight & real-time 3D reconstruction w/o synchronization Encode code for temporal dimension → decode by simple pixel-wise correlation 3D Rec Event+Projector Y.Sekikawa, イベントカメラの研究動向と,
  61. 61. イベントカメラの研究動向 第2部 ML-Based うれしさ 技術ポイント 62
  62. 62. ニューラルネットワークによる処理, SSII2020 ML-Based (End-to-end neural-network for event processing) 63 Event camera Pros: Can be utilize exiting architecture !! (e.g., CNN) Cons: Inefficient, Slow Frame-based processing !! ≔ x, y, p, t " fD (g(e)) = y Event-based processing Pros: Efficient (Process only for changed pixels) Cons: No established method like CNN Event to Frame "(⋅) fS (e) = y
  63. 63. ニューラルネットワークによる処理, SSII2020 ML-Based (End-to-end neural-network for event processing) 64 Event camera Pros: Can be utilize exiting architecture !! (e.g., CNN) Cons: Inefficient, Slow Frame-based processing !! ≔ x, y, p, t " fD (g(e)) = y Event-based processing Pros: Efficient (Process only for changed pixels) Cons: No established method like CNN Event to Frame "(⋅) fS (e) = y
  64. 64. ニューラルネットワークによる処理, SSII2020 Event-based Vision meets Deep Learning on Steering Prediction for Self-driving Cars Maqueda et.al., Dept. of Informatics and Neuroinformatics@ETH, CVPR2018 65 126 Sophisticated CNN can be used Convert sparse events to dense frame
  65. 65. ニューラルネットワークによる処理, SSII2020 Industrial DVS Design: Key Features and Applications Ryu et.al., Samsung, CVPR2019WS 66
  66. 66. ニューラルネットワークによる処理, SSII2020 Learning an event sequence embedding for dense event-based deep stereo Tulyakov et.al., EPLF, ICCV2019 67 Event camera Frame-based processing !! ≔ x, y, p, t " fD (g(e)) = y Event to Frame "(⋅) & Better than hand crafted conversion Learn to convert (temporal kernel ) sparse events to dense frame '&
  67. 67. ニューラルネットワークによる処理, SSII2020 End-to-End Learning of Representations for Asynchronous Event-Based Data Gehrig et.al., RPG@ETH, ICCV2019 68 !±[#", %#, &$] = (# ∗ %±)('", )#*$) = ∑ !(∈ℰ± #±(%%, '%, (%)*(%& − %%, '' − '%, (( − (%) 132 Trainable Kernel
  68. 68. ニューラルネットワークによる処理, SSII2020 Matrix-LSTM: a Differentiable Recurrent Surface for Asynchronous Event-Based Data Cannici et.al, arXiv 2020 69
  69. 69. ニューラルネットワークによる処理, SSII2020 Constant Velocity 3D Convolution Sekikawa et.al., 3DV IEEE Access 2018 70 [Background] 3D Convolution: Common strategy for capturing spatiotemporal feature § Problem: Computationally intensive ! ! % = ' ⊛ ) Time-surface representation of stream of events Red: Newer, Blue: Older [Key observation] Spatiotemporal event ≈ piece-wise linear movements of 2D feature Efficient 3D convolution to capture spatiotemporal features Decompose constant velocity 3D kernel into 2D conv+sum # # 2D kernel + linear motion
  70. 70. 71 constant velocity 3d kernel 3dconv >1,000x less MAP※ ※multiply–accumulate operation decompose cv3dconv(ours) =∑* 71 #
  71. 71. 7272
  72. 72. ニューラルネットワークによる処理, SSII2020 High Speed and High Dynamic Range Video with an Event Camera Rebecq et.al., UZH@ETH, CVPR 2019 PAMI 2019 73 Existing alg. can be readily applicable for challenging applications Leaning to convert sparse event to intensity frame
  73. 73. 7474
  74. 74. ニューラルネットワークによる処理, SSII2020 ML-Based (End-to-end neural-network for event processing) 75 Event camera Pros: Can be utilize exiting architecture !! (e.g., CNN) Cons: Inefficient, Slow Frame-based processing !! ≔ x, y, p, t " fD (g(e)) = y Event-based processing Pros: Efficient (Process only for changed pixels) Cons: No established method like CNN Event to Frame "(⋅) fS (e) = y
  75. 75. ニューラルネットワークによる処理, SSII2020 ML-Based (End-to-end neural-network for event processing) 76 Event camera Pros: Can be utilize exiting architecture !! (e.g., CNN) Cons: Inefficient, Slow Frame-based processing !! ≔ x, y, p, t " fD (g(e)) = y Event-based processing (Spike) Pros: Efficient (Process only for changed pixels) Cons: Requires Neuromorphic H/W (less neurons), Difficult to train Event to Frame "(⋅) Event-based processing (Continuous) fS (e) = y
  76. 76. ニューラルネットワークによる処理, SSII2020 SNN (Spikingk Neural Network): 3rd generation of neural network: Spiking Neural Network 77 Leaky and Integrate and Fire (LIF) Charge → Fire(10ms)→ Refractory(100ms)→ " # Activation: Non-differentiable Spike (ANN: Relu, Sigmoid) Asynchronous: MP※ > threshold → Fire (Similar to Event Camera) MP%(') Non-differentiable *+ *,! = *+ *," ⋅ … *,# *,! Chain Rule ※MP: membrane potential
  77. 77. ニューラルネットワークによる処理, SSII2020 SNN Hardware TrueNorth DYNAP Loihi Braindrop Manufacture IBM aiCTX Intel Stanford Type of neurons Digital LIF Analog LIF Digital LIF Analog Neurons per chip 1,0000,000 4096x4 130,000x8 4096 Year 2014 2017 2018 2018 Programing Corelet, Eedn libcaer /cAER in C/C++ Nengo/Brain/PyNN Nengo Training Outside chip On chip On chip Outside chip For more detailed review see Young et.al. A Review of Spiking Neuromorphic Hardware Communication Systems, IEEE Access 2019 78
  78. 78. ニューラルネットワークによる処理, SSII2020 Categorization of training SNN Supervised Rewarded-STDP ANN (Back-propagation) to SNN Unsupervised STDP Back-propagation • Continuous relaxation (Approximate gradient / Inefficient) • Temporal Coding (Exact / Dead neuron) • Random back propagation 79
  79. 79. ニューラルネットワークによる処理, SSII2020 Synaptic Modifications in Cultured Hippocampal Neurons: Dependence on Spike Timing, Synaptic Strength, and Postsynaptic Cell Type Bi et.al., Journal of Neuroscience, 1998 Images from arc-instruments Weight update Simple unsupervised training for non-differentiable spike. Neuroplausible Hebb rule: “who fire together, wire together” 80
  80. 80. 81 Image Diehl et.al, l. Unsupervised learning of digit recognition using spike-timing-dependent plasticity Simple mapping yealds 95% accuracy 81
  81. 81. ニューラルネットワークによる処理, SSII2020 A Low Power, Fully Event-Based Gesture Recognition System Amir et.al., IBM Research+UZH-ETH, CVPR2017 Realized efficient gesture recognition using real SNN H/W (TrueNorth) Convert trained ANN to SNN 82
  82. 82. 8383
  83. 83. ニューラルネットワークによる処理, SSII2020 Training Deep Spiking Neural Networks Using Backpropagation Lee et.al, Institute of Neuroinformatics@ETH, Frontiers in Neuroscience 2016 84 Events by emulation saccade " # 0 9 Non-differentiable 0 9 Error Est Ref 44 E2E training of SNN Approximate non-differentiable spike using differentiable low-passed spike
  84. 84. 85 " # 0 9 Spike rate Approximate with differentiable continuous function Spike rate GradientLow pass
  85. 85. ニューラルネットワークによる処理, SSII2020 Random synaptic feedback weights support error backpropagation for deep learning Lillicrap et.al., Univ.Oxford, Nature2016 Symmetric Backpropagation (Chain Rule on ANN) Random- Backpropagation Direct feedback For more detail see http://www.cs.toronto.edu/~tingwuwang/2546.pdf Asymmetric Backpropagation (Mammal neuron) Neuroplausible training DNN can be trained using Random matrix $ instead of symmetric weight Direct error feedback 86
  86. 86. ニューラルネットワークによる処理, SSII2020 Event-Driven Random Back-Propagation: Enabling Neuromorphic Deep Learning Machines Neftci et.al., Univ.California+Intel, Frontiers in Neuroscience 2016 Weight update Enables on-chip training & layer-by-layer parallelization Apply RBP to SNN. 87
  87. 87. ニューラルネットワークによる処理, SSII2020 ML-Based (End-to-end neural-network for event processing) Event camera Pros: Can be utilize exiting architecture !! (e.g., CNN) Cons: Inefficient, Slow Frame-based processing !! ≔ x, y, p, t " fD (g(e)) = y Event-based processing (Spike) Pros: Efficient (Process only for changed pixels) Cons: Requires Neuromorphic H/W (less neurons), Difficult to train Event to Frame "(⋅) Event-based processing (Continuous) 88 fS (e) = y
  88. 88. ニューラルネットワークによる処理, SSII2020 EventNet: Asynchronous recursive event processing Sekikawa et.al, CVPR 2019 89 Event camera Pros: Can be utilize exiting architecture !! (e.g., CNN) Cons: Inefficient, Slow Frame-based processing !! ≔ x, y, p, t " fD (g(e)) = y Event-based processing (Continuous) Pros: Efficient (Process only for changed pixels) Cons: Requires Neuromorphic H/W (less neurons), Difficult to train Event to Frame "(⋅) Event-based processing (Spike) 133 Real-time event-wise inference on CPU Recursive formulation & LUT to drastically reduce computational complexity fS (e) = y
  89. 89. Problem Statement: Asynchronously Model Event Stream 90 Requirements §Sparse Event-based Processing (No densification) §Recursive Processing (Real time processing) §Local Permutation Invariance (Order may change) e : (x, y, p, t)<latexit sha1_base64="6WSEWb1HC7XemksphOEtcHz4Qbc=">AAACsHicbZFNaxsxEIbl7UfS7UecFnrpRdQUUjBhNxRSegptDzkmECeGrDGz2rGjRh9babax2W4P/Sm9Nn+o/6Zax5Q6zoDg5Zl5pdFMXirpKUn+dKJ79x883Nh8FD9+8vTZVnf7+am3lRM4EFZZN8zBo5IGByRJ4bB0CDpXeJZffmrzZ9/QeWnNCc1LHGmYGjmRAiigcfclfuA7sz6f93nZ59lnVASc3o67vWQ3WQRfF+lS9NgyjsbbnZ9ZYUWl0ZBQ4P15mpQ0qsGRFAqbOKs8liAuYYrnQRrQ6Ef14gMNfxNIwSfWhWOIL+j/jhq093Odh0oNdOFv51p4Z05XiqSzVyvv17nDr6ZZLwzv9nkeRlkQzqiJV5umyftRLU1ZERpx0/OkUpwsbwfLC+lQkJoHAcLJcBsXF+BAUBh/nBm8ElZrMEWdgZtqmDV11jZsyzpzmgf2o4WZklqSb9Yd0tzhCPCfIw5bS2/vaF2c7u2mQR+/6x18XO5vk71ir9kOS9k+O2CH7IgNmGDf2S/2m11He9EwGkdwUxp1lp4XbCWiL38BgbjYcw==</latexit><latexit sha1_base64="6WSEWb1HC7XemksphOEtcHz4Qbc=">AAACsHicbZFNaxsxEIbl7UfS7UecFnrpRdQUUjBhNxRSegptDzkmECeGrDGz2rGjRh9babax2W4P/Sm9Nn+o/6Zax5Q6zoDg5Zl5pdFMXirpKUn+dKJ79x883Nh8FD9+8vTZVnf7+am3lRM4EFZZN8zBo5IGByRJ4bB0CDpXeJZffmrzZ9/QeWnNCc1LHGmYGjmRAiigcfclfuA7sz6f93nZ59lnVASc3o67vWQ3WQRfF+lS9NgyjsbbnZ9ZYUWl0ZBQ4P15mpQ0qsGRFAqbOKs8liAuYYrnQRrQ6Ef14gMNfxNIwSfWhWOIL+j/jhq093Odh0oNdOFv51p4Z05XiqSzVyvv17nDr6ZZLwzv9nkeRlkQzqiJV5umyftRLU1ZERpx0/OkUpwsbwfLC+lQkJoHAcLJcBsXF+BAUBh/nBm8ElZrMEWdgZtqmDV11jZsyzpzmgf2o4WZklqSb9Yd0tzhCPCfIw5bS2/vaF2c7u2mQR+/6x18XO5vk71ir9kOS9k+O2CH7IgNmGDf2S/2m11He9EwGkdwUxp1lp4XbCWiL38BgbjYcw==</latexit><latexit sha1_base64="6WSEWb1HC7XemksphOEtcHz4Qbc=">AAACsHicbZFNaxsxEIbl7UfS7UecFnrpRdQUUjBhNxRSegptDzkmECeGrDGz2rGjRh9babax2W4P/Sm9Nn+o/6Zax5Q6zoDg5Zl5pdFMXirpKUn+dKJ79x883Nh8FD9+8vTZVnf7+am3lRM4EFZZN8zBo5IGByRJ4bB0CDpXeJZffmrzZ9/QeWnNCc1LHGmYGjmRAiigcfclfuA7sz6f93nZ59lnVASc3o67vWQ3WQRfF+lS9NgyjsbbnZ9ZYUWl0ZBQ4P15mpQ0qsGRFAqbOKs8liAuYYrnQRrQ6Ef14gMNfxNIwSfWhWOIL+j/jhq093Odh0oNdOFv51p4Z05XiqSzVyvv17nDr6ZZLwzv9nkeRlkQzqiJV5umyftRLU1ZERpx0/OkUpwsbwfLC+lQkJoHAcLJcBsXF+BAUBh/nBm8ElZrMEWdgZtqmDV11jZsyzpzmgf2o4WZklqSb9Yd0tzhCPCfIw5bS2/vaF2c7u2mQR+/6x18XO5vk71ir9kOS9k+O2CH7IgNmGDf2S/2m11He9EwGkdwUxp1lp4XbCWiL38BgbjYcw==</latexit><latexit sha1_base64="6WSEWb1HC7XemksphOEtcHz4Qbc=">AAACsHicbZFNaxsxEIbl7UfS7UecFnrpRdQUUjBhNxRSegptDzkmECeGrDGz2rGjRh9babax2W4P/Sm9Nn+o/6Zax5Q6zoDg5Zl5pdFMXirpKUn+dKJ79x883Nh8FD9+8vTZVnf7+am3lRM4EFZZN8zBo5IGByRJ4bB0CDpXeJZffmrzZ9/QeWnNCc1LHGmYGjmRAiigcfclfuA7sz6f93nZ59lnVASc3o67vWQ3WQRfF+lS9NgyjsbbnZ9ZYUWl0ZBQ4P15mpQ0qsGRFAqbOKs8liAuYYrnQRrQ6Ef14gMNfxNIwSfWhWOIL+j/jhq093Odh0oNdOFv51p4Z05XiqSzVyvv17nDr6ZZLwzv9nkeRlkQzqiJV5umyftRLU1ZERpx0/OkUpwsbwfLC+lQkJoHAcLJcBsXF+BAUBh/nBm8ElZrMEWdgZtqmDV11jZsyzpzmgf2o4WZklqSb9Yd0tzhCPCfIw5bS2/vaF2c7u2mQR+/6x18XO5vk71ir9kOS9k+O2CH7IgNmGDf2S/2m11He9EwGkdwUxp1lp4XbCWiL38BgbjYcw==</latexit> yj = f(ej) ⇡ g(max(h(<latexit sha1_base64="fGAcWM//pBaoofooVUXaKfJbq34=">AAAC4XicbZHLihNBFIYr7W1sL5PRpZvCIHRjbLpF0I0w6EKXI5iZgekQqqtPJzVTl7aq2kloyoU7d+LWZ/BpXAn6LlYnQcxkDhT8fOf8dfuLmjNj0/RXL7hy9dr1Gzs3w1u379zd7e/dOzSq0RRGVHGljwtigDMJI8ssh+NaAxEFh6Pi7HXXP/oI2jAl39tFDWNBppJVjBLr0aT/ZjFpT93LKsoFsbOiasF1IM5JXWs1x9OuMY9mEXj8REan8ePMxcMkSYYr5uI4HobhpD9Ik3RZeFtkazFA6zqY7PU+56WijQBpKSfGnGRpbcct0ZZRDi7MGwM1oWdkCideSiLAjNvlix1+5EmJK6X9khYv6f+OlghjFqLwk927zMVeBy/tiYZbptX5xvltoeGDdNuD/twhLvzflxbm1oWbl7bVi3HLZN1YkHR156rh2CrcJYFLpoFavvCCUM38bpjOiCbU+rzCXMI5VUIQWbY50VOfgmuXIam6zbXAnn3qYM6ZYNa4bQeTlzg8/OfoUssuZrQtDp8mmdfvng32X63z20EP0EMUoQw9R/voLTpAI0TRD/QT/UZ/Ahp8Cb4G31ajQW/tuY82Kvj+Fy4w65s=</latexit><latexit sha1_base64="fGAcWM//pBaoofooVUXaKfJbq34=">AAAC4XicbZHLihNBFIYr7W1sL5PRpZvCIHRjbLpF0I0w6EKXI5iZgekQqqtPJzVTl7aq2kloyoU7d+LWZ/BpXAn6LlYnQcxkDhT8fOf8dfuLmjNj0/RXL7hy9dr1Gzs3w1u379zd7e/dOzSq0RRGVHGljwtigDMJI8ssh+NaAxEFh6Pi7HXXP/oI2jAl39tFDWNBppJVjBLr0aT/ZjFpT93LKsoFsbOiasF1IM5JXWs1x9OuMY9mEXj8REan8ePMxcMkSYYr5uI4HobhpD9Ik3RZeFtkazFA6zqY7PU+56WijQBpKSfGnGRpbcct0ZZRDi7MGwM1oWdkCideSiLAjNvlix1+5EmJK6X9khYv6f+OlghjFqLwk927zMVeBy/tiYZbptX5xvltoeGDdNuD/twhLvzflxbm1oWbl7bVi3HLZN1YkHR156rh2CrcJYFLpoFavvCCUM38bpjOiCbU+rzCXMI5VUIQWbY50VOfgmuXIam6zbXAnn3qYM6ZYNa4bQeTlzg8/OfoUssuZrQtDp8mmdfvng32X63z20EP0EMUoQw9R/voLTpAI0TRD/QT/UZ/Ahp8Cb4G31ajQW/tuY82Kvj+Fy4w65s=</latexit><latexit sha1_base64="fGAcWM//pBaoofooVUXaKfJbq34=">AAAC4XicbZHLihNBFIYr7W1sL5PRpZvCIHRjbLpF0I0w6EKXI5iZgekQqqtPJzVTl7aq2kloyoU7d+LWZ/BpXAn6LlYnQcxkDhT8fOf8dfuLmjNj0/RXL7hy9dr1Gzs3w1u379zd7e/dOzSq0RRGVHGljwtigDMJI8ssh+NaAxEFh6Pi7HXXP/oI2jAl39tFDWNBppJVjBLr0aT/ZjFpT93LKsoFsbOiasF1IM5JXWs1x9OuMY9mEXj8REan8ePMxcMkSYYr5uI4HobhpD9Ik3RZeFtkazFA6zqY7PU+56WijQBpKSfGnGRpbcct0ZZRDi7MGwM1oWdkCideSiLAjNvlix1+5EmJK6X9khYv6f+OlghjFqLwk927zMVeBy/tiYZbptX5xvltoeGDdNuD/twhLvzflxbm1oWbl7bVi3HLZN1YkHR156rh2CrcJYFLpoFavvCCUM38bpjOiCbU+rzCXMI5VUIQWbY50VOfgmuXIam6zbXAnn3qYM6ZYNa4bQeTlzg8/OfoUssuZrQtDp8mmdfvng32X63z20EP0EMUoQw9R/voLTpAI0TRD/QT/UZ/Ahp8Cb4G31ajQW/tuY82Kvj+Fy4w65s=</latexit><latexit sha1_base64="fGAcWM//pBaoofooVUXaKfJbq34=">AAAC4XicbZHLihNBFIYr7W1sL5PRpZvCIHRjbLpF0I0w6EKXI5iZgekQqqtPJzVTl7aq2kloyoU7d+LWZ/BpXAn6LlYnQcxkDhT8fOf8dfuLmjNj0/RXL7hy9dr1Gzs3w1u379zd7e/dOzSq0RRGVHGljwtigDMJI8ssh+NaAxEFh6Pi7HXXP/oI2jAl39tFDWNBppJVjBLr0aT/ZjFpT93LKsoFsbOiasF1IM5JXWs1x9OuMY9mEXj8REan8ePMxcMkSYYr5uI4HobhpD9Ik3RZeFtkazFA6zqY7PU+56WijQBpKSfGnGRpbcct0ZZRDi7MGwM1oWdkCideSiLAjNvlix1+5EmJK6X9khYv6f+OlghjFqLwk927zMVeBy/tiYZbptX5xvltoeGDdNuD/twhLvzflxbm1oWbl7bVi3HLZN1YkHR156rh2CrcJYFLpoFavvCCUM38bpjOiCbU+rzCXMI5VUIQWbY50VOfgmuXIam6zbXAnn3qYM6ZYNa4bQeTlzg8/OfoUssuZrQtDp8mmdfvng32X63z20EP0EMUoQw9R/voLTpAI0TRD/QT/UZ/Ahp8Cb4G31ajQW/tuY82Kvj+Fy4w65s=</latexit> yj =<latexit sha1_base64="fGAcWM//pBaoofooVUXaKfJbq34=">AAAC4XicbZHLihNBFIYr7W1sL5PRpZvCIHRjbLpF0I0w6EKXI5iZgekQqqtPJzVTl7aq2kloyoU7d+LWZ/BpXAn6LlYnQcxkDhT8fOf8dfuLmjNj0/RXL7hy9dr1Gzs3w1u379zd7e/dOzSq0RRGVHGljwtigDMJI8ssh+NaAxEFh6Pi7HXXP/oI2jAl39tFDWNBppJVjBLr0aT/ZjFpT93LKsoFsbOiasF1IM5JXWs1x9OuMY9mEXj8REan8ePMxcMkSYYr5uI4HobhpD9Ik3RZeFtkazFA6zqY7PU+56WijQBpKSfGnGRpbcct0ZZRDi7MGwM1oWdkCideSiLAjNvlix1+5EmJK6X9khYv6f+OlghjFqLwk927zMVeBy/tiYZbptX5xvltoeGDdNuD/twhLvzflxbm1oWbl7bVi3HLZN1YkHR156rh2CrcJYFLpoFavvCCUM38bpjOiCbU+rzCXMI5VUIQWbY50VOfgmuXIam6zbXAnn3qYM6ZYNa4bQeTlzg8/OfoUssuZrQtDp8mmdfvng32X63z20EP0EMUoQw9R/voLTpAI0TRD/QT/UZ/Ahp8Cb4G31ajQW/tuY82Kvj+Fy4w65s=</latexit><latexit sha1_base64="fGAcWM//pBaoofooVUXaKfJbq34=">AAAC4XicbZHLihNBFIYr7W1sL5PRpZvCIHRjbLpF0I0w6EKXI5iZgekQqqtPJzVTl7aq2kloyoU7d+LWZ/BpXAn6LlYnQcxkDhT8fOf8dfuLmjNj0/RXL7hy9dr1Gzs3w1u379zd7e/dOzSq0RRGVHGljwtigDMJI8ssh+NaAxEFh6Pi7HXXP/oI2jAl39tFDWNBppJVjBLr0aT/ZjFpT93LKsoFsbOiasF1IM5JXWs1x9OuMY9mEXj8REan8ePMxcMkSYYr5uI4HobhpD9Ik3RZeFtkazFA6zqY7PU+56WijQBpKSfGnGRpbcct0ZZRDi7MGwM1oWdkCideSiLAjNvlix1+5EmJK6X9khYv6f+OlghjFqLwk927zMVeBy/tiYZbptX5xvltoeGDdNuD/twhLvzflxbm1oWbl7bVi3HLZN1YkHR156rh2CrcJYFLpoFavvCCUM38bpjOiCbU+rzCXMI5VUIQWbY50VOfgmuXIam6zbXAnn3qYM6ZYNa4bQeTlzg8/OfoUssuZrQtDp8mmdfvng32X63z20EP0EMUoQw9R/voLTpAI0TRD/QT/UZ/Ahp8Cb4G31ajQW/tuY82Kvj+Fy4w65s=</latexit><latexit sha1_base64="fGAcWM//pBaoofooVUXaKfJbq34=">AAAC4XicbZHLihNBFIYr7W1sL5PRpZvCIHRjbLpF0I0w6EKXI5iZgekQqqtPJzVTl7aq2kloyoU7d+LWZ/BpXAn6LlYnQcxkDhT8fOf8dfuLmjNj0/RXL7hy9dr1Gzs3w1u379zd7e/dOzSq0RRGVHGljwtigDMJI8ssh+NaAxEFh6Pi7HXXP/oI2jAl39tFDWNBppJVjBLr0aT/ZjFpT93LKsoFsbOiasF1IM5JXWs1x9OuMY9mEXj8REan8ePMxcMkSYYr5uI4HobhpD9Ik3RZeFtkazFA6zqY7PU+56WijQBpKSfGnGRpbcct0ZZRDi7MGwM1oWdkCideSiLAjNvlix1+5EmJK6X9khYv6f+OlghjFqLwk927zMVeBy/tiYZbptX5xvltoeGDdNuD/twhLvzflxbm1oWbl7bVi3HLZN1YkHR156rh2CrcJYFLpoFavvCCUM38bpjOiCbU+rzCXMI5VUIQWbY50VOfgmuXIam6zbXAnn3qYM6ZYNa4bQeTlzg8/OfoUssuZrQtDp8mmdfvng32X63z20EP0EMUoQw9R/voLTpAI0TRD/QT/UZ/Ahp8Cb4G31ajQW/tuY82Kvj+Fy4w65s=</latexit><latexit sha1_base64="fGAcWM//pBaoofooVUXaKfJbq34=">AAAC4XicbZHLihNBFIYr7W1sL5PRpZvCIHRjbLpF0I0w6EKXI5iZgekQqqtPJzVTl7aq2kloyoU7d+LWZ/BpXAn6LlYnQcxkDhT8fOf8dfuLmjNj0/RXL7hy9dr1Gzs3w1u379zd7e/dOzSq0RRGVHGljwtigDMJI8ssh+NaAxEFh6Pi7HXXP/oI2jAl39tFDWNBppJVjBLr0aT/ZjFpT93LKsoFsbOiasF1IM5JXWs1x9OuMY9mEXj8REan8ePMxcMkSYYr5uI4HobhpD9Ik3RZeFtkazFA6zqY7PU+56WijQBpKSfGnGRpbcct0ZZRDi7MGwM1oWdkCideSiLAjNvlix1+5EmJK6X9khYv6f+OlghjFqLwk927zMVeBy/tiYZbptX5xvltoeGDdNuD/twhLvzflxbm1oWbl7bVi3HLZN1YkHR156rh2CrcJYFLpoFavvCCUM38bpjOiCbU+rzCXMI5VUIQWbY50VOfgmuXIam6zbXAnn3qYM6ZYNa4bQeTlzg8/OfoUssuZrQtDp8mmdfvng32X63z20EP0EMUoQw9R/voLTpAI0TRD/QT/UZ/Ahp8Cb4G31ajQW/tuY82Kvj+Fy4w65s=</latexit> tj<latexit sha1_base64="Kqg2Xj/aeDgXRzQXV5js0JJQFl0=">AAACnXicbVFNaxRBEO0do8bxI4kePdhkETxImAkBPQZF8CASwc0GMstS01O7adMfY3eNydKMh/yBXPWn+W/s2Swhm82Dhserel2PqrJW0lOW/esl99buP3i4/ih9/OTps43NreeH3jZO4EBYZd1RCR6VNDggSQqPaoegS4XD8vRjVx/+QuelNd9pVuNIw9TIiRRAURrQOPxox5v9bCebg6+SfEH6bIGD8VbvoqisaDQaEgq8P86zmkYBHEmhsE2LxmMN4hSmeBypAY1+FOZpW/46KhWfWBefIT5XbzoCaO9nuoydGujE36514p013SiSzp4tzQ+lw5+mXW2Mc9/yMu6tIjynNl0OTZP3oyBN3RAacZV50ihOlndb5JV0KEjNIgHhZPyNixNwICjuOi0MngmrNZgqFOCmGs7bUHSBbR0Kp3nUfndioaSW5NtVhzR3OKJ47Ujj1fLbN1olh7s7eeTf9vr7Hxb3W2cv2TZ7w3L2ju2zz+yADZhgkl2yP+xv8ir5lHxJvl61Jr2F5wVbQjL8D3Kd054=</latexit><latexit sha1_base64="Kqg2Xj/aeDgXRzQXV5js0JJQFl0=">AAACnXicbVFNaxRBEO0do8bxI4kePdhkETxImAkBPQZF8CASwc0GMstS01O7adMfY3eNydKMh/yBXPWn+W/s2Swhm82Dhserel2PqrJW0lOW/esl99buP3i4/ih9/OTps43NreeH3jZO4EBYZd1RCR6VNDggSQqPaoegS4XD8vRjVx/+QuelNd9pVuNIw9TIiRRAURrQOPxox5v9bCebg6+SfEH6bIGD8VbvoqisaDQaEgq8P86zmkYBHEmhsE2LxmMN4hSmeBypAY1+FOZpW/46KhWfWBefIT5XbzoCaO9nuoydGujE36514p013SiSzp4tzQ+lw5+mXW2Mc9/yMu6tIjynNl0OTZP3oyBN3RAacZV50ihOlndb5JV0KEjNIgHhZPyNixNwICjuOi0MngmrNZgqFOCmGs7bUHSBbR0Kp3nUfndioaSW5NtVhzR3OKJ47Ujj1fLbN1olh7s7eeTf9vr7Hxb3W2cv2TZ7w3L2ju2zz+yADZhgkl2yP+xv8ir5lHxJvl61Jr2F5wVbQjL8D3Kd054=</latexit><latexit sha1_base64="Kqg2Xj/aeDgXRzQXV5js0JJQFl0=">AAACnXicbVFNaxRBEO0do8bxI4kePdhkETxImAkBPQZF8CASwc0GMstS01O7adMfY3eNydKMh/yBXPWn+W/s2Swhm82Dhserel2PqrJW0lOW/esl99buP3i4/ih9/OTps43NreeH3jZO4EBYZd1RCR6VNDggSQqPaoegS4XD8vRjVx/+QuelNd9pVuNIw9TIiRRAURrQOPxox5v9bCebg6+SfEH6bIGD8VbvoqisaDQaEgq8P86zmkYBHEmhsE2LxmMN4hSmeBypAY1+FOZpW/46KhWfWBefIT5XbzoCaO9nuoydGujE36514p013SiSzp4tzQ+lw5+mXW2Mc9/yMu6tIjynNl0OTZP3oyBN3RAacZV50ihOlndb5JV0KEjNIgHhZPyNixNwICjuOi0MngmrNZgqFOCmGs7bUHSBbR0Kp3nUfndioaSW5NtVhzR3OKJ47Ujj1fLbN1olh7s7eeTf9vr7Hxb3W2cv2TZ7w3L2ju2zz+yADZhgkl2yP+xv8ir5lHxJvl61Jr2F5wVbQjL8D3Kd054=</latexit><latexit sha1_base64="Kqg2Xj/aeDgXRzQXV5js0JJQFl0=">AAACnXicbVFNaxRBEO0do8bxI4kePdhkETxImAkBPQZF8CASwc0GMstS01O7adMfY3eNydKMh/yBXPWn+W/s2Swhm82Dhserel2PqrJW0lOW/esl99buP3i4/ih9/OTps43NreeH3jZO4EBYZd1RCR6VNDggSQqPaoegS4XD8vRjVx/+QuelNd9pVuNIw9TIiRRAURrQOPxox5v9bCebg6+SfEH6bIGD8VbvoqisaDQaEgq8P86zmkYBHEmhsE2LxmMN4hSmeBypAY1+FOZpW/46KhWfWBefIT5XbzoCaO9nuoydGujE36514p013SiSzp4tzQ+lw5+mXW2Mc9/yMu6tIjynNl0OTZP3oyBN3RAacZV50ihOlndb5JV0KEjNIgHhZPyNixNwICjuOi0MngmrNZgqFOCmGs7bUHSBbR0Kp3nUfndioaSW5NtVhzR3OKJ47Ujj1fLbN1olh7s7eeTf9vr7Hxb3W2cv2TZ7w3L2ju2zz+yADZhgkl2yP+xv8ir5lHxJvl61Jr2F5wVbQjL8D3Kd054=</latexit> tj n(j)+1<latexit sha1_base64="YPdS0xIHAmK3ijRwSVpp5syn4JM=">AAACpHicbVFNTxRBEO0dP8DxC/TopeNGg1HJjCHBI9ELBw8YXSBh1k1NT+3S0B9Dd42w6YwHfoRX/Vv+G3uWjXFZXtLJy6t6XS9VZa2kpyz700tu3b5zd2X1Xnr/wcNHj9fWn+x72ziBA2GVdYcleFTS4IAkKTysHYIuFR6Upx+7+sF3dF5a85WmNQ41TIwcSwEUpYJG4eSt2Th59TpvR2v9bDObgS+TfE76bI690XrvsqisaDQaEgq8P8qzmoYBHEmhsE2LxmMN4hQmeBSpAY1+GGahW/4iKhUfWxefIT5T/3cE0N5PdRk7NdCxv17rxBtrulEknT1fmB9Kh2emXW6Mc9/wMq6vIrygNl0MTeP3wyBN3RAacZV53ChOlnfL5JV0KEhNIwHhZPyNi2NwICiuPC0MngurNZgqFOAmGi7aUHSBbR0Kp3nUfnRioaSW5NtlhzQ3OKL4z5HGq+XXb7RM9t9t5pF/3urvfJjfb5U9Y8/ZBsvZNtthu2yPDZhgNfvJfrHfycvkU/IlGVy1Jr255ylbQPLtL0uA1ZY=</latexit><latexit sha1_base64="YPdS0xIHAmK3ijRwSVpp5syn4JM=">AAACpHicbVFNTxRBEO0dP8DxC/TopeNGg1HJjCHBI9ELBw8YXSBh1k1NT+3S0B9Dd42w6YwHfoRX/Vv+G3uWjXFZXtLJy6t6XS9VZa2kpyz700tu3b5zd2X1Xnr/wcNHj9fWn+x72ziBA2GVdYcleFTS4IAkKTysHYIuFR6Upx+7+sF3dF5a85WmNQ41TIwcSwEUpYJG4eSt2Th59TpvR2v9bDObgS+TfE76bI690XrvsqisaDQaEgq8P8qzmoYBHEmhsE2LxmMN4hQmeBSpAY1+GGahW/4iKhUfWxefIT5T/3cE0N5PdRk7NdCxv17rxBtrulEknT1fmB9Kh2emXW6Mc9/wMq6vIrygNl0MTeP3wyBN3RAacZV53ChOlnfL5JV0KEhNIwHhZPyNi2NwICiuPC0MngurNZgqFOAmGi7aUHSBbR0Kp3nUfnRioaSW5NtlhzQ3OKL4z5HGq+XXb7RM9t9t5pF/3urvfJjfb5U9Y8/ZBsvZNtthu2yPDZhgNfvJfrHfycvkU/IlGVy1Jr255ylbQPLtL0uA1ZY=</latexit><latexit sha1_base64="YPdS0xIHAmK3ijRwSVpp5syn4JM=">AAACpHicbVFNTxRBEO0dP8DxC/TopeNGg1HJjCHBI9ELBw8YXSBh1k1NT+3S0B9Dd42w6YwHfoRX/Vv+G3uWjXFZXtLJy6t6XS9VZa2kpyz700tu3b5zd2X1Xnr/wcNHj9fWn+x72ziBA2GVdYcleFTS4IAkKTysHYIuFR6Upx+7+sF3dF5a85WmNQ41TIwcSwEUpYJG4eSt2Th59TpvR2v9bDObgS+TfE76bI690XrvsqisaDQaEgq8P8qzmoYBHEmhsE2LxmMN4hQmeBSpAY1+GGahW/4iKhUfWxefIT5T/3cE0N5PdRk7NdCxv17rxBtrulEknT1fmB9Kh2emXW6Mc9/wMq6vIrygNl0MTeP3wyBN3RAacZV53ChOlnfL5JV0KEhNIwHhZPyNi2NwICiuPC0MngurNZgqFOAmGi7aUHSBbR0Kp3nUfnRioaSW5NtlhzQ3OKL4z5HGq+XXb7RM9t9t5pF/3urvfJjfb5U9Y8/ZBsvZNtthu2yPDZhgNfvJfrHfycvkU/IlGVy1Jr255ylbQPLtL0uA1ZY=</latexit><latexit sha1_base64="YPdS0xIHAmK3ijRwSVpp5syn4JM=">AAACpHicbVFNTxRBEO0dP8DxC/TopeNGg1HJjCHBI9ELBw8YXSBh1k1NT+3S0B9Dd42w6YwHfoRX/Vv+G3uWjXFZXtLJy6t6XS9VZa2kpyz700tu3b5zd2X1Xnr/wcNHj9fWn+x72ziBA2GVdYcleFTS4IAkKTysHYIuFR6Upx+7+sF3dF5a85WmNQ41TIwcSwEUpYJG4eSt2Th59TpvR2v9bDObgS+TfE76bI690XrvsqisaDQaEgq8P8qzmoYBHEmhsE2LxmMN4hQmeBSpAY1+GGahW/4iKhUfWxefIT5T/3cE0N5PdRk7NdCxv17rxBtrulEknT1fmB9Kh2emXW6Mc9/wMq6vIrygNl0MTeP3wyBN3RAacZV53ChOlnfL5JV0KEhNIwHhZPyNi2NwICiuPC0MngurNZgqFOAmGi7aUHSBbR0Kp3nUfnRioaSW5NtlhzQ3OKL4z5HGq+XXb7RM9t9t5pF/3urvfJjfb5U9Y8/ZBsvZNtthu2yPDZhgNfvJfrHfycvkU/IlGVy1Jr255ylbQPLtL0uA1ZY=</latexit> t<latexit sha1_base64="9jOB8mtYM9Sb1ynIEBBIJNz1L8s=">AAACmXicbVFNSxxBEO0d86GTGN149NJkCXgQmZFAcjTJZfGkxFXBWaSmp3Zt7I+xuya6DJODV6/64/w36VmXkHV90PB4Va/rUZWXSnpKksdOtPTq9Zu3yyvxu/erH9bWux+Pva2cwIGwyrrTHDwqaXBAkhSelg5B5wpP8sufbf3kNzovrTmiSYlDDWMjR1IABemQztd7yU4yBV8k6Yz02AwH593ObVZYUWk0JBR4f5YmJQ1rcCSFwibOKo8liEsY41mgBjT6YT1N2vDPQSn4yLrwDPGp+r+jBu39ROehUwNd+Oe1VnyxpitF0tnrufl17vDKNIuNYe42z8POCsIbauL50DT6NqylKStCI54yjyrFyfJ2g7yQDgWpSSAgnAy/cXEBDgSFPceZwWthtQZT1Bm4sYabps7awLasM6d50P60YqakluSbRYc0LziC+M8Rh6ulz2+0SI53d9LAD7/09n7M7rfMNtkntsVS9pXtsT47YAMmGLI7ds8eos3oe9SP9p9ao87Ms8HmEP36C/iw0bU=</latexit><latexit sha1_base64="9jOB8mtYM9Sb1ynIEBBIJNz1L8s=">AAACmXicbVFNSxxBEO0d86GTGN149NJkCXgQmZFAcjTJZfGkxFXBWaSmp3Zt7I+xuya6DJODV6/64/w36VmXkHV90PB4Va/rUZWXSnpKksdOtPTq9Zu3yyvxu/erH9bWux+Pva2cwIGwyrrTHDwqaXBAkhSelg5B5wpP8sufbf3kNzovrTmiSYlDDWMjR1IABemQztd7yU4yBV8k6Yz02AwH593ObVZYUWk0JBR4f5YmJQ1rcCSFwibOKo8liEsY41mgBjT6YT1N2vDPQSn4yLrwDPGp+r+jBu39ROehUwNd+Oe1VnyxpitF0tnrufl17vDKNIuNYe42z8POCsIbauL50DT6NqylKStCI54yjyrFyfJ2g7yQDgWpSSAgnAy/cXEBDgSFPceZwWthtQZT1Bm4sYabps7awLasM6d50P60YqakluSbRYc0LziC+M8Rh6ulz2+0SI53d9LAD7/09n7M7rfMNtkntsVS9pXtsT47YAMmGLI7ds8eos3oe9SP9p9ao87Ms8HmEP36C/iw0bU=</latexit><latexit sha1_base64="9jOB8mtYM9Sb1ynIEBBIJNz1L8s=">AAACmXicbVFNSxxBEO0d86GTGN149NJkCXgQmZFAcjTJZfGkxFXBWaSmp3Zt7I+xuya6DJODV6/64/w36VmXkHV90PB4Va/rUZWXSnpKksdOtPTq9Zu3yyvxu/erH9bWux+Pva2cwIGwyrrTHDwqaXBAkhSelg5B5wpP8sufbf3kNzovrTmiSYlDDWMjR1IABemQztd7yU4yBV8k6Yz02AwH593ObVZYUWk0JBR4f5YmJQ1rcCSFwibOKo8liEsY41mgBjT6YT1N2vDPQSn4yLrwDPGp+r+jBu39ROehUwNd+Oe1VnyxpitF0tnrufl17vDKNIuNYe42z8POCsIbauL50DT6NqylKStCI54yjyrFyfJ2g7yQDgWpSSAgnAy/cXEBDgSFPceZwWthtQZT1Bm4sYabps7awLasM6d50P60YqakluSbRYc0LziC+M8Rh6ulz2+0SI53d9LAD7/09n7M7rfMNtkntsVS9pXtsT47YAMmGLI7ds8eos3oe9SP9p9ao87Ms8HmEP36C/iw0bU=</latexit><latexit sha1_base64="9jOB8mtYM9Sb1ynIEBBIJNz1L8s=">AAACmXicbVFNSxxBEO0d86GTGN149NJkCXgQmZFAcjTJZfGkxFXBWaSmp3Zt7I+xuya6DJODV6/64/w36VmXkHV90PB4Va/rUZWXSnpKksdOtPTq9Zu3yyvxu/erH9bWux+Pva2cwIGwyrrTHDwqaXBAkhSelg5B5wpP8sufbf3kNzovrTmiSYlDDWMjR1IABemQztd7yU4yBV8k6Yz02AwH593ObVZYUWk0JBR4f5YmJQ1rcCSFwibOKo8liEsY41mgBjT6YT1N2vDPQSn4yLrwDPGp+r+jBu39ROehUwNd+Oe1VnyxpitF0tnrufl17vDKNIuNYe42z8POCsIbauL50DT6NqylKStCI54yjyrFyfJ2g7yQDgWpSSAgnAy/cXEBDgSFPceZwWthtQZT1Bm4sYabps7awLasM6d50P60YqakluSbRYc0LziC+M8Rh6ulz2+0SI53d9LAD7/09n7M7rfMNtkntsVS9pXtsT47YAMmGLI7ds8eos3oe9SP9p9ao87Ms8HmEP36C/iw0bU=</latexit> ⌧<latexit sha1_base64="CAemzWtty/jfKAZcsg5D/TBMhqk=">AAACnHicbVFNSxxBEO0dEz/GaDQ5CqFxETyIzIiQHCXJQQgBA9lVcBap6andbeyPSXeNugzjwR+Qa/LX8m/Ssy4h6/qg4fGqXtejKi+V9JQkfzrR0ouXyyura/H6q43N11vbb/reVk5gT1hl3UUOHpU02CNJCi9Kh6Bzhef59ae2fn6DzktrvtOkxIGGkZFDKYBaKSOorra6yWEyBV8k6Yx02QxnV9udh6ywotJoSCjw/jJNShrU4EgKhU2cVR5LENcwwstADWj0g3oatuF7QSn40LrwDPGp+r+jBu39ROehUwON/dNaKz5b05Ui6ezt3Pw6d/jDNIuNYe4Bz8PaCsI7auL50DT8MKilKStCIx4zDyvFyfJ2ibyQDgWpSSAgnAy/cTEGB4LCquPM4K2wWoMp6gzcSMNdU2dtYFvWmdM8aPetmCmpJflm0SHNM44g/nPE4Wrp0xstkv7RYRr4t+PuycfZ/VbZDttl+yxl79kJO2VnrMcEG7Of7Bf7Hb2LPkdfoq+PrVFn5nnL5hD1/wIPaNMF</latexit><latexit sha1_base64="CAemzWtty/jfKAZcsg5D/TBMhqk=">AAACnHicbVFNSxxBEO0dEz/GaDQ5CqFxETyIzIiQHCXJQQgBA9lVcBap6andbeyPSXeNugzjwR+Qa/LX8m/Ssy4h6/qg4fGqXtejKi+V9JQkfzrR0ouXyyura/H6q43N11vbb/reVk5gT1hl3UUOHpU02CNJCi9Kh6Bzhef59ae2fn6DzktrvtOkxIGGkZFDKYBaKSOorra6yWEyBV8k6Yx02QxnV9udh6ywotJoSCjw/jJNShrU4EgKhU2cVR5LENcwwstADWj0g3oatuF7QSn40LrwDPGp+r+jBu39ROehUwON/dNaKz5b05Ui6ezt3Pw6d/jDNIuNYe4Bz8PaCsI7auL50DT8MKilKStCIx4zDyvFyfJ2ibyQDgWpSSAgnAy/cTEGB4LCquPM4K2wWoMp6gzcSMNdU2dtYFvWmdM8aPetmCmpJflm0SHNM44g/nPE4Wrp0xstkv7RYRr4t+PuycfZ/VbZDttl+yxl79kJO2VnrMcEG7Of7Bf7Hb2LPkdfoq+PrVFn5nnL5hD1/wIPaNMF</latexit><latexit sha1_base64="CAemzWtty/jfKAZcsg5D/TBMhqk=">AAACnHicbVFNSxxBEO0dEz/GaDQ5CqFxETyIzIiQHCXJQQgBA9lVcBap6andbeyPSXeNugzjwR+Qa/LX8m/Ssy4h6/qg4fGqXtejKi+V9JQkfzrR0ouXyyura/H6q43N11vbb/reVk5gT1hl3UUOHpU02CNJCi9Kh6Bzhef59ae2fn6DzktrvtOkxIGGkZFDKYBaKSOorra6yWEyBV8k6Yx02QxnV9udh6ywotJoSCjw/jJNShrU4EgKhU2cVR5LENcwwstADWj0g3oatuF7QSn40LrwDPGp+r+jBu39ROehUwON/dNaKz5b05Ui6ezt3Pw6d/jDNIuNYe4Bz8PaCsI7auL50DT8MKilKStCIx4zDyvFyfJ2ibyQDgWpSSAgnAy/cTEGB4LCquPM4K2wWoMp6gzcSMNdU2dtYFvWmdM8aPetmCmpJflm0SHNM44g/nPE4Wrp0xstkv7RYRr4t+PuycfZ/VbZDttl+yxl79kJO2VnrMcEG7Of7Bf7Hb2LPkdfoq+PrVFn5nnL5hD1/wIPaNMF</latexit><latexit sha1_base64="CAemzWtty/jfKAZcsg5D/TBMhqk=">AAACnHicbVFNSxxBEO0dEz/GaDQ5CqFxETyIzIiQHCXJQQgBA9lVcBap6andbeyPSXeNugzjwR+Qa/LX8m/Ssy4h6/qg4fGqXtejKi+V9JQkfzrR0ouXyyura/H6q43N11vbb/reVk5gT1hl3UUOHpU02CNJCi9Kh6Bzhef59ae2fn6DzktrvtOkxIGGkZFDKYBaKSOorra6yWEyBV8k6Yx02QxnV9udh6ywotJoSCjw/jJNShrU4EgKhU2cVR5LENcwwstADWj0g3oatuF7QSn40LrwDPGp+r+jBu39ROehUwON/dNaKz5b05Ui6ezt3Pw6d/jDNIuNYe4Bz8PaCsI7auL50DT8MKilKStCIx4zDyvFyfJ2ibyQDgWpSSAgnAy/cTEGB4LCquPM4K2wWoMp6gzcSMNdU2dtYFvWmdM8aPetmCmpJflm0SHNM44g/nPE4Wrp0xstkv7RYRr4t+PuycfZ/VbZDttl+yxl79kJO2VnrMcEG7Of7Bf7Hb2LPkdfoq+PrVFn5nnL5hD1/wIPaNMF</latexit> ej := {ei|i = j n(j) + 1, ..., j}<latexit sha1_base64="u2rFRK6vv1UnGQ6yiXMxNqGWQ7A=">AAACwnicbVFba9RAFJ6Nl9Z42+qjL4OLsOIaEluoCIVSffCxgtsWmiVMJifbSeeSzkxsl2n64A/w1/iq/8N/42S7iNvtgYGP75xvzuXLa86MjeM/veDO3Xv319YfhA8fPX7ytL/x7MCoRlMYU8WVPsqJAc4kjC2zHI5qDUTkHA7z049d/vAbaMOU/GpnNUwEmUpWMkqsp7L+MBXEnuSlgzZzVfthJ3WQsUu2U72Vw+r1m2QURdGoStusP4ijeB54FSQLMECL2M82et/TQtFGgLSUE2OOk7i2E0e0ZZRDG6aNgZrQUzKFYw8lEWAmbr5Si195psCl0v5Ji+fs/wpHhDEzkfvKbgFzM9eRt+ZEwy3T6nypv8s1nMl2tdD3HeHcH7ewcGHbcHloW76fOCbrxoKk1zOXDcdW4e7UuGAaqOUzDwjVzP+G6QnRhFpvSJhKOKdKCCILlxI9FeSidXM3VO1SLbDnrjoy5Uwwa9pVBZO3KDz5TxF615KbHq2Cg3dRshnFX7YGu3sL/9bRC/QSDVGCttEu+oz20RhR9AP9RL/Q7+BTUAVngbkuDXoLzXO0FMHlX+Fi4MQ=</latexit>
  90. 90. ニューラルネットワークによる処理, SSII2020 [Related work] PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation §Qi et.al., Stanford Univ., CVPR2016 {&!, . . . , &"} input points MLP: Multi-layer perceptron #*+,(⋅) ( = *({-/, . . . , -0}) = 1(23#({4/, . . . , 40})) '(⋅) Point Feature Embedding: 1$ = ℎ%&'(4$) mlp-e Nx3 # != (&,(,)) NxK max global feature ! mlp-c 1xK outputs embedded features " shared (64) shared (64) shared (64) shared (128) shared (1024) -+ 91 xs xy z [,, &, .] Direct Point Processing (Efficient ITO. Memory & Computation) Realize permutation-invariance using symmetric function
  91. 91. 92 6(7(#)) Pentagon Star 8,9({. . . }) {&!, . . . , &" (,) } {&!, . . . , &" (!) } ℎ*+,(⋅) ). = ℎ*+,(+.) 6(7(*)) 7(#) = 8,9({1#, . . . , 1+}) 7(*)
  92. 92. ニューラルネットワークによる処理, SSII2020 Idea 1: Recursive computation by temporal coding (t-code) MLP ℎ nx1024 shared global feature MLP g Batch-based synchronous architecture (PointNet) Requirements ✓ Sparse ✓ Recursive ✓ PI※ ※ Permutation invariant max "($) events (+ ms) /(1-) = 4(567({ℎ(9-./ - 0!), . . . ℎ(9-)})) t-code - Requirements ✓ Sparse ✓ Recursive ✓ PI t-code - MLP ℎ MLP gmax Event-based asynchronous architecture (EventNet) global feature 1x1024 ! events (+ ms) /(1-) = 4(567({:(;-.!, <=-), ℎ(9-)})) e : (x, y, p, t)<latexit sha1_base64="6WSEWb1HC7XemksphOEtcHz4Qbc=">AAACsHicbZFNaxsxEIbl7UfS7UecFnrpRdQUUjBhNxRSegptDzkmECeGrDGz2rGjRh9babax2W4P/Sm9Nn+o/6Zax5Q6zoDg5Zl5pdFMXirpKUn+dKJ79x883Nh8FD9+8vTZVnf7+am3lRM4EFZZN8zBo5IGByRJ4bB0CDpXeJZffmrzZ9/QeWnNCc1LHGmYGjmRAiigcfclfuA7sz6f93nZ59lnVASc3o67vWQ3WQRfF+lS9NgyjsbbnZ9ZYUWl0ZBQ4P15mpQ0qsGRFAqbOKs8liAuYYrnQRrQ6Ef14gMNfxNIwSfWhWOIL+j/jhq093Odh0oNdOFv51p4Z05XiqSzVyvv17nDr6ZZLwzv9nkeRlkQzqiJV5umyftRLU1ZERpx0/OkUpwsbwfLC+lQkJoHAcLJcBsXF+BAUBh/nBm8ElZrMEWdgZtqmDV11jZsyzpzmgf2o4WZklqSb9Yd0tzhCPCfIw5bS2/vaF2c7u2mQR+/6x18XO5vk71ir9kOS9k+O2CH7IgNmGDf2S/2m11He9EwGkdwUxp1lp4XbCWiL38BgbjYcw==</latexit><latexit sha1_base64="6WSEWb1HC7XemksphOEtcHz4Qbc=">AAACsHicbZFNaxsxEIbl7UfS7UecFnrpRdQUUjBhNxRSegptDzkmECeGrDGz2rGjRh9babax2W4P/Sm9Nn+o/6Zax5Q6zoDg5Zl5pdFMXirpKUn+dKJ79x883Nh8FD9+8vTZVnf7+am3lRM4EFZZN8zBo5IGByRJ4bB0CDpXeJZffmrzZ9/QeWnNCc1LHGmYGjmRAiigcfclfuA7sz6f93nZ59lnVASc3o67vWQ3WQRfF+lS9NgyjsbbnZ9ZYUWl0ZBQ4P15mpQ0qsGRFAqbOKs8liAuYYrnQRrQ6Ef14gMNfxNIwSfWhWOIL+j/jhq093Odh0oNdOFv51p4Z05XiqSzVyvv17nDr6ZZLwzv9nkeRlkQzqiJV5umyftRLU1ZERpx0/OkUpwsbwfLC+lQkJoHAcLJcBsXF+BAUBh/nBm8ElZrMEWdgZtqmDV11jZsyzpzmgf2o4WZklqSb9Yd0tzhCPCfIw5bS2/vaF2c7u2mQR+/6x18XO5vk71ir9kOS9k+O2CH7IgNmGDf2S/2m11He9EwGkdwUxp1lp4XbCWiL38BgbjYcw==</latexit><latexit sha1_base64="6WSEWb1HC7XemksphOEtcHz4Qbc=">AAACsHicbZFNaxsxEIbl7UfS7UecFnrpRdQUUjBhNxRSegptDzkmECeGrDGz2rGjRh9babax2W4P/Sm9Nn+o/6Zax5Q6zoDg5Zl5pdFMXirpKUn+dKJ79x883Nh8FD9+8vTZVnf7+am3lRM4EFZZN8zBo5IGByRJ4bB0CDpXeJZffmrzZ9/QeWnNCc1LHGmYGjmRAiigcfclfuA7sz6f93nZ59lnVASc3o67vWQ3WQRfF+lS9NgyjsbbnZ9ZYUWl0ZBQ4P15mpQ0qsGRFAqbOKs8liAuYYrnQRrQ6Ef14gMNfxNIwSfWhWOIL+j/jhq093Odh0oNdOFv51p4Z05XiqSzVyvv17nDr6ZZLwzv9nkeRlkQzqiJV5umyftRLU1ZERpx0/OkUpwsbwfLC+lQkJoHAcLJcBsXF+BAUBh/nBm8ElZrMEWdgZtqmDV11jZsyzpzmgf2o4WZklqSb9Yd0tzhCPCfIw5bS2/vaF2c7u2mQR+/6x18XO5vk71ir9kOS9k+O2CH7IgNmGDf2S/2m11He9EwGkdwUxp1lp4XbCWiL38BgbjYcw==</latexit><latexit sha1_base64="6WSEWb1HC7XemksphOEtcHz4Qbc=">AAACsHicbZFNaxsxEIbl7UfS7UecFnrpRdQUUjBhNxRSegptDzkmECeGrDGz2rGjRh9babax2W4P/Sm9Nn+o/6Zax5Q6zoDg5Zl5pdFMXirpKUn+dKJ79x883Nh8FD9+8vTZVnf7+am3lRM4EFZZN8zBo5IGByRJ4bB0CDpXeJZffmrzZ9/QeWnNCc1LHGmYGjmRAiigcfclfuA7sz6f93nZ59lnVASc3o67vWQ3WQRfF+lS9NgyjsbbnZ9ZYUWl0ZBQ4P15mpQ0qsGRFAqbOKs8liAuYYrnQRrQ6Ef14gMNfxNIwSfWhWOIL+j/jhq093Odh0oNdOFv51p4Z05XiqSzVyvv17nDr6ZZLwzv9nkeRlkQzqiJV5umyftRLU1ZERpx0/OkUpwsbwfLC+lQkJoHAcLJcBsXF+BAUBh/nBm8ElZrMEWdgZtqmDV11jZsyzpzmgf2o4WZklqSb9Yd0tzhCPCfIw5bS2/vaF2c7u2mQR+/6x18XO5vk71ir9kOS9k+O2CH7IgNmGDf2S/2m11He9EwGkdwUxp1lp4XbCWiL38BgbjYcw==</latexit> 93
  93. 93. ニューラルネットワークによる処理, SSII2020 mlp (64,64,64,128,1024) Idea 2: LUT※ Realization of MLP ℎ ※Look-up table > ? > ? ,(-@) = '(./0({2(3@AB, 56@), ℎ(7@)})) 94 Inputs to ℎ: discrete → Precompute MLP on LUT → 45× faster than MLP
  94. 94. 95 Triangle other Setup Output 95
  95. 95. ニューラルネットワークによる処理, SSII2020 Event-based Asynchronous Sparse Convolutional Networks Messikommer et.al, arXiv2020 Synchronous training = Asynchronous event-wise inference. 10x less FLOPS than dense conv Derived recursive alg. based on SSC※ ※Submanifold Sparse Convolutional Networks” (CVPR2018) 96 Image from github/btgraham CONV Sparcity is constant acrross layers =Fixed # of anctive site (spatial potision whicn contained none zero entry) SSC
  96. 96. 9797
  97. 97. まとめ 98
  98. 98. ニューラルネットワークによる処理, SSII2020 まとめ イベントカメラって︖ §明るさの変化を観測するカメラ §センサーとして良い特徴 § HDR・⾼速・ブラーレス・コンパクトデータ § うまく使えば難しい環境で動作する低計算量で⾼速レスポンスな⼿法が実現︕ Video from, Inivation 99
  99. 99. ニューラルネットワークによる処理, SSII2020 まとめ どうやって使うの︖ §データの形式や特性がフレーム画像と違うので,“フレーム画像処理“がそのまま使えない §⾮同期&スパース §動きで⾒えが変わる §イベントの特性を⽣かした処理で フレームカメラの適⽤が困難なシーンにも §HD環境での⾼速トラッキング §逐次型NNによる⾼速な認識 §将来 §Event型センサ&処理とフレーム型センサ&処理のハイブリッド §スパース性を活かした逐次型NNはまだ黎明期 発展が楽しみな分野 100
  100. 100. ニューラルネットワークによる処理, SSII2020 Reference • Gallego et.al., Event-based Vision: A Survey • Gallego et.al., Event-based Vision Resources • Scaramuzza et.al., Event-based Vision and Smart Cameras (CVPR2019 Workshop) 101
  101. 101. 102102

×