PPT for A Neural Autoregressive Approach to Collaborative Filtering (CF-NADE). I made ppt for explaining the paper. Abstract of the paper: This paper proposes CF-NADE, a neural autoregressive architecture for collaborative filtering (CF) tasks, which is inspired by the Restricted Boltzmann Machine (RBM) based CF model and the Neural Autoregressive Distribution Estimator (NADE). We first describe the basic CF-NADE model for CF tasks. Then we propose to improve the model by sharing parameters between different ratings. A factored version of CF-NADE is also proposed for better scalability. Furthermore, we take the ordinal nature of the preferences into consideration and propose an ordinal cost to optimize CF-NADE, which shows superior performance. Finally, CF-NADE can be extended to a deep model, with only moderately increased computational complexity. Experimental results show that CF-NADE with a single hidden layer beats all previous state-of-the-art methods on MovieLens 1M, MovieLens 10M, and Netflix datasets, and adding more hidden layers can further improve the performance.

1. CF-NADE
A NEURAL AUTOREGRESSIVE APPROACH
TO COLLABORATIVE FILTERING
2018. 10. 31.
JoonyoungYi
joonyoung.yi@kaist.ac.kr
*This slide is adopted from the author’s slide.

2. CONTENTS
1. Motivation
2. NADE
3. Basic Model of CF-NADE
4. Parameters Sharing
5. Ordinal Cost
6. Deep Models and Augmentation
7. Experiments
8. Future Works and Discussion

3. 3
• NADE (Neural Autoregressive Density Estimator)
• 2011 AISTAT.
• Alternative to Restricted Boltzmann Machine (RBM).
• CF-NADE (NADE to Collaborative Filtering)
• 2016 ICML.
• Apply NADE to Collaborative Filtering (CF) Problem.
• Matrix Completion is one of the popular algorithms to solve CF Problem.
TODAY’S PAPERS

4. 4
• The Netflix recommendation engine use hybrid models combined with
matrix factorization based models (Biased-MF, …) and deep networks [3].
• Not Netflix Prize.
• The main model used for deep networks is RBM-CF [9].
• The RBM-CF apply RBM to CF problem.
• An algorithm that works well with practical data.
• Authors of CF-NADE were in Hulu, which is a competitor of the Netflix.
• To enhance their recommendation engine.
• By replacing RBM-CF to CF-NADE.
• Personally, the CF-NADE paper is able to get several insights on engineering.
MOTIVATION

5. 5
MOTIVATION
RBM NADE
RBM-CF CF-NADE

6. CONTENTS
1. Motivation
2. NADE
3. Basic Model of CF-NADE
4. Parameters Sharing
5. Ordinal Cost
6. Deep Models and Augmentation
7. Experiments
8. Future Works and Discussion

7. 7
• RBM is not tractable by the partition function.
• The goal of NADE:
• Make RBM be tractable.
• To do this, the author of NADE convert RBM to a Bayesian network.
RBM TO NADE
Z is the partition function.
We should calculate Z by considering all possible v, h
The thing we need to train in NADE

8. 8
• The model of NADE:
• It is a tractable.There is no partition function or similar things.
• For fixed length binary vectors.
• Practical for high dimensional data.
• Practical version of NADE:
NADE
p(vi = 1|v<i) = sigm(bi + Vi,:hi)
hi = sigm(c +
X
j<i
W:,j)
<latexit sha1_base64="4YOzk+sFkqupXgiISHGBsvBHvLw=">AAACdHicbVFNaxsxENVuvxL3I05zCZTCNCZg02B2S6BtSCDQS6GXFOo44DWLVpZtJdLuIs2aGkX/IL+ut/6NXHKN7CwkdTogePPmPY00k5VSGIyiv0H45Omz5y/W1hsvX71+s9HcfHtqikoz3mOFLPRZRg2XIuc9FCj5Wak5VZnk/ezi26Len3FtRJH/wnnJh4pOcjEWjKKn0uZVCW2YpQKOIIZLSLJCjuzMpfZQuI4nE+S/0RoxUQ7amdd9rDWnXiP24MDV+dSlogNJ0rhPV+21krnFJaZSqT33XaCf2oM9OHfQSZutqBstAx6DuAYtUsdJ2vyTjApWKZ4jk9SYQRyVOLRUo2CSu0ZSGV5SdkEnfOBhThU3Q7ucmoNdz4xgXGh/coQl+9BhqTJmrjKvVBSnZrW2IP9XG1Q4/jK0Ii8r5Dm7azSuJGABixXASGjOUM49oEwL/1ZgU6opQ7+ohh9CvPrlx6D3qfu1G//cbx3/qKexRt6RHdImMflMjsl3ckJ6hJHrYDv4EOwEN+H7sBXu3knDoPZskX8i7N4CrD66zw==</latexit><latexit sha1_base64="4YOzk+sFkqupXgiISHGBsvBHvLw=">AAACdHicbVFNaxsxENVuvxL3I05zCZTCNCZg02B2S6BtSCDQS6GXFOo44DWLVpZtJdLuIs2aGkX/IL+ut/6NXHKN7CwkdTogePPmPY00k5VSGIyiv0H45Omz5y/W1hsvX71+s9HcfHtqikoz3mOFLPRZRg2XIuc9FCj5Wak5VZnk/ezi26Len3FtRJH/wnnJh4pOcjEWjKKn0uZVCW2YpQKOIIZLSLJCjuzMpfZQuI4nE+S/0RoxUQ7amdd9rDWnXiP24MDV+dSlogNJ0rhPV+21krnFJaZSqT33XaCf2oM9OHfQSZutqBstAx6DuAYtUsdJ2vyTjApWKZ4jk9SYQRyVOLRUo2CSu0ZSGV5SdkEnfOBhThU3Q7ucmoNdz4xgXGh/coQl+9BhqTJmrjKvVBSnZrW2IP9XG1Q4/jK0Ii8r5Dm7azSuJGABixXASGjOUM49oEwL/1ZgU6opQ7+ohh9CvPrlx6D3qfu1G//cbx3/qKexRt6RHdImMflMjsl3ckJ6hJHrYDv4EOwEN+H7sBXu3knDoPZskX8i7N4CrD66zw==</latexit><latexit sha1_base64="4YOzk+sFkqupXgiISHGBsvBHvLw=">AAACdHicbVFNaxsxENVuvxL3I05zCZTCNCZg02B2S6BtSCDQS6GXFOo44DWLVpZtJdLuIs2aGkX/IL+ut/6NXHKN7CwkdTogePPmPY00k5VSGIyiv0H45Omz5y/W1hsvX71+s9HcfHtqikoz3mOFLPRZRg2XIuc9FCj5Wak5VZnk/ezi26Len3FtRJH/wnnJh4pOcjEWjKKn0uZVCW2YpQKOIIZLSLJCjuzMpfZQuI4nE+S/0RoxUQ7amdd9rDWnXiP24MDV+dSlogNJ0rhPV+21krnFJaZSqT33XaCf2oM9OHfQSZutqBstAx6DuAYtUsdJ2vyTjApWKZ4jk9SYQRyVOLRUo2CSu0ZSGV5SdkEnfOBhThU3Q7ucmoNdz4xgXGh/coQl+9BhqTJmrjKvVBSnZrW2IP9XG1Q4/jK0Ii8r5Dm7azSuJGABixXASGjOUM49oEwL/1ZgU6opQ7+ohh9CvPrlx6D3qfu1G//cbx3/qKexRt6RHdImMflMjsl3ckJ6hJHrYDv4EOwEN+H7sBXu3knDoPZskX8i7N4CrD66zw==</latexit><latexit sha1_base64="4YOzk+sFkqupXgiISHGBsvBHvLw=">AAACdHicbVFNaxsxENVuvxL3I05zCZTCNCZg02B2S6BtSCDQS6GXFOo44DWLVpZtJdLuIs2aGkX/IL+ut/6NXHKN7CwkdTogePPmPY00k5VSGIyiv0H45Omz5y/W1hsvX71+s9HcfHtqikoz3mOFLPRZRg2XIuc9FCj5Wak5VZnk/ezi26Len3FtRJH/wnnJh4pOcjEWjKKn0uZVCW2YpQKOIIZLSLJCjuzMpfZQuI4nE+S/0RoxUQ7amdd9rDWnXiP24MDV+dSlogNJ0rhPV+21krnFJaZSqT33XaCf2oM9OHfQSZutqBstAx6DuAYtUsdJ2vyTjApWKZ4jk9SYQRyVOLRUo2CSu0ZSGV5SdkEnfOBhThU3Q7ucmoNdz4xgXGh/coQl+9BhqTJmrjKvVBSnZrW2IP9XG1Q4/jK0Ii8r5Dm7azSuJGABixXASGjOUM49oEwL/1ZgU6opQ7+ohh9CvPrlx6D3qfu1G//cbx3/qKexRt6RHdImMflMjsl3ckJ6hJHrYDv4EOwEN+H7sBXu3knDoPZskX8i7N4CrD66zw==</latexit>

9. CONTENTS
1. Motivation
2. NADE
3. Basic Model of CF-NADE
4. Parameters Sharing
5. Ordinal Cost
6. Deep Models and Augmentation
7. Experiments
8. Future Works and Discussion

10. 10
• The training case for user u:
• o is a D-tuple in the set of permutations of (1, 2, …, D).
• The items .
• .
• For simplicity, the index u of ru can be omitted.
• We want to model the below equation by NADE.
• : the first i-1 elements of r indexed by o.
• D is the number of ratings by user u.
• o: ordering.
NOTATION AND PROBABILISTIC MODEL

11. • The objective function:
• where
• g( ) : activation function like tanh.
• is a latent matrix. is a weight matrix.
• and are the bias terms.
11
THE BASIC MODEL
no activation function

12. 12
FIGURES OF THE MODEL
W1 W2 Wk…

13. 13
FIGURES OF THE MODEL
W1 W2 Wk…
+
W
rmo1
:,mo1<latexit sha1_base64="iJwv7B/8dh/Y1fnz6gNS8tSGnig=">AAACBXicbVDLSgMxFM34rPU16lKEYBFcSJkRwceq4EZwU8GxhXYcMmnahiaTIckIJczKjb/ixoWKW//BnX9j2s5CWw9cODnnXnLviVNGlfa8b2dufmFxabm0Ul5dW9/YdLe275TIJCYBFkzIZowUYTQhgaaakWYqCeIxI414cDnyGw9EKiqSWz1MSchRL6FdipG2UuTuNSJzcQR5ZETk5zC/NzIyxSvPI7fiVb0x4CzxC1IBBeqR+9XuCJxxkmjMkFIt30t1aJDUFDOSl9uZIinCA9QjLUsTxIkKzfiMHB5YpQO7QtpKNByrvycM4koNeWw7OdJ9Ne2NxP+8Vqa7Z6GhSZppkuDJR92MQS3gKBPYoZJgzYaWICyp3RXiPpIIa5tc2YbgT588S4Lj6nnVvzmp1K6LNEpgF+yDQ+CDU1ADV6AOAoDBI3gGr+DNeXJenHfnY9I65xQzO+APnM8fTteYow==</latexit><latexit sha1_base64="iJwv7B/8dh/Y1fnz6gNS8tSGnig=">AAACBXicbVDLSgMxFM34rPU16lKEYBFcSJkRwceq4EZwU8GxhXYcMmnahiaTIckIJczKjb/ixoWKW//BnX9j2s5CWw9cODnnXnLviVNGlfa8b2dufmFxabm0Ul5dW9/YdLe275TIJCYBFkzIZowUYTQhgaaakWYqCeIxI414cDnyGw9EKiqSWz1MSchRL6FdipG2UuTuNSJzcQR5ZETk5zC/NzIyxSvPI7fiVb0x4CzxC1IBBeqR+9XuCJxxkmjMkFIt30t1aJDUFDOSl9uZIinCA9QjLUsTxIkKzfiMHB5YpQO7QtpKNByrvycM4koNeWw7OdJ9Ne2NxP+8Vqa7Z6GhSZppkuDJR92MQS3gKBPYoZJgzYaWICyp3RXiPpIIa5tc2YbgT588S4Lj6nnVvzmp1K6LNEpgF+yDQ+CDU1ADV6AOAoDBI3gGr+DNeXJenHfnY9I65xQzO+APnM8fTteYow==</latexit><latexit sha1_base64="iJwv7B/8dh/Y1fnz6gNS8tSGnig=">AAACBXicbVDLSgMxFM34rPU16lKEYBFcSJkRwceq4EZwU8GxhXYcMmnahiaTIckIJczKjb/ixoWKW//BnX9j2s5CWw9cODnnXnLviVNGlfa8b2dufmFxabm0Ul5dW9/YdLe275TIJCYBFkzIZowUYTQhgaaakWYqCeIxI414cDnyGw9EKiqSWz1MSchRL6FdipG2UuTuNSJzcQR5ZETk5zC/NzIyxSvPI7fiVb0x4CzxC1IBBeqR+9XuCJxxkmjMkFIt30t1aJDUFDOSl9uZIinCA9QjLUsTxIkKzfiMHB5YpQO7QtpKNByrvycM4koNeWw7OdJ9Ne2NxP+8Vqa7Z6GhSZppkuDJR92MQS3gKBPYoZJgzYaWICyp3RXiPpIIa5tc2YbgT588S4Lj6nnVvzmp1K6LNEpgF+yDQ+CDU1ADV6AOAoDBI3gGr+DNeXJenHfnY9I65xQzO+APnM8fTteYow==</latexit><latexit sha1_base64="iJwv7B/8dh/Y1fnz6gNS8tSGnig=">AAACBXicbVDLSgMxFM34rPU16lKEYBFcSJkRwceq4EZwU8GxhXYcMmnahiaTIckIJczKjb/ixoWKW//BnX9j2s5CWw9cODnnXnLviVNGlfa8b2dufmFxabm0Ul5dW9/YdLe275TIJCYBFkzIZowUYTQhgaaakWYqCeIxI414cDnyGw9EKiqSWz1MSchRL6FdipG2UuTuNSJzcQR5ZETk5zC/NzIyxSvPI7fiVb0x4CzxC1IBBeqR+9XuCJxxkmjMkFIt30t1aJDUFDOSl9uZIinCA9QjLUsTxIkKzfiMHB5YpQO7QtpKNByrvycM4koNeWw7OdJ9Ne2NxP+8Vqa7Z6GhSZppkuDJR92MQS3gKBPYoZJgzYaWICyp3RXiPpIIa5tc2YbgT588S4Lj6nnVvzmp1K6LNEpgF+yDQ+CDU1ADV6AOAoDBI3gGr+DNeXJenHfnY9I65xQzO+APnM8fTteYow==</latexit>
+ + … +
W
rmo2
:,mo2<latexit sha1_base64="opdz4oAh5Obm9hxV3ZKKD/28H70=">AAACBXicbVDLSgMxFM3UV62vUZciBIvgQspMEXysCm4ENxUcW2jHIZNm2tDMZEgyQgmzcuOvuHGh4tZ/cOffmLaz0NYDF07OuZfce8KUUakc59sqLSwuLa+UVytr6xubW/b2zp3kmcDEw5xx0Q6RJIwmxFNUMdJOBUFxyEgrHF6O/dYDEZLy5FaNUuLHqJ/QiGKkjBTY+61AXxzDONA8qOcwv9ci0MUrzwO76tScCeA8cQtSBQWagf3V7XGcxSRRmCEpO66TKl8joShmJK90M0lShIeoTzqGJigm0teTM3J4aJQejLgwlSg4UX9PaBRLOYpD0xkjNZCz3lj8z+tkKjrzNU3STJEETz+KMgYVh+NMYI8KghUbGYKwoGZXiAdIIKxMchUTgjt78jzx6rXzmntzUm1cF2mUwR44AEfABaegAa5AE3gAg0fwDF7Bm/VkvVjv1se0tWQVM7vgD6zPH1HxmKU=</latexit><latexit sha1_base64="opdz4oAh5Obm9hxV3ZKKD/28H70=">AAACBXicbVDLSgMxFM3UV62vUZciBIvgQspMEXysCm4ENxUcW2jHIZNm2tDMZEgyQgmzcuOvuHGh4tZ/cOffmLaz0NYDF07OuZfce8KUUakc59sqLSwuLa+UVytr6xubW/b2zp3kmcDEw5xx0Q6RJIwmxFNUMdJOBUFxyEgrHF6O/dYDEZLy5FaNUuLHqJ/QiGKkjBTY+61AXxzDONA8qOcwv9ci0MUrzwO76tScCeA8cQtSBQWagf3V7XGcxSRRmCEpO66TKl8joShmJK90M0lShIeoTzqGJigm0teTM3J4aJQejLgwlSg4UX9PaBRLOYpD0xkjNZCz3lj8z+tkKjrzNU3STJEETz+KMgYVh+NMYI8KghUbGYKwoGZXiAdIIKxMchUTgjt78jzx6rXzmntzUm1cF2mUwR44AEfABaegAa5AE3gAg0fwDF7Bm/VkvVjv1se0tWQVM7vgD6zPH1HxmKU=</latexit><latexit sha1_base64="opdz4oAh5Obm9hxV3ZKKD/28H70=">AAACBXicbVDLSgMxFM3UV62vUZciBIvgQspMEXysCm4ENxUcW2jHIZNm2tDMZEgyQgmzcuOvuHGh4tZ/cOffmLaz0NYDF07OuZfce8KUUakc59sqLSwuLa+UVytr6xubW/b2zp3kmcDEw5xx0Q6RJIwmxFNUMdJOBUFxyEgrHF6O/dYDEZLy5FaNUuLHqJ/QiGKkjBTY+61AXxzDONA8qOcwv9ci0MUrzwO76tScCeA8cQtSBQWagf3V7XGcxSRRmCEpO66TKl8joShmJK90M0lShIeoTzqGJigm0teTM3J4aJQejLgwlSg4UX9PaBRLOYpD0xkjNZCz3lj8z+tkKjrzNU3STJEETz+KMgYVh+NMYI8KghUbGYKwoGZXiAdIIKxMchUTgjt78jzx6rXzmntzUm1cF2mUwR44AEfABaegAa5AE3gAg0fwDF7Bm/VkvVjv1se0tWQVM7vgD6zPH1HxmKU=</latexit><latexit sha1_base64="opdz4oAh5Obm9hxV3ZKKD/28H70=">AAACBXicbVDLSgMxFM3UV62vUZciBIvgQspMEXysCm4ENxUcW2jHIZNm2tDMZEgyQgmzcuOvuHGh4tZ/cOffmLaz0NYDF07OuZfce8KUUakc59sqLSwuLa+UVytr6xubW/b2zp3kmcDEw5xx0Q6RJIwmxFNUMdJOBUFxyEgrHF6O/dYDEZLy5FaNUuLHqJ/QiGKkjBTY+61AXxzDONA8qOcwv9ci0MUrzwO76tScCeA8cQtSBQWagf3V7XGcxSRRmCEpO66TKl8joShmJK90M0lShIeoTzqGJigm0teTM3J4aJQejLgwlSg4UX9PaBRLOYpD0xkjNZCz3lj8z+tkKjrzNU3STJEETz+KMgYVh+NMYI8KghUbGYKwoGZXiAdIIKxMchUTgjt78jzx6rXzmntzUm1cF2mUwR44AEfABaegAa5AE3gAg0fwDF7Bm/VkvVjv1se0tWQVM7vgD6zPH1HxmKU=</latexit>
W
rmoi 1
:,moi 1<latexit sha1_base64="vDWHlwlOmdQjyoSWksi1NThDM8A=">AAACDnicbVDLSgMxFM3UV62vUZdugkVxoWVGBB+rghvBTQXHFtpxyKRpG5pJhiQjlNA/cOOvuHGh4ta1O//GtB1Qq+dy4XDOvST3xCmjSnvep1OYmZ2bXygulpaWV1bX3PWNGyUyiUmABROyESNFGOUk0FQz0kglQUnMSD3un4/8+h2Rigp+rQcpCRPU5bRDMdJWitzdemTO9mESGREZeuAPoa1bIyPzLVlEbtmreGPAv8TPSRnkqEXuR6stcJYQrjFDSjV9L9WhQVJTzMiw1MoUSRHuoy5pWspRQlRoxvcM4Y5V2rAjpG2u4Vj9uWFQotQgie1kgnRPTXsj8T+vmenOSWgoTzNNOJ481MkY1AKOwoFtKgnWbGAJwpLav0LcQxJhbSMs2RD86ZP/kuCwclrxr47K1cs8jSLYAttgD/jgGFTBBaiBAGBwDx7BM3hxHpwn59V5m4wWnHxnE/yC8/4F64qcOQ==</latexit><latexit sha1_base64="vDWHlwlOmdQjyoSWksi1NThDM8A=">AAACDnicbVDLSgMxFM3UV62vUZdugkVxoWVGBB+rghvBTQXHFtpxyKRpG5pJhiQjlNA/cOOvuHGh4ta1O//GtB1Qq+dy4XDOvST3xCmjSnvep1OYmZ2bXygulpaWV1bX3PWNGyUyiUmABROyESNFGOUk0FQz0kglQUnMSD3un4/8+h2Rigp+rQcpCRPU5bRDMdJWitzdemTO9mESGREZeuAPoa1bIyPzLVlEbtmreGPAv8TPSRnkqEXuR6stcJYQrjFDSjV9L9WhQVJTzMiw1MoUSRHuoy5pWspRQlRoxvcM4Y5V2rAjpG2u4Vj9uWFQotQgie1kgnRPTXsj8T+vmenOSWgoTzNNOJ481MkY1AKOwoFtKgnWbGAJwpLav0LcQxJhbSMs2RD86ZP/kuCwclrxr47K1cs8jSLYAttgD/jgGFTBBaiBAGBwDx7BM3hxHpwn59V5m4wWnHxnE/yC8/4F64qcOQ==</latexit><latexit sha1_base64="vDWHlwlOmdQjyoSWksi1NThDM8A=">AAACDnicbVDLSgMxFM3UV62vUZdugkVxoWVGBB+rghvBTQXHFtpxyKRpG5pJhiQjlNA/cOOvuHGh4ta1O//GtB1Qq+dy4XDOvST3xCmjSnvep1OYmZ2bXygulpaWV1bX3PWNGyUyiUmABROyESNFGOUk0FQz0kglQUnMSD3un4/8+h2Rigp+rQcpCRPU5bRDMdJWitzdemTO9mESGREZeuAPoa1bIyPzLVlEbtmreGPAv8TPSRnkqEXuR6stcJYQrjFDSjV9L9WhQVJTzMiw1MoUSRHuoy5pWspRQlRoxvcM4Y5V2rAjpG2u4Vj9uWFQotQgie1kgnRPTXsj8T+vmenOSWgoTzNNOJ481MkY1AKOwoFtKgnWbGAJwpLav0LcQxJhbSMs2RD86ZP/kuCwclrxr47K1cs8jSLYAttgD/jgGFTBBaiBAGBwDx7BM3hxHpwn59V5m4wWnHxnE/yC8/4F64qcOQ==</latexit><latexit sha1_base64="vDWHlwlOmdQjyoSWksi1NThDM8A=">AAACDnicbVDLSgMxFM3UV62vUZdugkVxoWVGBB+rghvBTQXHFtpxyKRpG5pJhiQjlNA/cOOvuHGh4ta1O//GtB1Qq+dy4XDOvST3xCmjSnvep1OYmZ2bXygulpaWV1bX3PWNGyUyiUmABROyESNFGOUk0FQz0kglQUnMSD3un4/8+h2Rigp+rQcpCRPU5bRDMdJWitzdemTO9mESGREZeuAPoa1bIyPzLVlEbtmreGPAv8TPSRnkqEXuR6stcJYQrjFDSjV9L9WhQVJTzMiw1MoUSRHuoy5pWspRQlRoxvcM4Y5V2rAjpG2u4Vj9uWFQotQgie1kgnRPTXsj8T+vmenOSWgoTzNNOJ481MkY1AKOwoFtKgnWbGAJwpLav0LcQxJhbSMs2RD86ZP/kuCwclrxr47K1cs8jSLYAttgD/jgGFTBBaiBAGBwDx7BM3hxHpwn59V5m4wWnHxnE/yC8/4F64qcOQ==</latexit>

14. 14
FIGURES OF THE MODEL
+
W
rmo1
:,mo1<latexit sha1_base64="iJwv7B/8dh/Y1fnz6gNS8tSGnig=">AAACBXicbVDLSgMxFM34rPU16lKEYBFcSJkRwceq4EZwU8GxhXYcMmnahiaTIckIJczKjb/ixoWKW//BnX9j2s5CWw9cODnnXnLviVNGlfa8b2dufmFxabm0Ul5dW9/YdLe275TIJCYBFkzIZowUYTQhgaaakWYqCeIxI414cDnyGw9EKiqSWz1MSchRL6FdipG2UuTuNSJzcQR5ZETk5zC/NzIyxSvPI7fiVb0x4CzxC1IBBeqR+9XuCJxxkmjMkFIt30t1aJDUFDOSl9uZIinCA9QjLUsTxIkKzfiMHB5YpQO7QtpKNByrvycM4koNeWw7OdJ9Ne2NxP+8Vqa7Z6GhSZppkuDJR92MQS3gKBPYoZJgzYaWICyp3RXiPpIIa5tc2YbgT588S4Lj6nnVvzmp1K6LNEpgF+yDQ+CDU1ADV6AOAoDBI3gGr+DNeXJenHfnY9I65xQzO+APnM8fTteYow==</latexit><latexit sha1_base64="iJwv7B/8dh/Y1fnz6gNS8tSGnig=">AAACBXicbVDLSgMxFM34rPU16lKEYBFcSJkRwceq4EZwU8GxhXYcMmnahiaTIckIJczKjb/ixoWKW//BnX9j2s5CWw9cODnnXnLviVNGlfa8b2dufmFxabm0Ul5dW9/YdLe275TIJCYBFkzIZowUYTQhgaaakWYqCeIxI414cDnyGw9EKiqSWz1MSchRL6FdipG2UuTuNSJzcQR5ZETk5zC/NzIyxSvPI7fiVb0x4CzxC1IBBeqR+9XuCJxxkmjMkFIt30t1aJDUFDOSl9uZIinCA9QjLUsTxIkKzfiMHB5YpQO7QtpKNByrvycM4koNeWw7OdJ9Ne2NxP+8Vqa7Z6GhSZppkuDJR92MQS3gKBPYoZJgzYaWICyp3RXiPpIIa5tc2YbgT588S4Lj6nnVvzmp1K6LNEpgF+yDQ+CDU1ADV6AOAoDBI3gGr+DNeXJenHfnY9I65xQzO+APnM8fTteYow==</latexit><latexit sha1_base64="iJwv7B/8dh/Y1fnz6gNS8tSGnig=">AAACBXicbVDLSgMxFM34rPU16lKEYBFcSJkRwceq4EZwU8GxhXYcMmnahiaTIckIJczKjb/ixoWKW//BnX9j2s5CWw9cODnnXnLviVNGlfa8b2dufmFxabm0Ul5dW9/YdLe275TIJCYBFkzIZowUYTQhgaaakWYqCeIxI414cDnyGw9EKiqSWz1MSchRL6FdipG2UuTuNSJzcQR5ZETk5zC/NzIyxSvPI7fiVb0x4CzxC1IBBeqR+9XuCJxxkmjMkFIt30t1aJDUFDOSl9uZIinCA9QjLUsTxIkKzfiMHB5YpQO7QtpKNByrvycM4koNeWw7OdJ9Ne2NxP+8Vqa7Z6GhSZppkuDJR92MQS3gKBPYoZJgzYaWICyp3RXiPpIIa5tc2YbgT588S4Lj6nnVvzmp1K6LNEpgF+yDQ+CDU1ADV6AOAoDBI3gGr+DNeXJenHfnY9I65xQzO+APnM8fTteYow==</latexit><latexit sha1_base64="iJwv7B/8dh/Y1fnz6gNS8tSGnig=">AAACBXicbVDLSgMxFM34rPU16lKEYBFcSJkRwceq4EZwU8GxhXYcMmnahiaTIckIJczKjb/ixoWKW//BnX9j2s5CWw9cODnnXnLviVNGlfa8b2dufmFxabm0Ul5dW9/YdLe275TIJCYBFkzIZowUYTQhgaaakWYqCeIxI414cDnyGw9EKiqSWz1MSchRL6FdipG2UuTuNSJzcQR5ZETk5zC/NzIyxSvPI7fiVb0x4CzxC1IBBeqR+9XuCJxxkmjMkFIt30t1aJDUFDOSl9uZIinCA9QjLUsTxIkKzfiMHB5YpQO7QtpKNByrvycM4koNeWw7OdJ9Ne2NxP+8Vqa7Z6GhSZppkuDJR92MQS3gKBPYoZJgzYaWICyp3RXiPpIIa5tc2YbgT588S4Lj6nnVvzmp1K6LNEpgF+yDQ+CDU1ADV6AOAoDBI3gGr+DNeXJenHfnY9I65xQzO+APnM8fTteYow==</latexit>
+
c<latexit sha1_base64="8xJmIOcF0OzDWOLwsax2XAaqU70=">AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEUG8FL4KXCsYW2lA2m027dLMbdzdCCfkTXjyoePX3ePPfuG1z0NYHA4/3ZpiZF6acaeO6305lZXVtfaO6Wdva3tndq+8fPGiZKUJ9IrlU3RBrypmgvmGG026qKE5CTjvh+Hrqd56o0kyKezNJaZDgoWAxI9hYqdsPJY9yUgzqDbfpzoCWiVeSBpRoD+pf/UiSLKHCEI617nluaoIcK8MIp0Wtn2maYjLGQ9qzVOCE6iCf3VugE6tEKJbKljBopv6eyHGi9SQJbWeCzUgvelPxP6+XmfgyyJlIM0MFmS+KM46MRNPnUcQUJYZPLMFEMXsrIiOsMDE2opoNwVt8eZn4Z82rpnd33mjdlmlU4QiO4RQ8uIAW3EAbfCDA4Rle4c15dF6cd+dj3lpxyplD+APn8we9zo//</latexit><latexit sha1_base64="8xJmIOcF0OzDWOLwsax2XAaqU70=">AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEUG8FL4KXCsYW2lA2m027dLMbdzdCCfkTXjyoePX3ePPfuG1z0NYHA4/3ZpiZF6acaeO6305lZXVtfaO6Wdva3tndq+8fPGiZKUJ9IrlU3RBrypmgvmGG026qKE5CTjvh+Hrqd56o0kyKezNJaZDgoWAxI9hYqdsPJY9yUgzqDbfpzoCWiVeSBpRoD+pf/UiSLKHCEI617nluaoIcK8MIp0Wtn2maYjLGQ9qzVOCE6iCf3VugE6tEKJbKljBopv6eyHGi9SQJbWeCzUgvelPxP6+XmfgyyJlIM0MFmS+KM46MRNPnUcQUJYZPLMFEMXsrIiOsMDE2opoNwVt8eZn4Z82rpnd33mjdlmlU4QiO4RQ8uIAW3EAbfCDA4Rle4c15dF6cd+dj3lpxyplD+APn8we9zo//</latexit><latexit sha1_base64="8xJmIOcF0OzDWOLwsax2XAaqU70=">AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEUG8FL4KXCsYW2lA2m027dLMbdzdCCfkTXjyoePX3ePPfuG1z0NYHA4/3ZpiZF6acaeO6305lZXVtfaO6Wdva3tndq+8fPGiZKUJ9IrlU3RBrypmgvmGG026qKE5CTjvh+Hrqd56o0kyKezNJaZDgoWAxI9hYqdsPJY9yUgzqDbfpzoCWiVeSBpRoD+pf/UiSLKHCEI617nluaoIcK8MIp0Wtn2maYjLGQ9qzVOCE6iCf3VugE6tEKJbKljBopv6eyHGi9SQJbWeCzUgvelPxP6+XmfgyyJlIM0MFmS+KM46MRNPnUcQUJYZPLMFEMXsrIiOsMDE2opoNwVt8eZn4Z82rpnd33mjdlmlU4QiO4RQ8uIAW3EAbfCDA4Rle4c15dF6cd+dj3lpxyplD+APn8we9zo//</latexit><latexit sha1_base64="8xJmIOcF0OzDWOLwsax2XAaqU70=">AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEUG8FL4KXCsYW2lA2m027dLMbdzdCCfkTXjyoePX3ePPfuG1z0NYHA4/3ZpiZF6acaeO6305lZXVtfaO6Wdva3tndq+8fPGiZKUJ9IrlU3RBrypmgvmGG026qKE5CTjvh+Hrqd56o0kyKezNJaZDgoWAxI9hYqdsPJY9yUgzqDbfpzoCWiVeSBpRoD+pf/UiSLKHCEI617nluaoIcK8MIp0Wtn2maYjLGQ9qzVOCE6iCf3VugE6tEKJbKljBopv6eyHGi9SQJbWeCzUgvelPxP6+XmfgyyJlIM0MFmS+KM46MRNPnUcQUJYZPLMFEMXsrIiOsMDE2opoNwVt8eZn4Z82rpnd33mjdlmlU4QiO4RQ8uIAW3EAbfCDA4Rle4c15dF6cd+dj3lpxyplD+APn8we9zo//</latexit>
+ … +
W
rmo2
:,mo2<latexit sha1_base64="opdz4oAh5Obm9hxV3ZKKD/28H70=">AAACBXicbVDLSgMxFM3UV62vUZciBIvgQspMEXysCm4ENxUcW2jHIZNm2tDMZEgyQgmzcuOvuHGh4tZ/cOffmLaz0NYDF07OuZfce8KUUakc59sqLSwuLa+UVytr6xubW/b2zp3kmcDEw5xx0Q6RJIwmxFNUMdJOBUFxyEgrHF6O/dYDEZLy5FaNUuLHqJ/QiGKkjBTY+61AXxzDONA8qOcwv9ci0MUrzwO76tScCeA8cQtSBQWagf3V7XGcxSRRmCEpO66TKl8joShmJK90M0lShIeoTzqGJigm0teTM3J4aJQejLgwlSg4UX9PaBRLOYpD0xkjNZCz3lj8z+tkKjrzNU3STJEETz+KMgYVh+NMYI8KghUbGYKwoGZXiAdIIKxMchUTgjt78jzx6rXzmntzUm1cF2mUwR44AEfABaegAa5AE3gAg0fwDF7Bm/VkvVjv1se0tWQVM7vgD6zPH1HxmKU=</latexit><latexit sha1_base64="opdz4oAh5Obm9hxV3ZKKD/28H70=">AAACBXicbVDLSgMxFM3UV62vUZciBIvgQspMEXysCm4ENxUcW2jHIZNm2tDMZEgyQgmzcuOvuHGh4tZ/cOffmLaz0NYDF07OuZfce8KUUakc59sqLSwuLa+UVytr6xubW/b2zp3kmcDEw5xx0Q6RJIwmxFNUMdJOBUFxyEgrHF6O/dYDEZLy5FaNUuLHqJ/QiGKkjBTY+61AXxzDONA8qOcwv9ci0MUrzwO76tScCeA8cQtSBQWagf3V7XGcxSRRmCEpO66TKl8joShmJK90M0lShIeoTzqGJigm0teTM3J4aJQejLgwlSg4UX9PaBRLOYpD0xkjNZCz3lj8z+tkKjrzNU3STJEETz+KMgYVh+NMYI8KghUbGYKwoGZXiAdIIKxMchUTgjt78jzx6rXzmntzUm1cF2mUwR44AEfABaegAa5AE3gAg0fwDF7Bm/VkvVjv1se0tWQVM7vgD6zPH1HxmKU=</latexit><latexit sha1_base64="opdz4oAh5Obm9hxV3ZKKD/28H70=">AAACBXicbVDLSgMxFM3UV62vUZciBIvgQspMEXysCm4ENxUcW2jHIZNm2tDMZEgyQgmzcuOvuHGh4tZ/cOffmLaz0NYDF07OuZfce8KUUakc59sqLSwuLa+UVytr6xubW/b2zp3kmcDEw5xx0Q6RJIwmxFNUMdJOBUFxyEgrHF6O/dYDEZLy5FaNUuLHqJ/QiGKkjBTY+61AXxzDONA8qOcwv9ci0MUrzwO76tScCeA8cQtSBQWagf3V7XGcxSRRmCEpO66TKl8joShmJK90M0lShIeoTzqGJigm0teTM3J4aJQejLgwlSg4UX9PaBRLOYpD0xkjNZCz3lj8z+tkKjrzNU3STJEETz+KMgYVh+NMYI8KghUbGYKwoGZXiAdIIKxMchUTgjt78jzx6rXzmntzUm1cF2mUwR44AEfABaegAa5AE3gAg0fwDF7Bm/VkvVjv1se0tWQVM7vgD6zPH1HxmKU=</latexit><latexit sha1_base64="opdz4oAh5Obm9hxV3ZKKD/28H70=">AAACBXicbVDLSgMxFM3UV62vUZciBIvgQspMEXysCm4ENxUcW2jHIZNm2tDMZEgyQgmzcuOvuHGh4tZ/cOffmLaz0NYDF07OuZfce8KUUakc59sqLSwuLa+UVytr6xubW/b2zp3kmcDEw5xx0Q6RJIwmxFNUMdJOBUFxyEgrHF6O/dYDEZLy5FaNUuLHqJ/QiGKkjBTY+61AXxzDONA8qOcwv9ci0MUrzwO76tScCeA8cQtSBQWagf3V7XGcxSRRmCEpO66TKl8joShmJK90M0lShIeoTzqGJigm0teTM3J4aJQejLgwlSg4UX9PaBRLOYpD0xkjNZCz3lj8z+tkKjrzNU3STJEETz+KMgYVh+NMYI8KghUbGYKwoGZXiAdIIKxMchUTgjt78jzx6rXzmntzUm1cF2mUwR44AEfABaegAa5AE3gAg0fwDF7Bm/VkvVjv1se0tWQVM7vgD6zPH1HxmKU=</latexit>
W
rmoi 1
:,moi 1<latexit sha1_base64="vDWHlwlOmdQjyoSWksi1NThDM8A=">AAACDnicbVDLSgMxFM3UV62vUZdugkVxoWVGBB+rghvBTQXHFtpxyKRpG5pJhiQjlNA/cOOvuHGh4ta1O//GtB1Qq+dy4XDOvST3xCmjSnvep1OYmZ2bXygulpaWV1bX3PWNGyUyiUmABROyESNFGOUk0FQz0kglQUnMSD3un4/8+h2Rigp+rQcpCRPU5bRDMdJWitzdemTO9mESGREZeuAPoa1bIyPzLVlEbtmreGPAv8TPSRnkqEXuR6stcJYQrjFDSjV9L9WhQVJTzMiw1MoUSRHuoy5pWspRQlRoxvcM4Y5V2rAjpG2u4Vj9uWFQotQgie1kgnRPTXsj8T+vmenOSWgoTzNNOJ481MkY1AKOwoFtKgnWbGAJwpLav0LcQxJhbSMs2RD86ZP/kuCwclrxr47K1cs8jSLYAttgD/jgGFTBBaiBAGBwDx7BM3hxHpwn59V5m4wWnHxnE/yC8/4F64qcOQ==</latexit><latexit sha1_base64="vDWHlwlOmdQjyoSWksi1NThDM8A=">AAACDnicbVDLSgMxFM3UV62vUZdugkVxoWVGBB+rghvBTQXHFtpxyKRpG5pJhiQjlNA/cOOvuHGh4ta1O//GtB1Qq+dy4XDOvST3xCmjSnvep1OYmZ2bXygulpaWV1bX3PWNGyUyiUmABROyESNFGOUk0FQz0kglQUnMSD3un4/8+h2Rigp+rQcpCRPU5bRDMdJWitzdemTO9mESGREZeuAPoa1bIyPzLVlEbtmreGPAv8TPSRnkqEXuR6stcJYQrjFDSjV9L9WhQVJTzMiw1MoUSRHuoy5pWspRQlRoxvcM4Y5V2rAjpG2u4Vj9uWFQotQgie1kgnRPTXsj8T+vmenOSWgoTzNNOJ481MkY1AKOwoFtKgnWbGAJwpLav0LcQxJhbSMs2RD86ZP/kuCwclrxr47K1cs8jSLYAttgD/jgGFTBBaiBAGBwDx7BM3hxHpwn59V5m4wWnHxnE/yC8/4F64qcOQ==</latexit><latexit sha1_base64="vDWHlwlOmdQjyoSWksi1NThDM8A=">AAACDnicbVDLSgMxFM3UV62vUZdugkVxoWVGBB+rghvBTQXHFtpxyKRpG5pJhiQjlNA/cOOvuHGh4ta1O//GtB1Qq+dy4XDOvST3xCmjSnvep1OYmZ2bXygulpaWV1bX3PWNGyUyiUmABROyESNFGOUk0FQz0kglQUnMSD3un4/8+h2Rigp+rQcpCRPU5bRDMdJWitzdemTO9mESGREZeuAPoa1bIyPzLVlEbtmreGPAv8TPSRnkqEXuR6stcJYQrjFDSjV9L9WhQVJTzMiw1MoUSRHuoy5pWspRQlRoxvcM4Y5V2rAjpG2u4Vj9uWFQotQgie1kgnRPTXsj8T+vmenOSWgoTzNNOJ481MkY1AKOwoFtKgnWbGAJwpLav0LcQxJhbSMs2RD86ZP/kuCwclrxr47K1cs8jSLYAttgD/jgGFTBBaiBAGBwDx7BM3hxHpwn59V5m4wWnHxnE/yC8/4F64qcOQ==</latexit><latexit sha1_base64="vDWHlwlOmdQjyoSWksi1NThDM8A=">AAACDnicbVDLSgMxFM3UV62vUZdugkVxoWVGBB+rghvBTQXHFtpxyKRpG5pJhiQjlNA/cOOvuHGh4ta1O//GtB1Qq+dy4XDOvST3xCmjSnvep1OYmZ2bXygulpaWV1bX3PWNGyUyiUmABROyESNFGOUk0FQz0kglQUnMSD3un4/8+h2Rigp+rQcpCRPU5bRDMdJWitzdemTO9mESGREZeuAPoa1bIyPzLVlEbtmreGPAv8TPSRnkqEXuR6stcJYQrjFDSjV9L9WhQVJTzMiw1MoUSRHuoy5pWspRQlRoxvcM4Y5V2rAjpG2u4Vj9uWFQotQgie1kgnRPTXsj8T+vmenOSWgoTzNNOJ481MkY1AKOwoFtKgnWbGAJwpLav0LcQxJhbSMs2RD86ZP/kuCwclrxr47K1cs8jSLYAttgD/jgGFTBBaiBAGBwDx7BM3hxHpwn59V5m4wWnHxnE/yC8/4F64qcOQ==</latexit>
tanh

15. 15
FIGURES OF THE MODEL
V1 b1
b1
moi<latexit sha1_base64="khFMxWwj1mfcXUdBB7rNQr8Otg8=">AAAB83icbVBNS8NAEJ3Ur1q/qh69BIvgqSQiqLeCF8FLBWMLbQyb7bZduh9xd1MoIb/DiwcVr/4Zb/4bt20O2vpg4PHeDDPz4oRRbTzv2ymtrK6tb5Q3K1vbO7t71f2DBy1ThUmAJZOqHSNNGBUkMNQw0k4UQTxmpBWPrqd+a0yUplLcm0lCQo4GgvYpRsZKYfzoRxmPMhnRPI+qNa/uzeAuE78gNSjQjKpf3Z7EKSfCYIa07vheYsIMKUMxI3mlm2qSIDxCA9KxVCBOdJjNjs7dE6v03L5UtoRxZ+rviQxxrSc8tp0cmaFe9Kbif14nNf3LMKMiSQ0ReL6onzLXSHeagNujimDDJpYgrKi91cVDpBA2NqeKDcFffHmZBGf1q7p/d15r3BZplOEIjuEUfLiABtxAEwLA8ATP8Apvzth5cd6dj3lrySlmDuEPnM8fb2OSHA==</latexit><latexit sha1_base64="khFMxWwj1mfcXUdBB7rNQr8Otg8=">AAAB83icbVBNS8NAEJ3Ur1q/qh69BIvgqSQiqLeCF8FLBWMLbQyb7bZduh9xd1MoIb/DiwcVr/4Zb/4bt20O2vpg4PHeDDPz4oRRbTzv2ymtrK6tb5Q3K1vbO7t71f2DBy1ThUmAJZOqHSNNGBUkMNQw0k4UQTxmpBWPrqd+a0yUplLcm0lCQo4GgvYpRsZKYfzoRxmPMhnRPI+qNa/uzeAuE78gNSjQjKpf3Z7EKSfCYIa07vheYsIMKUMxI3mlm2qSIDxCA9KxVCBOdJjNjs7dE6v03L5UtoRxZ+rviQxxrSc8tp0cmaFe9Kbif14nNf3LMKMiSQ0ReL6onzLXSHeagNujimDDJpYgrKi91cVDpBA2NqeKDcFffHmZBGf1q7p/d15r3BZplOEIjuEUfLiABtxAEwLA8ATP8Apvzth5cd6dj3lrySlmDuEPnM8fb2OSHA==</latexit><latexit sha1_base64="khFMxWwj1mfcXUdBB7rNQr8Otg8=">AAAB83icbVBNS8NAEJ3Ur1q/qh69BIvgqSQiqLeCF8FLBWMLbQyb7bZduh9xd1MoIb/DiwcVr/4Zb/4bt20O2vpg4PHeDDPz4oRRbTzv2ymtrK6tb5Q3K1vbO7t71f2DBy1ThUmAJZOqHSNNGBUkMNQw0k4UQTxmpBWPrqd+a0yUplLcm0lCQo4GgvYpRsZKYfzoRxmPMhnRPI+qNa/uzeAuE78gNSjQjKpf3Z7EKSfCYIa07vheYsIMKUMxI3mlm2qSIDxCA9KxVCBOdJjNjs7dE6v03L5UtoRxZ+rviQxxrSc8tp0cmaFe9Kbif14nNf3LMKMiSQ0ReL6onzLXSHeagNujimDDJpYgrKi91cVDpBA2NqeKDcFffHmZBGf1q7p/d15r3BZplOEIjuEUfLiABtxAEwLA8ATP8Apvzth5cd6dj3lrySlmDuEPnM8fb2OSHA==</latexit><latexit sha1_base64="khFMxWwj1mfcXUdBB7rNQr8Otg8=">AAAB83icbVBNS8NAEJ3Ur1q/qh69BIvgqSQiqLeCF8FLBWMLbQyb7bZduh9xd1MoIb/DiwcVr/4Zb/4bt20O2vpg4PHeDDPz4oRRbTzv2ymtrK6tb5Q3K1vbO7t71f2DBy1ThUmAJZOqHSNNGBUkMNQw0k4UQTxmpBWPrqd+a0yUplLcm0lCQo4GgvYpRsZKYfzoRxmPMhnRPI+qNa/uzeAuE78gNSjQjKpf3Z7EKSfCYIa07vheYsIMKUMxI3mlm2qSIDxCA9KxVCBOdJjNjs7dE6v03L5UtoRxZ+rviQxxrSc8tp0cmaFe9Kbif14nNf3LMKMiSQ0ReL6onzLXSHeagNujimDDJpYgrKi91cVDpBA2NqeKDcFffHmZBGf1q7p/d15r3BZplOEIjuEUfLiABtxAEwLA8ATP8Apvzth5cd6dj3lrySlmDuEPnM8fb2OSHA==</latexit>
V1
moi
,:<latexit sha1_base64="6/wzl1Pz+AiS40dOdQg4tQAE2hI=">AAAB/3icbVDLSsNAFJ34rPUVdeHCzWARXEhJRPCxKrgR3FQwbaGNYTKZtENnMmFmIpSQjb/ixoWKW3/DnX/jtM1CWw9cOJxzL/feE6aMKu0439bC4tLyymplrbq+sbm1be/stpTIJCYeFkzITogUYTQhnqaakU4qCeIhI+1weD32249EKiqSez1Kic9RP6ExxUgbKbD3e6FgUd4qHtwg50EuAlqcwKsisGtO3ZkAzhO3JDVQohnYX71I4IyTRGOGlOq6Tqr9HElNMSNFtZcpkiI8RH3SNTRBnCg/nzxQwCOjRDAW0lSi4UT9PZEjrtSIh6aTIz1Qs95Y/M/rZjq+8HOapJkmCZ4uijMGtYDjNGBEJcGajQxBWFJzK8QDJBHWJrOqCcGdfXmeeKf1y7p7d1Zr3JZpVMABOATHwAXnoAFuQBN4AIMCPINX8GY9WS/Wu/UxbV2wypk98AfW5w/RaZYg</latexit><latexit sha1_base64="6/wzl1Pz+AiS40dOdQg4tQAE2hI=">AAAB/3icbVDLSsNAFJ34rPUVdeHCzWARXEhJRPCxKrgR3FQwbaGNYTKZtENnMmFmIpSQjb/ixoWKW3/DnX/jtM1CWw9cOJxzL/feE6aMKu0439bC4tLyymplrbq+sbm1be/stpTIJCYeFkzITogUYTQhnqaakU4qCeIhI+1weD32249EKiqSez1Kic9RP6ExxUgbKbD3e6FgUd4qHtwg50EuAlqcwKsisGtO3ZkAzhO3JDVQohnYX71I4IyTRGOGlOq6Tqr9HElNMSNFtZcpkiI8RH3SNTRBnCg/nzxQwCOjRDAW0lSi4UT9PZEjrtSIh6aTIz1Qs95Y/M/rZjq+8HOapJkmCZ4uijMGtYDjNGBEJcGajQxBWFJzK8QDJBHWJrOqCcGdfXmeeKf1y7p7d1Zr3JZpVMABOATHwAXnoAFuQBN4AIMCPINX8GY9WS/Wu/UxbV2wypk98AfW5w/RaZYg</latexit><latexit sha1_base64="6/wzl1Pz+AiS40dOdQg4tQAE2hI=">AAAB/3icbVDLSsNAFJ34rPUVdeHCzWARXEhJRPCxKrgR3FQwbaGNYTKZtENnMmFmIpSQjb/ixoWKW3/DnX/jtM1CWw9cOJxzL/feE6aMKu0439bC4tLyymplrbq+sbm1be/stpTIJCYeFkzITogUYTQhnqaakU4qCeIhI+1weD32249EKiqSez1Kic9RP6ExxUgbKbD3e6FgUd4qHtwg50EuAlqcwKsisGtO3ZkAzhO3JDVQohnYX71I4IyTRGOGlOq6Tqr9HElNMSNFtZcpkiI8RH3SNTRBnCg/nzxQwCOjRDAW0lSi4UT9PZEjrtSIh6aTIz1Qs95Y/M/rZjq+8HOapJkmCZ4uijMGtYDjNGBEJcGajQxBWFJzK8QDJBHWJrOqCcGdfXmeeKf1y7p7d1Zr3JZpVMABOATHwAXnoAFuQBN4AIMCPINX8GY9WS/Wu/UxbV2wypk98AfW5w/RaZYg</latexit><latexit sha1_base64="6/wzl1Pz+AiS40dOdQg4tQAE2hI=">AAAB/3icbVDLSsNAFJ34rPUVdeHCzWARXEhJRPCxKrgR3FQwbaGNYTKZtENnMmFmIpSQjb/ixoWKW3/DnX/jtM1CWw9cOJxzL/feE6aMKu0439bC4tLyymplrbq+sbm1be/stpTIJCYeFkzITogUYTQhnqaakU4qCeIhI+1weD32249EKiqSez1Kic9RP6ExxUgbKbD3e6FgUd4qHtwg50EuAlqcwKsisGtO3ZkAzhO3JDVQohnYX71I4IyTRGOGlOq6Tqr9HElNMSNFtZcpkiI8RH3SNTRBnCg/nzxQwCOjRDAW0lSi4UT9PZEjrtSIh6aTIz1Qs95Y/M/rZjq+8HOapJkmCZ4uijMGtYDjNGBEJcGajQxBWFJzK8QDJBHWJrOqCcGdfXmeeKf1y7p7d1Zr3JZpVMABOATHwAXnoAFuQBN4AIMCPINX8GY9WS/Wu/UxbV2wypk98AfW5w/RaZYg</latexit>
b1
moi
+ V1
moi
,:h
⇣
rmo<i
⌘
<latexit sha1_base64="LHhK5VPbhS1xg2EMwUUBfHHGwMc=">AAACP3icbVBLSwMxGMzWV62vVY9egkVQlLIrgg88FLwIXirYB7R1yabZNjS7WZJvhbLsT/PiT/Dm2YsHFa/eTNs9+BoITGa+4UvGjwXX4DhPVmFmdm5+obhYWlpeWV2z1zcaWiaKsjqVQqqWTzQTPGJ14CBYK1aMhL5gTX94Mfabd0xpLqMbGMWsG5J+xANOCRjJs5v+reuloZdKj2cZ3scdX4pe2si+yQf4LMv1gSGCBbCb31WWT6Xn43iGcUfx/gD2PLvsVJwJ8F/i5qSMctQ8+7HTkzQJWQRUEK3brhNDNyUKOBUsK3USzWJCh6TP2oZGJGS6m04KyPCOUXo4kMqcCPBE/Z5ISaj1KPTNZEhgoH97Y/E/r51AcNJNeRQnwCI6XRQkAoPE4zZxjytGQYwMIVRx81ZMB0QRCqbzkinB/f3lv6R+WDmtuNdH5epV3kYRbaFttItcdIyq6BLVUB1RdI+e0St6sx6sF+vd+piOFqw8s4l+wPr8AsyNsCo=</latexit><latexit sha1_base64="LHhK5VPbhS1xg2EMwUUBfHHGwMc=">AAACP3icbVBLSwMxGMzWV62vVY9egkVQlLIrgg88FLwIXirYB7R1yabZNjS7WZJvhbLsT/PiT/Dm2YsHFa/eTNs9+BoITGa+4UvGjwXX4DhPVmFmdm5+obhYWlpeWV2z1zcaWiaKsjqVQqqWTzQTPGJ14CBYK1aMhL5gTX94Mfabd0xpLqMbGMWsG5J+xANOCRjJs5v+reuloZdKj2cZ3scdX4pe2si+yQf4LMv1gSGCBbCb31WWT6Xn43iGcUfx/gD2PLvsVJwJ8F/i5qSMctQ8+7HTkzQJWQRUEK3brhNDNyUKOBUsK3USzWJCh6TP2oZGJGS6m04KyPCOUXo4kMqcCPBE/Z5ISaj1KPTNZEhgoH97Y/E/r51AcNJNeRQnwCI6XRQkAoPE4zZxjytGQYwMIVRx81ZMB0QRCqbzkinB/f3lv6R+WDmtuNdH5epV3kYRbaFttItcdIyq6BLVUB1RdI+e0St6sx6sF+vd+piOFqw8s4l+wPr8AsyNsCo=</latexit><latexit sha1_base64="LHhK5VPbhS1xg2EMwUUBfHHGwMc=">AAACP3icbVBLSwMxGMzWV62vVY9egkVQlLIrgg88FLwIXirYB7R1yabZNjS7WZJvhbLsT/PiT/Dm2YsHFa/eTNs9+BoITGa+4UvGjwXX4DhPVmFmdm5+obhYWlpeWV2z1zcaWiaKsjqVQqqWTzQTPGJ14CBYK1aMhL5gTX94Mfabd0xpLqMbGMWsG5J+xANOCRjJs5v+reuloZdKj2cZ3scdX4pe2si+yQf4LMv1gSGCBbCb31WWT6Xn43iGcUfx/gD2PLvsVJwJ8F/i5qSMctQ8+7HTkzQJWQRUEK3brhNDNyUKOBUsK3USzWJCh6TP2oZGJGS6m04KyPCOUXo4kMqcCPBE/Z5ISaj1KPTNZEhgoH97Y/E/r51AcNJNeRQnwCI6XRQkAoPE4zZxjytGQYwMIVRx81ZMB0QRCqbzkinB/f3lv6R+WDmtuNdH5epV3kYRbaFttItcdIyq6BLVUB1RdI+e0St6sx6sF+vd+piOFqw8s4l+wPr8AsyNsCo=</latexit><latexit sha1_base64="LHhK5VPbhS1xg2EMwUUBfHHGwMc=">AAACP3icbVBLSwMxGMzWV62vVY9egkVQlLIrgg88FLwIXirYB7R1yabZNjS7WZJvhbLsT/PiT/Dm2YsHFa/eTNs9+BoITGa+4UvGjwXX4DhPVmFmdm5+obhYWlpeWV2z1zcaWiaKsjqVQqqWTzQTPGJ14CBYK1aMhL5gTX94Mfabd0xpLqMbGMWsG5J+xANOCRjJs5v+reuloZdKj2cZ3scdX4pe2si+yQf4LMv1gSGCBbCb31WWT6Xn43iGcUfx/gD2PLvsVJwJ8F/i5qSMctQ8+7HTkzQJWQRUEK3brhNDNyUKOBUsK3USzWJCh6TP2oZGJGS6m04KyPCOUXo4kMqcCPBE/Z5ISaj1KPTNZEhgoH97Y/E/r51AcNJNeRQnwCI6XRQkAoPE4zZxjytGQYwMIVRx81ZMB0QRCqbzkinB/f3lv6R+WDmtuNdH5epV3kYRbaFttItcdIyq6BLVUB1RdI+e0St6sx6sF+vd+piOFqw8s4l+wPr8AsyNsCo=</latexit>
… …
s1
moi
⇣
rmo<i
⌘
<latexit sha1_base64="dEuQ73sOOGlAJordFhBO/PRTXnQ=">AAACHHicbVBNS8NAEN3Ur1q/qh69LBahXkoiBRU8FLwIXipYW2hq2Gw37dJNNuxOhBLyR7z4V7x4UPHiQfDfuG1z0NYHA4/3ZpiZ58eCa7Dtb6uwtLyyulZcL21sbm3vlHf37rRMFGUtKoVUHZ9oJnjEWsBBsE6sGAl9wdr+6HLitx+Y0lxGtzCOWS8kg4gHnBIwkleu63vHS0MvlR7PMuwKFkAVu74U/VRluZNeTLwMY1fxwRCOvXLFrtlT4EXi5KSCcjS98qfblzQJWQRUEK27jh1DLyUKOBUsK7mJZjGhIzJgXUMjEjLdS6ffZfjIKH0cSGUqAjxVf0+kJNR6HPqmMyQw1PPeRPzP6yYQnPVSHsUJsIjOFgWJwCDxJCrc54pREGNDCFXc3IrpkChCwQRaMiE48y8vktZJ7bzm3NQrjes8jSI6QIeoihx0ihroCjVRC1H0iJ7RK3qznqwX6936mLUWrHxmH/2B9fUDSeiiQw==</latexit><latexit sha1_base64="dEuQ73sOOGlAJordFhBO/PRTXnQ=">AAACHHicbVBNS8NAEN3Ur1q/qh69LBahXkoiBRU8FLwIXipYW2hq2Gw37dJNNuxOhBLyR7z4V7x4UPHiQfDfuG1z0NYHA4/3ZpiZ58eCa7Dtb6uwtLyyulZcL21sbm3vlHf37rRMFGUtKoVUHZ9oJnjEWsBBsE6sGAl9wdr+6HLitx+Y0lxGtzCOWS8kg4gHnBIwkleu63vHS0MvlR7PMuwKFkAVu74U/VRluZNeTLwMY1fxwRCOvXLFrtlT4EXi5KSCcjS98qfblzQJWQRUEK27jh1DLyUKOBUsK7mJZjGhIzJgXUMjEjLdS6ffZfjIKH0cSGUqAjxVf0+kJNR6HPqmMyQw1PPeRPzP6yYQnPVSHsUJsIjOFgWJwCDxJCrc54pREGNDCFXc3IrpkChCwQRaMiE48y8vktZJ7bzm3NQrjes8jSI6QIeoihx0ihroCjVRC1H0iJ7RK3qznqwX6936mLUWrHxmH/2B9fUDSeiiQw==</latexit><latexit sha1_base64="dEuQ73sOOGlAJordFhBO/PRTXnQ=">AAACHHicbVBNS8NAEN3Ur1q/qh69LBahXkoiBRU8FLwIXipYW2hq2Gw37dJNNuxOhBLyR7z4V7x4UPHiQfDfuG1z0NYHA4/3ZpiZ58eCa7Dtb6uwtLyyulZcL21sbm3vlHf37rRMFGUtKoVUHZ9oJnjEWsBBsE6sGAl9wdr+6HLitx+Y0lxGtzCOWS8kg4gHnBIwkleu63vHS0MvlR7PMuwKFkAVu74U/VRluZNeTLwMY1fxwRCOvXLFrtlT4EXi5KSCcjS98qfblzQJWQRUEK27jh1DLyUKOBUsK7mJZjGhIzJgXUMjEjLdS6ffZfjIKH0cSGUqAjxVf0+kJNR6HPqmMyQw1PPeRPzP6yYQnPVSHsUJsIjOFgWJwCDxJCrc54pREGNDCFXc3IrpkChCwQRaMiE48y8vktZJ7bzm3NQrjes8jSI6QIeoihx0ihroCjVRC1H0iJ7RK3qznqwX6936mLUWrHxmH/2B9fUDSeiiQw==</latexit><latexit sha1_base64="dEuQ73sOOGlAJordFhBO/PRTXnQ=">AAACHHicbVBNS8NAEN3Ur1q/qh69LBahXkoiBRU8FLwIXipYW2hq2Gw37dJNNuxOhBLyR7z4V7x4UPHiQfDfuG1z0NYHA4/3ZpiZ58eCa7Dtb6uwtLyyulZcL21sbm3vlHf37rRMFGUtKoVUHZ9oJnjEWsBBsE6sGAl9wdr+6HLitx+Y0lxGtzCOWS8kg4gHnBIwkleu63vHS0MvlR7PMuwKFkAVu74U/VRluZNeTLwMY1fxwRCOvXLFrtlT4EXi5KSCcjS98qfblzQJWQRUEK27jh1DLyUKOBUsK7mJZjGhIzJgXUMjEjLdS6ffZfjIKH0cSGUqAjxVf0+kJNR6HPqmMyQw1PPeRPzP6yYQnPVSHsUJsIjOFgWJwCDxJCrc54pREGNDCFXc3IrpkChCwQRaMiE48y8vktZJ7bzm3NQrjes8jSI6QIeoihx0ihroCjVRC1H0iJ7RK3qznqwX6936mLUWrHxmH/2B9fUDSeiiQw==</latexit>
sk
moi
⇣
rmo<i
⌘
<latexit sha1_base64="8yFvGakMoAgUOhWppwqPpdRpbLM=">AAACHHicbVBNS8NAEN3Ur1q/qh69LBahXkoiBRU8FLwIXipYW2hq2Gw37dJNNuxOhBLyR7z4V7x4UPHiQfDfuG1z0NYHA4/3ZpiZ58eCa7Dtb6uwtLyyulZcL21sbm3vlHf37rRMFGUtKoVUHZ9oJnjEWsBBsE6sGAl9wdr+6HLitx+Y0lxGtzCOWS8kg4gHnBIwkleu6/uRl4ZeKj2eZdgVLIAqdn0p+qnKcie9mHgZxq7igyEce+WKXbOnwIvEyUkF5Wh65U+3L2kSsgioIFp3HTuGXkoUcCpYVnITzWJCR2TAuoZGJGS6l06/y/CRUfo4kMpUBHiq/p5ISaj1OPRNZ0hgqOe9ifif100gOOulPIoTYBGdLQoSgUHiSVS4zxWjIMaGEKq4uRXTIVGEggm0ZEJw5l9eJK2T2nnNualXGtd5GkV0gA5RFTnoFDXQFWqiFqLoET2jV/RmPVkv1rv1MWstWPnMPvoD6+sHrLCifQ==</latexit><latexit sha1_base64="8yFvGakMoAgUOhWppwqPpdRpbLM=">AAACHHicbVBNS8NAEN3Ur1q/qh69LBahXkoiBRU8FLwIXipYW2hq2Gw37dJNNuxOhBLyR7z4V7x4UPHiQfDfuG1z0NYHA4/3ZpiZ58eCa7Dtb6uwtLyyulZcL21sbm3vlHf37rRMFGUtKoVUHZ9oJnjEWsBBsE6sGAl9wdr+6HLitx+Y0lxGtzCOWS8kg4gHnBIwkleu6/uRl4ZeKj2eZdgVLIAqdn0p+qnKcie9mHgZxq7igyEce+WKXbOnwIvEyUkF5Wh65U+3L2kSsgioIFp3HTuGXkoUcCpYVnITzWJCR2TAuoZGJGS6l06/y/CRUfo4kMpUBHiq/p5ISaj1OPRNZ0hgqOe9ifif100gOOulPIoTYBGdLQoSgUHiSVS4zxWjIMaGEKq4uRXTIVGEggm0ZEJw5l9eJK2T2nnNualXGtd5GkV0gA5RFTnoFDXQFWqiFqLoET2jV/RmPVkv1rv1MWstWPnMPvoD6+sHrLCifQ==</latexit><latexit sha1_base64="8yFvGakMoAgUOhWppwqPpdRpbLM=">AAACHHicbVBNS8NAEN3Ur1q/qh69LBahXkoiBRU8FLwIXipYW2hq2Gw37dJNNuxOhBLyR7z4V7x4UPHiQfDfuG1z0NYHA4/3ZpiZ58eCa7Dtb6uwtLyyulZcL21sbm3vlHf37rRMFGUtKoVUHZ9oJnjEWsBBsE6sGAl9wdr+6HLitx+Y0lxGtzCOWS8kg4gHnBIwkleu6/uRl4ZeKj2eZdgVLIAqdn0p+qnKcie9mHgZxq7igyEce+WKXbOnwIvEyUkF5Wh65U+3L2kSsgioIFp3HTuGXkoUcCpYVnITzWJCR2TAuoZGJGS6l06/y/CRUfo4kMpUBHiq/p5ISaj1OPRNZ0hgqOe9ifif100gOOulPIoTYBGdLQoSgUHiSVS4zxWjIMaGEKq4uRXTIVGEggm0ZEJw5l9eJK2T2nnNualXGtd5GkV0gA5RFTnoFDXQFWqiFqLoET2jV/RmPVkv1rv1MWstWPnMPvoD6+sHrLCifQ==</latexit><latexit sha1_base64="8yFvGakMoAgUOhWppwqPpdRpbLM=">AAACHHicbVBNS8NAEN3Ur1q/qh69LBahXkoiBRU8FLwIXipYW2hq2Gw37dJNNuxOhBLyR7z4V7x4UPHiQfDfuG1z0NYHA4/3ZpiZ58eCa7Dtb6uwtLyyulZcL21sbm3vlHf37rRMFGUtKoVUHZ9oJnjEWsBBsE6sGAl9wdr+6HLitx+Y0lxGtzCOWS8kg4gHnBIwkleu6/uRl4ZeKj2eZdgVLIAqdn0p+qnKcie9mHgZxq7igyEce+WKXbOnwIvEyUkF5Wh65U+3L2kSsgioIFp3HTuGXkoUcCpYVnITzWJCR2TAuoZGJGS6l06/y/CRUfo4kMpUBHiq/p5ISaj1OPRNZ0hgqOe9ifif100gOOulPIoTYBGdLQoSgUHiSVS4zxWjIMaGEKq4uRXTIVGEggm0ZEJw5l9eJK2T2nnNualXGtd5GkV0gA5RFTnoFDXQFWqiFqLoET2jV/RmPVkv1rv1MWstWPnMPvoD6+sHrLCifQ==</latexit>
sK
moi
⇣
rmo<i
⌘
<latexit sha1_base64="TEHfIFWYNFymBdy4hOMMgNi9FCk=">AAACHHicbVBNS8NAEN34WetX1aOXxSLUS0mkoIKHghehlwrWFpoaNttNu3STDbsToYT8ES/+FS8eVLx4EPw3btsctPXBwOO9GWbm+bHgGmz721paXlldWy9sFDe3tnd2S3v7d1omirIWlUKqjk80EzxiLeAgWCdWjIS+YG1/dDXx2w9MaS6jWxjHrBeSQcQDTgkYySvV9H3DS0MvlR7PMuwKFkAFu74U/VRluZNeTrwMY1fxwRBOvFLZrtpT4EXi5KSMcjS90qfblzQJWQRUEK27jh1DLyUKOBUsK7qJZjGhIzJgXUMjEjLdS6ffZfjYKH0cSGUqAjxVf0+kJNR6HPqmMyQw1PPeRPzP6yYQnPdSHsUJsIjOFgWJwCDxJCrc54pREGNDCFXc3IrpkChCwQRaNCE48y8vktZp9aLq3NTK9UaeRgEdoiNUQQ46Q3V0jZqohSh6RM/oFb1ZT9aL9W59zFqXrHzmAP2B9fUDdjCiXQ==</latexit><latexit sha1_base64="TEHfIFWYNFymBdy4hOMMgNi9FCk=">AAACHHicbVBNS8NAEN34WetX1aOXxSLUS0mkoIKHghehlwrWFpoaNttNu3STDbsToYT8ES/+FS8eVLx4EPw3btsctPXBwOO9GWbm+bHgGmz721paXlldWy9sFDe3tnd2S3v7d1omirIWlUKqjk80EzxiLeAgWCdWjIS+YG1/dDXx2w9MaS6jWxjHrBeSQcQDTgkYySvV9H3DS0MvlR7PMuwKFkAFu74U/VRluZNeTrwMY1fxwRBOvFLZrtpT4EXi5KSMcjS90qfblzQJWQRUEK27jh1DLyUKOBUsK7qJZjGhIzJgXUMjEjLdS6ffZfjYKH0cSGUqAjxVf0+kJNR6HPqmMyQw1PPeRPzP6yYQnPdSHsUJsIjOFgWJwCDxJCrc54pREGNDCFXc3IrpkChCwQRaNCE48y8vktZp9aLq3NTK9UaeRgEdoiNUQQ46Q3V0jZqohSh6RM/oFb1ZT9aL9W59zFqXrHzmAP2B9fUDdjCiXQ==</latexit><latexit sha1_base64="TEHfIFWYNFymBdy4hOMMgNi9FCk=">AAACHHicbVBNS8NAEN34WetX1aOXxSLUS0mkoIKHghehlwrWFpoaNttNu3STDbsToYT8ES/+FS8eVLx4EPw3btsctPXBwOO9GWbm+bHgGmz721paXlldWy9sFDe3tnd2S3v7d1omirIWlUKqjk80EzxiLeAgWCdWjIS+YG1/dDXx2w9MaS6jWxjHrBeSQcQDTgkYySvV9H3DS0MvlR7PMuwKFkAFu74U/VRluZNeTrwMY1fxwRBOvFLZrtpT4EXi5KSMcjS90qfblzQJWQRUEK27jh1DLyUKOBUsK7qJZjGhIzJgXUMjEjLdS6ffZfjYKH0cSGUqAjxVf0+kJNR6HPqmMyQw1PPeRPzP6yYQnPdSHsUJsIjOFgWJwCDxJCrc54pREGNDCFXc3IrpkChCwQRaNCE48y8vktZp9aLq3NTK9UaeRgEdoiNUQQ46Q3V0jZqohSh6RM/oFb1ZT9aL9W59zFqXrHzmAP2B9fUDdjCiXQ==</latexit><latexit sha1_base64="TEHfIFWYNFymBdy4hOMMgNi9FCk=">AAACHHicbVBNS8NAEN34WetX1aOXxSLUS0mkoIKHghehlwrWFpoaNttNu3STDbsToYT8ES/+FS8eVLx4EPw3btsctPXBwOO9GWbm+bHgGmz721paXlldWy9sFDe3tnd2S3v7d1omirIWlUKqjk80EzxiLeAgWCdWjIS+YG1/dDXx2w9MaS6jWxjHrBeSQcQDTgkYySvV9H3DS0MvlR7PMuwKFkAFu74U/VRluZNeTrwMY1fxwRBOvFLZrtpT4EXi5KSMcjS90qfblzQJWQRUEK27jh1DLyUKOBUsK7qJZjGhIzJgXUMjEjLdS6ffZfjYKH0cSGUqAjxVf0+kJNR6HPqmMyQw1PPeRPzP6yYQnPdSHsUJsIjOFgWJwCDxJCrc54pREGNDCFXc3IrpkChCwQRaNCE48y8vktZp9aLq3NTK9UaeRgEdoiNUQQ46Q3V0jZqohSh6RM/oFb1ZT9aL9W59zFqXrHzmAP2B9fUDdjCiXQ==</latexit>

16. 16
FIGURES OF THE MODEL
V1 b1
b1
moi<latexit sha1_base64="khFMxWwj1mfcXUdBB7rNQr8Otg8=">AAAB83icbVBNS8NAEJ3Ur1q/qh69BIvgqSQiqLeCF8FLBWMLbQyb7bZduh9xd1MoIb/DiwcVr/4Zb/4bt20O2vpg4PHeDDPz4oRRbTzv2ymtrK6tb5Q3K1vbO7t71f2DBy1ThUmAJZOqHSNNGBUkMNQw0k4UQTxmpBWPrqd+a0yUplLcm0lCQo4GgvYpRsZKYfzoRxmPMhnRPI+qNa/uzeAuE78gNSjQjKpf3Z7EKSfCYIa07vheYsIMKUMxI3mlm2qSIDxCA9KxVCBOdJjNjs7dE6v03L5UtoRxZ+rviQxxrSc8tp0cmaFe9Kbif14nNf3LMKMiSQ0ReL6onzLXSHeagNujimDDJpYgrKi91cVDpBA2NqeKDcFffHmZBGf1q7p/d15r3BZplOEIjuEUfLiABtxAEwLA8ATP8Apvzth5cd6dj3lrySlmDuEPnM8fb2OSHA==</latexit><latexit sha1_base64="khFMxWwj1mfcXUdBB7rNQr8Otg8=">AAAB83icbVBNS8NAEJ3Ur1q/qh69BIvgqSQiqLeCF8FLBWMLbQyb7bZduh9xd1MoIb/DiwcVr/4Zb/4bt20O2vpg4PHeDDPz4oRRbTzv2ymtrK6tb5Q3K1vbO7t71f2DBy1ThUmAJZOqHSNNGBUkMNQw0k4UQTxmpBWPrqd+a0yUplLcm0lCQo4GgvYpRsZKYfzoRxmPMhnRPI+qNa/uzeAuE78gNSjQjKpf3Z7EKSfCYIa07vheYsIMKUMxI3mlm2qSIDxCA9KxVCBOdJjNjs7dE6v03L5UtoRxZ+rviQxxrSc8tp0cmaFe9Kbif14nNf3LMKMiSQ0ReL6onzLXSHeagNujimDDJpYgrKi91cVDpBA2NqeKDcFffHmZBGf1q7p/d15r3BZplOEIjuEUfLiABtxAEwLA8ATP8Apvzth5cd6dj3lrySlmDuEPnM8fb2OSHA==</latexit><latexit sha1_base64="khFMxWwj1mfcXUdBB7rNQr8Otg8=">AAAB83icbVBNS8NAEJ3Ur1q/qh69BIvgqSQiqLeCF8FLBWMLbQyb7bZduh9xd1MoIb/DiwcVr/4Zb/4bt20O2vpg4PHeDDPz4oRRbTzv2ymtrK6tb5Q3K1vbO7t71f2DBy1ThUmAJZOqHSNNGBUkMNQw0k4UQTxmpBWPrqd+a0yUplLcm0lCQo4GgvYpRsZKYfzoRxmPMhnRPI+qNa/uzeAuE78gNSjQjKpf3Z7EKSfCYIa07vheYsIMKUMxI3mlm2qSIDxCA9KxVCBOdJjNjs7dE6v03L5UtoRxZ+rviQxxrSc8tp0cmaFe9Kbif14nNf3LMKMiSQ0ReL6onzLXSHeagNujimDDJpYgrKi91cVDpBA2NqeKDcFffHmZBGf1q7p/d15r3BZplOEIjuEUfLiABtxAEwLA8ATP8Apvzth5cd6dj3lrySlmDuEPnM8fb2OSHA==</latexit><latexit sha1_base64="khFMxWwj1mfcXUdBB7rNQr8Otg8=">AAAB83icbVBNS8NAEJ3Ur1q/qh69BIvgqSQiqLeCF8FLBWMLbQyb7bZduh9xd1MoIb/DiwcVr/4Zb/4bt20O2vpg4PHeDDPz4oRRbTzv2ymtrK6tb5Q3K1vbO7t71f2DBy1ThUmAJZOqHSNNGBUkMNQw0k4UQTxmpBWPrqd+a0yUplLcm0lCQo4GgvYpRsZKYfzoRxmPMhnRPI+qNa/uzeAuE78gNSjQjKpf3Z7EKSfCYIa07vheYsIMKUMxI3mlm2qSIDxCA9KxVCBOdJjNjs7dE6v03L5UtoRxZ+rviQxxrSc8tp0cmaFe9Kbif14nNf3LMKMiSQ0ReL6onzLXSHeagNujimDDJpYgrKi91cVDpBA2NqeKDcFffHmZBGf1q7p/d15r3BZplOEIjuEUfLiABtxAEwLA8ATP8Apvzth5cd6dj3lrySlmDuEPnM8fb2OSHA==</latexit>
V1
moi
,:<latexit sha1_base64="6/wzl1Pz+AiS40dOdQg4tQAE2hI=">AAAB/3icbVDLSsNAFJ34rPUVdeHCzWARXEhJRPCxKrgR3FQwbaGNYTKZtENnMmFmIpSQjb/ixoWKW3/DnX/jtM1CWw9cOJxzL/feE6aMKu0439bC4tLyymplrbq+sbm1be/stpTIJCYeFkzITogUYTQhnqaakU4qCeIhI+1weD32249EKiqSez1Kic9RP6ExxUgbKbD3e6FgUd4qHtwg50EuAlqcwKsisGtO3ZkAzhO3JDVQohnYX71I4IyTRGOGlOq6Tqr9HElNMSNFtZcpkiI8RH3SNTRBnCg/nzxQwCOjRDAW0lSi4UT9PZEjrtSIh6aTIz1Qs95Y/M/rZjq+8HOapJkmCZ4uijMGtYDjNGBEJcGajQxBWFJzK8QDJBHWJrOqCcGdfXmeeKf1y7p7d1Zr3JZpVMABOATHwAXnoAFuQBN4AIMCPINX8GY9WS/Wu/UxbV2wypk98AfW5w/RaZYg</latexit><latexit sha1_base64="6/wzl1Pz+AiS40dOdQg4tQAE2hI=">AAAB/3icbVDLSsNAFJ34rPUVdeHCzWARXEhJRPCxKrgR3FQwbaGNYTKZtENnMmFmIpSQjb/ixoWKW3/DnX/jtM1CWw9cOJxzL/feE6aMKu0439bC4tLyymplrbq+sbm1be/stpTIJCYeFkzITogUYTQhnqaakU4qCeIhI+1weD32249EKiqSez1Kic9RP6ExxUgbKbD3e6FgUd4qHtwg50EuAlqcwKsisGtO3ZkAzhO3JDVQohnYX71I4IyTRGOGlOq6Tqr9HElNMSNFtZcpkiI8RH3SNTRBnCg/nzxQwCOjRDAW0lSi4UT9PZEjrtSIh6aTIz1Qs95Y/M/rZjq+8HOapJkmCZ4uijMGtYDjNGBEJcGajQxBWFJzK8QDJBHWJrOqCcGdfXmeeKf1y7p7d1Zr3JZpVMABOATHwAXnoAFuQBN4AIMCPINX8GY9WS/Wu/UxbV2wypk98AfW5w/RaZYg</latexit><latexit sha1_base64="6/wzl1Pz+AiS40dOdQg4tQAE2hI=">AAAB/3icbVDLSsNAFJ34rPUVdeHCzWARXEhJRPCxKrgR3FQwbaGNYTKZtENnMmFmIpSQjb/ixoWKW3/DnX/jtM1CWw9cOJxzL/feE6aMKu0439bC4tLyymplrbq+sbm1be/stpTIJCYeFkzITogUYTQhnqaakU4qCeIhI+1weD32249EKiqSez1Kic9RP6ExxUgbKbD3e6FgUd4qHtwg50EuAlqcwKsisGtO3ZkAzhO3JDVQohnYX71I4IyTRGOGlOq6Tqr9HElNMSNFtZcpkiI8RH3SNTRBnCg/nzxQwCOjRDAW0lSi4UT9PZEjrtSIh6aTIz1Qs95Y/M/rZjq+8HOapJkmCZ4uijMGtYDjNGBEJcGajQxBWFJzK8QDJBHWJrOqCcGdfXmeeKf1y7p7d1Zr3JZpVMABOATHwAXnoAFuQBN4AIMCPINX8GY9WS/Wu/UxbV2wypk98AfW5w/RaZYg</latexit><latexit sha1_base64="6/wzl1Pz+AiS40dOdQg4tQAE2hI=">AAAB/3icbVDLSsNAFJ34rPUVdeHCzWARXEhJRPCxKrgR3FQwbaGNYTKZtENnMmFmIpSQjb/ixoWKW3/DnX/jtM1CWw9cOJxzL/feE6aMKu0439bC4tLyymplrbq+sbm1be/stpTIJCYeFkzITogUYTQhnqaakU4qCeIhI+1weD32249EKiqSez1Kic9RP6ExxUgbKbD3e6FgUd4qHtwg50EuAlqcwKsisGtO3ZkAzhO3JDVQohnYX71I4IyTRGOGlOq6Tqr9HElNMSNFtZcpkiI8RH3SNTRBnCg/nzxQwCOjRDAW0lSi4UT9PZEjrtSIh6aTIz1Qs95Y/M/rZjq+8HOapJkmCZ4uijMGtYDjNGBEJcGajQxBWFJzK8QDJBHWJrOqCcGdfXmeeKf1y7p7d1Zr3JZpVMABOATHwAXnoAFuQBN4AIMCPINX8GY9WS/Wu/UxbV2wypk98AfW5w/RaZYg</latexit>
b1
moi
+ V1
moi
,:h
⇣
rmo<i
⌘
<latexit sha1_base64="LHhK5VPbhS1xg2EMwUUBfHHGwMc=">AAACP3icbVBLSwMxGMzWV62vVY9egkVQlLIrgg88FLwIXirYB7R1yabZNjS7WZJvhbLsT/PiT/Dm2YsHFa/eTNs9+BoITGa+4UvGjwXX4DhPVmFmdm5+obhYWlpeWV2z1zcaWiaKsjqVQqqWTzQTPGJ14CBYK1aMhL5gTX94Mfabd0xpLqMbGMWsG5J+xANOCRjJs5v+reuloZdKj2cZ3scdX4pe2si+yQf4LMv1gSGCBbCb31WWT6Xn43iGcUfx/gD2PLvsVJwJ8F/i5qSMctQ8+7HTkzQJWQRUEK3brhNDNyUKOBUsK3USzWJCh6TP2oZGJGS6m04KyPCOUXo4kMqcCPBE/Z5ISaj1KPTNZEhgoH97Y/E/r51AcNJNeRQnwCI6XRQkAoPE4zZxjytGQYwMIVRx81ZMB0QRCqbzkinB/f3lv6R+WDmtuNdH5epV3kYRbaFttItcdIyq6BLVUB1RdI+e0St6sx6sF+vd+piOFqw8s4l+wPr8AsyNsCo=</latexit><latexit sha1_base64="LHhK5VPbhS1xg2EMwUUBfHHGwMc=">AAACP3icbVBLSwMxGMzWV62vVY9egkVQlLIrgg88FLwIXirYB7R1yabZNjS7WZJvhbLsT/PiT/Dm2YsHFa/eTNs9+BoITGa+4UvGjwXX4DhPVmFmdm5+obhYWlpeWV2z1zcaWiaKsjqVQqqWTzQTPGJ14CBYK1aMhL5gTX94Mfabd0xpLqMbGMWsG5J+xANOCRjJs5v+reuloZdKj2cZ3scdX4pe2si+yQf4LMv1gSGCBbCb31WWT6Xn43iGcUfx/gD2PLvsVJwJ8F/i5qSMctQ8+7HTkzQJWQRUEK3brhNDNyUKOBUsK3USzWJCh6TP2oZGJGS6m04KyPCOUXo4kMqcCPBE/Z5ISaj1KPTNZEhgoH97Y/E/r51AcNJNeRQnwCI6XRQkAoPE4zZxjytGQYwMIVRx81ZMB0QRCqbzkinB/f3lv6R+WDmtuNdH5epV3kYRbaFttItcdIyq6BLVUB1RdI+e0St6sx6sF+vd+piOFqw8s4l+wPr8AsyNsCo=</latexit><latexit sha1_base64="LHhK5VPbhS1xg2EMwUUBfHHGwMc=">AAACP3icbVBLSwMxGMzWV62vVY9egkVQlLIrgg88FLwIXirYB7R1yabZNjS7WZJvhbLsT/PiT/Dm2YsHFa/eTNs9+BoITGa+4UvGjwXX4DhPVmFmdm5+obhYWlpeWV2z1zcaWiaKsjqVQqqWTzQTPGJ14CBYK1aMhL5gTX94Mfabd0xpLqMbGMWsG5J+xANOCRjJs5v+reuloZdKj2cZ3scdX4pe2si+yQf4LMv1gSGCBbCb31WWT6Xn43iGcUfx/gD2PLvsVJwJ8F/i5qSMctQ8+7HTkzQJWQRUEK3brhNDNyUKOBUsK3USzWJCh6TP2oZGJGS6m04KyPCOUXo4kMqcCPBE/Z5ISaj1KPTNZEhgoH97Y/E/r51AcNJNeRQnwCI6XRQkAoPE4zZxjytGQYwMIVRx81ZMB0QRCqbzkinB/f3lv6R+WDmtuNdH5epV3kYRbaFttItcdIyq6BLVUB1RdI+e0St6sx6sF+vd+piOFqw8s4l+wPr8AsyNsCo=</latexit><latexit sha1_base64="LHhK5VPbhS1xg2EMwUUBfHHGwMc=">AAACP3icbVBLSwMxGMzWV62vVY9egkVQlLIrgg88FLwIXirYB7R1yabZNjS7WZJvhbLsT/PiT/Dm2YsHFa/eTNs9+BoITGa+4UvGjwXX4DhPVmFmdm5+obhYWlpeWV2z1zcaWiaKsjqVQqqWTzQTPGJ14CBYK1aMhL5gTX94Mfabd0xpLqMbGMWsG5J+xANOCRjJs5v+reuloZdKj2cZ3scdX4pe2si+yQf4LMv1gSGCBbCb31WWT6Xn43iGcUfx/gD2PLvsVJwJ8F/i5qSMctQ8+7HTkzQJWQRUEK3brhNDNyUKOBUsK3USzWJCh6TP2oZGJGS6m04KyPCOUXo4kMqcCPBE/Z5ISaj1KPTNZEhgoH97Y/E/r51AcNJNeRQnwCI6XRQkAoPE4zZxjytGQYwMIVRx81ZMB0QRCqbzkinB/f3lv6R+WDmtuNdH5epV3kYRbaFttItcdIyq6BLVUB1RdI+e0St6sx6sF+vd+piOFqw8s4l+wPr8AsyNsCo=</latexit>
… …
s1
moi
⇣
rmo<i
⌘
<latexit sha1_base64="dEuQ73sOOGlAJordFhBO/PRTXnQ=">AAACHHicbVBNS8NAEN3Ur1q/qh69LBahXkoiBRU8FLwIXipYW2hq2Gw37dJNNuxOhBLyR7z4V7x4UPHiQfDfuG1z0NYHA4/3ZpiZ58eCa7Dtb6uwtLyyulZcL21sbm3vlHf37rRMFGUtKoVUHZ9oJnjEWsBBsE6sGAl9wdr+6HLitx+Y0lxGtzCOWS8kg4gHnBIwkleu63vHS0MvlR7PMuwKFkAVu74U/VRluZNeTLwMY1fxwRCOvXLFrtlT4EXi5KSCcjS98qfblzQJWQRUEK27jh1DLyUKOBUsK7mJZjGhIzJgXUMjEjLdS6ffZfjIKH0cSGUqAjxVf0+kJNR6HPqmMyQw1PPeRPzP6yYQnPVSHsUJsIjOFgWJwCDxJCrc54pREGNDCFXc3IrpkChCwQRaMiE48y8vktZJ7bzm3NQrjes8jSI6QIeoihx0ihroCjVRC1H0iJ7RK3qznqwX6936mLUWrHxmH/2B9fUDSeiiQw==</latexit><latexit sha1_base64="dEuQ73sOOGlAJordFhBO/PRTXnQ=">AAACHHicbVBNS8NAEN3Ur1q/qh69LBahXkoiBRU8FLwIXipYW2hq2Gw37dJNNuxOhBLyR7z4V7x4UPHiQfDfuG1z0NYHA4/3ZpiZ58eCa7Dtb6uwtLyyulZcL21sbm3vlHf37rRMFGUtKoVUHZ9oJnjEWsBBsE6sGAl9wdr+6HLitx+Y0lxGtzCOWS8kg4gHnBIwkleu63vHS0MvlR7PMuwKFkAVu74U/VRluZNeTLwMY1fxwRCOvXLFrtlT4EXi5KSCcjS98qfblzQJWQRUEK27jh1DLyUKOBUsK7mJZjGhIzJgXUMjEjLdS6ffZfjIKH0cSGUqAjxVf0+kJNR6HPqmMyQw1PPeRPzP6yYQnPVSHsUJsIjOFgWJwCDxJCrc54pREGNDCFXc3IrpkChCwQRaMiE48y8vktZJ7bzm3NQrjes8jSI6QIeoihx0ihroCjVRC1H0iJ7RK3qznqwX6936mLUWrHxmH/2B9fUDSeiiQw==</latexit><latexit sha1_base64="dEuQ73sOOGlAJordFhBO/PRTXnQ=">AAACHHicbVBNS8NAEN3Ur1q/qh69LBahXkoiBRU8FLwIXipYW2hq2Gw37dJNNuxOhBLyR7z4V7x4UPHiQfDfuG1z0NYHA4/3ZpiZ58eCa7Dtb6uwtLyyulZcL21sbm3vlHf37rRMFGUtKoVUHZ9oJnjEWsBBsE6sGAl9wdr+6HLitx+Y0lxGtzCOWS8kg4gHnBIwkleu63vHS0MvlR7PMuwKFkAVu74U/VRluZNeTLwMY1fxwRCOvXLFrtlT4EXi5KSCcjS98qfblzQJWQRUEK27jh1DLyUKOBUsK7mJZjGhIzJgXUMjEjLdS6ffZfjIKH0cSGUqAjxVf0+kJNR6HPqmMyQw1PPeRPzP6yYQnPVSHsUJsIjOFgWJwCDxJCrc54pREGNDCFXc3IrpkChCwQRaMiE48y8vktZJ7bzm3NQrjes8jSI6QIeoihx0ihroCjVRC1H0iJ7RK3qznqwX6936mLUWrHxmH/2B9fUDSeiiQw==</latexit><latexit sha1_base64="dEuQ73sOOGlAJordFhBO/PRTXnQ=">AAACHHicbVBNS8NAEN3Ur1q/qh69LBahXkoiBRU8FLwIXipYW2hq2Gw37dJNNuxOhBLyR7z4V7x4UPHiQfDfuG1z0NYHA4/3ZpiZ58eCa7Dtb6uwtLyyulZcL21sbm3vlHf37rRMFGUtKoVUHZ9oJnjEWsBBsE6sGAl9wdr+6HLitx+Y0lxGtzCOWS8kg4gHnBIwkleu63vHS0MvlR7PMuwKFkAVu74U/VRluZNeTLwMY1fxwRCOvXLFrtlT4EXi5KSCcjS98qfblzQJWQRUEK27jh1DLyUKOBUsK7mJZjGhIzJgXUMjEjLdS6ffZfjIKH0cSGUqAjxVf0+kJNR6HPqmMyQw1PPeRPzP6yYQnPVSHsUJsIjOFgWJwCDxJCrc54pREGNDCFXc3IrpkChCwQRaMiE48y8vktZJ7bzm3NQrjes8jSI6QIeoihx0ihroCjVRC1H0iJ7RK3qznqwX6936mLUWrHxmH/2B9fUDSeiiQw==</latexit>
sk
moi
⇣
rmo<i
⌘
<latexit sha1_base64="8yFvGakMoAgUOhWppwqPpdRpbLM=">AAACHHicbVBNS8NAEN3Ur1q/qh69LBahXkoiBRU8FLwIXipYW2hq2Gw37dJNNuxOhBLyR7z4V7x4UPHiQfDfuG1z0NYHA4/3ZpiZ58eCa7Dtb6uwtLyyulZcL21sbm3vlHf37rRMFGUtKoVUHZ9oJnjEWsBBsE6sGAl9wdr+6HLitx+Y0lxGtzCOWS8kg4gHnBIwkleu6/uRl4ZeKj2eZdgVLIAqdn0p+qnKcie9mHgZxq7igyEce+WKXbOnwIvEyUkF5Wh65U+3L2kSsgioIFp3HTuGXkoUcCpYVnITzWJCR2TAuoZGJGS6l06/y/CRUfo4kMpUBHiq/p5ISaj1OPRNZ0hgqOe9ifif100gOOulPIoTYBGdLQoSgUHiSVS4zxWjIMaGEKq4uRXTIVGEggm0ZEJw5l9eJK2T2nnNualXGtd5GkV0gA5RFTnoFDXQFWqiFqLoET2jV/RmPVkv1rv1MWstWPnMPvoD6+sHrLCifQ==</latexit><latexit sha1_base64="8yFvGakMoAgUOhWppwqPpdRpbLM=">AAACHHicbVBNS8NAEN3Ur1q/qh69LBahXkoiBRU8FLwIXipYW2hq2Gw37dJNNuxOhBLyR7z4V7x4UPHiQfDfuG1z0NYHA4/3ZpiZ58eCa7Dtb6uwtLyyulZcL21sbm3vlHf37rRMFGUtKoVUHZ9oJnjEWsBBsE6sGAl9wdr+6HLitx+Y0lxGtzCOWS8kg4gHnBIwkleu6/uRl4ZeKj2eZdgVLIAqdn0p+qnKcie9mHgZxq7igyEce+WKXbOnwIvEyUkF5Wh65U+3L2kSsgioIFp3HTuGXkoUcCpYVnITzWJCR2TAuoZGJGS6l06/y/CRUfo4kMpUBHiq/p5ISaj1OPRNZ0hgqOe9ifif100gOOulPIoTYBGdLQoSgUHiSVS4zxWjIMaGEKq4uRXTIVGEggm0ZEJw5l9eJK2T2nnNualXGtd5GkV0gA5RFTnoFDXQFWqiFqLoET2jV/RmPVkv1rv1MWstWPnMPvoD6+sHrLCifQ==</latexit><latexit sha1_base64="8yFvGakMoAgUOhWppwqPpdRpbLM=">AAACHHicbVBNS8NAEN3Ur1q/qh69LBahXkoiBRU8FLwIXipYW2hq2Gw37dJNNuxOhBLyR7z4V7x4UPHiQfDfuG1z0NYHA4/3ZpiZ58eCa7Dtb6uwtLyyulZcL21sbm3vlHf37rRMFGUtKoVUHZ9oJnjEWsBBsE6sGAl9wdr+6HLitx+Y0lxGtzCOWS8kg4gHnBIwkleu6/uRl4ZeKj2eZdgVLIAqdn0p+qnKcie9mHgZxq7igyEce+WKXbOnwIvEyUkF5Wh65U+3L2kSsgioIFp3HTuGXkoUcCpYVnITzWJCR2TAuoZGJGS6l06/y/CRUfo4kMpUBHiq/p5ISaj1OPRNZ0hgqOe9ifif100gOOulPIoTYBGdLQoSgUHiSVS4zxWjIMaGEKq4uRXTIVGEggm0ZEJw5l9eJK2T2nnNualXGtd5GkV0gA5RFTnoFDXQFWqiFqLoET2jV/RmPVkv1rv1MWstWPnMPvoD6+sHrLCifQ==</latexit><latexit sha1_base64="8yFvGakMoAgUOhWppwqPpdRpbLM=">AAACHHicbVBNS8NAEN3Ur1q/qh69LBahXkoiBRU8FLwIXipYW2hq2Gw37dJNNuxOhBLyR7z4V7x4UPHiQfDfuG1z0NYHA4/3ZpiZ58eCa7Dtb6uwtLyyulZcL21sbm3vlHf37rRMFGUtKoVUHZ9oJnjEWsBBsE6sGAl9wdr+6HLitx+Y0lxGtzCOWS8kg4gHnBIwkleu6/uRl4ZeKj2eZdgVLIAqdn0p+qnKcie9mHgZxq7igyEce+WKXbOnwIvEyUkF5Wh65U+3L2kSsgioIFp3HTuGXkoUcCpYVnITzWJCR2TAuoZGJGS6l06/y/CRUfo4kMpUBHiq/p5ISaj1OPRNZ0hgqOe9ifif100gOOulPIoTYBGdLQoSgUHiSVS4zxWjIMaGEKq4uRXTIVGEggm0ZEJw5l9eJK2T2nnNualXGtd5GkV0gA5RFTnoFDXQFWqiFqLoET2jV/RmPVkv1rv1MWstWPnMPvoD6+sHrLCifQ==</latexit>
sK
moi
⇣
rmo<i
⌘
<latexit sha1_base64="TEHfIFWYNFymBdy4hOMMgNi9FCk=">AAACHHicbVBNS8NAEN34WetX1aOXxSLUS0mkoIKHghehlwrWFpoaNttNu3STDbsToYT8ES/+FS8eVLx4EPw3btsctPXBwOO9GWbm+bHgGmz721paXlldWy9sFDe3tnd2S3v7d1omirIWlUKqjk80EzxiLeAgWCdWjIS+YG1/dDXx2w9MaS6jWxjHrBeSQcQDTgkYySvV9H3DS0MvlR7PMuwKFkAFu74U/VRluZNeTrwMY1fxwRBOvFLZrtpT4EXi5KSMcjS90qfblzQJWQRUEK27jh1DLyUKOBUsK7qJZjGhIzJgXUMjEjLdS6ffZfjYKH0cSGUqAjxVf0+kJNR6HPqmMyQw1PPeRPzP6yYQnPdSHsUJsIjOFgWJwCDxJCrc54pREGNDCFXc3IrpkChCwQRaNCE48y8vktZp9aLq3NTK9UaeRgEdoiNUQQ46Q3V0jZqohSh6RM/oFb1ZT9aL9W59zFqXrHzmAP2B9fUDdjCiXQ==</latexit><latexit sha1_base64="TEHfIFWYNFymBdy4hOMMgNi9FCk=">AAACHHicbVBNS8NAEN34WetX1aOXxSLUS0mkoIKHghehlwrWFpoaNttNu3STDbsToYT8ES/+FS8eVLx4EPw3btsctPXBwOO9GWbm+bHgGmz721paXlldWy9sFDe3tnd2S3v7d1omirIWlUKqjk80EzxiLeAgWCdWjIS+YG1/dDXx2w9MaS6jWxjHrBeSQcQDTgkYySvV9H3DS0MvlR7PMuwKFkAFu74U/VRluZNeTrwMY1fxwRBOvFLZrtpT4EXi5KSMcjS90qfblzQJWQRUEK27jh1DLyUKOBUsK7qJZjGhIzJgXUMjEjLdS6ffZfjYKH0cSGUqAjxVf0+kJNR6HPqmMyQw1PPeRPzP6yYQnPdSHsUJsIjOFgWJwCDxJCrc54pREGNDCFXc3IrpkChCwQRaNCE48y8vktZp9aLq3NTK9UaeRgEdoiNUQQ46Q3V0jZqohSh6RM/oFb1ZT9aL9W59zFqXrHzmAP2B9fUDdjCiXQ==</latexit><latexit sha1_base64="TEHfIFWYNFymBdy4hOMMgNi9FCk=">AAACHHicbVBNS8NAEN34WetX1aOXxSLUS0mkoIKHghehlwrWFpoaNttNu3STDbsToYT8ES/+FS8eVLx4EPw3btsctPXBwOO9GWbm+bHgGmz721paXlldWy9sFDe3tnd2S3v7d1omirIWlUKqjk80EzxiLeAgWCdWjIS+YG1/dDXx2w9MaS6jWxjHrBeSQcQDTgkYySvV9H3DS0MvlR7PMuwKFkAFu74U/VRluZNeTrwMY1fxwRBOvFLZrtpT4EXi5KSMcjS90qfblzQJWQRUEK27jh1DLyUKOBUsK7qJZjGhIzJgXUMjEjLdS6ffZfjYKH0cSGUqAjxVf0+kJNR6HPqmMyQw1PPeRPzP6yYQnPdSHsUJsIjOFgWJwCDxJCrc54pREGNDCFXc3IrpkChCwQRaNCE48y8vktZp9aLq3NTK9UaeRgEdoiNUQQ46Q3V0jZqohSh6RM/oFb1ZT9aL9W59zFqXrHzmAP2B9fUDdjCiXQ==</latexit><latexit sha1_base64="TEHfIFWYNFymBdy4hOMMgNi9FCk=">AAACHHicbVBNS8NAEN34WetX1aOXxSLUS0mkoIKHghehlwrWFpoaNttNu3STDbsToYT8ES/+FS8eVLx4EPw3btsctPXBwOO9GWbm+bHgGmz721paXlldWy9sFDe3tnd2S3v7d1omirIWlUKqjk80EzxiLeAgWCdWjIS+YG1/dDXx2w9MaS6jWxjHrBeSQcQDTgkYySvV9H3DS0MvlR7PMuwKFkAFu74U/VRluZNeTrwMY1fxwRBOvFLZrtpT4EXi5KSMcjS90qfblzQJWQRUEK27jh1DLyUKOBUsK7qJZjGhIzJgXUMjEjLdS6ffZfjYKH0cSGUqAjxVf0+kJNR6HPqmMyQw1PPeRPzP6yYQnPdSHsUJsIjOFgWJwCDxJCrc54pREGNDCFXc3IrpkChCwQRaNCE48y8vktZp9aLq3NTK9UaeRgEdoiNUQQ46Q3V0jZqohSh6RM/oFb1ZT9aL9W59zFqXrHzmAP2B9fUDdjCiXQ==</latexit>
Then, we can calculate by softmax.
It can be viewed as single-hidden-layer MLP.

17. 17
• At fist, the authors used timestamps to make order.
• The benchmark datasets (e.g,. Movielens, Netflix) contains timestamps information.
• They found that a random order works well in practice.
• For fair comparison with other algorithms.
• There was no significant difference from using timestamp information.
• The order has to be prefixed before training.
THE ORDER OF THE RATINGS

18. 18
• Training phase:
• Test phase:
• Given a user’s past behavior r = .
• The user’s rating of new item m* can be predicted as:
• where
TEST PHASE FOR BASIC MODEL

19. CONTENTS
1. Motivation
2. NADE
3. Basic Model of CF-NADE
4. Parameters Sharing
5. Ordinal Cost
6. Deep Models and Augmentation
7. Experiments
8. Future Works and Discussion

20. 20
• 2 kinds of weight sharing in this paper.
• Between users
• Between ratings
• Between users:
• The CF (Collaborative Filtering) has sparsity problem.
• The training data of CF problem is too sparse.
• W, K, c, b are shared between users.
• Between ratings:
• Share weighs between different ratings. Why? and How?
WEIGHT SHARING

21. 21
• Why?
• Some items may not be rated much.
• Items that are less rated are less optimized.
• To enhance optimization of items that are less rated
• How? using other weights of other ratings.
WEIGHT SHARING BETWEEN RATINGS
[ Basic model ] [ Weight sharing between ratings ]

22. 22
• When calculating the parameter of rating k in weight sharing,
the parameters of rating 1, ... , k of the basic model are added.
• It is a kind of regularization,
which encourages the model to use as many parameters as possible to
explain the data.
WHY WEIGHT SHARING BETWEEN RATINGS WORKS?

23. CONTENTS
1. Motivation
2. NADE
3. Basic Model of CF-NADE
4. Parameters Sharing
5. Ordinal Cost
6. Deep Models and Augmentation
7. Experiments
8. Future Works and Discussion

24. 24
• In the basic model, we use softmax function to calculate cost.
• We call it as regular cost Creg.
• However, rating data has ordinal nature.
• If user u want to give the movie m 4-star score,
the PMF by ordinal cost is more reasonable compared to that by regular cost.
• Ordinal cost could be calculated by ranking loss.
REGULAR COST CREG AND ORDINAL COST CORD
0.125
0.25
0.375
0.5
1 2 3 4 5
0.15
0.3
0.45
0.6
1 2 3 4 5
[ PMF by Regular Cost Creg ] [ PMF by Ordinal Cost Cord ]

25. 25
• What is ordinal nature?
• Suppose .
• the ranking of preferences over all the possible ratings can be expressed as:
• denotes the preference of rating k over k-1.
• It is used by multiplying two ranking losses (kind of heuristic I think).
• To capture this ordinal nature:
ORDINAL COST CORD
regular to ordinal

26. 26
• 1. Ordinal cost is inspired by Xia et al (2008) [4].
• 2. the equation of PMF by ordinal cost is not PMF.
• We have to normalize!
• 3. the ordinal cost can not always capture ordinal nature.
• ex. s1, s2, s3, s4, s5 = 1, 2.7, 2, 3, 1
SOME NOTES FOR ORDINAL COST CORD
0.125
0.25
0.375
0.5
1 2 3 4 5

27. 27
• Let hybrid cost:
• CF-NADE: the basic model.
• CF-NADE-S: the basic model with shared parameters.
THE EFFECTS OF ORDINAL COST
• The model with weight sharing
outperforms the basic model.
• The ordinal cost outperforms
the regular cost.

28. CONTENTS
1. Motivation
2. NADE
3. Basic Model of CF-NADE
4. Parameters Sharing
5. Ordinal Cost
6. Deep Models and Augmentation
7. Experiments
8. Future Works and Discussion

29. 29
• The basic CF-NADE model can be viewed as single-hidden-layer MLP.
• To make CF-NADE model be deep,
all we need is to define the relation between hidden layers.
• This architecture is adopted from Uria et al. (2014), which makes original NADE be
deep networks [5].
MAKE CF-NADE BE DEEP
The basic model of CF-NADE

30. 30
• Training deep networks is difficult compared to training the basic model.
• It has the quite large numbers of free parameters.
• This paper suggest 2 tricks to handle the situation training deep networks.
• 1. Data augmentation
• 2. Reduction of free parameters in the networks.
TO TRAIN DEEP NETWORKS

31. 31
• As I mentioned earlier, a random order works well in practice.
• Different order is an different instantiation of CF-NADE for the same user.
• This is key to extend CF-NADE to a deep model.
• Training all possible orderings:
• Given a context , CF-NADE be equally good at modeling .
• Cost function is rewritten as:
DATA AUGMENTATION
C = Eo2O
DX
i=1
log p
⇣
rmoi
|rmo<i
, o
⌘
<latexit sha1_base64="L5bgHKhYO1EJQoX09/swzqPbmB0=">AAACaXicbVFdaxQxFM2Mttbth9sKKu3LxaVQoS4zUlDBQqEVhD60gmsLO+uQyWZ2QzPJkNwRlpgH/6Jv/gJf/BHNbhex214IOTnn3pvck6KWwmKS/I7iBw+Xlh+tPG6trq1vPGlvbn21ujGM95iW2lwW1HIpFO+hQMkva8NpVUh+UVwdT/WL79xYodUXnNR8UNGREqVgFAOVt39mFcUxo9IdeziE2ako3EefOw2ZUPBPhzMPHjLbVLkTh6n/dgKvIZN6BHXYeIl7kBVaDp0JtSFH58J7+LFIug+B9vsQuhsxGuOrvN1Jusks4C5I56BD5nGet39lQ82aiitkklrbT5MaB44aFExy38oay2vKruiI9wNUtOJ24GZWedgNzBBKbcJSCDP2/wpHK2snVREyp4PbRW1K3qf1GyzfDZxQdYNcsZuLykYCapj6DkNhOEM5CYAyI8JbgY2poQzD77SCCeniyHdB7033fTf9fNA5Op27sUJ2yEuyR1LylhyRT+Sc9Agjf6L16Fn0PPobb8Uv4u2b1Dia1zwltyLuXANyBLoP</latexit><latexit sha1_base64="L5bgHKhYO1EJQoX09/swzqPbmB0=">AAACaXicbVFdaxQxFM2Mttbth9sKKu3LxaVQoS4zUlDBQqEVhD60gmsLO+uQyWZ2QzPJkNwRlpgH/6Jv/gJf/BHNbhex214IOTnn3pvck6KWwmKS/I7iBw+Xlh+tPG6trq1vPGlvbn21ujGM95iW2lwW1HIpFO+hQMkva8NpVUh+UVwdT/WL79xYodUXnNR8UNGREqVgFAOVt39mFcUxo9IdeziE2ako3EefOw2ZUPBPhzMPHjLbVLkTh6n/dgKvIZN6BHXYeIl7kBVaDp0JtSFH58J7+LFIug+B9vsQuhsxGuOrvN1Jusks4C5I56BD5nGet39lQ82aiitkklrbT5MaB44aFExy38oay2vKruiI9wNUtOJ24GZWedgNzBBKbcJSCDP2/wpHK2snVREyp4PbRW1K3qf1GyzfDZxQdYNcsZuLykYCapj6DkNhOEM5CYAyI8JbgY2poQzD77SCCeniyHdB7033fTf9fNA5Op27sUJ2yEuyR1LylhyRT+Sc9Agjf6L16Fn0PPobb8Uv4u2b1Dia1zwltyLuXANyBLoP</latexit><latexit sha1_base64="L5bgHKhYO1EJQoX09/swzqPbmB0=">AAACaXicbVFdaxQxFM2Mttbth9sKKu3LxaVQoS4zUlDBQqEVhD60gmsLO+uQyWZ2QzPJkNwRlpgH/6Jv/gJf/BHNbhex214IOTnn3pvck6KWwmKS/I7iBw+Xlh+tPG6trq1vPGlvbn21ujGM95iW2lwW1HIpFO+hQMkva8NpVUh+UVwdT/WL79xYodUXnNR8UNGREqVgFAOVt39mFcUxo9IdeziE2ako3EefOw2ZUPBPhzMPHjLbVLkTh6n/dgKvIZN6BHXYeIl7kBVaDp0JtSFH58J7+LFIug+B9vsQuhsxGuOrvN1Jusks4C5I56BD5nGet39lQ82aiitkklrbT5MaB44aFExy38oay2vKruiI9wNUtOJ24GZWedgNzBBKbcJSCDP2/wpHK2snVREyp4PbRW1K3qf1GyzfDZxQdYNcsZuLykYCapj6DkNhOEM5CYAyI8JbgY2poQzD77SCCeniyHdB7033fTf9fNA5Op27sUJ2yEuyR1LylhyRT+Sc9Agjf6L16Fn0PPobb8Uv4u2b1Dia1zwltyLuXANyBLoP</latexit><latexit sha1_base64="L5bgHKhYO1EJQoX09/swzqPbmB0=">AAACaXicbVFdaxQxFM2Mttbth9sKKu3LxaVQoS4zUlDBQqEVhD60gmsLO+uQyWZ2QzPJkNwRlpgH/6Jv/gJf/BHNbhex214IOTnn3pvck6KWwmKS/I7iBw+Xlh+tPG6trq1vPGlvbn21ujGM95iW2lwW1HIpFO+hQMkva8NpVUh+UVwdT/WL79xYodUXnNR8UNGREqVgFAOVt39mFcUxo9IdeziE2ako3EefOw2ZUPBPhzMPHjLbVLkTh6n/dgKvIZN6BHXYeIl7kBVaDp0JtSFH58J7+LFIug+B9vsQuhsxGuOrvN1Jusks4C5I56BD5nGet39lQ82aiitkklrbT5MaB44aFExy38oay2vKruiI9wNUtOJ24GZWedgNzBBKbcJSCDP2/wpHK2snVREyp4PbRW1K3qf1GyzfDZxQdYNcsZuLykYCapj6DkNhOEM5CYAyI8JbgY2poQzD77SCCeniyHdB7033fTf9fNA5Op27sUJ2yEuyR1LylhyRT+Sc9Agjf6L16Fn0PPobb8Uv4u2b1Dia1zwltyLuXANyBLoP</latexit>
rmo<i<latexit sha1_base64="u17hOyMmvjjd0W+DvViZwKnRctk=">AAAB/XicbVBNSwMxEM3Wr1q/VsWTl2ARPJVdEVTwUPAieKng2kK7LNls2oZmkyXJCiUs+Fe8eFDx6v/w5r8xbfegrQ8GHu/NMDMvzhhV2vO+ncrS8srqWnW9trG5tb3j7u49KJFLTAIsmJCdGCnCKCeBppqRTiYJSmNG2vHoeuK3H4lUVPB7Pc5ImKIBp32KkbZS5B70YsESI4vIpJERkbmiRVFEbt1reFPAReKXpA5KtCL3q5cInKeEa8yQUl3fy3RokNQUM1LUerkiGcIjNCBdSzlKiQrN9PwCHlslgX0hbXENp+rvCYNSpcZpbDtTpIdq3puI/3ndXPcvQkN5lmvC8WxRP2dQCzjJAiZUEqzZ2BKEJbW3QjxEEmFtE6vZEPz5lxdJcNq4bPh3Z/XmbZlGFRyCI3ACfHAOmuAGtEAAMDDgGbyCN+fJeXHenY9Za8UpZ/bBHzifP9Sflkc=</latexit><latexit sha1_base64="u17hOyMmvjjd0W+DvViZwKnRctk=">AAAB/XicbVBNSwMxEM3Wr1q/VsWTl2ARPJVdEVTwUPAieKng2kK7LNls2oZmkyXJCiUs+Fe8eFDx6v/w5r8xbfegrQ8GHu/NMDMvzhhV2vO+ncrS8srqWnW9trG5tb3j7u49KJFLTAIsmJCdGCnCKCeBppqRTiYJSmNG2vHoeuK3H4lUVPB7Pc5ImKIBp32KkbZS5B70YsESI4vIpJERkbmiRVFEbt1reFPAReKXpA5KtCL3q5cInKeEa8yQUl3fy3RokNQUM1LUerkiGcIjNCBdSzlKiQrN9PwCHlslgX0hbXENp+rvCYNSpcZpbDtTpIdq3puI/3ndXPcvQkN5lmvC8WxRP2dQCzjJAiZUEqzZ2BKEJbW3QjxEEmFtE6vZEPz5lxdJcNq4bPh3Z/XmbZlGFRyCI3ACfHAOmuAGtEAAMDDgGbyCN+fJeXHenY9Za8UpZ/bBHzifP9Sflkc=</latexit><latexit sha1_base64="u17hOyMmvjjd0W+DvViZwKnRctk=">AAAB/XicbVBNSwMxEM3Wr1q/VsWTl2ARPJVdEVTwUPAieKng2kK7LNls2oZmkyXJCiUs+Fe8eFDx6v/w5r8xbfegrQ8GHu/NMDMvzhhV2vO+ncrS8srqWnW9trG5tb3j7u49KJFLTAIsmJCdGCnCKCeBppqRTiYJSmNG2vHoeuK3H4lUVPB7Pc5ImKIBp32KkbZS5B70YsESI4vIpJERkbmiRVFEbt1reFPAReKXpA5KtCL3q5cInKeEa8yQUl3fy3RokNQUM1LUerkiGcIjNCBdSzlKiQrN9PwCHlslgX0hbXENp+rvCYNSpcZpbDtTpIdq3puI/3ndXPcvQkN5lmvC8WxRP2dQCzjJAiZUEqzZ2BKEJbW3QjxEEmFtE6vZEPz5lxdJcNq4bPh3Z/XmbZlGFRyCI3ACfHAOmuAGtEAAMDDgGbyCN+fJeXHenY9Za8UpZ/bBHzifP9Sflkc=</latexit><latexit sha1_base64="u17hOyMmvjjd0W+DvViZwKnRctk=">AAAB/XicbVBNSwMxEM3Wr1q/VsWTl2ARPJVdEVTwUPAieKng2kK7LNls2oZmkyXJCiUs+Fe8eFDx6v/w5r8xbfegrQ8GHu/NMDMvzhhV2vO+ncrS8srqWnW9trG5tb3j7u49KJFLTAIsmJCdGCnCKCeBppqRTiYJSmNG2vHoeuK3H4lUVPB7Pc5ImKIBp32KkbZS5B70YsESI4vIpJERkbmiRVFEbt1reFPAReKXpA5KtCL3q5cInKeEa8yQUl3fy3RokNQUM1LUerkiGcIjNCBdSzlKiQrN9PwCHlslgX0hbXENp+rvCYNSpcZpbDtTpIdq3puI/3ndXPcvQkN5lmvC8WxRP2dQCzjJAiZUEqzZ2BKEJbW3QjxEEmFtE6vZEPz5lxdJcNq4bPh3Z/XmbZlGFRyCI3ACfHAOmuAGtEAAMDDgGbyCN+fJeXHenY9Za8UpZ/bBHzifP9Sflkc=</latexit>
rmoi<latexit sha1_base64="ue6ZLKhuXfSzu96sGd7/xMCwn5E=">AAAB+3icbVDLSsNAFJ3UV62vaJduBovgqiQiqLuCG8FNBaOFNoTJZNIOnUeYmQglxF9x40LFrT/izr9x2mahrQcuHM65l3vviTNGtfG8b6e2srq2vlHfbGxt7+zuufsH91rmCpMASyZVL0aaMCpIYKhhpJcpgnjMyEM8vpr6D49EaSrFnZlkJORoKGhKMTJWitzmIJYsKVQZFTwqZETLEkZuy2t7M8Bl4lekBSp0I/drkEiccyIMZkjrvu9lJiyQMhQzUjYGuSYZwmM0JH1LBeJEh8Xs+BIeWyWBqVS2hIEz9fdEgbjWEx7bTo7MSC96U/E/r5+b9CIsqMhyQwSeL0pzBo2E0yRgQhXBhk0sQVhReyvEI6QQNjavhg3BX3x5mQSn7cu2f3vW6txUadTBITgCJ8AH56ADrkEXBACDCXgGr+DNeXJenHfnY95ac6qZJvgD5/MH2PSVHw==</latexit><latexit sha1_base64="ue6ZLKhuXfSzu96sGd7/xMCwn5E=">AAAB+3icbVDLSsNAFJ3UV62vaJduBovgqiQiqLuCG8FNBaOFNoTJZNIOnUeYmQglxF9x40LFrT/izr9x2mahrQcuHM65l3vviTNGtfG8b6e2srq2vlHfbGxt7+zuufsH91rmCpMASyZVL0aaMCpIYKhhpJcpgnjMyEM8vpr6D49EaSrFnZlkJORoKGhKMTJWitzmIJYsKVQZFTwqZETLEkZuy2t7M8Bl4lekBSp0I/drkEiccyIMZkjrvu9lJiyQMhQzUjYGuSYZwmM0JH1LBeJEh8Xs+BIeWyWBqVS2hIEz9fdEgbjWEx7bTo7MSC96U/E/r5+b9CIsqMhyQwSeL0pzBo2E0yRgQhXBhk0sQVhReyvEI6QQNjavhg3BX3x5mQSn7cu2f3vW6txUadTBITgCJ8AH56ADrkEXBACDCXgGr+DNeXJenHfnY95ac6qZJvgD5/MH2PSVHw==</latexit><latexit sha1_base64="ue6ZLKhuXfSzu96sGd7/xMCwn5E=">AAAB+3icbVDLSsNAFJ3UV62vaJduBovgqiQiqLuCG8FNBaOFNoTJZNIOnUeYmQglxF9x40LFrT/izr9x2mahrQcuHM65l3vviTNGtfG8b6e2srq2vlHfbGxt7+zuufsH91rmCpMASyZVL0aaMCpIYKhhpJcpgnjMyEM8vpr6D49EaSrFnZlkJORoKGhKMTJWitzmIJYsKVQZFTwqZETLEkZuy2t7M8Bl4lekBSp0I/drkEiccyIMZkjrvu9lJiyQMhQzUjYGuSYZwmM0JH1LBeJEh8Xs+BIeWyWBqVS2hIEz9fdEgbjWEx7bTo7MSC96U/E/r5+b9CIsqMhyQwSeL0pzBo2E0yRgQhXBhk0sQVhReyvEI6QQNjavhg3BX3x5mQSn7cu2f3vW6txUadTBITgCJ8AH56ADrkEXBACDCXgGr+DNeXJenHfnY95ac6qZJvgD5/MH2PSVHw==</latexit><latexit sha1_base64="ue6ZLKhuXfSzu96sGd7/xMCwn5E=">AAAB+3icbVDLSsNAFJ3UV62vaJduBovgqiQiqLuCG8FNBaOFNoTJZNIOnUeYmQglxF9x40LFrT/izr9x2mahrQcuHM65l3vviTNGtfG8b6e2srq2vlHfbGxt7+zuufsH91rmCpMASyZVL0aaMCpIYKhhpJcpgnjMyEM8vpr6D49EaSrFnZlkJORoKGhKMTJWitzmIJYsKVQZFTwqZETLEkZuy2t7M8Bl4lekBSp0I/drkEiccyIMZkjrvu9lJiyQMhQzUjYGuSYZwmM0JH1LBeJEh8Xs+BIeWyWBqVS2hIEz9fdEgbjWEx7bTo7MSC96U/E/r5+b9CIsqMhyQwSeL0pzBo2E0yRgQhXBhk0sQVhReyvEI6QQNjavhg3BX3x5mQSn7cu2f3vW6txUadTBITgCJ8AH56ADrkEXBACDCXgGr+DNeXJenHfnY95ac6qZJvgD5/MH2PSVHw==</latexit>

32. 32
• Too many parameters…
•
• Netflix dataset, M=17700, H=500, K=5, then 89 million free parameters.
• Weight decay and dropout is ok, but…
• Solution: Factorize W, V by a product of 2 low-rank matrices.
• e.g. J=50, M=17700, H=500, K=5, then only 9-million free parameters
for Netflix dataset.
• Is this the best way? I think there is better way to make scalability.
REDUCTION OF FREE PARAMETERS
k 2 {1, 2, ..., K}<latexit sha1_base64="mB5r8UlHuNdIEU0I2gHnSZUyPKY=">AAACC3icbVDLSsNAFJ3UV62vqks3g0VwUUJSBHVXcCN0U8FYoQllMp20QyeTMHMjlNAPcOOvuHGh4tYfcOffOG2z0NYDA4dzzuXOPWEquAbH+bZKK6tr6xvlzcrW9s7uXnX/4E4nmaLMo4lI1H1INBNcMg84CHafKkbiULBOOLqa+p0HpjRP5C2MUxbEZCB5xCkBI/WqtRH2ucS+YBH4OXbruFHHtm3XcQv7ig+G4E9MyrGdGfAycQtSQwXaveqX309oFjMJVBCtu66TQpATBZwKNqn4mWYpoSMyYF1DJYmZDvLZMRN8YpQ+jhJlngQ8U39P5CTWehyHJhkTGOpFbyr+53UziC6CnMs0AybpfFGUCQwJnjaD+1wxCmJsCKGKm79iOiSKUDD9VUwJ7uLJy8Rr2Je2e3NWa7aKNsroCB2jU+Sic9RE16iNPETRI3pGr+jNerJerHfrYx4tWcXMIfoD6/MHdu2YUQ==</latexit><latexit sha1_base64="mB5r8UlHuNdIEU0I2gHnSZUyPKY=">AAACC3icbVDLSsNAFJ3UV62vqks3g0VwUUJSBHVXcCN0U8FYoQllMp20QyeTMHMjlNAPcOOvuHGh4tYfcOffOG2z0NYDA4dzzuXOPWEquAbH+bZKK6tr6xvlzcrW9s7uXnX/4E4nmaLMo4lI1H1INBNcMg84CHafKkbiULBOOLqa+p0HpjRP5C2MUxbEZCB5xCkBI/WqtRH2ucS+YBH4OXbruFHHtm3XcQv7ig+G4E9MyrGdGfAycQtSQwXaveqX309oFjMJVBCtu66TQpATBZwKNqn4mWYpoSMyYF1DJYmZDvLZMRN8YpQ+jhJlngQ8U39P5CTWehyHJhkTGOpFbyr+53UziC6CnMs0AybpfFGUCQwJnjaD+1wxCmJsCKGKm79iOiSKUDD9VUwJ7uLJy8Rr2Je2e3NWa7aKNsroCB2jU+Sic9RE16iNPETRI3pGr+jNerJerHfrYx4tWcXMIfoD6/MHdu2YUQ==</latexit><latexit sha1_base64="mB5r8UlHuNdIEU0I2gHnSZUyPKY=">AAACC3icbVDLSsNAFJ3UV62vqks3g0VwUUJSBHVXcCN0U8FYoQllMp20QyeTMHMjlNAPcOOvuHGh4tYfcOffOG2z0NYDA4dzzuXOPWEquAbH+bZKK6tr6xvlzcrW9s7uXnX/4E4nmaLMo4lI1H1INBNcMg84CHafKkbiULBOOLqa+p0HpjRP5C2MUxbEZCB5xCkBI/WqtRH2ucS+YBH4OXbruFHHtm3XcQv7ig+G4E9MyrGdGfAycQtSQwXaveqX309oFjMJVBCtu66TQpATBZwKNqn4mWYpoSMyYF1DJYmZDvLZMRN8YpQ+jhJlngQ8U39P5CTWehyHJhkTGOpFbyr+53UziC6CnMs0AybpfFGUCQwJnjaD+1wxCmJsCKGKm79iOiSKUDD9VUwJ7uLJy8Rr2Je2e3NWa7aKNsroCB2jU+Sic9RE16iNPETRI3pGr+jNerJerHfrYx4tWcXMIfoD6/MHdu2YUQ==</latexit><latexit sha1_base64="mB5r8UlHuNdIEU0I2gHnSZUyPKY=">AAACC3icbVDLSsNAFJ3UV62vqks3g0VwUUJSBHVXcCN0U8FYoQllMp20QyeTMHMjlNAPcOOvuHGh4tYfcOffOG2z0NYDA4dzzuXOPWEquAbH+bZKK6tr6xvlzcrW9s7uXnX/4E4nmaLMo4lI1H1INBNcMg84CHafKkbiULBOOLqa+p0HpjRP5C2MUxbEZCB5xCkBI/WqtRH2ucS+YBH4OXbruFHHtm3XcQv7ig+G4E9MyrGdGfAycQtSQwXaveqX309oFjMJVBCtu66TQpATBZwKNqn4mWYpoSMyYF1DJYmZDvLZMRN8YpQ+jhJlngQ8U39P5CTWehyHJhkTGOpFbyr+53UziC6CnMs0AybpfFGUCQwJnjaD+1wxCmJsCKGKm79iOiSKUDD9VUwJ7uLJy8Rr2Je2e3NWa7aKNsroCB2jU+Sic9RE16iNPETRI3pGr+jNerJerHfrYx4tWcXMIfoD6/MHdu2YUQ==</latexit>

33. CONTENTS
1. Motivation
2. NADE
3. Basic Model of CF-NADE
4. Parameters Sharing
5. Ordinal Cost
6. Deep Models and Augmentation
7. Experiments
8. Future Works and Discussion

34. 34
• Metric:
• is the i-th true rating and is the predicted rating by the model.
• S is the total number of ratings in the test set.
• 90% training set and 10% test set.
• Benchmark dataset:
EXPERIMENTS
ri<latexit sha1_base64="bZrBveIyIMjF/b94D20eOjf2Xro=">AAAB6XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEUG8FL4KXisYW2lA22027dLMJuxOhhP4ELx5UvPqPvPlv3LY5aOuDgcd7M8zMC1MpDLrut1NaWV1b3yhvVra2d3b3qvsHjybJNOM+S2Si2yE1XArFfRQoeTvVnMah5K1wdD31W09cG5GoBxynPIjpQIlIMIpWutc90avW3Lo7A1kmXkFqUKDZq351+wnLYq6QSWpMx3NTDHKqUTDJJ5VuZnhK2YgOeMdSRWNugnx26oScWKVPokTbUkhm6u+JnMbGjOPQdsYUh2bRm4r/eZ0Mo8sgFyrNkCs2XxRlkmBCpn+TvtCcoRxbQpkW9lbChlRThjadig3BW3x5mfhn9au6d3dea9wWaZThCI7hFDy4gAbcQBN8YDCAZ3iFN0c6L8678zFvLTnFzCH8gfP5A8e0ja8=</latexit><latexit sha1_base64="bZrBveIyIMjF/b94D20eOjf2Xro=">AAAB6XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEUG8FL4KXisYW2lA22027dLMJuxOhhP4ELx5UvPqPvPlv3LY5aOuDgcd7M8zMC1MpDLrut1NaWV1b3yhvVra2d3b3qvsHjybJNOM+S2Si2yE1XArFfRQoeTvVnMah5K1wdD31W09cG5GoBxynPIjpQIlIMIpWutc90avW3Lo7A1kmXkFqUKDZq351+wnLYq6QSWpMx3NTDHKqUTDJJ5VuZnhK2YgOeMdSRWNugnx26oScWKVPokTbUkhm6u+JnMbGjOPQdsYUh2bRm4r/eZ0Mo8sgFyrNkCs2XxRlkmBCpn+TvtCcoRxbQpkW9lbChlRThjadig3BW3x5mfhn9au6d3dea9wWaZThCI7hFDy4gAbcQBN8YDCAZ3iFN0c6L8678zFvLTnFzCH8gfP5A8e0ja8=</latexit><latexit sha1_base64="bZrBveIyIMjF/b94D20eOjf2Xro=">AAAB6XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEUG8FL4KXisYW2lA22027dLMJuxOhhP4ELx5UvPqPvPlv3LY5aOuDgcd7M8zMC1MpDLrut1NaWV1b3yhvVra2d3b3qvsHjybJNOM+S2Si2yE1XArFfRQoeTvVnMah5K1wdD31W09cG5GoBxynPIjpQIlIMIpWutc90avW3Lo7A1kmXkFqUKDZq351+wnLYq6QSWpMx3NTDHKqUTDJJ5VuZnhK2YgOeMdSRWNugnx26oScWKVPokTbUkhm6u+JnMbGjOPQdsYUh2bRm4r/eZ0Mo8sgFyrNkCs2XxRlkmBCpn+TvtCcoRxbQpkW9lbChlRThjadig3BW3x5mfhn9au6d3dea9wWaZThCI7hFDy4gAbcQBN8YDCAZ3iFN0c6L8678zFvLTnFzCH8gfP5A8e0ja8=</latexit><latexit sha1_base64="bZrBveIyIMjF/b94D20eOjf2Xro=">AAAB6XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEUG8FL4KXisYW2lA22027dLMJuxOhhP4ELx5UvPqPvPlv3LY5aOuDgcd7M8zMC1MpDLrut1NaWV1b3yhvVra2d3b3qvsHjybJNOM+S2Si2yE1XArFfRQoeTvVnMah5K1wdD31W09cG5GoBxynPIjpQIlIMIpWutc90avW3Lo7A1kmXkFqUKDZq351+wnLYq6QSWpMx3NTDHKqUTDJJ5VuZnhK2YgOeMdSRWNugnx26oScWKVPokTbUkhm6u+JnMbGjOPQdsYUh2bRm4r/eZ0Mo8sgFyrNkCs2XxRlkmBCpn+TvtCcoRxbQpkW9lbChlRThjadig3BW3x5mfhn9au6d3dea9wWaZThCI7hFDy4gAbcQBN8YDCAZ3iFN0c6L8678zFvLTnFzCH8gfP5A8e0ja8=</latexit>
˜ri<latexit sha1_base64="x+b/iaYbr09zkeeDQuYZoNK4MVw=">AAAB8XicbVBNS8NAEJ3Ur1q/qh69BIvgqSQiqLeCF8FLBWMLbSibzaZdutkNuxOhhP4MLx5UvPpvvPlv3LY5aPXBwOO9GWbmRZngBj3vy6msrK6tb1Q3a1vbO7t79f2DB6NyTVlAlVC6GxHDBJcsQI6CdTPNSBoJ1onG1zO/88i04Ure4yRjYUqGkiecErRSr49cxKzQ0wEf1Bte05vD/Uv8kjSgRHtQ/+zHiuYpk0gFMabnexmGBdHIqWDTWj83LCN0TIasZ6kkKTNhMT956p5YJXYTpW1JdOfqz4mCpMZM0sh2pgRHZtmbif95vRyTy7DgMsuRSbpYlOTCReXO/ndjrhlFMbGEUM3trS4dEU0o2pRqNgR/+eW/JDhrXjX9u/NG67ZMowpHcAyn4MMFtOAG2hAABQVP8AKvDjrPzpvzvmitOOXMIfyC8/ENKFCRZQ==</latexit><latexit sha1_base64="x+b/iaYbr09zkeeDQuYZoNK4MVw=">AAAB8XicbVBNS8NAEJ3Ur1q/qh69BIvgqSQiqLeCF8FLBWMLbSibzaZdutkNuxOhhP4MLx5UvPpvvPlv3LY5aPXBwOO9GWbmRZngBj3vy6msrK6tb1Q3a1vbO7t79f2DB6NyTVlAlVC6GxHDBJcsQI6CdTPNSBoJ1onG1zO/88i04Ure4yRjYUqGkiecErRSr49cxKzQ0wEf1Bte05vD/Uv8kjSgRHtQ/+zHiuYpk0gFMabnexmGBdHIqWDTWj83LCN0TIasZ6kkKTNhMT956p5YJXYTpW1JdOfqz4mCpMZM0sh2pgRHZtmbif95vRyTy7DgMsuRSbpYlOTCReXO/ndjrhlFMbGEUM3trS4dEU0o2pRqNgR/+eW/JDhrXjX9u/NG67ZMowpHcAyn4MMFtOAG2hAABQVP8AKvDjrPzpvzvmitOOXMIfyC8/ENKFCRZQ==</latexit><latexit sha1_base64="x+b/iaYbr09zkeeDQuYZoNK4MVw=">AAAB8XicbVBNS8NAEJ3Ur1q/qh69BIvgqSQiqLeCF8FLBWMLbSibzaZdutkNuxOhhP4MLx5UvPpvvPlv3LY5aPXBwOO9GWbmRZngBj3vy6msrK6tb1Q3a1vbO7t79f2DB6NyTVlAlVC6GxHDBJcsQI6CdTPNSBoJ1onG1zO/88i04Ure4yRjYUqGkiecErRSr49cxKzQ0wEf1Bte05vD/Uv8kjSgRHtQ/+zHiuYpk0gFMabnexmGBdHIqWDTWj83LCN0TIasZ6kkKTNhMT956p5YJXYTpW1JdOfqz4mCpMZM0sh2pgRHZtmbif95vRyTy7DgMsuRSbpYlOTCReXO/ndjrhlFMbGEUM3trS4dEU0o2pRqNgR/+eW/JDhrXjX9u/NG67ZMowpHcAyn4MMFtOAG2hAABQVP8AKvDjrPzpvzvmitOOXMIfyC8/ENKFCRZQ==</latexit><latexit sha1_base64="x+b/iaYbr09zkeeDQuYZoNK4MVw=">AAAB8XicbVBNS8NAEJ3Ur1q/qh69BIvgqSQiqLeCF8FLBWMLbSibzaZdutkNuxOhhP4MLx5UvPpvvPlv3LY5aPXBwOO9GWbmRZngBj3vy6msrK6tb1Q3a1vbO7t79f2DB6NyTVlAlVC6GxHDBJcsQI6CdTPNSBoJ1onG1zO/88i04Ure4yRjYUqGkiecErRSr49cxKzQ0wEf1Bte05vD/Uv8kjSgRHtQ/+zHiuYpk0gFMabnexmGBdHIqWDTWj83LCN0TIasZ6kkKTNhMT956p5YJXYTpW1JdOfqz4mCpMZM0sh2pgRHZtmbif95vRyTy7DgMsuRSbpYlOTCReXO/ndjrhlFMbGEUM3trS4dEU0o2pRqNgR/+eW/JDhrXjX9u/NG67ZMowpHcAyn4MMFtOAG2hAABQVP8AKvDjrPzpvzvmitOOXMIfyC8/ENKFCRZQ==</latexit>
dataset #Users #Items #Scales #Ratings
Movielens 100k 1000 1700 5 105
Movielens 1M 6040 3952 5 106
Movielens 10M 71567 10681 10 107
Netflix 480189 17770 5 108

35. 35
• The model described in previous slides are user-based model.
• We model rating vectors of user: ru.
• Similary, item-based CF-NADE model rating vectors of item: ri.
• Some CF papers said item-based CF model outperforms user-based CF.
USER-BASED VS. ITEM-BASED

36. 36
• In Movielens 1M dataset, CF-NADE outperforms all state-of-arts algorithms.
• The double hidden layer model outperforms single hidden layer model.
• Item based CF model outperforms user based CF model.
EXPERIMENT RESULTS - MOVIELENS 1M

37. 37
• CF-NADE shows the best performance even on the larger data sets.
• Now, it is one of the best performing algorithms except MRMA (NIPS 2017).
• MRMA model is not deep networks, but a matrix factorization based model.
• They didn’t use item based CF-NADE because of the computational issue.
• I think that the performance will be better if we use item-based CF-NADE model.
EXPERIMENT RESULTS - OTHER DATASETS

38. 38
• At first, the RBM-CF paper [9] said that item based CF model outperforms
user based CF model.
• The AAE paper [8],AutoRec [7] and CF-NADE [2] paper presented the
experimental results supporting the claim.
• There are pros and cons: performance vs computational cost.
• Why item based model outperforms user based model?
• The AutoRec [7] paper said that
• The average number of ratings per item is much more than those per user
• It is inconsistent with the experimental results with movie lens 100k data of
other papers.
• High variance in the number of user ratings leads to less reliable prediction for
user-based methods.
ITEM-BASED VS USER-BASED

39. 39
• tSNE of the representations of the movies on Movielens 1M dataset.
VISUALIZATION OF THE LEARNED MODEL
RED: Documentary
BLUE: Children’s

40. 40
• They picked 5 most similar movies of the left side movies
COSINE SIMILARITIES OF THE W VECTOR
Star TrekVI:
The Undiscovered Country
Star Trek III:
The Search for Spock
Star Trek:
Generations
Star Trek:
Insurrection
Star Trek:
First Contact
Star Trek IV:
TheVoyage Home
The Lion King Aladdin
Beauty and
the Beast
The Little
Mermaid
A Bug’s Life Lady and the Tramp

41. CONTENTS
1. Motivation
2. NADE
3. Basic Model of CF-NADE
4. Parameters Sharing
5. Ordinal Cost
6. Deep Models and Augmentation
7. Experiments
8. Future Works and Discussion

42. 42
• CF-NADE:
• An efficient and powerful architecture for CF tasks.
• Sharing parameters is a good way to improve performance.
• Better scalability by factorization.
• Can be extended to a deep version and performs well.
• Ordinal cost for rating data
• Meaningful representations
• Source code at: https://github.com/Ian09/CF-NADE
CONCLUSION

43. • Implicit Feedback vs Explicit Feedback
• The recommendation using the implicit feedback data (e.g., click or view).
• Generally, implicit feedback problem is more difficult than explicit feedback
problem.
• Because, the view or click can not tell the positive preference.
• Implicit CF-NADE [6]:
• where t is implicit feedback data and c is confidence level.
43
A FOLLOW-UP STUDY

44. 44
• [1] Larochelle, Hugo and Murray, Iain.The neural autoregressive distribution estimator. In International
Conference on Artificial Intelligence and Statistics, pp. 29–37, 2011.
• [2] Zheng,Y.,Tang, B., Ding,W., & Zhou, H. (2016).A neural autoregressive approach to collaborative
filtering. arXiv preprint arXiv:1605.09477.
• [3] https://www.youtube.com/watch?v=JBP1xUIlH5I&t=4s
• [4] Xia, Fen, Liu,Tie-Yan,Wang, Jue, Zhang,Wensheng, and Li, Hang. Listwise approach to learning to
rank: theory and algorithm. In Proceedings of the 25th international conference on Machine learning,
pp. 1192–1199.ACM, 2008
• [5] Uria, Benigno, Murray, Iain, and Larochelle, Hugo.A deep and tractable density estimator. JMLR:
W&CP, 32(1): 467–475, 2014.
• [6] Zheng,Y., Liu, C.,Tang, B., & Zhou, H. (2016, September). Neural autoregressive collaborative filtering
for implicit feedback. In Proceedings of the 1st workshop on deep learning for recommender systems (pp. 2-
6).ACM.
• [7] Sedhain, S., Menon,A. K., Sanner, S., & Xie, L. (2015, May).Autorec:Autoencoders meet collaborative
filtering. In Proceedings of the 24th International Conference onWorldWideWeb (pp. 111-112).ACM.
• [8] Lee, K., Lee,Y. H., & Suh, C. (2018, June).Alternating autoencoders for matrix completion. In 2018
IEEE Data ScienceWorkshop (DSW) (pp. 130-134). IEEE.
• [9] Salakhutdinov, R., Mnih,A., & Hinton, G. (2007, June). Restricted Boltzmann machines for
collaborative filtering. In Proceedings of the 24th international conference on Machine learning (pp. 791-798).
ACM.
REFERENCE