O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.
Taegyun Jeon
TensorFlow-KR / 2016.06.18
Gwangju Institute of Science and Technology
Electricity Price Forecasting
with Rec...
Who is a speaker?
 Taegyun Jeon (GIST)
▫ Research Scientist in Machine Learning and Biomedical Engineering
tgjeon@gist.ac...
What you will learn about RNN
How to:
 Build a prediction model
▫ Easy case study: sine function
▫ Practical case study: ...
Contents
 Overview of TensorFlow
 Recurrent Neural Networks (RNN)
 RNN Implementation
 Case studies
▫ Case study #1: s...
Contents
 Overview of TensorFlow
 Recurrent Neural Networks (RNN)
 RNN Implementation
 Case studies
▫ Case study #1: s...
TensorFlow
 Open Source Software Library for Machine Intelligence
[TensorFlow-KR Advanced Track] Electricity Price Foreca...
Prerequisite
 Software
▫ TensorFlow (r0.9)
▫ Python (3.4.4)
▫ Numpy (1.11.0)
▫ Pandas (0.16.2)
 Tutorials
▫ “Recurrent N...
Contents
 Overview of TensorFlow
 Recurrent Neural Networks (RNN)
 RNN Implementation
 Case studies
▫ Case study #1: s...
Recurrent Neural Networks
 Neural Networks
▫ Inputs and outputs are independent
Page 9[TensorFlow-KR Advanced Track] Elec...
Recurrent Neural Networks (RNN)
 𝒙 𝒕: the input at time step 𝑡
 𝒔 𝒕: the hidden state at time 𝑡
 𝒐 𝒕: the output state ...
Overall procedure: RNN
 Initialization
▫ All zeros
▫ Random values (dependent on activation function)
▫ Xavier initializa...
Overall procedure: RNN
 Initialization
 Forward Propagation
▫ 𝑠𝑡 = 𝑓 𝑈𝑥 𝑡 + 𝑊𝑠𝑡−1
• Function 𝑓 usually is a nonlinearity...
Overall procedure: RNN
 Initialization
 Forward Propagation
 Calculating the loss
▫ 𝑦: the labeled data
▫ 𝑜: the output...
Overall procedure: RNN
 Initialization
 Forward Propagation
 Calculating the loss
 Stochastic Gradient Descent (SGD)
▫...
Overall procedure: RNN
 Initialization
 Forward Propagation
 Calculating the loss
 Stochastic Gradient Descent (SGD)
...
Vanishing gradient over time
 Conventional RNN with sigmoid
▫ The sensitivity of the input values
decays over time
▫ The ...
Design Patterns for RNN
 RNN Sequences
Page 17[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent...
Design Pattern for Time Series Prediction
Page 18[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurre...
Contents
 Overview of TensorFlow
 Recurrent Neural Networks (RNN)
 RNN Implementation
 Case studies
▫ Case study #1: s...
RNN Implementation using TensorFlow
 How we design RNN model
for time series prediction?
[TensorFlow-KR Advanced Track] E...
Regression models in Scikit-Learn
[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Netwo...
RNN Implementation
 Recurrent States
▫ Choose RNN cell type
▫ Use multiple RNN cells
 Input layer
▫ Prepare time series ...
1) Choose the RNN cell type
 Neural Network RNN Cells (tf.nn.rnn_cell)
▫ BasicRNNCell (tf.nn.rnn_cell.BasicRNNCell)
• act...
LAB-1) Choose the RNN Cell type
[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Network...
2) Use the multiple RNN cells
▫ RNN Cell wrapper (tf.nn.rnn_cell.MultiRNNCell)
• Create a RNN cell composed sequentially o...
LAB-2) Use the multiple RNN cells
[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Netwo...
3) Prepare the time series data
 Split raw data into train, validation, and test dataset
▫ split_data [6]
• data : raw da...
LAB-3) Prepare the time series data
[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Net...
3) Prepare the time series data
 Generate sequence pair (x, y)
▫ rnn_data [6]
• labels : True for input data (x) / False ...
LAB-3) Prepare the time series data
[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Net...
4) Split our data
 Split time series data into smaller tensors
▫ split (tf.split)
• split_dim : batch_size
• num_split : ...
LAB-4) Split our data
[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks Page 32
...
5) Connect input and recurrent layers
 Create a recurrent neural network specified by RNNCell
▫ rnn (tf.nn.rnn)
• Args:
◦...
LAB-5) Connect input and recurrent layers
[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neur...
6) Output Layer
 Add DNN layer
▫ dnn (tf.contrib.learn.ops.dnn)
• input_layer
• hidden units
 Add Linear Regression
▫ li...
LAB-6) Output Layer
[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks Page 36
dn...
7) Create RNN model for regression
 TensorFlowEstimator (tf.contrib.learn.TensorFlowEstimator)
[TensorFlow-KR Advanced Tr...
Contents
 Overview of TensorFlow
 Recurrent Neural Networks (RNN)
 RNN Implementation
 Case studies
▫ Case study #1: s...
Case study #1: sine function
 Libraries
▫ numpy: package for scientific computing
▫ matplotlib: 2D plotting library
▫ ten...
Case study #1: sine function
 Parameter definitions
▫ LOG_DIR: log file
▫ TIMESTEPS: RNN time steps
▫ RNN_LAYERS: RNN lay...
Case study #1: sine function
 Generate waveform
▫ fct: function
▫ x: observation
▫ time_steps: timesteps
▫ seperate: chec...
Case study #1: sine function
 Create a regressor with TF Learn
▫ model_fn: regression model
▫ n_classes: 0 for regression...
Case study #1: sine function
Page 43[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Net...
Case study #1: sine function
Page 44[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Net...
Contents
 Overview of TensorFlow
 Recurrent Neural Networks (RNN)
 RNN Implementation
 Case studies
▫ Case study #1: s...
Energy forecasting problems
Current timeEnergy signal
(e.g. load, price, generation)
Signal forecast
External signal
(e.g....
Electricity Price Forecasting (EPF)
[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Net...
EEM2016: Price Forecasting Competition
[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural ...
MIBEL: Iberian Electricity Market
[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Netwo...
Dataset
 Historical
Data (2015)
 Daily
Rolling
Data
[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Re...
Dataset: Historical Data (2015-16) – Prices
 Prices ( € / MWh )
▫ Hourly real electricity price for MIBEL (the Portuguese...
Dataset: Historical Data (2015-16) – Prices
 Monthly data (Jan, 2015)
[TensorFlow-KR Advanced Track] Electricity Price Fo...
Dataset: Historical Data (2015-16) – Prices
date (UTC) Price
01/01/2015 0:00 48.1
01/01/2015 1:00 47.33
01/01/2015 2:00 42...
Electricity market
[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks Page 54
Case study #2: Electricity Price Forecasting
Page 55[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recu...
Tensorboard: Main Graph
[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks Page 56
Tensorboard: RNN
[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks Page 57
Tensorboard: DNN
[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks Page 58
Tensorboard: Linear Regression
[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks...
Tensorboard: loss
[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks Page 60
Tensorboard: Histogram
[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks Page 61
Experiment results
Page 62[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks
Experiment results
 LSTM + DNN + LinearRegression
[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recur...
Experiment results
Models Mean Absolute Error (euro/MWh)
LinearRegression 4.04
RidgeRegression 4.04
LassoRegression 3.73
E...
Competition Ranking (Official)
 Check the website of EPF2016 competition
▫ http://complatt.smartwatt.net/
[TensorFlow-KR ...
Contents
 Overview of TensorFlow
 Recurrent Neural Networks (RNN)
 RNN Implementation
 Case studies
▫ Case study #1: s...
Implementation issues
 Issues for Future Works
▫ About mathematical models
• It was used wind or solar generation forecas...
Contents
 Overview of TensorFlow
 Recurrent Neural Networks (RNN)
 RNN Implementation
 Case studies
▫ Case study #1: s...
Q & A
Any Questions?
[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks Page 69
Próximos SlideShares
Carregando em…5
×

Electricity price forecasting with Recurrent Neural Networks

31.917 visualizações

Publicada em

Title: Electricity price forecasting with Recurrent Neural Networks
TensorFlow-KR
2016.06.18

Publicada em: Dados e análise
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE Format, ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE Format, ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui
  • Women usually act out of emotion, not logic. Take advantage of this and get your Ex back today! See how to  http://goo.gl/nkXEkK
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui

Electricity price forecasting with Recurrent Neural Networks

  1. 1. Taegyun Jeon TensorFlow-KR / 2016.06.18 Gwangju Institute of Science and Technology Electricity Price Forecasting with Recurrent Neural Networks RNN을 이용한 전력 가격 예측 TensorFlow-KR Advanced Track
  2. 2. Who is a speaker?  Taegyun Jeon (GIST) ▫ Research Scientist in Machine Learning and Biomedical Engineering tgjeon@gist.ac.kr linkedin.com/in/tgjeon tgjeon.github.io Page 2[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks  Github for this tutorial: https://github.com/tgjeon/TensorFlow-Tutorials-for-Time-Series
  3. 3. What you will learn about RNN How to:  Build a prediction model ▫ Easy case study: sine function ▫ Practical case study: electricity price forecasting  Manipulate time series data ▫ For RNN models  Run and evaluate graph  Predict using RNN as regressor Page 3[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks
  4. 4. Contents  Overview of TensorFlow  Recurrent Neural Networks (RNN)  RNN Implementation  Case studies ▫ Case study #1: sine function ▫ Case study #2: electricity price forecasting  Conclusions  Q & A Page 4[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks
  5. 5. Contents  Overview of TensorFlow  Recurrent Neural Networks (RNN)  RNN Implementation  Case studies ▫ Case study #1: sine function ▫ Case study #2: electricity price forecasting  Conclusions  Q & A Page 5[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks
  6. 6. TensorFlow  Open Source Software Library for Machine Intelligence [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks Page 6
  7. 7. Prerequisite  Software ▫ TensorFlow (r0.9) ▫ Python (3.4.4) ▫ Numpy (1.11.0) ▫ Pandas (0.16.2)  Tutorials ▫ “Recurrent Neural Networks”, TensorFlow Tutorials ▫ “Sequence-to-Sequence Models”, TensorFlow Tutorials  Blog Posts ▫ Understanding LSTM Networks (Chris Olah @ colah.github.io) ▫ Introduction to Recurrent Networks in TensorFlow (Danijar Hafner @ danijar.com)  Book ▫ “Deep Learning”, I. Goodfellow, Y. Bengio, and A. Courville, MIT Press, 2016 Page 7[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks
  8. 8. Contents  Overview of TensorFlow  Recurrent Neural Networks (RNN)  RNN Implementation  Case studies ▫ Case study #1: sine function ▫ Case study #2: electricity price forecasting  Conclusions  Q & A Page 8[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks
  9. 9. Recurrent Neural Networks  Neural Networks ▫ Inputs and outputs are independent Page 9[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks  Recurrent Neural Networks ▫ Sequential inputs and outputs ... 𝑥 𝑥 𝑥 𝑜 𝑠𝑠 𝑠𝑠 𝑜 𝑜 ... 𝑥𝑡−1 𝑥𝑡 𝑥𝑡+1 𝑜𝑡−1 𝑠𝑠 𝑠𝑠 𝑜𝑡 𝑜𝑡+1
  10. 10. Recurrent Neural Networks (RNN)  𝒙 𝒕: the input at time step 𝑡  𝒔 𝒕: the hidden state at time 𝑡  𝒐 𝒕: the output state at time 𝑡 Page 10[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks Image from WILDML.com: “RECURRENT NEURAL NETWORKS TUTORIAL, PART 1 – INTRODUCTION TO RNNS”
  11. 11. Overall procedure: RNN  Initialization ▫ All zeros ▫ Random values (dependent on activation function) ▫ Xavier initialization [1]: Random values in the interval from − 1 𝑛 , 1 𝑛 , where n is the number of incoming connections from the previous layer Page 11[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks [1] X. Glorot and Y. Bengio, “Understanding the difficulty of training deep feedforward neural networks” (2010)
  12. 12. Overall procedure: RNN  Initialization  Forward Propagation ▫ 𝑠𝑡 = 𝑓 𝑈𝑥 𝑡 + 𝑊𝑠𝑡−1 • Function 𝑓 usually is a nonlinearity such as tanh or ReLU ▫ 𝑜𝑡 = 𝑠𝑜𝑓𝑡𝑚𝑎𝑥 𝑉𝑠𝑡 Page 12[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks
  13. 13. Overall procedure: RNN  Initialization  Forward Propagation  Calculating the loss ▫ 𝑦: the labeled data ▫ 𝑜: the output data ▫ Cross-entropy loss: 𝐿 𝑦, 𝑜 = − 1 𝑁 𝑛∈𝑁 𝑦𝑛log(𝑜 𝑛) Page 13[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks
  14. 14. Overall procedure: RNN  Initialization  Forward Propagation  Calculating the loss  Stochastic Gradient Descent (SGD) ▫ Push the parameters into a direction that reduced the error ▫ The directions: the gradients on the loss : 𝜕𝐿 𝜕𝑈 , 𝜕𝐿 𝜕𝑉 , 𝜕𝐿 𝜕𝑊 Page 14[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks
  15. 15. Overall procedure: RNN  Initialization  Forward Propagation  Calculating the loss  Stochastic Gradient Descent (SGD)  Backpropagation Through Time (BPTT) ▫ Long-term dependencies → vanishing/exploding gradient problem Page 15[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks
  16. 16. Vanishing gradient over time  Conventional RNN with sigmoid ▫ The sensitivity of the input values decays over time ▫ The network forgets the previous input  Long-Short Term Memory (LSTM) [2] ▫ The cell remember the input as long as it wants ▫ The output can be used anytime it wants [2] A. Graves. “Supervised Sequence Labelling with Recurrent Neural Networks” (2012) Page 16[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks
  17. 17. Design Patterns for RNN  RNN Sequences Page 17[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks Blog post by A. Karpathy. “The Unreasonable Effectiveness of Recurrent Neural Networks” (2015) Task Input Output Image classification fixed-sized image fixed-sized class Image captioning image input sentence of words Sentiment analysis sentence positive or negative sentiment Machine translation sentence in English sentence in French Video classification video sequence label each frame
  18. 18. Design Pattern for Time Series Prediction Page 18[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks RNN DNN Linear Regression
  19. 19. Contents  Overview of TensorFlow  Recurrent Neural Networks (RNN)  RNN Implementation  Case studies ▫ Case study #1: sine function ▫ Case study #2: electricity price forecasting  Conclusions  Q & A Page 19[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks
  20. 20. RNN Implementation using TensorFlow  How we design RNN model for time series prediction? [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks Page 20  How manipulate our time series data as input of RNN?
  21. 21. Regression models in Scikit-Learn [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks Page 21 X = np.atleast_2d([0., 1., 2., 3., 5., 6., 7., 8., 9.5]).T y = (X*np.sin(x)).ravel() x = np.atleast_2d(np.linspace(0, 10, 1000)).T gp = GaussianProcess(corr='cubic', theta0=1e-2, thetaL=1e-4, thetaU=1e-1, random_start=100) gp.fit(X, y) y_pred, MSE = gp.predict(x, eval_MSE=True)
  22. 22. RNN Implementation  Recurrent States ▫ Choose RNN cell type ▫ Use multiple RNN cells  Input layer ▫ Prepare time series data as RNN input ▫ Data splitting ▫ Connect input and recurrent layers  Output layer ▫ Add DNN layer ▫ Add regression model  Create RNN model for regression ▫ Train & Prediction [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks Page 22
  23. 23. 1) Choose the RNN cell type  Neural Network RNN Cells (tf.nn.rnn_cell) ▫ BasicRNNCell (tf.nn.rnn_cell.BasicRNNCell) • activation : tanh() • num_units : The number of units in the RNN cell ▫ BasicLSTMCell (tf.nn.rnn_cell.BasicLSTMCell) • The implementation is based on RNN Regularization[3] • activation : tanh() • state_is_tuple : 2-tuples of the accepted and returned states ▫ GRUCell (tf.nn.rnn_cell.GRUCell) • Gated Recurrent Unit cell[4] • activation : tanh() ▫ LSTMCell (tf.nn.rnn_cell.LSTMCell) • use_peepholes (bool) : diagonal/peephole connections[5]. • cell_clip (float) : the cell state is clipped by this value prior to the cell output activation. • num_proj (int): The output dimensionality for the projection matrices Page 23[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks [3] W. Zaremba, L. Sutskever, and O. Vinyals, “Recurrent Neural Network Regularization” (2014) [4] K. Cho et al., “Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation” (2014) [5] H. Sak et al., “Long short-term memory recurrent neural network architectures for large scale acoustic modeling” (2014)
  24. 24. LAB-1) Choose the RNN Cell type [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks Page 24 Import tensorflow as tf rnn_cell = tf.nn.rnn_cell.BasicRNNCell(num_units) rnn_cell = tf.nn.rnn_cell.BasicLSTMCell(num_units) rnn_cell = tf.nn.rnn_cell.GRUCell(num_units) rnn_cell = tf.nn.rnn_cell.LSTMCell(num_units) BasicRNNCell BasicLSTMCell GRUCell LSTMCell
  25. 25. 2) Use the multiple RNN cells ▫ RNN Cell wrapper (tf.nn.rnn_cell.MultiRNNCell) • Create a RNN cell composed sequentially of a number of RNN Cells. ▫ RNN Dropout (tf.nn.rnn_cell.Dropoutwrapper) • Add dropout to inputs and outputs of the given cell. ▫ RNN Embedding wrapper (tf.nn.rnn_cell.EmbeddingWrapper) • Add input embedding to the given cell. • Ex) word2vec, GloVe ▫ RNN Input Projection wrapper (tf.nn.rnn_cell.InputProjectionWrapper) • Add input projection to the given cell. ▫ RNN Output Projection wrapper (tf.nn.rnn_cell.OutputProjectionWrapper) • Add output projection to the given cell. Page 25[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks
  26. 26. LAB-2) Use the multiple RNN cells [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks Page 26 rnn_cell = tf.nn.rnn_cell.DropoutWrapper (rnn_cell, input_keep_prob=0.8, output_keep_prob=0.8) GRU/LSTM Input_keep_prob=0.8 output_keep_prob=0.8 GRU/LSTM GRU/LSTM GRU/LSTM Stacked_lstm = tf.nn.rnn_cell.MultiRNNCell([rnn_cell] * depth) depth
  27. 27. 3) Prepare the time series data  Split raw data into train, validation, and test dataset ▫ split_data [6] • data : raw data • val_size : the ratio of validation set (ex. val_size=0.2) • test_size : the ratio of test set (ex. test_size=0.2) Page 27[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks [6] M. Mourafiq, “tensorflow-lstm-regression” (code: https://github.com/mouradmourafiq/tensorflow-lstm-regression) def split_data(data, val_size=0.2, test_size=0.2): ntest = int(round(len(data) * (1 - test_size))) nval = int(round(len(data.iloc[:ntest]) * (1 - val_size))) df_train, df_val, df_test = data.iloc[:nval], data.iloc[nval:ntest], data.iloc[ntest:] return df_train, df_val, df_test
  28. 28. LAB-3) Prepare the time series data [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks Page 28 train, val, test = split_data(raw_data, val_size=0.2, test_size=0.2) Raw data (100%) Train (80%) Validation (20%) Test (20%) Test (20%) Train (80%) 16%64% 20%
  29. 29. 3) Prepare the time series data  Generate sequence pair (x, y) ▫ rnn_data [6] • labels : True for input data (x) / False for target data (y) • num_split : time_steps • data : our data Page 29[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks def rnn_data(data, time_steps, labels=False): """ creates new data frame based on previous observation * example: l = [1, 2, 3, 4, 5] time_steps = 2 -> labels == False [[1, 2], [2, 3], [3, 4]] -> labels == True [3, 4, 5] """ rnn_df = [] for i in range(len(data) - time_steps): if labels: try: rnn_df.append(data.iloc[i + time_steps].as_matrix()) except AttributeError: rnn_df.append(data.iloc[i + time_steps]) else: data_ = data.iloc[i: i + time_steps].as_matrix() rnn_df.append(data_ if len(data_.shape) > 1 else [[i] for i in data_]) return np.array(rnn_df)
  30. 30. LAB-3) Prepare the time series data [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks Page 30 time_steps = 10 train_x = rnn_data(df_train, time_steps, labels=false) train_y = rnn_data(df_train, time_steps, labels=true) df_train [1:10000] x #01 [1, 2, 3, …,10] y #01 11 … … train_x train_y x #02 [2, 3, 4, …,11] y #02 12 x #9990 [9990, 9991, 9992, …,9999] y #9990 10000
  31. 31. 4) Split our data  Split time series data into smaller tensors ▫ split (tf.split) • split_dim : batch_size • num_split : time_steps • value : our data ▫ split_squeeze (tf.contrib.learn.ops.split_squeeze) • Splits input on given dimension and then squeezes that dimension. • dim • num_split • tensor_in Page 31[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks
  32. 32. LAB-4) Split our data [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks Page 32 time_step = 10 x_split = split_squeeze(1, time_steps, x_data) split_squeeze 1 2 3 10 𝑥𝑡−9 𝑥𝑡−8 𝑥 𝑡−7 … 𝑥𝑡 … x #01 [1, 2, 3, …,10]
  33. 33. 5) Connect input and recurrent layers  Create a recurrent neural network specified by RNNCell ▫ rnn (tf.nn.rnn) • Args: ◦ cell : an instance of RNNCell ◦ inputs : list of inputs, tensor shape = [batch_size, input_size] • Returns: ◦ (outputs, state) ◦ outputs : list of outputs ◦ state : the final state [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks Page 33
  34. 34. LAB-5) Connect input and recurrent layers [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks Page 34 rnn_cell = tf.nn.rnn_cell.BasicLSTMCell(num_units) stacked_lstm = tf.nn.rnn_cell.MultiRNNCell([rnn_cell] * depth) x_split = tf.split(batch_size, time_steps, x_data) output, state = tf.nn.rnn(stacked_lstm, x_split) 𝑥𝑡−9 𝑥𝑡−8 𝑥𝑡−7 … 𝑥𝑡 LSTM LSTM LSTM LSTM LSTM LSTM LSTM LSTM LSTM LSTM LSTM LSTM … 𝑜𝑡−9 𝑜𝑡−8 𝑜𝑡−7 … 𝑜𝑡
  35. 35. 6) Output Layer  Add DNN layer ▫ dnn (tf.contrib.learn.ops.dnn) • input_layer • hidden units  Add Linear Regression ▫ linear_regression (tf.contrib.learn.models.linear_regression) • X • y [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks Page 35
  36. 36. LAB-6) Output Layer [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks Page 36 dnn_output = dnn(rnn_output, [10, 10]) LSTM_Regressor = linear_regression(dnn_output, y) LSTM LSTM LSTM LSTM… DNN Layer 1 with 10 hidden units DNN Layer 2 with 10 hidden units Linear regression
  37. 37. 7) Create RNN model for regression  TensorFlowEstimator (tf.contrib.learn.TensorFlowEstimator) [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks Page 37 regressor = learn.TensorFlowEstimator(model_fn=LSTM_Regressor, n_classes=0, verbose=1, steps=TRAINING_STEPS, optimizer='Adagrad', learning_rate=0.03, batch_size=BATCH_SIZE) regressor.fit(X['train'], y['train'] predicted = regressor.predict(X['test']) mse = mean_squared_error(y['test'], predicted)
  38. 38. Contents  Overview of TensorFlow  Recurrent Neural Networks (RNN)  RNN Implementation  Case studies ▫ Case study #1: sine function ▫ Case study #2: electricity price forecasting  Conclusions  Q & A Page 38[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks
  39. 39. Case study #1: sine function  Libraries ▫ numpy: package for scientific computing ▫ matplotlib: 2D plotting library ▫ tensorflow: open source software library for machine intelligence ▫ learn: Simplified interface for TensorFlow (mimicking Scikit Learn) for Deep Learning ▫ mse: "mean squared error" as evaluation metric ▫ lstm_predictor: our lstm class Page 39[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks %matplotlib inline import numpy as np from matplotlib import pyplot as plt from tensorflow.contrib import learn from sklearn.metrics import mean_squared_error, mean_absolute_error from lstm_predictor import generate_data, lstm_model
  40. 40. Case study #1: sine function  Parameter definitions ▫ LOG_DIR: log file ▫ TIMESTEPS: RNN time steps ▫ RNN_LAYERS: RNN layer information ▫ DENSE_LAYERS: Size of DNN[10, 10]: Two dense layer with 10 hidden units ▫ TRAINING_STEPS ▫ BATCH_SIZE ▫ PRINT_STEPS Page 40[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks LOG_DIR = './ops_logs' TIMESTEPS = 5 RNN_LAYERS = [{'steps': TIMESTEPS}] DENSE_LAYERS = [10, 10] TRAINING_STEPS = 100000 BATCH_SIZE = 100 PRINT_STEPS = TRAINING_STEPS / 100
  41. 41. Case study #1: sine function  Generate waveform ▫ fct: function ▫ x: observation ▫ time_steps: timesteps ▫ seperate: check multimodality Page 41[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks X, y = generate_data(np.sin, np.linspace(0, 100, 10000), TIMESTEPS, seperate=False)
  42. 42. Case study #1: sine function  Create a regressor with TF Learn ▫ model_fn: regression model ▫ n_classes: 0 for regression ▫ verbose: ▫ steps: training steps ▫ optimizer: ("SGD", "Adam", "Adagrad") ▫ learning_rate ▫ batch_size Page 42[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks regressor = learn.TensorFlowEstimator(model_fn=lstm_model(TIMESTEPS, RNN_LAYERS, DENSE_LAYERS), n_classes=0, verbose=1, steps=TRAINING_STEPS, optimizer='Adagrad', learning_rate=0.03, batch_size=BATCH_SIZE)
  43. 43. Case study #1: sine function Page 43[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks validation_monitor = learn.monitors.ValidationMonitor( X['val'], y['val'], every_n_steps=PRINT_STEPS, early_stopping_rounds=1000) regressor.fit(X['train'], y['train'], monitors=[validation_monitor], logdir=LOG_DIR) predicted = regressor.predict(X['test']) mse = mean_squared_error(y['test'], predicted) print ("Error: %f" % mse) Error: 0.000294
  44. 44. Case study #1: sine function Page 44[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks plot_predicted, = plt.plot(predicted, label='predicted') plot_test, = plt.plot(y['test'], label='test') plt.legend(handles=[plot_predicted, plot_test])
  45. 45. Contents  Overview of TensorFlow  Recurrent Neural Networks (RNN)  RNN Implementation  Case studies ▫ Case study #1: sine function ▫ Case study #2: electricity price forecasting  Conclusions  Q & A Page 45[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks
  46. 46. Energy forecasting problems Current timeEnergy signal (e.g. load, price, generation) Signal forecast External signal (e.g. Weather) External forecast (e.g. Weather forecast) Page 46[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks
  47. 47. Electricity Price Forecasting (EPF) [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks Page 47 Current timeEnergy signal (Price) External signal (e.g. Weather, load, generation)
  48. 48. EEM2016: Price Forecasting Competition [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks Page 48
  49. 49. MIBEL: Iberian Electricity Market [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks Page 49
  50. 50. Dataset  Historical Data (2015)  Daily Rolling Data [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks Page 50
  51. 51. Dataset: Historical Data (2015-16) – Prices  Prices ( € / MWh ) ▫ Hourly real electricity price for MIBEL (the Portuguese (PT) area) ▫ Duration: Jan 1st, 2015 (UTC 00:00) – Feb 2nd, 2016 (UTC 23:00) [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks Page 51
  52. 52. Dataset: Historical Data (2015-16) – Prices  Monthly data (Jan, 2015) [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks Page 52
  53. 53. Dataset: Historical Data (2015-16) – Prices date (UTC) Price 01/01/2015 0:00 48.1 01/01/2015 1:00 47.33 01/01/2015 2:00 42.27 01/01/2015 3:00 38.41 01/01/2015 4:00 35.72 01/01/2015 5:00 35.13 01/01/2015 6:00 36.22 01/01/2015 7:00 32.4 01/01/2015 8:00 36.6 01/01/2015 9:00 43.1 01/01/2015 10:00 45.14 01/01/2015 11:00 45.14 01/01/2015 12:00 47.35 01/01/2015 13:00 47.35 01/01/2015 14:00 43.61 01/01/2015 15:00 44.91 01/01/2015 16:00 48.1 01/01/2015 17:00 58.02 01/01/2015 18:00 61.01 01/01/2015 19:00 62.69 01/01/2015 20:00 60.41 01/01/2015 21:00 58.15 01/01/2015 22:00 53.6 01/01/2015 23:00 47.34 [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks Page 53
  54. 54. Electricity market [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks Page 54
  55. 55. Case study #2: Electricity Price Forecasting Page 55[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks dateparse = lambda dates: pd.datetime.strptime(dates, '%d/%m/%Y %H:%M') rawdata = pd.read_csv("./input/ElectricityPrice/RealMarketPriceDataPT.csv", parse_dates={'timeline': ['date', '(UTC)']}, index_col='timeline', date_parser=dateparse) X, y = load_csvdata(rawdata, TIMESTEPS, seperate=False)
  56. 56. Tensorboard: Main Graph [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks Page 56
  57. 57. Tensorboard: RNN [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks Page 57
  58. 58. Tensorboard: DNN [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks Page 58
  59. 59. Tensorboard: Linear Regression [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks Page 59
  60. 60. Tensorboard: loss [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks Page 60
  61. 61. Tensorboard: Histogram [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks Page 61
  62. 62. Experiment results Page 62[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks
  63. 63. Experiment results  LSTM + DNN + LinearRegression [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks Page 63 predicted test hour price (euro/MWh)
  64. 64. Experiment results Models Mean Absolute Error (euro/MWh) LinearRegression 4.04 RidgeRegression 4.04 LassoRegression 3.73 ElasticNet 3.57 LeastAngleRegression 6.27 LSTM+DNN+LinearRegression 2.13 [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks Page 64
  65. 65. Competition Ranking (Official)  Check the website of EPF2016 competition ▫ http://complatt.smartwatt.net/ [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks Page 65
  66. 66. Contents  Overview of TensorFlow  Recurrent Neural Networks (RNN)  RNN Implementation  Case studies ▫ Case study #1: sine function ▫ Case study #2: electricity price forecasting  Conclusions  Q & A Page 66[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks
  67. 67. Implementation issues  Issues for Future Works ▫ About mathematical models • It was used wind or solar generation forecast models? • It was used load generation forecast models? • It was used ensemble of mathematical models or ensemble average of multiple runs? ▫ About information used • There are a cascading usage of the forecast in your price model? For instance, you use your forecast (D+1) as input for model (D+2)? • You adjusted the models based on previous forecasts of other forecasters ? If yes, whish forecast you usually follow? ▫ About training period • What time period was used to train your model? • The model was updated with recent data? • In which days you update the models? Page 67[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks
  68. 68. Contents  Overview of TensorFlow  Recurrent Neural Networks (RNN)  RNN Implementation  Case studies ▫ Case study #1: sine function ▫ Case study #2: electricity price forecasting  Conclusions  Q & A Page 68[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks
  69. 69. Q & A Any Questions? [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks Page 69

×