3. BIG DATA SPAIN 2016 2
What is Energy Forecasting?
From Wikipedia:
Energy forecasting is a broad term that refers to
"forecasting in the energy industry".
It includes - but is not limited to - forecasting demand
(load) and price of electricity, fossil fuels (natural
gas, oil, coal) and renewable energy sources (RES;
hydro, wind, solar).
4. BIG DATA SPAIN 2016 3
What is Data Analytics?
• What happened?Descriptive
• Why did it happen?Diagnostics
• What will happen?Predictive
• What should be done?Prescriptive
Turn large volumes of complex data into actionable information
Data Decisions
5. BIG DATA SPAIN 2016 4
Data Analytics – Using Data to Make Better Decisions
Develop Predictive
Models
Access and Explore
Data
Preprocess Data
Integrate Analytics with
Systems
6. BIG DATA SPAIN 2016 5
Goal:
Implement a tool for easy and accurate computation of day-ahead system load forecast
Requirements:
Acquire and clean data from multiple
sources
Accurate predictive model
Easily deploy to production environment
Case Study: Day-Ahead Energy Load Forecasting
7. BIG DATA SPAIN 2016 6
The Data
mis.nyiso.com/public/
NYISO Energy Load Data
cdo.ncdc.noaa.gov/qclcd_ascii/
National Climatic Data Center Weather Data
8. BIG DATA SPAIN 2016 7
Data Analytics Workflow
Integrate Analytics with
Systems
Desktop Apps
Enterprise Scale
Systems
Embedded Devices
and Hardware
Files
Databases
Sensors
Access and Explore
Data
Develop Predictive
Models
Model Creation e.g.
Machine Learning
Model
Validation
Parameter
Optimization
Preprocess Data
Working with
Messy Data
Data Reduction/
Transformation
Feature
Extraction
9. BIG DATA SPAIN 2016 8
Data Analytics Workflow
Integrate Analytics with
Systems
Desktop Apps
Enterprise Scale
Systems
Embedded Devices
and Hardware
Files
Databases
Sensors
Access and Explore
Data
Develop Predictive
Models
Model Creation e.g.
Machine Learning
Model
Validation
Parameter
Optimization
Preprocess Data
Working with
Messy Data
Data Reduction/
Transformation
Feature
Extraction
1
10. BIG DATA SPAIN 2016 9
Data Analytics Workflow
Files
Databases
Sensors
Access and Explore
Data
Preprocess Data
Working with
Messy Data
Data Reduction/
Transformation
Feature
Extraction
Repositories – SQL, NoSQL, etc.
File I/O – Text, Spreadsheet, etc.
Web Sources – RESTful, JSON, etc.
Business and Transactional Data
Engineering, Scientific and Field Data
Real-Time Sources – Sensors, GPS, etc.
File I/O – Image, Audio, etc.
Communication Protocols – OPC (OLE for
Process Control), CAN (Controller Area
Network), etc.
11. BIG DATA SPAIN 2016 10
Data Analytics Workflow
Files
Databases
Sensors
Access and Explore
Data
Preprocess Data
Working with
Messy Data
Data Reduction/
Transformation
Feature
Extraction
Data aggregation
– Different sources (files, web, etc.)
– Different types (images, text, audio, etc.)
Data clean up
– Poorly formatted files
– Irregularly sampled data
– Redundant data, outliers, missing data etc.
Data specific processing
– Signals: Smoothing, resampling, denoising,
Wavelet transforms, etc.
– Images: Image registration, morphological
filtering, deblurring, etc.
Dealing with out of memory data (big data)
Challenges
12. BIG DATA SPAIN 2016 11
Data Analytics Workflow
Files
Databases
Sensors
Access and Explore
Data
Preprocess Data
Working with
Messy Data
Data Reduction/
Transformation
Feature
Extraction
Point and click tools to access
variety of data sources
High-performance environment
for big data
Files
Signals
Databases
Images
Built-in algorithms for data
preprocessing including sensor,
image, audio, video and other
real-time data
MATLAB Analytics work
with business and
engineering data
1
13. BIG DATA SPAIN 2016 12
Data Analytics Workflow
Integrate Analytics with
Systems
Desktop Apps
Enterprise Scale
Systems
Embedded Devices
and Hardware
Files
Databases
Sensors
Access and Explore
Data
Develop Predictive
Models
Model Creation e.g.
Machine Learning
Model
Validation
Parameter
Optimization
Preprocess Data
Working with
Messy Data
Data Reduction/
Transformation
Feature
Extraction
1 2
14. BIG DATA SPAIN 2016 13
Data Analytics Workflow
Develop Predictive
Models
Model Creation e.g.
Machine Learning
Model
Validation
Parameter
Optimization
Challenges
Lack of data science expertise
Feature Extraction – How to transform
data to best represent the system?
– Requires subject matter expertise
– No right way of designing features
Feature Selection – What attributes or
subset of data to use?
– Entails a lot of iteration – Trial and error
– Difficult to evaluate features
Model Development
– Many different models
– Model Validation and Tuning
Time required to conduct the analysis
Preprocess Data
Working with
Messy Data
Data Reduction/
Transformation
Feature
Extraction
15. BIG DATA SPAIN 2016 14
Data Analytics Workflow
Develop Predictive
Models
Model Creation e.g.
Machine Learning
Model
Validation
Parameter
Optimization
Preprocess Data
Working with
Messy Data
Data Reduction/
Transformation
Feature
Extraction
MATLAB enables
domain experts to
do Data Science
2
Apps Language
Easy to use apps
Wide breadth of tools to facilitate
domain specific analysis
Examples/videos to get started
Automatic MATLAB code
generation
High speed processing of large
data sets
16. BIG DATA SPAIN 2016 15
Data Analytics Workflow
Integrate Analytics with
Systems
Desktop Apps
Enterprise Scale
Systems
Embedded Devices
and Hardware
Files
Databases
Sensors
Access and Explore
Data
Develop Predictive
Models
Model Creation e.g.
Machine Learning
Model
Validation
Parameter
Optimization
Preprocess Data
Working with
Messy Data
Data Reduction/
Transformation
Feature
Extraction
1 2 3
17. BIG DATA SPAIN 2016 16
Data Analytics Workflow
Integrate Analytics with
Systems
Desktop Apps
Enterprise Scale
Systems
Embedded Devices
and Hardware
Develop Predictive
Models
Model Creation e.g.
Machine Learning
Model
Validation
Parameter
Optimization
End user: Operators, Analysts,
Administrative Staff, customers etc.
Different target platforms:
– Cluster or Cloud environment
– Standalone desktop applications
– Server based Web and enterprise systems
– Embedded hardware
Different Interfaces: C++, Java, Python,
.NET etc.
Need to translate analytics to production
environment
Challenges
18. BIG DATA SPAIN 2016 17
Integrate analytics with systems
MATLAB
Runtime
C, C++ HDL PLC
Embedded Hardware
C/C++ ++
Excel
Add-in Java
Hadoop/
Spark
.NET
MATLAB
Production
Server
Standalone
Application
Enterprise Systems
Python
MATLAB Analytics
run anywhere
3
19. BIG DATA SPAIN 2016 18
MATLAB
Desktop
Deployed Analytics
MATLAB Production Server
MATLAB
Production
Server
Web
Application
Server
MATLAB
Production Server
RequestBroker
CTF
Apache Tomcat
Web Server/
Webservice
Weather
Data
Energy
Data
Predictive
Models
Train in
MATLAB
20. BIG DATA SPAIN 2016 19
Key Takeaways
Utilize all of your data
Apply advanced analytics techniques
Operationalize analytics to enterprise
systems and embedded devices
MATLAB Analytics work
with business and
engineering data
1
MATLAB enables
domain experts to do
Data Science
2
3MATLAB Analytics
run anywhere
21. BIG DATA SPAIN 2016 20
Thank you!
Stay tuned: Twitter: @MATLAB | LinkedIn: https://www.linkedin.com/company/the-mathworks_2
% Send me your feedback:
% lucas.garcia@mathworks.com
% Twitter: @mathinking