Presentation for University of Helsinki event Openness in Education and Science: http://avointiede.fi/osaajakoulutuksen-ohjelma#avoimettoimintatavat

1. Openness in research
Experiences at Finnish Meteorological
Institute
28.10.2016
Finnish Meteorological Institute
Mikko Strahlendorff
Roope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari

2. Background for my
experience
Mostly employed by Finnish Meteorological Institute
 Weatherforecasting 1995-1997
 Public service development 1998--99
 Commercial service development and head of it
2000-2003
 Head of Informations system services 2004-
2007
 Seconded Expert to European Commission
Earth Observation program 2008-2010
 Ministry of Transport and Communications
Ministerial adviser 2011-2012
 EU research policy and international
organisation 2012-
 Helping to build European Research
Infrastructure Consortias
 ICOS, EuroARGO, SIOS, COPAL
 Executive commiittee member in Group on Earth
Observation
28.10.2016 Openness experience at FMI – Mikko Strahlendorff 2

3. FOSS revolution and
the internet
• Researcher Linus Torvalds at Helsinki
university releases Linux as Free and
Open-Source Software
• 25 years anniversary now and
• 96% of all web-servers run Linux
• 69% of all mobile devices run Android
(developed from Linux)
• Sharing ones work with a huge
network of like-minded ensures high
quality results faster than anything
• Motivation comes from the common
utility of the result
28.10.2016
Openness experience at FMI – Mikko Strahlendorff
3

4. • Openness requires extra efforts
• Free and open doesn’t generate revenue
• How to motivate financing? How to share expenses
fairly?
• Sharing efforts is making the list of contributors long
and single persons are less visible
• How to divide the fame?
Solutions:
• Application of the common to many challenges in
many places
• Analysis, understanding and education for sale
• Fame to the whole system:
• Data production is publishing!
• Each use of data should be rewarded!
• FOSS tools and internet technologies reduce costs
Challenges with
Openness –
rewarding effort
28.10.2016 4Openness experience at FMI – Mikko Strahlendorff

5. Data is a great
common motivation
o Analysis needs data and it gets better with
more data
o Selling scientific data is rarely lucrative
and data lying in databases unused is only
expenses - no gain
o Data is good to be collected in similar
ways for similar applications – common
interest naturally networks collectors
o Quality control and technical solutions are
perfect tasks for sharing
o Data dissemination has costs that are
usually not planned in research
o Tracking usage is good for development
and for finance motivation
28.10.2016 5

6. Finnish Meteorological Institute opened its data in 2013 after
government decision for all the Finnish public sector.
Basically everything that FMI has property rights on was opened.
Data is provided freely in machine readable format over the internet.
28.10.2016 Meteorological and Aviation Weather Open Data implementation OGC standards
Roope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari
6
FMI Open Data
https://en.ilmatieteenlaitos.fi/open-data

7. Data set Description Time
Interval
Estimated
publish date
Weather
Observations
Temperature, Wind,
Humidity, Ground
Temperature…
10 min Open,
older data to be
added
Sun Radiation UV, Short and Long
Term Radiation…
1 min Open
Marine
Observations
Waves, Sea
Temperature, Sea
Level…
1 h Open
Weather Radars Precipitation Rate,
Precipitation Amount…
5 min Open,
older data to be
added
Lightning Thunder Strikes in
Finland
5 min Open
Example Data Sets
28.10.2016 7Meteorological and Aviation Weather Open Data implementation OGC standards
Roope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari

8. Example of Data Sets
Data set Description Time Interval Estimated
publish date
Real Time
Observations
Real Time Observations from
specific location(s)
AWS 2010 –
Soundings 1959 –
Flashes 1998 –
Sea Level 1971 –
Waves 2005 –
Open
older data will
be added
Climatological
Observations
Dayly and monthly
temperature mean and
extreme values from weather
stations
1959 - Open
Climatological
Observations
Monthly temperature and
precipitation rate mean
values interpolated to grid
1961 - Open
Climatological
Reference
Climatological Reference.
Temperature, humidity,
pressure, precipitation
amount and snow depth.
Reference seasons:
1971-2000 1981-
2010
Open
28.10.2016 8Meteorological and Aviation Weather Open Data implementation OGC standards
Roope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari

9. Example Data Sets
Data set Description Time Interval Estimated
publish date
Weather forecast
model HIRLAM RCR
Point forecasts and grid
data
Latest model
run
(4 times a day)
0…54 h
Open
Sea forecast models Sea level point
forecasts, Wave (WAM)
and current (HBM) as
grid data
Latest model
run
(4 times a day)
0...54 h
Open
Environmental
Monitoring Facilities
Weather observation
stations, radars…
2015
Aviation
Observations
METAR 30 min open
Ground & mast
observations
Special observations
from ground and masts
2016 /Open
28.10.2016 9Meteorological and Aviation Weather Open Data implementation OGC standards
Roope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari

10. Example Data Sets
Data set Description Time Interval Estimated
publish date
Air Quality
Observations
Air Quality Observations 1h open
Silam Model Dispersion Model for Air
Quality, Forest Fire and
Pollen
Latest model
run (once a day)
0…96h
2015
HELMI Ice Model Ice forecast model Latest model
run
(4 times a day)
0...54 h
open
Soundings Temperature, Humidity,
Pressure, Wind from
ground to 25 km height
2 times a day open
Road Weather
Observations (LIVI)
Road Weather
Observations
10 min open
28.10.2016 10Meteorological and Aviation Weather Open Data implementation OGC standards
Roope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari

11. FMI Open Data Portal follows INSPIRE requirements.
FMI Open Data Portal
Meta data
Data
Models
Services
The very same data portal works as Open Data and
INSPIRE portal.
28.10.2016 11Meteorological and Aviation Weather Open Data implementation OGC standards
Roope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari

12. Registration
o Registration is required to use View and Download
Services
o Working email address is the only mandatory
information
o After registration the user gets an API key which have to
be added into all requests
o GET parameter fmi-apikey=…&
o Header fmi-apikey; …
o Part of url http://wms.fmi.fi/fmi-apikey/…/wms?
o One can create several API keys with one email
28.10.2016 12Meteorological and Aviation Weather Open Data implementation OGC standards
Roope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari

13. Usage Limits
With one API key it’s allowed to
o do at most 20 000 requests per day to Download Service
o do at most 10 000 requests per day to View Service
o do at most 600 requests per 5 minutes to both services
o If all observations from one time step is calculated to as one,
little over 17 000 new data sets are published daily
o So, with one API key it’s allowed to load everything once
o View service can be used for testing but can not be used as a
back end for popular clients
28.10.2016 13Meteorological and Aviation Weather Open Data implementation OGC standards
Roope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari

14. And a little over
450 000 data
downloads
per day
(5.2 req/s)
At the moment
about 11 700
registered users
3 year experience
28.10.2016 14Meteorological and Aviation Weather Open Data implementation OGC standards
Roope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari

15. Industry is
happy to use
standardized
services
Amateur and
freelancer coders
would prefer simple
JSON API
Some Experiences
28.10.2016 15Meteorological and Aviation Weather Open Data implementation OGC standards
Roope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari

16. MetoLib
o Open source JavaScript library produced by Finnish
Meteorological Institute
o Helps users to load and use the data
o Supports multi point coverage data format
o Python version is on the list
Easy
requests Cache
Parse the data
to as JSON
28.10.2016 16Meteorological and Aviation Weather Open Data implementation OGC standards
Roope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari

17. SmartMet Server
• High capacity & availability Data server for MetOcean data
• Data is extracted and products generated always on-demand
• INSPIRE Compliant
• Operational use since 2008
• FMI client services (since 2008)
• Finnish Meteorological Institute (FMI) Open Data Portal (since 2013)
• Several input sources
• GRIB-, NetCDF-, etc. files via internal data format querydata
• PostGIS database (vectors)
• SQL database (point observations)
• Several output interfaces and formats
• WMS, WFS 2.0
• JSON, XML, ASCII, HTML, SERIAL
• GRIB1, GRIB2, NetCDF
10/28/2016
Open Meteorological Data with OGC and INSPIRE /
Roope Tervo
17

18. SmartMet Server goes
Open Source
• Going to be published as Open Source in Q4/2016
• MIT license
• https://github.com/fmidev
10/28/2016
Open Meteorological Data with OGC and INSPIRE /
Roope Tervo
18

19. Open Science vision
Science WIKIPEDIA
o Data is collected in standard ways and
published immediately
o Papers are written collaboratively
o Science is mostly about writing
scripts/models to analyze data for answers
to questions
o Analysis scripts are published with the
answers so readers can reproduce results
and evaluate conclusions themselves
o Donors get to ask questions and use the
knowledge
o Sharing the fame is visible thru linking all the
scientists, from every data set and every
model/script
28.10.2016 19Openness experience at FMI – Mikko Strahlendorff

20. • Governments: What data do we need for public interests?
• How do private entities support data production?
• Science interest
• Commercial interest
• Research funding should be two-fold:
• longer time plans and grants for data production and modeling
• short term grants for answers to societal questions for scripts and analysis
• How does a new Science approach conquer the world like Linux?
• Thanks and please comment!
What do we need to do for
this vision?
28.10.2016 20Openness experience at FMI – Mikko Strahlendorff

21. www.fmi.fi
http://www.slideshare.net/MikkoStrahlendorff
https://en.ilmatieteenlaitos.fi/open-data