O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.
© 2014 Mohammad Fadli
Python for Earth
Mohammad Fadli
Days before python
(back in college era)
• Using fortran for numerical programming
• Using matlab for matrix operational o...
Create matrix
do j = 1, 3
do i = 1, 3
A(i,j) = real(i)/real(j)
blah.. Blah ..
read text file ..
write text file ..
enddo
e...
Matlab
• Matrix Operation
• a = [ 1 2 3 4 5; 2 3 4 5 6; 3 4 5 6 7; 4 5 6 7 8]
• a(2,5)
Courtesy of Mathworks Inc.
Plot stuff
Courtesy of Golden Software Inc.
Courtesy of Embarcadero Technologies, Inc.
Courtesy of ESRI, Inc.
Hydrodynamics model
Courtesy of TideTech, Inc.
Courtesy of Kementerian Kelautan dan Perikanan
Courtesy of Kementerian Kelautan dan Perikanan
Courtesy of Kementerian Kelautan dan Perikanan
Courtesy of Kementerian Kelautan dan Perikanan
Courtesy van oord nv
• Build model for
pollutant trajectory
Grid Analysis and Display System (GrADS)
Courtesy of Weather & Climate Prediction Lab. ITB
Do something different!
• Install freeBSD and runs model on freeBSD
• Install mapserver on freeBSD
• Using grads for plott...
First time using python ArcGIS ArcPy
Courtesy of ESRI Inc.
Heboh Python (Python Fuss) :p
• A clear and readable syntax.
• Modules can be easily written in C, C++.
• It can be used o...
Nowadays
• ANUGA for Hydrodynamics model
• EQRM for Earthquakes model
• Python for Climate Data Analysis Tools (CDAT)
• GD...
Science with Python: NumPy, SciPy and Matplotlib Courtesy of Andrew Walker
Science with Python: NumPy, SciPy and Matplotlib Courtesy of Andrew Walker
SciPy: a collection of useful modules
Courtesy of Andrew Walker
• Python enables users to do more and better science.
• Python is a modern, interpreted, object-oriented, open-source
lang...
Another advantages
• Several packages exist that handle file formats used by
atmospheric and oceanic sciences users, inclu...
Disadvantages
• To be fair, Python has real disadvantages, including the fact that
pure Python code runs much slower than ...
What we are doing with python now?
• Try to runs (and optimize) anuga on 128
core HPC
• Optimize and automate earthquake m...
Flood Research 2013:
Domain Pasar Minggu to Manggarai (Segment 1)
• Criteria:
– Major factor only from discharge and
local...
Model Hidrodinamika 2D
AnuGA
ANUGA adalah model hidrodinamika Free and Open Source (FOSS) yang dikembangkan
oleh Australia...
Anuga for Jakarta Flood 2013
2013 flood simulation
Bukit Duri dan Kampung Melayu Kalibata
Courtesy of PT Reasuransi Maipar...
Hasil Simulasi Banjir Jakarta
Simulasi Banjir Jakarta 2013
Hadi, dkk., 2013
Input Curah Hujan Spasial, Robert, 2013
Anuga ...
Courtesy of LAPAN
Courtesy of LAPAN
Rendaman Max. vs Data UAV LAPAN
Courtesy of LAPAN, PT Reasuransi Maipark Indonesia
Courtesy of PT Reasuransi Maipark Indonesia
K.H. Abdullah Syafei
Foto: Kantor Berita ANTARA
(19/01/2014)
Courtesy of PT Reasuransi Maipark Indonesia
Depo, Bukit Duri
(19/01/2014)
Foto: Kantor Berita ANTARA
Bukit Duri
Bukit Duri (Depo & SMA 8)
Depo
SMA 8
Depo
Depo
Depo
SMA 8
SMA 8
(16/01/2013) Foto: Kantor Berita ANTARA
(19/01/2014) Foto ...
K.H. Abdullah Syafei, (19/01/2014), Foto: ANTARA
Courtesy of PT Reasuransi Maipark Indonesia
Jembatan Layang Kalibata, 19/1/2014, Foto: ANTARA
Courtesy of PT Reasuransi Maipark Indonesia
Jl. Jatinegara
barat raya
Depo
Depo
Jatinegara
Bukit Duri
Courtesy of PT Reasuransi Maipark Indonesia
Rawajati
Kampung Melayu
Diambil dari helikopter PMI, 16/1/2013
Thunder System
Summary Specification
Nodes 4 nodes
Cores 16 cores (4 cores/node)
Processors
(per nodes)
AMD FX (tm) – 8350...
National Computational Infrastructure (NCI), Australia
Summary Specification
Nodes 3592 node
Cores 57,472 cores (16 cores/...
Reference
• http://www.johnny-lin.com/pyintro/
• http://jakevdp.github.io/blog/2013/03/23/matplotlib-and-the-future-of-
vi...
grads
• The Grid Analysis and Display System (GrADS) is an
interactive desktop tool that is used for easy access,
manipula...
Python for Earth
Python for Earth
Próximos SlideShares
Carregando em…5
×

Python for Earth

975 visualizações

Publicada em

Slides from Python Indonesia meetup
http://www.python.or.id/2014/05/kopi-darat-python-indonesia-juni-2014.html

Publicada em: Tecnologia
  • Seja o primeiro a comentar

Python for Earth

  1. 1. © 2014 Mohammad Fadli Python for Earth Mohammad Fadli
  2. 2. Days before python (back in college era) • Using fortran for numerical programming • Using matlab for matrix operational or data plotting • Using excel for data processing • Using surfer or matlab for plotting • Using arcmap or or for spatial data processing • Using delphi for UI • No database (all textfile matrix)
  3. 3. Create matrix do j = 1, 3 do i = 1, 3 A(i,j) = real(i)/real(j) blah.. Blah .. read text file .. write text file .. enddo enddo
  4. 4. Matlab • Matrix Operation • a = [ 1 2 3 4 5; 2 3 4 5 6; 3 4 5 6 7; 4 5 6 7 8] • a(2,5) Courtesy of Mathworks Inc.
  5. 5. Plot stuff Courtesy of Golden Software Inc.
  6. 6. Courtesy of Embarcadero Technologies, Inc.
  7. 7. Courtesy of ESRI, Inc.
  8. 8. Hydrodynamics model Courtesy of TideTech, Inc.
  9. 9. Courtesy of Kementerian Kelautan dan Perikanan
  10. 10. Courtesy of Kementerian Kelautan dan Perikanan
  11. 11. Courtesy of Kementerian Kelautan dan Perikanan
  12. 12. Courtesy of Kementerian Kelautan dan Perikanan
  13. 13. Courtesy van oord nv • Build model for pollutant trajectory
  14. 14. Grid Analysis and Display System (GrADS) Courtesy of Weather & Climate Prediction Lab. ITB
  15. 15. Do something different! • Install freeBSD and runs model on freeBSD • Install mapserver on freeBSD • Using grads for plotting • Using fortran for data processing • Using php for web (still) • Parallel computing (not yet)
  16. 16. First time using python ArcGIS ArcPy Courtesy of ESRI Inc.
  17. 17. Heboh Python (Python Fuss) :p • A clear and readable syntax. • Modules can be easily written in C, C++. • It can be used on all major platforms (Windows, Linux/Unix, Mac etc.) • Easy to learn. • And it is free. • Python this, python that
  18. 18. Nowadays • ANUGA for Hydrodynamics model • EQRM for Earthquakes model • Python for Climate Data Analysis Tools (CDAT) • GDAL library for spatial data processing • Django, Grok, Pylons, web2py (and other tons of web library) for web • Psycopg for postgre database connection • PyMPI for Clusters computing and parallelization
  19. 19. Science with Python: NumPy, SciPy and Matplotlib Courtesy of Andrew Walker
  20. 20. Science with Python: NumPy, SciPy and Matplotlib Courtesy of Andrew Walker
  21. 21. SciPy: a collection of useful modules Courtesy of Andrew Walker
  22. 22. • Python enables users to do more and better science. • Python is a modern, interpreted, object-oriented, open-source language used in all kinds of software engineering. • Python is now a robust integration platform for all kinds of atmospheric sciences work, from data analysis to distributed computing, and graphical user interfaces to geographical information systems. • Python has a concise but natural syntax for both arrays and nonarrays, making programs exceedingly clear and easy to read; as the saying goes, “Python is executable pseudocode.” • Include visualization, numerical libraries, interconnection with compiled and other languages, memory caching, Web services, mobile and desktop graphical user interface programming, and others.
  23. 23. Another advantages • Several packages exist that handle file formats used by atmospheric and oceanic sciences users, including netCDF, HDF4, HDF-EOS 2, and GRIB 1 and 2 (e.g., via the netCDF4- Python, UV-CDAT, and PyNIO packages). • Python can be used as a scripting language for applications such as Unidata’s Integrated Data Viewer, the NCAR Command Language (via PyNGL), GrADS (via OpenGrADS), and Esri’s ArcGIS.
  24. 24. Disadvantages • To be fair, Python has real disadvantages, including the fact that pure Python code runs much slower than compiled code, there are fewer scientific libraries compared to Fortran, and documentation and support for new science users is relatively sparse. • There are tools to overcome the speed penalty, the collection of scientific libraries is growing, and science support resources are becoming more robust; nevertheless, these are real issues. • For most Earth science applications, the strengths of Python outweigh the weaknesses.
  25. 25. What we are doing with python now? • Try to runs (and optimize) anuga on 128 core HPC • Optimize and automate earthquake model • Create our own spatial analysis framework using GDAL lib • Prove what John-Wei said
  26. 26. Flood Research 2013: Domain Pasar Minggu to Manggarai (Segment 1) • Criteria: – Major factor only from discharge and local rainfall – Homogenous Hydrologic characteristic 27 Manggarai Detail Domain Katulampa Courtesy of PT Reasuransi Maipark Indonesia 2013
  27. 27. Model Hidrodinamika 2D AnuGA ANUGA adalah model hidrodinamika Free and Open Source (FOSS) yang dikembangkan oleh Australian National University dan Geoscience Australian; metode pemecahan persamaan menggunakan finite volume; dapat diunduh dari http://anuga.anu.edu.au/; Salah satu kelebihan ANUGA : dapat meng-input-kan hujan pada domain Kenapa Anuga: 1. Public Domain 2. Open Source 3. Parallel 4. Finite Volume  Shock Capturing Capability  Hydraulic Jump
  28. 28. Anuga for Jakarta Flood 2013 2013 flood simulation Bukit Duri dan Kampung Melayu Kalibata Courtesy of PT Reasuransi Maipark Indonesia 2014
  29. 29. Hasil Simulasi Banjir Jakarta Simulasi Banjir Jakarta 2013 Hadi, dkk., 2013 Input Curah Hujan Spasial, Robert, 2013 Anuga dapat mengakomodasi simulasi banjir dengan input curah hujan spasial, simulasi paralel Dan input debit serta TMA Courtesy of PT Reasuransi Maipark Indonesia 2014
  30. 30. Courtesy of LAPAN
  31. 31. Courtesy of LAPAN
  32. 32. Rendaman Max. vs Data UAV LAPAN Courtesy of LAPAN, PT Reasuransi Maipark Indonesia
  33. 33. Courtesy of PT Reasuransi Maipark Indonesia
  34. 34. K.H. Abdullah Syafei Foto: Kantor Berita ANTARA (19/01/2014) Courtesy of PT Reasuransi Maipark Indonesia
  35. 35. Depo, Bukit Duri (19/01/2014) Foto: Kantor Berita ANTARA Bukit Duri
  36. 36. Bukit Duri (Depo & SMA 8) Depo SMA 8 Depo Depo Depo SMA 8 SMA 8 (16/01/2013) Foto: Kantor Berita ANTARA (19/01/2014) Foto diambil dari Heli PMI Courtesy of PT Reasuransi Maipark Indonesia
  37. 37. K.H. Abdullah Syafei, (19/01/2014), Foto: ANTARA Courtesy of PT Reasuransi Maipark Indonesia
  38. 38. Jembatan Layang Kalibata, 19/1/2014, Foto: ANTARA Courtesy of PT Reasuransi Maipark Indonesia
  39. 39. Jl. Jatinegara barat raya Depo Depo Jatinegara Bukit Duri Courtesy of PT Reasuransi Maipark Indonesia
  40. 40. Rawajati Kampung Melayu Diambil dari helikopter PMI, 16/1/2013
  41. 41. Thunder System Summary Specification Nodes 4 nodes Cores 16 cores (4 cores/node) Processors (per nodes) AMD FX (tm) – 8350, 4 Ghz Memory (per nodes) 8 GB DDR 3, 800 Mhz Scenario: Spin up scenario for 16.67 hours Domain : Full Jakarta consist of Ciliwung River (Bogor-Pluit) Even though faster simulation could be performed, we have to notify that low resolution mesh is used in this case Scenario: 15-18 January 2013 (3 days) Domain : Ciliwung River (Pasar Minggu-Manggarai) “The computation speed may increase as the water volume inside of the domain is increasing” Courtesy of WCPL-ITB
  42. 42. National Computational Infrastructure (NCI), Australia Summary Specification Nodes 3592 node Cores 57,472 cores (16 cores/node) Model Fujitsu Server Primergy Cx 250 S2 Processors 2 x intel Sandy E5-2600 (8-core, 2.6 GHz) Memory 8 GB DDR 3, 1600 MHz 2395 nodes ->32 GB 1125 nodes -> 64 GB 72 nodes -> 128 GB TOTAL : 158 TB Harddisk 500 GB 7200 rpm SATA www.nci.org.au, www.fujitsu.com Cluster Raijin, NCI Domain 2 :Jakarta Approximately 10 rivers will be included in our scenario Approximately total mesh element number: 3.300.940 triangles Problem in computation time while using large number of mesh elements could be tackled by employing large core number of machine used for simulation NCI Building Australia National University Contoh :
  43. 43. Reference • http://www.johnny-lin.com/pyintro/ • http://jakevdp.github.io/blog/2013/03/23/matplotlib-and-the-future-of- visualization-in-python/ • http://greenteapress.com/pythonhydro/pythonhydro.html • http://resources.arcgis.com/en/communities/python/ • http://arcpy.wordpress.com/page/3/ • http://pyaos.johnny-lin.com/?p=1271 • http://www1.gly.bris.ac.uk/~walker/PythonEarthSci/ • http://conferences.ncl.ac.uk/sciprog/Presentations/python_processing_spyder_v1.0.2. pdf • http://numpy.sourceforge.net/numdoc/HTML/numdoc.htm • http://arcpy.wordpress.com/
  44. 44. grads • The Grid Analysis and Display System (GrADS) is an interactive desktop tool that is used for easy access, manipulation, and visualization of earth science data. • Support binary (stream or sequential), GRIB (version 1 and 2), NetCDF, HDF (version 4 and 5), and BUFR (for station data). • Windows & Unix support

×