SlideShare uma empresa Scribd logo
1 de 23
Baixar para ler offline
Process Automation for
Hydrological Data Mapping
over GIS Software


By Rohan Jain (08AG1016)
Introduction
● Weather data is available from various
  organisations like IMD, CWC through their
  stations spanning all over the country,
  periodically.
● The data available from these places can be
  used for further processing.
● Processing is done via various GIS Software
  available.
● ArcGIS is one such popular software. It is
  used for this project
Introduction: Problem
● Data is not available in format ArcGIS support
● So it cannot be directly imported
● Manually importing 10s of thousands of data
  is not possible.
● Hence data needs to be automatically
  converted into an ArcGIS format.
● But again data from all the sources is not in a
  standardised format.
● So each data source needs special attention
Objectives
● Automatic conversion of existing
  hydrological data of Mahanadi river basin
  into a universal time-series format
● Mapping of the data into ArcHydro model of
  the ArcGIS software
Study Area: Description
● Mahandi river basin, located between
  longitudes 800 30' and 870 E, and latitudes
  190 21' and 230 35' N
● 4.3% of the total geographical area of India
● Mahanadi was notorious for its devastating
  floods.
● Hirakud Dam, one of the longest dams
  improved the situation greatly.
Mahanadi river
basin
Study Area: Data Available
● Data from India Meteorological Department
  and Central Water Commission (CWC)
● Rainfall data
● Escape Discharge data
● Water Level Data
● Data from remote sensing
Methodology: Requirements
● ArcGIS (Version 9.3)
● ArcHydro tools (Version 1.4) and ArcHydro
  data model
● Python Programming Language (Version >
  2.6)
● External Python Libraries
  ○ xlrd (for reading spreadsheets)
  ○ dbfpy (for writing dBase files)
Methodology: Study Material
● Book: ArcHydro - GIS for Water Resources
  by David R. Maidment[7]
● Book: Arc Hydro Tools - Tutorials
● GIS Course Content - University of Texas
● Web Resources, Lectures made available by
  ESRI[8] (ArcGIS Developer organisation)
Methodology
● For interfacing with ArcGIS dBase (*.dbf)
  database file format used
● dBase is a popular database and ArcGIS
  relies on it itself for storing data, so a good
  choice for using it for our task
● Python libraries available (dbfpy)
● For data model to store the time series, used
  the TimeSeries model from ArcHydro data
  models.
Methodology: Data Model
● FeatureID: ID of the feature for which this
  time series data exists. IMD Stations, CWC
  Gauges etc.
● TSTypeID: ID of the time series type. We
  have Precipitaion, Discharge, Water Level
  etc defined
● TSDateTime: The date and time of individual
  data
● TSValue: Individual data value
Methodology: Automation
1. The data obtained from various organisations
   is converted into a format which follows
   python data structures.
2. Separate (dBase) files contain information
   about HydroIDs (which will help find
   FeatureID). The information is extracted and
   used to find FeatureIDs for station names
3. Time Series is generated and then further
   published as dBase files for use with ArcGIS
   software.
The Data Conversion
Process
Methodology: Code Written
● Modules
  ○ These are for generic tasks which are applicable to
      all data sources
   ○ timeseries.py
     ■ Takes care of timeseries related internal tasks
      ■ Also generates the dBase files
   ○ stations.py:
     ■ Process the HydroIDs (FeatureIDs in Time
         Series database)
      ■ Fetches ID - Name info about the stations
Methodology: Code Written
● Individual Data Source Scripts
  ○ Since each data source provides information in a
    different format, they all need a separate script.
  ○ These scripts process the raw data to pythonic
    format and then generate time series database
● Written in Python Programming Language
● Total roughly 450 lines of python code
● A C/Java equivalent will easily measure 2-3
  times
Results
● Set up an initial project with correct directory
  hierarchy and install python + the required
  libraries
● Then, on execution of the scripts the time
  series files are generated automatically
● The time series files can then be imported
  into ArcGIS table
Results: Loading Data




Loading data
into a Time
Series table in
ArcCatalog
Result: Loading Data




ArcCatalog data loading dialogs
Result: Loading Data
                       Displaying data
                       after being
                       imported.
Result: Processing Data
                   Processing the
                   data in
                   ArcMap using
                   ArcHydro tools
Result: Processing Data




ArcMap Processing the Discharge Time Series
Future Work
● Rewrite the modules using Object Oriented
  Approach to improve the code quality and
  future additions of code easier
● Apart from this Rainfall, Discharge, Water
  Level series more data can be obtained and
  added
Thank You

Mais conteúdo relacionado

Destaque

APPLICATIONS OF REMOTE SENSING AND GIS IN WATERSHED MANAGEMENT
APPLICATIONS OF REMOTE SENSING AND GIS IN WATERSHED MANAGEMENTAPPLICATIONS OF REMOTE SENSING AND GIS IN WATERSHED MANAGEMENT
APPLICATIONS OF REMOTE SENSING AND GIS IN WATERSHED MANAGEMENT
Sriram Chakravarthy
 
Application of gis and remote sensing in agriculture
Application of gis and remote sensing in agricultureApplication of gis and remote sensing in agriculture
Application of gis and remote sensing in agriculture
Rehana Qureshi
 

Destaque (20)

JHydro - an implementation of the digital watershed
JHydro - an implementation of the digital watershedJHydro - an implementation of the digital watershed
JHydro - an implementation of the digital watershed
 
Jaysukh C Songara
Jaysukh C SongaraJaysukh C Songara
Jaysukh C Songara
 
Spatial Earth Profile2
Spatial Earth Profile2Spatial Earth Profile2
Spatial Earth Profile2
 
Impacts of landuse change on sediment transport in the yali reservoir catchment
Impacts of landuse change on sediment transport in the yali reservoir catchmentImpacts of landuse change on sediment transport in the yali reservoir catchment
Impacts of landuse change on sediment transport in the yali reservoir catchment
 
Climate Smart Landscape-Based Integrated Watershed Management: Experiences fr...
Climate Smart Landscape-Based Integrated Watershed Management: Experiences fr...Climate Smart Landscape-Based Integrated Watershed Management: Experiences fr...
Climate Smart Landscape-Based Integrated Watershed Management: Experiences fr...
 
Thesis Presentation
Thesis PresentationThesis Presentation
Thesis Presentation
 
Gnd water
Gnd waterGnd water
Gnd water
 
Application of RS and GIS in Groundwater Prospects Zonation
Application of RS and GIS in Groundwater Prospects ZonationApplication of RS and GIS in Groundwater Prospects Zonation
Application of RS and GIS in Groundwater Prospects Zonation
 
Intergrated Water Resourcses system
Intergrated Water Resourcses systemIntergrated Water Resourcses system
Intergrated Water Resourcses system
 
Presentation on Aerosols, cloud properties
Presentation on Aerosols, cloud properties Presentation on Aerosols, cloud properties
Presentation on Aerosols, cloud properties
 
Analysis of runoff for vishwamitri river watershed using scs cn method and ge...
Analysis of runoff for vishwamitri river watershed using scs cn method and ge...Analysis of runoff for vishwamitri river watershed using scs cn method and ge...
Analysis of runoff for vishwamitri river watershed using scs cn method and ge...
 
Soil Erosion for Vishwamitri River watershed using RS and GIS
Soil Erosion for Vishwamitri River watershed using RS and GISSoil Erosion for Vishwamitri River watershed using RS and GIS
Soil Erosion for Vishwamitri River watershed using RS and GIS
 
Presentation on remote sensing & gis and watershed copy
Presentation on remote sensing & gis and watershed   copyPresentation on remote sensing & gis and watershed   copy
Presentation on remote sensing & gis and watershed copy
 
REMOTE SENSING & GIS APPLICATIONS IN WATERSHED MANAGEMENT
REMOTE SENSING & GIS APPLICATIONS IN WATERSHED MANAGEMENT REMOTE SENSING & GIS APPLICATIONS IN WATERSHED MANAGEMENT
REMOTE SENSING & GIS APPLICATIONS IN WATERSHED MANAGEMENT
 
Iirs overview -Remote sensing and GIS application in Water Resources Management
Iirs overview -Remote sensing and GIS application in Water Resources ManagementIirs overview -Remote sensing and GIS application in Water Resources Management
Iirs overview -Remote sensing and GIS application in Water Resources Management
 
APPLICATIONS OF REMOTE SENSING AND GIS IN WATERSHED MANAGEMENT
APPLICATIONS OF REMOTE SENSING AND GIS IN WATERSHED MANAGEMENTAPPLICATIONS OF REMOTE SENSING AND GIS IN WATERSHED MANAGEMENT
APPLICATIONS OF REMOTE SENSING AND GIS IN WATERSHED MANAGEMENT
 
Application of gis and remote sensing in agriculture
Application of gis and remote sensing in agricultureApplication of gis and remote sensing in agriculture
Application of gis and remote sensing in agriculture
 
Application of Remote Sensing in Agriculture
Application of Remote Sensing in AgricultureApplication of Remote Sensing in Agriculture
Application of Remote Sensing in Agriculture
 
Introduction to remote sensing and gis
Introduction to remote sensing and gisIntroduction to remote sensing and gis
Introduction to remote sensing and gis
 
Integration of the MODFLOW Lak7 package in the FREEWAT GIS modelling environment
Integration of the MODFLOW Lak7 package in the FREEWAT GIS modelling environmentIntegration of the MODFLOW Lak7 package in the FREEWAT GIS modelling environment
Integration of the MODFLOW Lak7 package in the FREEWAT GIS modelling environment
 

Semelhante a Btp presentation

Semelhante a Btp presentation (20)

Understanding Hadoop
Understanding HadoopUnderstanding Hadoop
Understanding Hadoop
 
Geospatial Data Abstraction Library (GDAL) Enhancement for ESDIS (GEE)
Geospatial Data Abstraction Library (GDAL) Enhancement for ESDIS (GEE)Geospatial Data Abstraction Library (GDAL) Enhancement for ESDIS (GEE)
Geospatial Data Abstraction Library (GDAL) Enhancement for ESDIS (GEE)
 
HDF and netCDF Data Support in ArcGIS
HDF and netCDF Data Support in ArcGISHDF and netCDF Data Support in ArcGIS
HDF and netCDF Data Support in ArcGIS
 
SF Big Analytics 20191112: How to performance-tune Spark applications in larg...
SF Big Analytics 20191112: How to performance-tune Spark applications in larg...SF Big Analytics 20191112: How to performance-tune Spark applications in larg...
SF Big Analytics 20191112: How to performance-tune Spark applications in larg...
 
HDF-EOS Data Product Developer's Guide
HDF-EOS Data Product Developer's GuideHDF-EOS Data Product Developer's Guide
HDF-EOS Data Product Developer's Guide
 
11. From Hadoop to Spark 1:2
11. From Hadoop to Spark 1:211. From Hadoop to Spark 1:2
11. From Hadoop to Spark 1:2
 
Analyzing Data at Scale with Apache Spark
Analyzing Data at Scale with Apache SparkAnalyzing Data at Scale with Apache Spark
Analyzing Data at Scale with Apache Spark
 
Hoodie: How (And Why) We built an analytical datastore on Spark
Hoodie: How (And Why) We built an analytical datastore on SparkHoodie: How (And Why) We built an analytical datastore on Spark
Hoodie: How (And Why) We built an analytical datastore on Spark
 
Incremental Processing on Large Analytical Datasets with Prasanna Rajaperumal...
Incremental Processing on Large Analytical Datasets with Prasanna Rajaperumal...Incremental Processing on Large Analytical Datasets with Prasanna Rajaperumal...
Incremental Processing on Large Analytical Datasets with Prasanna Rajaperumal...
 
Spark Meetup at Uber
Spark Meetup at UberSpark Meetup at Uber
Spark Meetup at Uber
 
design_doc
design_docdesign_doc
design_doc
 
Data Analytics and Machine Learning: From Node to Cluster on ARM64
Data Analytics and Machine Learning: From Node to Cluster on ARM64Data Analytics and Machine Learning: From Node to Cluster on ARM64
Data Analytics and Machine Learning: From Node to Cluster on ARM64
 
BKK16-404B Data Analytics and Machine Learning- from Node to Cluster
BKK16-404B Data Analytics and Machine Learning- from Node to ClusterBKK16-404B Data Analytics and Machine Learning- from Node to Cluster
BKK16-404B Data Analytics and Machine Learning- from Node to Cluster
 
BKK16-408B Data Analytics and Machine Learning From Node to Cluster
BKK16-408B Data Analytics and Machine Learning From Node to ClusterBKK16-408B Data Analytics and Machine Learning From Node to Cluster
BKK16-408B Data Analytics and Machine Learning From Node to Cluster
 
Streamsets and spark in Retail
Streamsets and spark in RetailStreamsets and spark in Retail
Streamsets and spark in Retail
 
Analytic Insights in Retail Using Apache Spark with Hari Shreedharan
Analytic Insights in Retail Using Apache Spark with Hari ShreedharanAnalytic Insights in Retail Using Apache Spark with Hari Shreedharan
Analytic Insights in Retail Using Apache Spark with Hari Shreedharan
 
Big Data Processing
Big Data ProcessingBig Data Processing
Big Data Processing
 
Spark Driven Big Data Analytics
Spark Driven Big Data AnalyticsSpark Driven Big Data Analytics
Spark Driven Big Data Analytics
 
Working with HDF and netCDF Data in ArcGIS: Tools and Case Studies
Working with HDF and netCDF Data in ArcGIS: Tools and Case StudiesWorking with HDF and netCDF Data in ArcGIS: Tools and Case Studies
Working with HDF and netCDF Data in ArcGIS: Tools and Case Studies
 
Dache: A Data Aware Caching for Big-Data using Map Reduce framework
Dache: A Data Aware Caching for Big-Data using Map Reduce frameworkDache: A Data Aware Caching for Big-Data using Map Reduce framework
Dache: A Data Aware Caching for Big-Data using Map Reduce framework
 

Último

Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
vu2urc
 

Último (20)

[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
 
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
 
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
 
What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
 
GenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdfGenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdf
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
 
🐬 The future of MySQL is Postgres 🐘
🐬  The future of MySQL is Postgres   🐘🐬  The future of MySQL is Postgres   🐘
🐬 The future of MySQL is Postgres 🐘
 
Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Script
 
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organization
 
Real Time Object Detection Using Open CV
Real Time Object Detection Using Open CVReal Time Object Detection Using Open CV
Real Time Object Detection Using Open CV
 
Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024
 

Btp presentation

  • 1. Process Automation for Hydrological Data Mapping over GIS Software By Rohan Jain (08AG1016)
  • 2. Introduction ● Weather data is available from various organisations like IMD, CWC through their stations spanning all over the country, periodically. ● The data available from these places can be used for further processing. ● Processing is done via various GIS Software available. ● ArcGIS is one such popular software. It is used for this project
  • 3. Introduction: Problem ● Data is not available in format ArcGIS support ● So it cannot be directly imported ● Manually importing 10s of thousands of data is not possible. ● Hence data needs to be automatically converted into an ArcGIS format. ● But again data from all the sources is not in a standardised format. ● So each data source needs special attention
  • 4. Objectives ● Automatic conversion of existing hydrological data of Mahanadi river basin into a universal time-series format ● Mapping of the data into ArcHydro model of the ArcGIS software
  • 5. Study Area: Description ● Mahandi river basin, located between longitudes 800 30' and 870 E, and latitudes 190 21' and 230 35' N ● 4.3% of the total geographical area of India ● Mahanadi was notorious for its devastating floods. ● Hirakud Dam, one of the longest dams improved the situation greatly.
  • 7. Study Area: Data Available ● Data from India Meteorological Department and Central Water Commission (CWC) ● Rainfall data ● Escape Discharge data ● Water Level Data ● Data from remote sensing
  • 8. Methodology: Requirements ● ArcGIS (Version 9.3) ● ArcHydro tools (Version 1.4) and ArcHydro data model ● Python Programming Language (Version > 2.6) ● External Python Libraries ○ xlrd (for reading spreadsheets) ○ dbfpy (for writing dBase files)
  • 9. Methodology: Study Material ● Book: ArcHydro - GIS for Water Resources by David R. Maidment[7] ● Book: Arc Hydro Tools - Tutorials ● GIS Course Content - University of Texas ● Web Resources, Lectures made available by ESRI[8] (ArcGIS Developer organisation)
  • 10. Methodology ● For interfacing with ArcGIS dBase (*.dbf) database file format used ● dBase is a popular database and ArcGIS relies on it itself for storing data, so a good choice for using it for our task ● Python libraries available (dbfpy) ● For data model to store the time series, used the TimeSeries model from ArcHydro data models.
  • 11. Methodology: Data Model ● FeatureID: ID of the feature for which this time series data exists. IMD Stations, CWC Gauges etc. ● TSTypeID: ID of the time series type. We have Precipitaion, Discharge, Water Level etc defined ● TSDateTime: The date and time of individual data ● TSValue: Individual data value
  • 12. Methodology: Automation 1. The data obtained from various organisations is converted into a format which follows python data structures. 2. Separate (dBase) files contain information about HydroIDs (which will help find FeatureID). The information is extracted and used to find FeatureIDs for station names 3. Time Series is generated and then further published as dBase files for use with ArcGIS software.
  • 14. Methodology: Code Written ● Modules ○ These are for generic tasks which are applicable to all data sources ○ timeseries.py ■ Takes care of timeseries related internal tasks ■ Also generates the dBase files ○ stations.py: ■ Process the HydroIDs (FeatureIDs in Time Series database) ■ Fetches ID - Name info about the stations
  • 15. Methodology: Code Written ● Individual Data Source Scripts ○ Since each data source provides information in a different format, they all need a separate script. ○ These scripts process the raw data to pythonic format and then generate time series database ● Written in Python Programming Language ● Total roughly 450 lines of python code ● A C/Java equivalent will easily measure 2-3 times
  • 16. Results ● Set up an initial project with correct directory hierarchy and install python + the required libraries ● Then, on execution of the scripts the time series files are generated automatically ● The time series files can then be imported into ArcGIS table
  • 17. Results: Loading Data Loading data into a Time Series table in ArcCatalog
  • 18. Result: Loading Data ArcCatalog data loading dialogs
  • 19. Result: Loading Data Displaying data after being imported.
  • 20. Result: Processing Data Processing the data in ArcMap using ArcHydro tools
  • 21. Result: Processing Data ArcMap Processing the Discharge Time Series
  • 22. Future Work ● Rewrite the modules using Object Oriented Approach to improve the code quality and future additions of code easier ● Apart from this Rainfall, Discharge, Water Level series more data can be obtained and added