HDF OPeNDAP project update and demo
•Transferir como PPT, PDF•
0 gostou•565 visualizações
In this presentation, we will give an update on the HDF OPeNDAP project. We will update the new features inside the HDF5 OPeNDAP data handler. We will also introduce the enhanced HDF4 OPeNDAP data handler and demonstrate how it can help users to view and analyze remote HDF-EOS2 data. A demo that uses OPeNDAP client tools to handle AIRS and MODIS Grid/Swath data with the enhanced handler will be presented.
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Semelhante a HDF OPeNDAP project update and demo
Semelhante a HDF OPeNDAP project update and demo (20)
Mais de The HDF-EOS Tools and Information Center
Mais de The HDF-EOS Tools and Information Center (20)
Último
Último (20)
HDF OPeNDAP project update and demo
- 1. The HDF Group HDF-OPeNDAP Project Update Joe Lee and Kent Yang The HDF Group James Gallagher OPeNDAP, Inc. The 13th HDF and HDF-EOS Workshop November 3-5, 2009 November 3-5, 2009 HDF/HDF-EOS Workshop XIII 1 www.hdfgroup.org
- 2. Outline • Introduction to OPeNDAP • What’s New? o HDF5-OPeNDAP Handler (minor) o HDF4-OPeNDAP Handler (major) November 3-5, 2009 HDF/HDF-EOS Workshop XIII 2 www.hdfgroup.org
- 3. Access HDF via FTP Aqua/Aura Users IDL AIRS/OMI FTP Local HDF Files November 3-5, 2009 FTP Clients FTP Servers HDF/HDF-EOS Workshop XIII HDF Files 3 www.hdfgroup.org
- 4. OPeNDAP Concept Typical Remote Access View Data (User) FTP/HTTP Remote Data (HDF4/5) Visualization Tools tools (e.g., grads) (gradsdap) Handler (hdf4/5_handler) Client Library (libnc_dap) Server (Hyrax) Data Access Protocol (DAP) via Internet November 3-5, 2009 HDF/HDF-EOS Workshop XIII 4 www.hdfgroup.org
- 5. Access HDF via OPeNDAP Aqua/Aura Users IDV HyraxHANDLER HDF AIRS/OMI libnc-dap DAP Visualization Tools November 3-5, 2009 OPeNDAP Clients OPeNDAP Servers HDF/HDF-EOS Workshop XIII HDF Files 5 www.hdfgroup.org
- 6. What’s New? HDF5-OPeNDAP Handler Work November 3-5, 2009 HDF/HDF-EOS Workshop XIII 6 www.hdfgroup.org
- 7. HDF5 Handler Update Aura Users IDV HDF5 HANDLER Update Hyrax OMI Swath Grid libnc-dap DAP Visualization Tools November 3-5, 2009 OPeNDAP Clients OPeNDAP Servers HDF/HDF-EOS Workshop XIII HDF5 Files 7 www.hdfgroup.org
- 8. HDF5-OPeNDAP Handler Update • Support Aura HDF-EOS5 OMI Swath • Access some files that follow HDF5 Dimension Scale Specification • Change in Short Name Option RadiativeCloudFraction A3RadiativeClou November 3-5, 2009 HDF/HDF-EOS Workshop XIII 8 www.hdfgroup.org
- 9. Summary: Updated HDF5 Handler • Official Release: October 6, 2009 • http://opendap.org/hdf5_handler November 3-5, 2009 HDF/HDF-EOS Workshop XIII 9 www.hdfgroup.org
- 10. What’s New? HDF4-OPeNDAP Handler Work November 3-5, 2009 HDF/HDF-EOS Workshop XIII 10 www.hdfgroup.org
- 11. HDF4-OPeNDAP Handler Work • Problems with the old HDF4 Handler • Our Work • Limitations • Future Work November 3-5, 2009 HDF/HDF-EOS Workshop XIII 11 www.hdfgroup.org
- 12. Old HDF4 Handler Problem • OPeNDAP Visualization Tools CAN NOT display most NASA HDF-EOS2 and HDF4 data November 3-5, 2009 HDF/HDF-EOS Workshop XIII 12 www.hdfgroup.org
- 13. Old HDF4 Handler Problem Aqua Users IDV Hyrax Old HDF4 HANDLER AIRS Grid libnc-dap DAP Visualization Tools November 3-5, 2009 OPeNDAP Clients OPeNDAP Servers HDF/HDF-EOS Workshop XIII HDF4 Files 13 www.hdfgroup.org
- 14. Our Work • Let Visualization Clients Access HDFEOS2 and HDF4 files November 3-5, 2009 HDF/HDF-EOS Workshop XIII 14 www.hdfgroup.org
- 15. What’s Missing? • My house costs 1. • Data – Computer Science • int my_house = 1; // Use boolean to save? • My house in Hawaii costs $1M. • Information – Earth Science • Geo-Location, Geo-Location, Geo-Location • Unit, Scale factor November 3-5, 2009 HDF/HDF-EOS Workshop XIII 15 www.hdfgroup.org
- 16. What’s Missing? • Clients expect certain data structures o DAP Grids and Shared dimensions • Clients expect some key attributes o CF conventions November 3-5, 2009 HDF/HDF-EOS Workshop XIII 16 www.hdfgroup.org
- 17. Solution: Metadata Parser • Goal: Meet the Client’s Requirements • Parse the structMetadata to o generate the DAP Grids and Shared Dims o generate the CF-compliant attributes November 3-5, 2009 HDF/HDF-EOS Workshop XIII 17 www.hdfgroup.org
- 18. New HDF4 Handler (parser) Aqua Users IDV libnc-dap New HDF4 HANDLER (parser) Hyrax Old HDF4 HANDLER AIRS Grid DAP Visualization Tools November 3-5, 2009 OPeNDAP Clients OPeNDAP Servers HDF/HDF-EOS Workshop XIII HDF4 Files 18 www.hdfgroup.org
- 19. Demo: Parser Solution Problem: Only a few HDF-EOS2 files work! AIRS Grid AIRS Swath MODIS Swath MODIS Grid AIRS.2008.10.27.L3.RetStd001.v5.2.2.0.G08303124144.hdf ftp://airspar1u.ecs.nasa.gov/data/s4pa//Aqua_AIRS_Level2/AIRX2RET.005//2008/330/AIRS.2008.11.25.143.L2.RetStd.v5.2.2.0.G0 http://daac.gsfc.nasa.gov/daac-bin/whom/mk_page_cgi.pl?PATH=datapool/AIRS/Level3_V005/01_AIRX3STD/2008/10 MOD08_D3.A2008284.005.2008289105354.hdf MOD06_L2.A2001222.0025.004.2003127111841.hdf 8331121849.hdf http://modis-atmos.gsfc.nasa.gov/MOD08_D3/acquiring.html XIII November 3-5, 2009 HDF/HDF-EOS Workshop 19 www.hdfgroup.org http://modis-atmos.gsfc.nasa.gov/MOD06_L2/acquiring.html
- 20. New Challenges for HDF4 Handler • Hundreds of HDF-EOS2 and HDF4 products o Grid Projections o Swath Dimension Map November 3-5, 2009 HDF/HDF-EOS Workshop XIII 20 www.hdfgroup.org
- 21. New Challenges HDF-EOS2 Grid projections • • • • Geographic Sinusoidal Polar Stereographic etc. HDF-EOS2 Swath Dimension Map November 3-5, 2009 HDF/HDF-EOS Workshop XIII 21 www.hdfgroup.org
- 22. Solution: Use HDF-EOS2 Library • Goal: Support more HDF-EOS2 products • Use HDF-EOS2 Library to o handle different Grid projections o handle Swath dimension map November 3-5, 2009 HDF/HDF-EOS Workshop XIII 22 www.hdfgroup.org
- 23. New HDF4 Handler (Parser) New HDF4 Handler (EOS2 library) New HDF4 HANDLER (parser) (EOS2 library) Aura Users IDV MODIS Swath w/ Dim. Map MODIS Swath Hyrax libnc-dap DAP Visualization Tools November 3-5, 2009 OPeNDAP Clients OPeNDAP Servers HDF/HDF-EOS Workshop XIII HDF4 Files 23 www.hdfgroup.org
- 24. Demo: HDF-EOS2 Library Solution Swath with Dimension Map November 3-5, 2009 HDF/HDF-EOS Workshop XIII 24 www.hdfgroup.org
- 25. Two Solutions Metadata Parser HDF-EOS2 Library • Pro: Simple installation • Pro: More products • Con: Limited products • Con: HE2 library installation November 3-5, 2009 HDF/HDF-EOS Workshop XIII 25 www.hdfgroup.org
- 26. Two Solutions - Implementation • Adapted the implementation of previous software packages Metadata Parser HDF-EOS2 Library • hdf5_handler • h4toh5 tool November 3-5, 2009 HDF/HDF-EOS Workshop XIII 26 www.hdfgroup.org
- 27. Solution Comparison November 3-5, 2009 HDF/HDF-EOS Workshop XIII 27 www.hdfgroup.org
- 28. Live Demo • Tutorial Example o http://hdfeos.org/software/idv.php • Demo Servers o http://hdfgroup.org/projects/opendap o http://hdfgroup.org/projects/opendap4 November 3-5, 2009 HDF/HDF-EOS Workshop XIII 28 www.hdfgroup.org
- 29. Limitations • 7 Reasons for Visualization Failures • Sample Files for Visualization Failures o http://hdfgroup.org/projects/opendap4/limits.html November 3-5, 2009 HDF/HDF-EOS Workshop XIII 29 www.hdfgroup.org
- 30. The Tip of the Iceberg The current HDF4 handler CAN NOT visualize most of NASA HDF-EOS2 / HDF4 files November 3-5, 2009 HDF/HDF-EOS Workshop XIII 30 www.hdfgroup.org
- 31. More Challenges for HDF4 Handler • Tackle limitations and other challenges. • Handle NASA HDF4 (e.g.,CERES) files. • In 2010 November 3-5, 2009 HDF/HDF-EOS Workshop XIII 31 www.hdfgroup.org
- 32. Summary: Enhanced HDF4 Handler • Official Release on Oct. 6, 2009. • http://opendap.org/hdf4_handler • Support some HDF-EOS2 Grid / Swath files • Use HDF-EOS2 library • Future work: support more NASA HDF-EOS2 / HDF4 files November 3-5, 2009 HDF/HDF-EOS Workshop XIII 32 www.hdfgroup.org
- 33. The HDF Group Thank You! November 3-5, 2009 HDF/HDF-EOS Workshop XIII 33 www.hdfgroup.org
- 34. Acknowledgements This work was supported by cooperative agreement number NNX06AC83A, NNX08AO77A and NNX06AG75A from the National Aeronautics and Space Administration (NASA). Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author[s] and do not necessarily reflect the views of the National Aeronautics and Space Administration. November 3-5, 2009 HDF/HDF-EOS Workshop XIII 34 www.hdfgroup.org
- 35. The HDF Group Questions/comments? November 3-5, 2009 HDF/HDF-EOS Workshop XIII 35 www.hdfgroup.org
Notas do Editor
- Hi, I’m Joe Lee from The HDF Group and I’m a software developer for HDF-OPeNDAP project at the group. I’ve been working on this project for last two and half years and I’ll present the updates from HDF-OPeNDAP Project. James Gallagher gave a tremendous help. Thanks to Chris Lynnes, Fan Fang and James Johnson and Daniel Kahn.
- For those who missed the OPeNDAP tutorial on the first day workshop, I’ll give a brief introduction to OPeNDAP again. I’ll give a short update for HDF5 OPeNDAP handler work first and then HDF4 OPeNDAP handler wok, which was the major work of the last year. At the end, I’ll go over the future work necessary for HDF4 handler.
- When it comes to the network access of data, people think of this manner. OPeNDAP is an open implementation of easy data access through visualization tools like IDV. Our work bridges the existing OPeNDAP framework and HDF so that visualization is possible over the Internet.
- This is how remote data can be accessed via OPeNDAP. Normally, users can view local data file using scientific data using visualization tools like GrADS. In DAP, this can be done for remote file. First DAP provides both generic server and client. For different data format like HDF and NetCDF, it needs an additional server that can map such data into a standard DAP format. Next, by modifying the existing visualization tools to adopt OPeNDAP client library, they can show the remote data on demand. The benefit of OPeNDAP is clear when you want to view a small dataset; you don’t have to download the entire data from the server. DAP is a low-level medium that implementers can impose any additional restrictions on top of DAP model. The bottom line is that web browser can access any HDF data.
- For people who are not familiar with OPeNDAP, I’ll explain how OPeNDAP works with HDF. OPeNDAP is an open implementation of easy data access through visualization tools like IDV. Our work bridges the existing OPeNDAP framework and HDF so that visualization is possible over the Internet.
- From now on, I will mainly focus on hdf4 handler since we made a lot of efforts on this projection. First, I’ll describe the problems of the old handlers and then what we did, limitations of our solution and future work.
- I will quickly go over hdf5 handler update. First, I’ll describe the problems of the old handlers and then what we did, limitations of our solution and future work.
- For HDF5 handler, we added support for some OMI swath file. Next, we added a support for some files that follow HDF5 dimension scale models. Finally, we changed short name option. This helps GrADs users to visualize data more quickly using the simple ‘sdfopen’ command.
- From now on, I will mainly focus on hdf4 handler since we made a lot of efforts on this projection. First, I’ll describe the problems of the old handlers and then what we did, limitations of our solution and future work.
- From now on, I will mainly focus on hdf4 handler since we made a lot of efforts on this projection. First, I’ll describe the problems of the old handlers and then what we did, limitations of our solution and future work.
- Yes, the HDF4 handler is more than 10 years old and it was part of the OPeNDAP ever since. The biggest problem is that we can’t see anything with visualization clients for HDF-EOS2 and HDF4 data.
- Here’s the illustration of problem.
- Therefore, our primary goal is to enhance it so that it can visualize data instantly.
- We asked ourselves. Why didn’t the clients work?
- We asked ourselves. Why didn’t the clients work? Many clients are built around the NetCDF-style data structures. Also, they expected some key attributes that follow CF conventions CF stands for Climate and Forecast. It’s because there are some missing information that clients expect and it’s the handler’s job to provide them.
- After identifying the missing components, we hacked the handler to provide them. Where can I find the missing information in the HDF-EOS? One solution is to create a parser that can analyze structMetadata to generate the Grids and shared dims. Also, we inserted and formatted attributes to make the clients to work. This was done efficiently because we’ve accumulated knowledge through the previous hdf5 handler work.
- I will quickly go over hdf5 handler update. First, I’ll describe the problems of the old handlers and then what we did, limitations of our solution and future work.
- By adding the metadata parser component, we could visualize some of the HDF-EOS2 files. However, this is just the tip of iceberg. There are tons of HDF-EOS files that cannot be visualized this way.
- The key challenge originates from the diversity of HDF4 and HDF-EOS2 products. For example, these are the new hurdles for meeting the convention.
- The parser could handle only the Geographic case. We could see many of HDF-EOS2 files use different projections.
- In order to support more files, we tried to use HDF-EOS2 library since it can handle different Grid projections and swath dimension maps. Again, this was done efficiently because we’ve accumulated knowledge through the previous h4toh5 tool work.
- I will quickly go over hdf5 handler update. First, I’ll describe the problems of the old handlers and then what we did, limitations of our solution and future work. HE2 to EOS2.
- Here’s one proof that using HDF-EOS2 library really helps to visualize something that wasn’t possible with a parser-based solution.
- Here’s a quick comparison.
- Here’s a quick comparison. Correct dates.
- Although installation might require an additional step, the benefit of using HDF-EOS2 library is huge.
- IDV Examples from hdfeos.org. Demo server from hdfgroup.org.
- We have summarized the top 7 reasons why some HDF-EOS will fail. We also listed some sample files that will not work with HDF4 handler. Here are screenshots of website. Make animation – screen shots in between texts.
- NASA AURA Guidelines.
- Not only limitations, but there are more challenges to overcome if you want to support hundreds of NASA HDF4 files.
- Here’s the summary of our HDF4 handler work.
- Please visit our website for further information.