SlideShare uma empresa Scribd logo

Workflows in the Virtual Observatory

•
•
Jose Enrique Ruiz
Jose Enrique Ruiz
EducaçãoTecnologia
1 de 15
Baixar para ler offline
Workflows in the VO! Grid and Web Services Session! Jose Enrique Ruiz! IAA-CSIC! ! October 19th 2011! 2011 IVOA Fall Interop Meeting - Pune! 1
What are workflows ?! Combination of data and processes into a configurable and structured set of steps that implement semi- automated computational solutions in problem solving! Types of workflows in Astronomy! •  Personal script-based recipes ! Python, IDL, Software..! •  Multi-archive VO recipes! •  Internal group developments ! GRID, Clusters..! •  Processing pipelines! Provide Data, Computing Infrastructure, Tools..! Scientifically exploitable results vs. scientific insight ! Wfs on Easily accessible and reproducible (Shared)! steroids ! 2
Tools! Taverna! •  Strongly typed bioinformatics! •  Taverna Engine! •  Taverna Server! •  Taverna Workbench! Kepler! •  Generic Science! •  Workflow System! Triana! •  Local execution! •  Clusters RMI! •  GRID! •  Web Services! 3 !
Tools! ESO Reflex! ! Finland’s in-kind contribution to ESO! •  Prototype/feasibility study! •  Initially based on Taverna 1! ! Current implementation based on Kepler! •  Main intent: replace CLI ! (pipeline orchestrator)! ! ! 4
Tools! AstroTaverna! ! AstroGrid Development! Prototype, marrying of VO Desktop & Taverna 1! Library of Taverna functions to access VO Desktop’s API! ! Status! Wrapper libraries only for Taverna 1! ! ! 5
Tools! Aladin JLOW Plugin! Aladin plugin API permits graphical replacement of Aladin tools! 6
Tools! Aladin JLOW Plugin! 7
Related Initiatives! Wf4Ever! 2011 – 2013 EU funded FP7 STREP Project! Preservation of Workflows! ! •  Preservation of workflows and associated material! •  Archival, classification, indexing in semantic repositories! •  Provide advanced access and recommendation capabilities! •  Collaborative working platform! •  Sharing, re-use, re-purpose! •  Digital Libraries! ! ! ! 8
Related Initiatives! Cyber-SKA! Provide infrastructure that will be required to address the needs of future radio telescopes such as the Square Kilometre Array! ! Web based workflow builder ! •  Image segmentation! •  Image mosaicking (Montage)! •  Spatial reprojection! •  Plane extraction from data cubes! 9
Related Initiatives! Montage! •  FITS Image Mosaicking! •  Toolkit for Desktops, Clusters and Grids! ! Astro-WISE! •  Distributed data storage and computing infrastructure! •  Track process provenance of final data products! •  Calibration and analysis of images! ! Helio-VO! •  Solar physics Virtual Observatory! •  Enable workflow execution via Taverna Server! ! Workflows VO France! •  Provide use cases mainly oriented VO ! •  AÏDA Workflow System implements FITS validation with CharDM ! 10
The upcoming context! The next generation of archives! ! Much wider FoV and spectral coverage! •  Huge sized datasets (~100 TB)! •  Big Data science highly dependent on I/O data rates! •  Subproducts as virtual data generated on-the-fly! Automated surveys! •  Huge amount of tabular data! •  Services for Knowledge Discovery in Databases! ! 11
The upcoming context! We are moving into a world where ! •  computing and storage are cheap ! •  data movement is death! Archives should evolve from data providers into virtual data and services providers, where web services may help to solve bandwidth issues.! ! Archives speaking web services! •  Smaller virtual data subproducts! •  Distributed, multi-archive, multi-wavelength astronomy! 12
The upcoming context! Web services based workflows as a disruptive working methodology ! •  Reproducible! •  Repeatable results ! •  Encourage best practices! •  Modular nature allows ! •  Re-use! •  Re-purpose! •  Expose ! •  Provenance ! •  Scientific method! •  Formative! •  Foster collaborative work! 13
Workflows & IVOA! Distributed data analysis in the VO! •  Panchromatic, multi-archive, multi-facility! •  Executes in the VO Infrastructure! •  Orchestration of simple services! ! Workflows VO Characterization! Present processing pipelines! •  Inputs! •  Produce exploitable data! •  Outputs! •  Provenance modelling! •  Processes! •  Descriptions! •  VO compliant data ! •  Metadata! ! •  Etc..! Data processing from the VO! •  Provide custom re-processing to VO users! •  Virtual data generation through UWS in VOSpace! 14
Related activities in the VO! IVOA Working Groups! •  Data Modeling! Characterization, Provenance..! •  Semantics! Ontologies, Vocabularies, Annotations..! •  Data Access Layer! TAP, self-descriptive Protocols..! ! •  Grid and Web Services! IVOA Note! UWS, VOSpace, SSO..! Scientific Workflows in the VO! •  Applications! workflow@ivoa.net! SAMP! •  IG . KDD! Knowledge Discovery and Data Mining! •  IG . Data Curation and Preservation! Persistent Identifiers, Curation of VO Resources..! Wf4Ever Project! 15

Recomendados

Workflows to access and massage VOData
Workflows to access and massage VODataWorkflows to access and massage VOData
Workflows to access and massage VOData
Jose Enrique Ruiz
 
Research Objects in Wf4Ever
Research Objects in Wf4EverResearch Objects in Wf4Ever
Research Objects in Wf4Ever
Jose Enrique Ruiz
 
Implementing a VO archive for datacubes of galaxies
Implementing a VO archive for datacubes of galaxiesImplementing a VO archive for datacubes of galaxies
Implementing a VO archive for datacubes of galaxies
Jose Enrique Ruiz
 
Digital Science: Towards the executable paper
Digital Science: Towards the executable paperDigital Science: Towards the executable paper
Digital Science: Towards the executable paper
Jose Enrique Ruiz
 
Velocity cubes of galaxies
Velocity cubes of galaxiesVelocity cubes of galaxies
Velocity cubes of galaxies
Jose Enrique Ruiz
 
Digital Science: Reproducibility and Visibility in Astronomy
Digital Science: Reproducibility and Visibility in AstronomyDigital Science: Reproducibility and Visibility in Astronomy
Digital Science: Reproducibility and Visibility in Astronomy
Jose Enrique Ruiz
 
IPython Notebooks - Hacia los papers ejecutables
IPython Notebooks - Hacia los papers ejecutablesIPython Notebooks - Hacia los papers ejecutables
IPython Notebooks - Hacia los papers ejecutables
Jose Enrique Ruiz
 
VO web-services-based astronomy workflows
VO web-services-based astronomy workflowsVO web-services-based astronomy workflows
VO web-services-based astronomy workflows
Jose Enrique Ruiz
 

Recomendados

Workflows to access and massage VOData
Workflows to access and massage VODataWorkflows to access and massage VOData
Workflows to access and massage VOData
Jose Enrique Ruiz
 
Research Objects in Wf4Ever
Research Objects in Wf4EverResearch Objects in Wf4Ever
Research Objects in Wf4Ever
Jose Enrique Ruiz
 
Implementing a VO archive for datacubes of galaxies
Implementing a VO archive for datacubes of galaxiesImplementing a VO archive for datacubes of galaxies
Implementing a VO archive for datacubes of galaxies
Jose Enrique Ruiz
 
Digital Science: Towards the executable paper
Digital Science: Towards the executable paperDigital Science: Towards the executable paper
Digital Science: Towards the executable paper
Jose Enrique Ruiz
 
Velocity cubes of galaxies
Velocity cubes of galaxiesVelocity cubes of galaxies
Velocity cubes of galaxies
Jose Enrique Ruiz
 
Digital Science: Reproducibility and Visibility in Astronomy
Digital Science: Reproducibility and Visibility in AstronomyDigital Science: Reproducibility and Visibility in Astronomy
Digital Science: Reproducibility and Visibility in Astronomy
Jose Enrique Ruiz
 
IPython Notebooks - Hacia los papers ejecutables
IPython Notebooks - Hacia los papers ejecutablesIPython Notebooks - Hacia los papers ejecutables
IPython Notebooks - Hacia los papers ejecutables
Jose Enrique Ruiz
 
VO web-services-based astronomy workflows
VO web-services-based astronomy workflowsVO web-services-based astronomy workflows
VO web-services-based astronomy workflows
Jose Enrique Ruiz
 
Virtual Science in the Cloud
Virtual Science in the CloudVirtual Science in the Cloud
Virtual Science in the Cloud
thetfoot
 
What to Expect of the LSST Archive: The LSST Science Platform
What to Expect of the LSST Archive: The LSST Science PlatformWhat to Expect of the LSST Archive: The LSST Science Platform
What to Expect of the LSST Archive: The LSST Science Platform
Mario Juric
 
LSST Education and Public Outreach (EPO)
LSST Education and Public Outreach (EPO) LSST Education and Public Outreach (EPO)
LSST Education and Public Outreach (EPO)
Amanda Bauer
 
What's New in Cytoscape
What's New in CytoscapeWhat's New in Cytoscape
What's New in Cytoscape
Keiichiro Ono
 
Scaling collaborative data science with Globus and Jupyter
Scaling collaborative data science with Globus and JupyterScaling collaborative data science with Globus and Jupyter
Scaling collaborative data science with Globus and Jupyter
Ian Foster
 
Big Data Modeling Challenges and Machine Learning with No Code
Big Data Modeling Challenges and Machine Learning with No CodeBig Data Modeling Challenges and Machine Learning with No Code
Big Data Modeling Challenges and Machine Learning with No Code
Liana Ye
 
A Recommender Story: Improving Backend Data Quality While Reducing Costs
A Recommender Story: Improving Backend Data Quality While Reducing CostsA Recommender Story: Improving Backend Data Quality While Reducing Costs
A Recommender Story: Improving Backend Data Quality While Reducing Costs
Databricks
 
Semantically-Enabling the Web of Things: The W3C Semantic Sensor Network Onto...
Semantically-Enabling the Web of Things: The W3C Semantic Sensor Network Onto...Semantically-Enabling the Web of Things: The W3C Semantic Sensor Network Onto...
Semantically-Enabling the Web of Things: The W3C Semantic Sensor Network Onto...
Laurent Lefort
 
DSD-INT 2015 - Data management with open earth datalabs - Gerben de Boer, van...
DSD-INT 2015 - Data management with open earth datalabs - Gerben de Boer, van...DSD-INT 2015 - Data management with open earth datalabs - Gerben de Boer, van...
DSD-INT 2015 - Data management with open earth datalabs - Gerben de Boer, van...
Deltares
 
Networking Materials Data
Networking Materials DataNetworking Materials Data
Networking Materials Data
Ian Foster
 
A Biological Internet?: Eywa
A Biological Internet?: EywaA Biological Internet?: Eywa
A Biological Internet?: Eywa
Eugene Siow
 
Astronomical Data Processing on the LSST Scale with Apache Spark
Astronomical Data Processing on the LSST Scale with Apache SparkAstronomical Data Processing on the LSST Scale with Apache Spark
Astronomical Data Processing on the LSST Scale with Apache Spark
Databricks
 
Data Trajectories: tracking the reuse of published data for transitive credi...
Data Trajectories: tracking the reuse of published data for transitive credi...Data Trajectories: tracking the reuse of published data for transitive credi...
Data Trajectories: tracking the reuse of published data for transitive credi...
Paolo Missier
 
Scaling People, Not Just Systems, to Take On Big Data Challenges
Scaling People, Not Just Systems, to Take On Big Data ChallengesScaling People, Not Just Systems, to Take On Big Data Challenges
Scaling People, Not Just Systems, to Take On Big Data Challenges
Matthew Vaughn
 
07 data structures_and_representations
07 data structures_and_representations07 data structures_and_representations
07 data structures_and_representations
Marco Quartulli
 
Reproducible Research and the Cloud
Reproducible Research and the CloudReproducible Research and the Cloud
Reproducible Research and the Cloud
Microsoft Azure for Research
 
Building Reproducible Network Data Analysis / Visualization Workflows
Building Reproducible Network Data Analysis / Visualization WorkflowsBuilding Reproducible Network Data Analysis / Visualization Workflows
Building Reproducible Network Data Analysis / Visualization Workflows
Keiichiro Ono
 
cyREST: Cytoscape as a Service
cyREST: Cytoscape as a ServicecyREST: Cytoscape as a Service
cyREST: Cytoscape as a Service
Keiichiro Ono
 
Data Enthusiasts London: Scalable and Interoperable data services. Applied to...
Data Enthusiasts London: Scalable and Interoperable data services. Applied to...Data Enthusiasts London: Scalable and Interoperable data services. Applied to...
Data Enthusiasts London: Scalable and Interoperable data services. Applied to...
Andy Petrella
 
Network Visualization and Analysis with Cytoscape
Network Visualization and Analysis with CytoscapeNetwork Visualization and Analysis with Cytoscape
Network Visualization and Analysis with Cytoscape
Alexander Pico
 
Curating and Preserving Collaborative Digital Experiments
Curating and Preserving Collaborative Digital ExperimentsCurating and Preserving Collaborative Digital Experiments
Curating and Preserving Collaborative Digital Experiments
Jose Enrique Ruiz
 
Workflow Preservation
Workflow PreservationWorkflow Preservation
Workflow Preservation
Jose Enrique Ruiz
 

Mais conteúdo relacionado

Mais procurados

Virtual Science in the Cloud
Virtual Science in the CloudVirtual Science in the Cloud
Virtual Science in the Cloud
thetfoot
 
Big Data Modeling Challenges and Machine Learning with No Code
Big Data Modeling Challenges and Machine Learning with No CodeBig Data Modeling Challenges and Machine Learning with No Code
Big Data Modeling Challenges and Machine Learning with No Code
Liana Ye
 
A Recommender Story: Improving Backend Data Quality While Reducing Costs
A Recommender Story: Improving Backend Data Quality While Reducing CostsA Recommender Story: Improving Backend Data Quality While Reducing Costs
A Recommender Story: Improving Backend Data Quality While Reducing Costs
Databricks
 
Networking Materials Data
Networking Materials DataNetworking Materials Data
Networking Materials Data
Ian Foster
 
Astronomical Data Processing on the LSST Scale with Apache Spark
Astronomical Data Processing on the LSST Scale with Apache SparkAstronomical Data Processing on the LSST Scale with Apache Spark
Astronomical Data Processing on the LSST Scale with Apache Spark
Databricks
 

Mais procurados (20)

Virtual Science in the Cloud
Virtual Science in the CloudVirtual Science in the Cloud
Virtual Science in the Cloud
 
What to Expect of the LSST Archive: The LSST Science Platform
What to Expect of the LSST Archive: The LSST Science PlatformWhat to Expect of the LSST Archive: The LSST Science Platform
What to Expect of the LSST Archive: The LSST Science Platform
 
LSST Education and Public Outreach (EPO)
LSST Education and Public Outreach (EPO) LSST Education and Public Outreach (EPO)
LSST Education and Public Outreach (EPO)
 
What's New in Cytoscape
What's New in CytoscapeWhat's New in Cytoscape
What's New in Cytoscape
 
Scaling collaborative data science with Globus and Jupyter
Scaling collaborative data science with Globus and JupyterScaling collaborative data science with Globus and Jupyter
Scaling collaborative data science with Globus and Jupyter
 
Big Data Modeling Challenges and Machine Learning with No Code
Big Data Modeling Challenges and Machine Learning with No CodeBig Data Modeling Challenges and Machine Learning with No Code
Big Data Modeling Challenges and Machine Learning with No Code
 
A Recommender Story: Improving Backend Data Quality While Reducing Costs
A Recommender Story: Improving Backend Data Quality While Reducing CostsA Recommender Story: Improving Backend Data Quality While Reducing Costs
A Recommender Story: Improving Backend Data Quality While Reducing Costs
 
Semantically-Enabling the Web of Things: The W3C Semantic Sensor Network Onto...
Semantically-Enabling the Web of Things: The W3C Semantic Sensor Network Onto...Semantically-Enabling the Web of Things: The W3C Semantic Sensor Network Onto...
Semantically-Enabling the Web of Things: The W3C Semantic Sensor Network Onto...
 
DSD-INT 2015 - Data management with open earth datalabs - Gerben de Boer, van...
DSD-INT 2015 - Data management with open earth datalabs - Gerben de Boer, van...DSD-INT 2015 - Data management with open earth datalabs - Gerben de Boer, van...
DSD-INT 2015 - Data management with open earth datalabs - Gerben de Boer, van...
 
Networking Materials Data
Networking Materials DataNetworking Materials Data
Networking Materials Data
 
A Biological Internet?: Eywa
A Biological Internet?: EywaA Biological Internet?: Eywa
A Biological Internet?: Eywa
 
Astronomical Data Processing on the LSST Scale with Apache Spark
Astronomical Data Processing on the LSST Scale with Apache SparkAstronomical Data Processing on the LSST Scale with Apache Spark
Astronomical Data Processing on the LSST Scale with Apache Spark
 
Data Trajectories: tracking the reuse of published data for transitive credi...
Data Trajectories: tracking the reuse of published data for transitive credi...Data Trajectories: tracking the reuse of published data for transitive credi...
Data Trajectories: tracking the reuse of published data for transitive credi...
 
Scaling People, Not Just Systems, to Take On Big Data Challenges
Scaling People, Not Just Systems, to Take On Big Data ChallengesScaling People, Not Just Systems, to Take On Big Data Challenges
Scaling People, Not Just Systems, to Take On Big Data Challenges
 
07 data structures_and_representations
07 data structures_and_representations07 data structures_and_representations
07 data structures_and_representations
 
Reproducible Research and the Cloud
Reproducible Research and the CloudReproducible Research and the Cloud
Reproducible Research and the Cloud
 
Building Reproducible Network Data Analysis / Visualization Workflows
Building Reproducible Network Data Analysis / Visualization WorkflowsBuilding Reproducible Network Data Analysis / Visualization Workflows
Building Reproducible Network Data Analysis / Visualization Workflows
 
cyREST: Cytoscape as a Service
cyREST: Cytoscape as a ServicecyREST: Cytoscape as a Service
cyREST: Cytoscape as a Service
 
Data Enthusiasts London: Scalable and Interoperable data services. Applied to...
Data Enthusiasts London: Scalable and Interoperable data services. Applied to...Data Enthusiasts London: Scalable and Interoperable data services. Applied to...
Data Enthusiasts London: Scalable and Interoperable data services. Applied to...
 
Network Visualization and Analysis with Cytoscape
Network Visualization and Analysis with CytoscapeNetwork Visualization and Analysis with Cytoscape
Network Visualization and Analysis with Cytoscape
 

Semelhante a Workflows in the Virtual Observatory

Curating and Preserving Collaborative Digital Experiments
Curating and Preserving Collaborative Digital ExperimentsCurating and Preserving Collaborative Digital Experiments
Curating and Preserving Collaborative Digital Experiments
Jose Enrique Ruiz
 
Wf4Ever: Workflow Preservation
Wf4Ever: Workflow PreservationWf4Ever: Workflow Preservation
Wf4Ever: Workflow Preservation
Jose Enrique Ruiz
 
Web services based workflows to deal with 3D data
Web services based workflows to deal with 3D dataWeb services based workflows to deal with 3D data
Web services based workflows to deal with 3D data
Jose Enrique Ruiz
 
Collaborative Digital Experiments
Collaborative Digital ExperimentsCollaborative Digital Experiments
Collaborative Digital Experiments
Jose Enrique Ruiz
 
Curation and Characterization of Web Services
Curation and Characterization of Web ServicesCuration and Characterization of Web Services
Curation and Characterization of Web Services
Jose Enrique Ruiz
 
Central Registry for Digitized Objects: Linking Production and Bibliographic ...
Central Registry for Digitized Objects: Linking Production and Bibliographic ...Central Registry for Digitized Objects: Linking Production and Bibliographic ...
Central Registry for Digitized Objects: Linking Production and Bibliographic ...
Ralf Stockmann
 
Best Practices to create High Load Websites
Best Practices to create High Load WebsitesBest Practices to create High Load Websites
Best Practices to create High Load Websites
Manfred Furuholmen
 
BP-8 Global Federation and Search
BP-8 Global Federation and SearchBP-8 Global Federation and Search
BP-8 Global Federation and Search
Alfresco Software
 

Semelhante a Workflows in the Virtual Observatory (20)

Curating and Preserving Collaborative Digital Experiments
Curating and Preserving Collaborative Digital ExperimentsCurating and Preserving Collaborative Digital Experiments
Curating and Preserving Collaborative Digital Experiments
 
Workflow Preservation
Workflow PreservationWorkflow Preservation
Workflow Preservation
 
Wf4Ever: Workflow Preservation
Wf4Ever: Workflow PreservationWf4Ever: Workflow Preservation
Wf4Ever: Workflow Preservation
 
Web services based workflows to deal with 3D data
Web services based workflows to deal with 3D dataWeb services based workflows to deal with 3D data
Web services based workflows to deal with 3D data
 
Collaborative Digital Experiments
Collaborative Digital ExperimentsCollaborative Digital Experiments
Collaborative Digital Experiments
 
Amplexor drupalcamp-gent-2012 - kinepolis platform
Amplexor drupalcamp-gent-2012 - kinepolis platformAmplexor drupalcamp-gent-2012 - kinepolis platform
Amplexor drupalcamp-gent-2012 - kinepolis platform
 
2012 09 aos-workshop-johanneskeizer
2012 09 aos-workshop-johanneskeizer2012 09 aos-workshop-johanneskeizer
2012 09 aos-workshop-johanneskeizer
 
Liferay & Big Data Dev Con 2014
Liferay & Big Data Dev Con 2014Liferay & Big Data Dev Con 2014
Liferay & Big Data Dev Con 2014
 
Networking in the Cloud Age (LISA 2012 Tutorial)
Networking in the Cloud Age (LISA 2012 Tutorial)Networking in the Cloud Age (LISA 2012 Tutorial)
Networking in the Cloud Age (LISA 2012 Tutorial)
 
Snakes on the Web; Developing web applications in python
Snakes on the Web; Developing web applications in pythonSnakes on the Web; Developing web applications in python
Snakes on the Web; Developing web applications in python
 
Making your data work for you: Scratchpads, publishing & the biodiversity dat...
Making your data work for you: Scratchpads, publishing & the biodiversity dat...Making your data work for you: Scratchpads, publishing & the biodiversity dat...
Making your data work for you: Scratchpads, publishing & the biodiversity dat...
 
agINFRA - Elements for an Information Infrastructure in Agricultural Resear...
agINFRA - Elements for an Information Infrastructure in Agricultural Resear...agINFRA - Elements for an Information Infrastructure in Agricultural Resear...
agINFRA - Elements for an Information Infrastructure in Agricultural Resear...
 
Curation and Characterization of Web Services
Curation and Characterization of Web ServicesCuration and Characterization of Web Services
Curation and Characterization of Web Services
 
Central Registry for Digitized Objects: Linking Production and Bibliographic ...
Central Registry for Digitized Objects: Linking Production and Bibliographic ...Central Registry for Digitized Objects: Linking Production and Bibliographic ...
Central Registry for Digitized Objects: Linking Production and Bibliographic ...
 
Best Practices to create High Load Websites
Best Practices to create High Load WebsitesBest Practices to create High Load Websites
Best Practices to create High Load Websites
 
2012 09 caas-ag_infra
2012 09 caas-ag_infra2012 09 caas-ag_infra
2012 09 caas-ag_infra
 
Caliber 2009 Tutorial Mgsree
Caliber 2009 Tutorial MgsreeCaliber 2009 Tutorial Mgsree
Caliber 2009 Tutorial Mgsree
 
BP-8 Global Federation and Search
BP-8 Global Federation and SearchBP-8 Global Federation and Search
BP-8 Global Federation and Search
 
Taverna and myExperiment. SCAPE presentation at a Hack-a-thon
Taverna and myExperiment. SCAPE presentation at a Hack-a-thonTaverna and myExperiment. SCAPE presentation at a Hack-a-thon
Taverna and myExperiment. SCAPE presentation at a Hack-a-thon
 
Context-Aware Access Control for RDF Graph Stores
Context-Aware Access Control for RDF Graph StoresContext-Aware Access Control for RDF Graph Stores
Context-Aware Access Control for RDF Graph Stores
 

Mais de Jose Enrique Ruiz

Jupyter notebooks on steroids
Jupyter notebooks on steroidsJupyter notebooks on steroids
Jupyter notebooks on steroids
Jose Enrique Ruiz
 
Use of CharDM in an archive of velocity cubes
Use of CharDM in an archive of velocity cubesUse of CharDM in an archive of velocity cubes
Use of CharDM in an archive of velocity cubes
Jose Enrique Ruiz
 
El Observatorio Virtual - eCA
El Observatorio Virtual - eCAEl Observatorio Virtual - eCA
El Observatorio Virtual - eCA
Jose Enrique Ruiz
 
Multidimensional Data in the VO
Multidimensional Data in the VOMultidimensional Data in the VO
Multidimensional Data in the VO
Jose Enrique Ruiz
 
B0DEGA 3D VO Archive - IVOA 2010 Fall Interop
B0DEGA 3D VO Archive - IVOA 2010 Fall InteropB0DEGA 3D VO Archive - IVOA 2010 Fall Interop
B0DEGA 3D VO Archive - IVOA 2010 Fall Interop
Jose Enrique Ruiz
 
Wf4Ever: Advanced Workflow Preservation Technologies for Enhanced Science i
Wf4Ever: Advanced Workflow Preservation Technologies for Enhanced Science iWf4Ever: Advanced Workflow Preservation Technologies for Enhanced Science i
Wf4Ever: Advanced Workflow Preservation Technologies for Enhanced Science i
Jose Enrique Ruiz
 

Mais de Jose Enrique Ruiz (9)

Jupyter notebooks on steroids
Jupyter notebooks on steroidsJupyter notebooks on steroids
Jupyter notebooks on steroids
 
Open Science and Executable Papers
Open Science and Executable PapersOpen Science and Executable Papers
Open Science and Executable Papers
 
Digital Science
Digital ScienceDigital Science
Digital Science
 
Use of CharDM in an archive of velocity cubes
Use of CharDM in an archive of velocity cubesUse of CharDM in an archive of velocity cubes
Use of CharDM in an archive of velocity cubes
 
SVO Activities - SEA 2008
SVO Activities - SEA 2008SVO Activities - SEA 2008
SVO Activities - SEA 2008
 
El Observatorio Virtual - eCA
El Observatorio Virtual - eCAEl Observatorio Virtual - eCA
El Observatorio Virtual - eCA
 
Multidimensional Data in the VO
Multidimensional Data in the VOMultidimensional Data in the VO
Multidimensional Data in the VO
 
B0DEGA 3D VO Archive - IVOA 2010 Fall Interop
B0DEGA 3D VO Archive - IVOA 2010 Fall InteropB0DEGA 3D VO Archive - IVOA 2010 Fall Interop
B0DEGA 3D VO Archive - IVOA 2010 Fall Interop
 
Wf4Ever: Advanced Workflow Preservation Technologies for Enhanced Science i
Wf4Ever: Advanced Workflow Preservation Technologies for Enhanced Science iWf4Ever: Advanced Workflow Preservation Technologies for Enhanced Science i
Wf4Ever: Advanced Workflow Preservation Technologies for Enhanced Science i
 

Último

Vishram Singh - Textbook of Anatomy Upper Limb and Thorax.. Volume 1 (1).pdf
Vishram Singh - Textbook of Anatomy Upper Limb and Thorax.. Volume 1 (1).pdfVishram Singh - Textbook of Anatomy Upper Limb and Thorax.. Volume 1 (1).pdf
Vishram Singh - Textbook of Anatomy Upper Limb and Thorax.. Volume 1 (1).pdf
ssuserdda66b
 
1029-Danh muc Sach Giao Khoa khoi 6.pdf
1029-Danh muc Sach Giao Khoa khoi 6.pdf1029-Danh muc Sach Giao Khoa khoi 6.pdf
1029-Danh muc Sach Giao Khoa khoi 6.pdf
QucHHunhnh
 
Making communications land - Are they received and understood as intended? we...
Making communications land - Are they received and understood as intended? we...Making communications land - Are they received and understood as intended? we...
Making communications land - Are they received and understood as intended? we...
Association for Project Management
 

Último (20)

How to Give a Domain for a Field in Odoo 17
How to Give a Domain for a Field in Odoo 17How to Give a Domain for a Field in Odoo 17
How to Give a Domain for a Field in Odoo 17
 
Vishram Singh - Textbook of Anatomy Upper Limb and Thorax.. Volume 1 (1).pdf
Vishram Singh - Textbook of Anatomy Upper Limb and Thorax.. Volume 1 (1).pdfVishram Singh - Textbook of Anatomy Upper Limb and Thorax.. Volume 1 (1).pdf
Vishram Singh - Textbook of Anatomy Upper Limb and Thorax.. Volume 1 (1).pdf
 
SKILL OF INTRODUCING THE LESSON MICRO SKILLS.pptx
SKILL OF INTRODUCING THE LESSON MICRO SKILLS.pptxSKILL OF INTRODUCING THE LESSON MICRO SKILLS.pptx
SKILL OF INTRODUCING THE LESSON MICRO SKILLS.pptx
 
Introduction to Nonprofit Accounting: The Basics
Introduction to Nonprofit Accounting: The BasicsIntroduction to Nonprofit Accounting: The Basics
Introduction to Nonprofit Accounting: The Basics
 
How to Create and Manage Wizard in Odoo 17
How to Create and Manage Wizard in Odoo 17How to Create and Manage Wizard in Odoo 17
How to Create and Manage Wizard in Odoo 17
 
On National Teacher Day, meet the 2024-25 Kenan Fellows
On National Teacher Day, meet the 2024-25 Kenan FellowsOn National Teacher Day, meet the 2024-25 Kenan Fellows
On National Teacher Day, meet the 2024-25 Kenan Fellows
 
UGC NET Paper 1 Mathematical Reasoning & Aptitude.pdf
UGC NET Paper 1 Mathematical Reasoning & Aptitude.pdfUGC NET Paper 1 Mathematical Reasoning & Aptitude.pdf
UGC NET Paper 1 Mathematical Reasoning & Aptitude.pdf
 
Dyslexia AI Workshop for Slideshare.pptx
Dyslexia AI Workshop for Slideshare.pptxDyslexia AI Workshop for Slideshare.pptx
Dyslexia AI Workshop for Slideshare.pptx
 
TỔNG ÔN TẬP THI VÀO LỚP 10 MÔN TIẾNG ANH NĂM HỌC 2023 - 2024 CÓ ĐÁP ÁN (NGỮ Â...
TỔNG ÔN TẬP THI VÀO LỚP 10 MÔN TIẾNG ANH NĂM HỌC 2023 - 2024 CÓ ĐÁP ÁN (NGỮ Â...TỔNG ÔN TẬP THI VÀO LỚP 10 MÔN TIẾNG ANH NĂM HỌC 2023 - 2024 CÓ ĐÁP ÁN (NGỮ Â...
TỔNG ÔN TẬP THI VÀO LỚP 10 MÔN TIẾNG ANH NĂM HỌC 2023 - 2024 CÓ ĐÁP ÁN (NGỮ Â...
 
Towards a code of practice for AI in AT.pptx
Towards a code of practice for AI in AT.pptxTowards a code of practice for AI in AT.pptx
Towards a code of practice for AI in AT.pptx
 
1029-Danh muc Sach Giao Khoa khoi 6.pdf
1029-Danh muc Sach Giao Khoa khoi 6.pdf1029-Danh muc Sach Giao Khoa khoi 6.pdf
1029-Danh muc Sach Giao Khoa khoi 6.pdf
 
Python Notes for mca i year students osmania university.docx
Python Notes for mca i year students osmania university.docxPython Notes for mca i year students osmania university.docx
Python Notes for mca i year students osmania university.docx
 
Single or Multiple melodic lines structure
Single or Multiple melodic lines structureSingle or Multiple melodic lines structure
Single or Multiple melodic lines structure
 
Kodo Millet PPT made by Ghanshyam bairwa college of Agriculture kumher bhara...
Kodo Millet PPT made by Ghanshyam bairwa college of Agriculture kumher bhara...Kodo Millet PPT made by Ghanshyam bairwa college of Agriculture kumher bhara...
Kodo Millet PPT made by Ghanshyam bairwa college of Agriculture kumher bhara...
 
2024-NATIONAL-LEARNING-CAMP-AND-OTHER.pptx
2024-NATIONAL-LEARNING-CAMP-AND-OTHER.pptx2024-NATIONAL-LEARNING-CAMP-AND-OTHER.pptx
2024-NATIONAL-LEARNING-CAMP-AND-OTHER.pptx
 
Key note speaker Neum_Admir Softic_ENG.pdf
Key note speaker Neum_Admir Softic_ENG.pdfKey note speaker Neum_Admir Softic_ENG.pdf
Key note speaker Neum_Admir Softic_ENG.pdf
 
How to Manage Global Discount in Odoo 17 POS
How to Manage Global Discount in Odoo 17 POSHow to Manage Global Discount in Odoo 17 POS
How to Manage Global Discount in Odoo 17 POS
 
Making communications land - Are they received and understood as intended? we...
Making communications land - Are they received and understood as intended? we...Making communications land - Are they received and understood as intended? we...
Making communications land - Are they received and understood as intended? we...
 
Explore beautiful and ugly buildings. Mathematics helps us create beautiful d...
Explore beautiful and ugly buildings. Mathematics helps us create beautiful d...Explore beautiful and ugly buildings. Mathematics helps us create beautiful d...
Explore beautiful and ugly buildings. Mathematics helps us create beautiful d...
 
FSB Advising Checklist - Orientation 2024
FSB Advising Checklist - Orientation 2024FSB Advising Checklist - Orientation 2024
FSB Advising Checklist - Orientation 2024
 

Workflows in the Virtual Observatory

  • 1. Workflows in the VO! Grid and Web Services Session! Jose Enrique Ruiz! IAA-CSIC! ! October 19th 2011! 2011 IVOA Fall Interop Meeting - Pune! 1
  • 2. What are workflows ?! Combination of data and processes into a configurable and structured set of steps that implement semi- automated computational solutions in problem solving! Types of workflows in Astronomy! •  Personal script-based recipes ! Python, IDL, Software..! •  Multi-archive VO recipes! •  Internal group developments ! GRID, Clusters..! •  Processing pipelines! Provide Data, Computing Infrastructure, Tools..! Scientifically exploitable results vs. scientific insight ! Wfs on Easily accessible and reproducible (Shared)! steroids ! 2
  • 3. Tools! Taverna! •  Strongly typed bioinformatics! •  Taverna Engine! •  Taverna Server! •  Taverna Workbench! Kepler! •  Generic Science! •  Workflow System! Triana! •  Local execution! •  Clusters RMI! •  GRID! •  Web Services! 3 !
  • 4. Tools! ESO Reflex! ! Finland’s in-kind contribution to ESO! •  Prototype/feasibility study! •  Initially based on Taverna 1! ! Current implementation based on Kepler! •  Main intent: replace CLI ! (pipeline orchestrator)! ! ! 4
  • 5. Tools! AstroTaverna! ! AstroGrid Development! Prototype, marrying of VO Desktop & Taverna 1! Library of Taverna functions to access VO Desktop’s API! ! Status! Wrapper libraries only for Taverna 1! ! ! 5
  • 6. Tools! Aladin JLOW Plugin! Aladin plugin API permits graphical replacement of Aladin tools! 6
  • 8. Related Initiatives! Wf4Ever! 2011 – 2013 EU funded FP7 STREP Project! Preservation of Workflows! ! •  Preservation of workflows and associated material! •  Archival, classification, indexing in semantic repositories! •  Provide advanced access and recommendation capabilities! •  Collaborative working platform! •  Sharing, re-use, re-purpose! •  Digital Libraries! ! ! ! 8
  • 9. Related Initiatives! Cyber-SKA! Provide infrastructure that will be required to address the needs of future radio telescopes such as the Square Kilometre Array! ! Web based workflow builder ! •  Image segmentation! •  Image mosaicking (Montage)! •  Spatial reprojection! •  Plane extraction from data cubes! 9
  • 10. Related Initiatives! Montage! •  FITS Image Mosaicking! •  Toolkit for Desktops, Clusters and Grids! ! Astro-WISE! •  Distributed data storage and computing infrastructure! •  Track process provenance of final data products! •  Calibration and analysis of images! ! Helio-VO! •  Solar physics Virtual Observatory! •  Enable workflow execution via Taverna Server! ! Workflows VO France! •  Provide use cases mainly oriented VO ! •  AÏDA Workflow System implements FITS validation with CharDM ! 10
  • 11. The upcoming context! The next generation of archives! ! Much wider FoV and spectral coverage! •  Huge sized datasets (~100 TB)! •  Big Data science highly dependent on I/O data rates! •  Subproducts as virtual data generated on-the-fly! Automated surveys! •  Huge amount of tabular data! •  Services for Knowledge Discovery in Databases! ! 11
  • 12. The upcoming context! We are moving into a world where ! •  computing and storage are cheap ! •  data movement is death! Archives should evolve from data providers into virtual data and services providers, where web services may help to solve bandwidth issues.! ! Archives speaking web services! •  Smaller virtual data subproducts! •  Distributed, multi-archive, multi-wavelength astronomy! 12
  • 13. The upcoming context! Web services based workflows as a disruptive working methodology ! •  Reproducible! •  Repeatable results ! •  Encourage best practices! •  Modular nature allows ! •  Re-use! •  Re-purpose! •  Expose ! •  Provenance ! •  Scientific method! •  Formative! •  Foster collaborative work! 13
  • 14. Workflows & IVOA! Distributed data analysis in the VO! •  Panchromatic, multi-archive, multi-facility! •  Executes in the VO Infrastructure! •  Orchestration of simple services! ! Workflows VO Characterization! Present processing pipelines! •  Inputs! •  Produce exploitable data! •  Outputs! •  Provenance modelling! •  Processes! •  Descriptions! •  VO compliant data ! •  Metadata! ! •  Etc..! Data processing from the VO! •  Provide custom re-processing to VO users! •  Virtual data generation through UWS in VOSpace! 14
  • 15. Related activities in the VO! IVOA Working Groups! •  Data Modeling! Characterization, Provenance..! •  Semantics! Ontologies, Vocabularies, Annotations..! •  Data Access Layer! TAP, self-descriptive Protocols..! ! •  Grid and Web Services! IVOA Note! UWS, VOSpace, SSO..! Scientific Workflows in the VO! •  Applications! workflow@ivoa.net! SAMP! •  IG . KDD! Knowledge Discovery and Data Mining! •  IG . Data Curation and Preservation! Persistent Identifiers, Curation of VO Resources..! Wf4Ever Project! 15