Sdmx-EDI and Sdmx-ML
•Transferir como PPT, PDF•
0 gostou•1,237 visualizações
How implement a "data sharing model" and a pull mode using SDMX standards
![[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recomendados
Recomendados
Mais conteúdo relacionado
Semelhante a Sdmx-EDI and Sdmx-ML
Semelhante a Sdmx-EDI and Sdmx-ML (20)
Mais de Vincenzo Patruno
Mais de Vincenzo Patruno (20)
Último
Último (20)
Sdmx-EDI and Sdmx-ML
- 1. IPA 2007 - Tirana - INSTAT SDMX-EDI SDMX-ML Skopje 15 Jan 2009
- 7. SDMX SDMX is primarily focused on the exchange and dissemination of statistical data and metadata. We have normally two different approach to exchange data: PUSH and PULL
- 8. SDMX PUSH mode means that the data provider takes action to send the data to the party collecting the data. PULL mode implies that the data provider makes the data available via the Internet. The data consumer then fetches the data on his own initiative.
- 9. SDMX SDMX promotes a “ data sharing ” model to facilitate low-cost, high-quality statistical data and metadata exchange. Data Providers publishes the availability of data/metadata to Data Consumers and the latter are responsible for fetching the data/metadata at will. .
- 11. NSI 1 2 3
- 13. An easy way to understand the SDMX-IM 10369
- 17. Data Set
- 20. Structural Definitions Topic A Brady Bonds B Bank Loans C Debt Securities Country AR Argentina MX Mexico SA South Africa Stock/Flow 1 Stock 2 Flow Concepts TOPIC COUNTRY FLOW
- 21. Data Makes Sense SA,B,1,1999-06-30=16547 16457
- 26. Data Structure Definition ………… ..………. Structure ………… ..... ComponentList ……………………… .... Component
- 27. SDMX V1 and V2
- 28. STS Sample Dataset EXAMPLE DATASET1
- 29. STS Sample Dataset Dimensions Measure Attributes Dimensions
- 30. STS DSD components Dataflow : STSRTD_TURN_M
- 31. Demography Sample Dataset EXAMPLE DATASET2
- 32. Demography Sample Dimensions Attributes Measures
- 33. Demography DSD components Dataflow : DEMOGRAPHY_RQ
- 34. IPA 2007 - Tirana - INSTAT Data Set Identifier Variables Form Description STSIND_PROD (_M, _Q) 110 I Production in industry STSIND_TURN (_M, _Q) 120, 121, 122 N, I Turnover in industry, total, domestic and non-domestic (total, Euro-zone, non-Euro-zone) STSIND_ORD (_M, _Q) 130, 131, 132 N, I New orders received in industry, total, domestic and non-domestic (total, Euro-zone, non-Euro-zone) STSIND_EMPL (_M, _Q) 210 N, I Number of persons employed, Number of employees, in industry STSIND_HOUR (_M, _Q) 220 N, I Hours worked in industry STSIND_EARN (_M, _Q) 230 N, I Gross wages and salaries in industry STSIND_PRIC (_M, _Q) 310, 311, 312, 340 I Output prices in industry, total, domestic market, non-domestic market (total, Euro-zone, non Euro-zone), import prices (total, Euro-zone, non-Euro-zone) STSCONS_PROD (_M, _Q) 110, 115, 116 I Production in construction, total, building construction, civil engineering STSCONS_ORD (_M, _Q) 130, 135, 136 N, I New orders received in construction, total, building construction and civil engineering STSCONS_EMPL (_M, _Q) 210, 211 N, I Number of persons employed, Number of employees, in construction STSCONS_HOUR (_M, _Q) 220 N, I Hours worked in construction STSCONS_EARN (_M, _Q) 230 N, I Gross wages and salaries in construction STSCONS_PRIC (_M, _Q) 310, 320, 321, 322 I Output prices in construction, construction costs, material costs, labour costs STSCONS_PERM (_M, _Q) 411, 412 N, I Building permits, number of dwellings or square metres of useful floor area STSRTD_TURN (_M, _Q) 120, 123 N, I Turnover in retail trade, value or deflated STSRTD_EMPL (_M, _Q) 210, 211 N, I Number of persons employed, Number of employees, in retail trade STSSERV_TURN (_M, _Q) 120, 123 N, I Turnover in repair and other services, value or deflated STSSERV_PRIC (_M, _Q) 310 I Outut prices in other services STSSERV_EMPL (_M, _Q) 210, 211 N, I Number of persons employed, Number of employees, in repair and other services STSSERV_CAR (_M, _Q) Number of car registrations STSOTHER_OTH (_M, _Q) Any other indicator not mentioned in the list above
- 35. IPA 2007 - Tirana - INSTAT Concept Mnemonic Concept Name Format Description Code list ADJUSTMENT Adjustment AN1 Code defining the adjustment of data such as working day or seasonally adjusted, etc. CL_ADJUSTMENT FREQ Frequency AN1 Frequency of the series (e.g. A, Q, M). CL_FREQ OBS_CONF Confidentiality flag AN1 Confidentiality status of the observation CL_OBS_CONF OBS_PRE_BREAK Pre-break observation value AN…15 Observation value if the reason of the "break" did not show up. [Conditional] OBS_STATUS Status flag AN1 S tatus of the observation, such as normal, estimated or provisional CL_OBS_STATUS OBS_VALUE Value AN…15 The value of the index. ORGANISATION Organisation AN3 Reporting/sending or receiving organisation used in the message administration section. CL_ORGANISATION REF_AREA Reference area AN2 Reporting Country in ISO code (The country, or geographical/political group of countries that the measured economic phenomenon relates to) CL_AREA_EE STS_ACTIVITY Economic Activity code AN6 NACE Rev. 1.1 & special STS aggregates CL_STS_ACTIVITY STS_BASE_YEAR Series variation in short-term stats context AN4 Concept to distinguish series variations in a short-term stats context CL_STS_BASE_YEAR STS_INDICATOR STS Indicator AN4 Type of indicator, such as production, turnover, etc. CL_STS_INDICATOR STS_INSTITUTION Institution originating STS dataflow AN1 Institution originating STS dataflow CL_STS_INSTITUTION TIME_FORMAT Time Format Code AN3 Technical use in message. TIME_PERIOD Time Period AN…35 The time period of the data.
- 36. IPA 2007 - Tirana - INSTAT Code List Mnemonic Code List Name Format CL_ADJUSTMENT Adjustment code AN1 CL_AREA_EE Country code AN2 CL_FREQ Frequency code AN1 CL_OBS_CONF Confidentiality flag AN1 CL_OBS_STATUS Observation status flag AN1 CL_ORGANISATION Organisation code list AN3 CL_STS_ACTIVITY STS Economic Activity code list AN6 CL_STS_BASE_YEAR Suffix in short-term stats context code list AN4 CL_STS_INDICATOR Indicators index code AN4 CL_STS_INSTITUTION Institution originating STS dataflow code list AN1
- 37. IPA 2007 - Tirana - INSTAT Value Description Variable PROD Production 110, 115, 116 TOVT Turnover (total turnover, non-deflated) 120 TOVD Turnover, domestic market (non-deflated) 121 TOVE Turnover, non-domestic market (non-deflated) 122 TOVV Turnover deflated (volume of sales) 123 TOVX Turnover, non-domestic market (non-deflated) (non-Euro-zone) 122 TOVZ Turnover, non-domestic market (non-deflated) (Euro-zone) 122 DEFL Deflator of sales 330 ORDT New orders received (total) 130, 135, 136 ORDD New orders received, domestic market 131 ORDE New orders received, non-domestic market 132 ORDX New orders received, non-domestic market (non-Euro-zone) 132 PRON Output prices for industry and services (total) 310 PRIN Output prices, domestic market 311 PREN Output prices, non-domestic market (can be approximated by unit value index , variable 313) 312, 313 PREX Output prices, non-domestic market (non-Euro-zone) 312 PREZ Output prices, non-domestic market (Euro-zone) 312 IMPR Import prices (total) 340 IMPX Import prices (non-Euro-zone) 340 IMPZ Import prices (Euro-zone) 340 EMPL Number of persons employed (can be approximated by number of employees, variable 211) 210, 211 HOWK Hours worked 220 WAGE Gross wages and salaries 230 PNUM Building permits, number of dwellings 411 PSQM Building permits: square metres of useful floor area 412 CSTI Construction costs (total) 320 CSTM Construction costs, material costs 321 CSTL Construction costs, labour costs 322 CSTO Output prices for construction (approximation for construction costs, variable 320) 310 CREG Car registrations (not in STS Regulation)
- 38. From SDMX-IM to messages SDMX-IM SDMX-EDI SDMX-ML
- 40. UNA:+.? ' UNB+UNOC:3+EUROSTAT+Unknown+060627:0000+IREF000001++GESMES/TS++++1' UNH+MREF000001+GESMES:2:1:E6' BGM+74' NAD+Z02+EUROSTAT NAD+MR+Unknown' NAD+MS+EUROSTAT' CTA+CC+:V. Patruno' DSI+SODI_IPI_PROD_M' STS+3+7' DTM+242:200601010000:203' DTM+Z02:200501200503:710' IDE+5+STS' GIS+AR3' GIS+1:::-' ARR++M:GR:W:PROD:NS0020:1:2000:200501200503:710:111.11:A:F:+222.22:A:F:+333.33:A:F:+444.44:A:F:+555.55:A:F:+666.66:A:F:+777.77:A:F:+888.88:A:F:+99.99:A:F:+123.45:A:F:+212.21:A:F:+234.56:A:F:' FNS+Attributes:10' REL+Z01+4' ARR+7+M:GR:W:PROD:NS0020:1:2000' IDE+Z10+COLLECTION' CDV+A' IDE+Z10+AVAILABILITY' CDV+A' UNT+26+MREF000001' UNZ+1+IREF000001' SDMX-EDI
- 41. SDMX-ML XML format for the exchange of SDMX-structured data and metadata.
- 42. <?xml version="1.0" encoding="UTF-8"?> <!-- Created with SDMX Converter v2.1 --> <CompactData xmlns="http://www.SDMX.org/resources/SDMXML/schemas/v2_0/message" xmlns:sts="urn:sdmx:org.sdmx.infomodel.keyfamily.KeyFamily=ESTAT:STS:compact" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.SDMX.org/resources/SDMXML/schemas/v2_0/message SDMXMessage.xsd urn:sdmx:org.sdmx.infomodel.keyfamily.KeyFamily=ESTAT:STS:compact ESTAT_STS_Compact.xsd"> <Header> <ID>SODI_IPI_PROD_M</ID> <Test>true</Test> <Name xml:lang="en">SDMX Tutorial Message</Name> <Prepared>2006-06-27T00:00:00</Prepared> <Sender id="EUROSTAT"> <Contact> <Name xml:lang="en">V. Patruno</Name> <Department xml:lang="en">IT Dept</Department> <Role xml:lang="en">Maintainer</Role> <Email>something@gmail.com</Email> </Contact> </Sender> <DataSetAgency>EUSTAT</DataSetAgency> <DataSetID>SODI_IPI_PROD_M_02</DataSetID> <DataSetAction>Append</DataSetAction> <Extracted>2006-01-01T00:00:00</Extracted> <ReportingBegin>2005-01-01T00:00:00</ReportingBegin> <ReportingEnd>2005-03-31T00:00:00</ReportingEnd> </Header> <sts:DataSet> <sts:SiblingGroup REF_AREA="GR" ADJUSTMENT="W" STS_INDICATOR="PROD" STS_ACTIVITY="NS0020" STS_INSTITUTION="1" STS_BASE_YEAR="2000" UNIT="PC" UNIT_MULT="0" DECIMALS="2" TITLE_COMPL="Elements of the full national etc."/> <sts:Series FREQ="M" REF_AREA="GR" ADJUSTMENT="W" STS_INDICATOR="PROD" STS_ACTIVITY="NS0020" STS_INSTITUTION="1" STS_BASE_YEAR="2000" COLLECTION="A" AVAILABILITY="A" TIME_FORMAT="P1M"> <sts:Obs TIME_PERIOD="2005-01" OBS_VALUE="111.11" OBS_STATUS="A" OBS_CONF="F"/> <sts:Obs TIME_PERIOD="2005-02" OBS_VALUE="222.22" OBS_STATUS="A" OBS_CONF="F"/> <sts:Obs TIME_PERIOD="2005-03" OBS_VALUE="333.33" OBS_STATUS="A" OBS_CONF="F"/> <sts:Obs TIME_PERIOD="2005-04" OBS_VALUE="444.44" OBS_STATUS="A" OBS_CONF="F"/> <sts:Obs TIME_PERIOD="2005-05" OBS_VALUE="555.55" OBS_STATUS="A" OBS_CONF="F"/> <sts:Obs TIME_PERIOD="2005-06" OBS_VALUE="666.66" OBS_STATUS="A" OBS_CONF="F"/> <sts:Obs TIME_PERIOD="2005-07" OBS_VALUE="777.77" OBS_STATUS="A" OBS_CONF="F"/> <sts:Obs TIME_PERIOD="2005-08" OBS_VALUE="888.88" OBS_STATUS="A" OBS_CONF="F"/> <sts:Obs TIME_PERIOD="2005-09" OBS_VALUE="99.99" OBS_STATUS="A" OBS_CONF="F"/> <sts:Obs TIME_PERIOD="2005-10" OBS_VALUE="123.45" OBS_STATUS="A" OBS_CONF="F"/> <sts:Obs TIME_PERIOD="2005-11" OBS_VALUE="212.21" OBS_STATUS="A" OBS_CONF="F"/> <sts:Obs TIME_PERIOD="2005-12" OBS_VALUE="234.56" OBS_STATUS="A" OBS_CONF="F"/> </sts:Series> </sts:DataSet> </CompactData>
- 46. SDMX-ML: Six standard messages Fixed To query a database to obtain an SDMX-ML message as the result Query message 6 Derived from data structure definition message Exchange of many observation types in a data structure definition-dependent form Cross-sectional Data Message 5 Derived from data structure definition message For schema-based functions, such as validation, in a data structure definition-dependent form Utility Data Message 4 Derived from data structure definition message Exchange of large data sets in a data structure definition-dependent form Compact Data Message 3 Fixed Conveys data in a form independent of a data structure definition. It is designed for data provision on websites and in any scenario where applications receiving the data may not have detailed understanding of the data set's structure before they obtain the data set itself. Generic Data Message 2 Fixed Contains a data structure definition Structure Definition Message 1 Schema file Short description Name of message
- 47. Cross-Sectional Data Set <demo:DataSet REV_NUM = "1" TAB_NUM = "RQFI05V1" > < demo:Group COUNTRY = "FI" FREQ = "A" TIME = "2005" TIME_FORMAT = "P1Y" > < demo:Section DECI = "0" UNIT = "PERS" UNIT_MULT = "0" > < demo:ADJT OBS_STATUS = "P" SEX = "F" value = "35" /> < demo:DEATHST OBS_STATUS = "P" SEX = "F" value = "23871" /> < demo:LBIRTHST OBS_STATUS = "P" SEX = "F" value = "28345" /> < demo:NETMT OBS_STATUS = "P" SEX = "F" value = "4187" /> < demo:PJAN1T OBS_STATUS = "P" SEX = "F" value = "2683230" /> < demo:PJANT OBS_STATUS = "P" SEX = "F" value = "2674534" /> < demo:ADJT OBS_STATUS = "P" SEX = "M" value = "131" /> < demo:DEATHST OBS_STATUS = "P" SEX = "M" value = "24057" /> < demo:LBIRTHST OBS_STATUS = "P" SEX = "M" value = "29400" /> < demo:NETMT OBS_STATUS = "P" SEX = "M" value = "4799" /> < demo:PJAN1T OBS_STATUS = "P" SEX = "M" value = "2572350" /> < demo:PJANT OBS_STATUS = "P" SEX = "M" value = "2562077" /> < demo:ADJT OBS_STATUS = "P" SEX = "T" value = "166" /> < demo:DEATHST OBS_STATUS = "P" SEX = "T" value = "47928" /> < demo:LBIRTHST OBS_STATUS = "P" SEX = "T" value = "57745" /> < demo:NETMT OBS_STATUS = "P" SEX = "T" value = "8986" /> < demo:PJAN1T OBS_STATUS = "P" SEX = "T" value = "5255580" /> < demo:PJANT OBS_STATUS = "P" SEX = "T" value = "5236611" /> < /demo:Section> < demo:Section DECI = "0" UNIT = "PURE_NUMB" UNIT_MULT = "0" > < demo:DIV OBS_STATUS = "P" SEX = "T" value = "13383" /> < demo:MAR OBS_STATUS = "P" SEX = "T" value = "29283" /> < /demo:Section> < demo:Section DECI = "3" UNIT = "PURE_NUMB" UNIT_MULT = "0" > < demo:TFRNSI SEX = "T" value = "1800" /> < /demo:Section> < /demo:Group> </demo:DataSet>
- 48. SDMX-ML “Model-Driven” XML Approach
- 53. Query Message
- 54. Query SDMX Data SDMX-ML RSS WS NSI
- 55. sodi.istat.it con.istat.it Query SDMX Dati (SDMX-ML) RSS WS demo.istat.it
- 56. DB WS demogr DEMO SODI RSS script
- 57. sodi.istat.it con.istat.it Query SDMX Dati (SDMX-ML) RSS WS demogr
- 58. Thank you for your attention Vincenzo Patruno: [email_address]
Notas do Editor
- <pagebreak> This diagram depicts one of the most important SDMX Artefacts, the Data Structure Definition (aka Key Family). NOTE: The three classes (DataSet, DataflowDefintion, Category) above the DSD class are not part of the Data Structure Definition, but are included in this diagram in order to show how the DSD is used by Dataflows that define DataSets and are linked to Categories within a Category Scheme. Three conceptual levels can be identified within the DSD diagram. The first level is the Structure level where the DSD is identified (id, version and other attributes not shown in this diagram). The second level comprises ComponentLists. These are conceptual groupings of components (i.e. Key, Groups). The third level includes the Components used by the DSD in order to define the structure of DataSet(s). Moreover, Components reference Concepts residing in ConceptSchemes and may have specific Roles within the DSD. Finally, a Dimension may be a MeasureType dimension, thus defining a Cross-Sectional Measure (XSMeasure class) for every Code included in the Codelist related to this Dimension.
- <pagebreak> The SDMX standard specifies a single Information Model in order to describe what and how data and metadata can be exchanged in the context of SDMX. Based on this Information Model, SDMX defines two ways of representing its messages. Two different formats are available for expressing SDMX messages. The first format is SDMX-EDI and is equivalent to GESMES/TS. This format is based on an EDIFACT syntax and is Time-Series oriented. This means that only one type of observation throughout time can be carried in a single DataSet. Moreover, the DataSet messages have only one format. This format covers a subset of the SDMX-IM. For example, non time-series messages or reference metadata messages are not supported. The second format is SDMX-ML. It is an XML format that covers the whole SDMX-IM. The SDMX-ML format supports four different (although equivalent) formats, for data messages, in order to serve different purposes. It also supports reference metadata messages as well as messages for querying SDMX Web Services and Registry Interface messages. Two simple “rules” in order to go from the SDMX-IM to the SDMX-ML implementation are that: (concrete) classes become XML elements and their attributes become XML attributes. Of course, there are exceptions in these “rules”. In the SDMX-ML implementation there are XML elements that do not correspond directly to classes from the SDMX-IM and vice versa.
- <pagebreak> The Cross-Sectional format is the only one capable of expressing non-time series DSDs. In case a TimeDimension is defined in the DSD, the Cross-Sectional measure is equivalent to the rest of the formats. In the namespace declarations of this message, a DSD specific XSD should be included in order to enable syntactical validation of the message. This XSD file can be downloaded from Eurostat’s SDMX Registry (https://webgate.ec.europa.eu/sdmxregistry). The layout of the Cross-Sectional type adopts the nesting used in the Generic format but expressing SDMX Artefacts like the Compact format. A major difference in this format is that the elements have different semantic than the previous formats. Although four levels are also defined in this format, only the <DataSet> element is used in the same way. The <Group> element is independent of the Groups included in the DSD. The <Section> element is different from the <Series> element in the sense that it does not include a time-series but a vertical slice across a time-series. Finally, the observation values are now included in DSD specific elements that correspond to the CrossSectionalMeasures defined in the DSD. The SDMX Dimensions and Attributes can be attached at any of the four aforementioned levels as defined in the DSD. Moreover, in the DSD more than one possible levels can be defined per SDMX Component. Of course, only one instance of each SDMX Component should exist in each combination of the four “level” elements. Apart from the capability of reporting data without Time, the Cross-Sectional format is also a useful message for reporting more than one measures at a single DataSet. Thus, in some cases the produced file is even smaller in size than the equivalent Compact message.
- <pagebreak> The SDMX standard, in its 5 th document of specifications (http://www.sdmx.org/docs/2_0/SDMX_2_0%20SECTION_05_RegistrySpecification.pdf), gives details on the interfaces that an SDMX compliant Registry should implement. Based on these specifications, Eurostat’s SDMX Registry has been developed and is now deployed in the European Commission’s production environment ( https://webgate.ec.europa.eu/sdmxregistry/ ). Eurostat’s SDMX Registry has been developed in order to be used a central repository for: Structural metadata: Code Lists, Concept Schemes, Data Structure Definitions, Metadata Structure Definitions, Category Schemes, Organisation Schemes, Hierarchical Code Lists Provisioning metadata: Data flows, Metadata flows, Provision Agreements The major interface of Eurostat’s SDMX Registry is a Web Service implementing the SDMX Registry Interface messages. These are SDMX-ML messages and are specified within the standard and by the following XSD: http://www.sdmx.org/docs/2_0/SDMXRegistry.xsd A Graphical User Interface (GUI) has been also implemented in order to enable human interaction of the World Wide Web. The user-authentication is realized using CIRCA accounts. A standalone tool implementing almost all functionality of the SDMX registry is the DSW (already presented).