How implement a "data sharing model" and a pull mode using SDMX standards

1. IPA 2007 - Tirana - INSTAT SDMX-EDI SDMX-ML Skopje 15 Jan 2009

7. SDMX SDMX is primarily focused on the exchange and dissemination of statistical data and metadata. We have normally two different approach to exchange data: PUSH and PULL

8. SDMX PUSH mode means that the data provider takes action to send the data to the party collecting the data. PULL mode implies that the data provider makes the data available via the Internet. The data consumer then fetches the data on his own initiative.

9. SDMX SDMX promotes a “ data sharing ” model to facilitate low-cost, high-quality statistical data and metadata exchange. Data Providers publishes the availability of data/metadata to Data Consumers and the latter are responsible for fetching the data/metadata at will. .

10. Data-Sharing Exchange YOU

11. NSI 1 2 3

13. An easy way to understand the SDMX-IM 10369

17. Data Set

18. Data Set: Structure

20. Structural Definitions Topic A Brady Bonds B Bank Loans C Debt Securities Country AR Argentina MX Mexico SA South Africa Stock/Flow 1 Stock 2 Flow Concepts TOPIC COUNTRY FLOW

21. Data Makes Sense SA,B,1,1999-06-30=16547 16457

26. Data Structure Definition ………… ..………. Structure ………… ..... ComponentList ……………………… .... Component

27. SDMX V1 and V2

28. STS Sample Dataset EXAMPLE DATASET1

29. STS Sample Dataset Dimensions Measure Attributes Dimensions

30. STS DSD components Dataflow : STSRTD_TURN_M

31. Demography Sample Dataset EXAMPLE DATASET2

32. Demography Sample Dimensions Attributes Measures

33. Demography DSD components Dataflow : DEMOGRAPHY_RQ

34. IPA 2007 - Tirana - INSTAT Data Set Identifier Variables Form Description STSIND_PROD (_M, _Q) 110 I Production in industry STSIND_TURN (_M, _Q) 120, 121, 122 N, I Turnover in industry, total, domestic and non-domestic (total, Euro-zone, non-Euro-zone) STSIND_ORD (_M, _Q) 130, 131, 132 N, I New orders received in industry, total, domestic and non-domestic (total, Euro-zone, non-Euro-zone) STSIND_EMPL (_M, _Q) 210 N, I Number of persons employed, Number of employees, in industry STSIND_HOUR (_M, _Q) 220 N, I Hours worked in industry STSIND_EARN (_M, _Q) 230 N, I Gross wages and salaries in industry STSIND_PRIC (_M, _Q) 310, 311, 312, 340 I Output prices in industry, total, domestic market, non-domestic market (total, Euro-zone, non Euro-zone), import prices (total, Euro-zone, non-Euro-zone) STSCONS_PROD (_M, _Q) 110, 115, 116 I Production in construction, total, building construction, civil engineering STSCONS_ORD (_M, _Q) 130, 135, 136 N, I New orders received in construction, total, building construction and civil engineering STSCONS_EMPL (_M, _Q) 210, 211 N, I Number of persons employed, Number of employees, in construction STSCONS_HOUR (_M, _Q) 220 N, I Hours worked in construction STSCONS_EARN (_M, _Q) 230 N, I Gross wages and salaries in construction STSCONS_PRIC (_M, _Q) 310, 320, 321, 322 I Output prices in construction, construction costs, material costs, labour costs STSCONS_PERM (_M, _Q) 411, 412 N, I Building permits, number of dwellings or square metres of useful floor area STSRTD_TURN (_M, _Q) 120, 123 N, I Turnover in retail trade, value or deflated STSRTD_EMPL (_M, _Q) 210, 211 N, I Number of persons employed, Number of employees, in retail trade STSSERV_TURN (_M, _Q) 120, 123 N, I Turnover in repair and other services, value or deflated STSSERV_PRIC (_M, _Q) 310 I Outut prices in other services STSSERV_EMPL (_M, _Q) 210, 211 N, I Number of persons employed, Number of employees, in repair and other services STSSERV_CAR (_M, _Q) Number of car registrations STSOTHER_OTH (_M, _Q) Any other indicator not mentioned in the list above

35. IPA 2007 - Tirana - INSTAT Concept Mnemonic Concept Name Format Description Code list ADJUSTMENT Adjustment AN1 Code defining the adjustment of data such as working day or seasonally adjusted, etc. CL_ADJUSTMENT FREQ Frequency AN1 Frequency of the series (e.g. A, Q, M). CL_FREQ OBS_CONF Confidentiality flag AN1 Confidentiality status of the observation CL_OBS_CONF OBS_PRE_BREAK Pre-break observation value AN…15 Observation value if the reason of the "break" did not show up. [Conditional] OBS_STATUS Status flag AN1 S tatus of the observation, such as normal, estimated or provisional CL_OBS_STATUS OBS_VALUE Value AN…15 The value of the index. ORGANISATION Organisation AN3 Reporting/sending or receiving organisation used in the message administration section. CL_ORGANISATION REF_AREA Reference area AN2 Reporting Country in ISO code (The country, or geographical/political group of countries that the measured economic phenomenon relates to) CL_AREA_EE STS_ACTIVITY Economic Activity code AN6 NACE Rev. 1.1 & special STS aggregates CL_STS_ACTIVITY STS_BASE_YEAR Series variation in short-term stats context AN4 Concept to distinguish series variations in a short-term stats context CL_STS_BASE_YEAR STS_INDICATOR STS Indicator AN4 Type of indicator, such as production, turnover, etc. CL_STS_INDICATOR STS_INSTITUTION Institution originating STS dataflow AN1 Institution originating STS dataflow CL_STS_INSTITUTION TIME_FORMAT Time Format Code AN3 Technical use in message. TIME_PERIOD Time Period AN…35 The time period of the data.

36. IPA 2007 - Tirana - INSTAT Code List Mnemonic Code List Name Format CL_ADJUSTMENT Adjustment code AN1 CL_AREA_EE Country code AN2 CL_FREQ Frequency code AN1 CL_OBS_CONF Confidentiality flag AN1 CL_OBS_STATUS Observation status flag AN1 CL_ORGANISATION Organisation code list AN3 CL_STS_ACTIVITY STS Economic Activity code list AN6 CL_STS_BASE_YEAR Suffix in short-term stats context code list AN4 CL_STS_INDICATOR Indicators index code AN4 CL_STS_INSTITUTION Institution originating STS dataflow code list AN1

37. IPA 2007 - Tirana - INSTAT Value Description Variable PROD Production 110, 115, 116 TOVT Turnover (total turnover, non-deflated) 120 TOVD Turnover, domestic market (non-deflated) 121 TOVE Turnover, non-domestic market (non-deflated) 122 TOVV Turnover deflated (volume of sales) 123 TOVX Turnover, non-domestic market (non-deflated) (non-Euro-zone) 122 TOVZ Turnover, non-domestic market (non-deflated) (Euro-zone) 122 DEFL Deflator of sales 330 ORDT New orders received (total) 130, 135, 136 ORDD New orders received, domestic market 131 ORDE New orders received, non-domestic market 132 ORDX New orders received, non-domestic market (non-Euro-zone) 132 PRON Output prices for industry and services (total) 310 PRIN Output prices, domestic market 311 PREN Output prices, non-domestic market (can be approximated by unit value index , variable 313) 312, 313 PREX Output prices, non-domestic market (non-Euro-zone) 312 PREZ Output prices, non-domestic market (Euro-zone) 312 IMPR Import prices (total) 340 IMPX Import prices (non-Euro-zone) 340 IMPZ Import prices (Euro-zone) 340 EMPL Number of persons employed (can be approximated by number of employees, variable 211) 210, 211 HOWK Hours worked 220 WAGE Gross wages and salaries 230 PNUM Building permits, number of dwellings 411 PSQM Building permits: square metres of useful floor area 412 CSTI Construction costs (total) 320 CSTM Construction costs, material costs 321 CSTL Construction costs, labour costs 322 CSTO Output prices for construction (approximation for construction costs, variable 320) 310 CREG Car registrations (not in STS Regulation)

38. From SDMX-IM to messages SDMX-IM SDMX-EDI SDMX-ML

40. UNA:+.? ' UNB+UNOC:3+EUROSTAT+Unknown+060627:0000+IREF000001++GESMES/TS++++1' UNH+MREF000001+GESMES:2:1:E6' BGM+74' NAD+Z02+EUROSTAT NAD+MR+Unknown' NAD+MS+EUROSTAT' CTA+CC+:V. Patruno' DSI+SODI_IPI_PROD_M' STS+3+7' DTM+242:200601010000:203' DTM+Z02:200501200503:710' IDE+5+STS' GIS+AR3' GIS+1:::-' ARR++M:GR:W:PROD:NS0020:1:2000:200501200503:710:111.11:A:F:+222.22:A:F:+333.33:A:F:+444.44:A:F:+555.55:A:F:+666.66:A:F:+777.77:A:F:+888.88:A:F:+99.99:A:F:+123.45:A:F:+212.21:A:F:+234.56:A:F:' FNS+Attributes:10' REL+Z01+4' ARR+7+M:GR:W:PROD:NS0020:1:2000' IDE+Z10+COLLECTION' CDV+A' IDE+Z10+AVAILABILITY' CDV+A' UNT+26+MREF000001' UNZ+1+IREF000001' SDMX-EDI

41. SDMX-ML XML format for the exchange of SDMX-structured data and metadata.

42. <?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?> <!-- Created with SDMX Converter v2.1 --> <CompactData xmlns=&quot;http://www.SDMX.org/resources/SDMXML/schemas/v2_0/message&quot; xmlns:sts=&quot;urn:sdmx:org.sdmx.infomodel.keyfamily.KeyFamily=ESTAT:STS:compact&quot; xmlns:xsi=&quot;http://www.w3.org/2001/XMLSchema-instance&quot; xsi:schemaLocation=&quot;http://www.SDMX.org/resources/SDMXML/schemas/v2_0/message SDMXMessage.xsd urn:sdmx:org.sdmx.infomodel.keyfamily.KeyFamily=ESTAT:STS:compact ESTAT_STS_Compact.xsd&quot;> <Header> <ID>SODI_IPI_PROD_M</ID> <Test>true</Test> <Name xml:lang=&quot;en&quot;>SDMX Tutorial Message</Name> <Prepared>2006-06-27T00:00:00</Prepared> <Sender id=&quot;EUROSTAT&quot;> <Contact> <Name xml:lang=&quot;en&quot;>V. Patruno</Name> <Department xml:lang=&quot;en&quot;>IT Dept</Department> <Role xml:lang=&quot;en&quot;>Maintainer</Role> <Email>something@gmail.com</Email> </Contact> </Sender> <DataSetAgency>EUSTAT</DataSetAgency> <DataSetID>SODI_IPI_PROD_M_02</DataSetID> <DataSetAction>Append</DataSetAction> <Extracted>2006-01-01T00:00:00</Extracted> <ReportingBegin>2005-01-01T00:00:00</ReportingBegin> <ReportingEnd>2005-03-31T00:00:00</ReportingEnd> </Header> <sts:DataSet> <sts:SiblingGroup REF_AREA=&quot;GR&quot; ADJUSTMENT=&quot;W&quot; STS_INDICATOR=&quot;PROD&quot; STS_ACTIVITY=&quot;NS0020&quot; STS_INSTITUTION=&quot;1&quot; STS_BASE_YEAR=&quot;2000&quot; UNIT=&quot;PC&quot; UNIT_MULT=&quot;0&quot; DECIMALS=&quot;2&quot; TITLE_COMPL=&quot;Elements of the full national etc.&quot;/> <sts:Series FREQ=&quot;M&quot; REF_AREA=&quot;GR&quot; ADJUSTMENT=&quot;W&quot; STS_INDICATOR=&quot;PROD&quot; STS_ACTIVITY=&quot;NS0020&quot; STS_INSTITUTION=&quot;1&quot; STS_BASE_YEAR=&quot;2000&quot; COLLECTION=&quot;A&quot; AVAILABILITY=&quot;A&quot; TIME_FORMAT=&quot;P1M&quot;> <sts:Obs TIME_PERIOD=&quot;2005-01&quot; OBS_VALUE=&quot;111.11&quot; OBS_STATUS=&quot;A&quot; OBS_CONF=&quot;F&quot;/> <sts:Obs TIME_PERIOD=&quot;2005-02&quot; OBS_VALUE=&quot;222.22&quot; OBS_STATUS=&quot;A&quot; OBS_CONF=&quot;F&quot;/> <sts:Obs TIME_PERIOD=&quot;2005-03&quot; OBS_VALUE=&quot;333.33&quot; OBS_STATUS=&quot;A&quot; OBS_CONF=&quot;F&quot;/> <sts:Obs TIME_PERIOD=&quot;2005-04&quot; OBS_VALUE=&quot;444.44&quot; OBS_STATUS=&quot;A&quot; OBS_CONF=&quot;F&quot;/> <sts:Obs TIME_PERIOD=&quot;2005-05&quot; OBS_VALUE=&quot;555.55&quot; OBS_STATUS=&quot;A&quot; OBS_CONF=&quot;F&quot;/> <sts:Obs TIME_PERIOD=&quot;2005-06&quot; OBS_VALUE=&quot;666.66&quot; OBS_STATUS=&quot;A&quot; OBS_CONF=&quot;F&quot;/> <sts:Obs TIME_PERIOD=&quot;2005-07&quot; OBS_VALUE=&quot;777.77&quot; OBS_STATUS=&quot;A&quot; OBS_CONF=&quot;F&quot;/> <sts:Obs TIME_PERIOD=&quot;2005-08&quot; OBS_VALUE=&quot;888.88&quot; OBS_STATUS=&quot;A&quot; OBS_CONF=&quot;F&quot;/> <sts:Obs TIME_PERIOD=&quot;2005-09&quot; OBS_VALUE=&quot;99.99&quot; OBS_STATUS=&quot;A&quot; OBS_CONF=&quot;F&quot;/> <sts:Obs TIME_PERIOD=&quot;2005-10&quot; OBS_VALUE=&quot;123.45&quot; OBS_STATUS=&quot;A&quot; OBS_CONF=&quot;F&quot;/> <sts:Obs TIME_PERIOD=&quot;2005-11&quot; OBS_VALUE=&quot;212.21&quot; OBS_STATUS=&quot;A&quot; OBS_CONF=&quot;F&quot;/> <sts:Obs TIME_PERIOD=&quot;2005-12&quot; OBS_VALUE=&quot;234.56&quot; OBS_STATUS=&quot;A&quot; OBS_CONF=&quot;F&quot;/> </sts:Series> </sts:DataSet> </CompactData>

46. SDMX-ML: Six standard messages Fixed To query a database to obtain an SDMX-ML message as the result Query message 6 Derived from data structure definition message Exchange of many observation types in a data structure definition-dependent form Cross-sectional Data Message 5 Derived from data structure definition message For schema-based functions, such as validation, in a data structure definition-dependent form Utility Data Message 4 Derived from data structure definition message Exchange of large data sets in a data structure definition-dependent form Compact Data Message 3 Fixed Conveys data in a form independent of a data structure definition. It is designed for data provision on websites and in any scenario where applications receiving the data may not have detailed understanding of the data set's structure before they obtain the data set itself. Generic Data Message 2 Fixed Contains a data structure definition Structure Definition Message 1 Schema file Short description Name of message

47. Cross-Sectional Data Set <demo:DataSet REV_NUM = &quot;1&quot; TAB_NUM = &quot;RQFI05V1&quot; > < demo:Group COUNTRY = &quot;FI&quot; FREQ = &quot;A&quot; TIME = &quot;2005&quot; TIME_FORMAT = &quot;P1Y&quot; > < demo:Section DECI = &quot;0&quot; UNIT = &quot;PERS&quot; UNIT_MULT = &quot;0&quot; > < demo:ADJT OBS_STATUS = &quot;P&quot; SEX = &quot;F&quot; value = &quot;35&quot; /> < demo:DEATHST OBS_STATUS = &quot;P&quot; SEX = &quot;F&quot; value = &quot;23871&quot; /> < demo:LBIRTHST OBS_STATUS = &quot;P&quot; SEX = &quot;F&quot; value = &quot;28345&quot; /> < demo:NETMT OBS_STATUS = &quot;P&quot; SEX = &quot;F&quot; value = &quot;4187&quot; /> < demo:PJAN1T OBS_STATUS = &quot;P&quot; SEX = &quot;F&quot; value = &quot;2683230&quot; /> < demo:PJANT OBS_STATUS = &quot;P&quot; SEX = &quot;F&quot; value = &quot;2674534&quot; /> < demo:ADJT OBS_STATUS = &quot;P&quot; SEX = &quot;M&quot; value = &quot;131&quot; /> < demo:DEATHST OBS_STATUS = &quot;P&quot; SEX = &quot;M&quot; value = &quot;24057&quot; /> < demo:LBIRTHST OBS_STATUS = &quot;P&quot; SEX = &quot;M&quot; value = &quot;29400&quot; /> < demo:NETMT OBS_STATUS = &quot;P&quot; SEX = &quot;M&quot; value = &quot;4799&quot; /> < demo:PJAN1T OBS_STATUS = &quot;P&quot; SEX = &quot;M&quot; value = &quot;2572350&quot; /> < demo:PJANT OBS_STATUS = &quot;P&quot; SEX = &quot;M&quot; value = &quot;2562077&quot; /> < demo:ADJT OBS_STATUS = &quot;P&quot; SEX = &quot;T&quot; value = &quot;166&quot; /> < demo:DEATHST OBS_STATUS = &quot;P&quot; SEX = &quot;T&quot; value = &quot;47928&quot; /> < demo:LBIRTHST OBS_STATUS = &quot;P&quot; SEX = &quot;T&quot; value = &quot;57745&quot; /> < demo:NETMT OBS_STATUS = &quot;P&quot; SEX = &quot;T&quot; value = &quot;8986&quot; /> < demo:PJAN1T OBS_STATUS = &quot;P&quot; SEX = &quot;T&quot; value = &quot;5255580&quot; /> < demo:PJANT OBS_STATUS = &quot;P&quot; SEX = &quot;T&quot; value = &quot;5236611&quot; /> < /demo:Section> < demo:Section DECI = &quot;0&quot; UNIT = &quot;PURE_NUMB&quot; UNIT_MULT = &quot;0&quot; > < demo:DIV OBS_STATUS = &quot;P&quot; SEX = &quot;T&quot; value = &quot;13383&quot; /> < demo:MAR OBS_STATUS = &quot;P&quot; SEX = &quot;T&quot; value = &quot;29283&quot; /> < /demo:Section> < demo:Section DECI = &quot;3&quot; UNIT = &quot;PURE_NUMB&quot; UNIT_MULT = &quot;0&quot; > < demo:TFRNSI SEX = &quot;T&quot; value = &quot;1800&quot; /> < /demo:Section> < /demo:Group> </demo:DataSet>

48. SDMX-ML “Model-Driven” XML Approach

53. Query Message

54. Query SDMX Data SDMX-ML RSS WS NSI

55. sodi.istat.it con.istat.it Query SDMX Dati (SDMX-ML)‏ RSS WS demo.istat.it

56. DB WS demogr DEMO SODI RSS script

57. sodi.istat.it con.istat.it Query SDMX Dati (SDMX-ML)‏ RSS WS demogr

58. Thank you for your attention Vincenzo Patruno: [email_address]