1. Opening up Government Data
linux.conf.au
January 2014
Pia Waugh
Open Source|Data|Government Geek
@piawaugh

2. The stars are aligned
eGovernment and Digital Economy Policy 2013
1) Public consultation for high value data to publish
datagovau.ideascale.com
2) Review Big Data Strategy
3) Seek proposals for public/private partnerships using
big data for improved services
4) Dashboard and league tables on ICT performance/metrics,
Including online engagement, platform-agnostic services,
availability of data sets and customer satisfaction
Government interactions that occur more than 50,000 times
per year to achieved online by 2017

3. The stars have been aligning for a while
Gov 2.0 Taskforce Report 2009 (based on PoIT UK)
1) Declaration of Open Government
2) CC-BY as default
3) Information Commissioner
4) data.gov.au and social media support/policy
5) Cloud/shared services
Myriad other supporting tech and copyright policies
States/Territories, Federal and Local largely aligned

4. The APS Policy Landscape
Others:
•
Publishing Public Sector Information & National Standards Framework
•
Open Public Sector Information: From Principles to Practice Report
•
Declaration of Open Government
•
Gov 2.0 Taskforce Report
•
Statement of IP Principles for Government (CC-BY)
•
Ahead of the Game
•
Digital Transition Policy & Accessibility Policy
•
Emerging Open Research Policies
•
Open Government Partnership (TBD)

5. State and Territory Policies
For more see OpenGov State of the Nation talk from Monday :)

6. Why open up government data?
Transparency
“Innovation” & value-add
Shared and open
data
Better public services & data
Participatory
government
Citizen centric
services

7. Benefits to Community in Opening Data
Transparency
Visibility to government spending, projects, effectiveness, etc
Increases incentive to follow evidence based approach
Builds trust in government services and information
Participatory Democracy
Enables greater participation in planning and decision making
More informed public → better decision making
Improvements to data → better policy and decisions
Innovation
New opportunities and innovation in industry, research, civil society
Economic
Creates opportunities for industry to value-add to government data

8. Benefits to Government in Opening Data
Cuts red tape
More efficient to share data across government and with public
Proactive automated publishing
Improves Government Operations
Enables collaboration and consistency across gov and with public
Improves policy analysis, development, implementation, reporting
Government as an API improves service delivery:
enables thematic personalised approach to info & mobile services
Improves data quality through verifiable public contributions
Improved opportunities for evidence based and iterative policy
Innovation
Enables government to tap into internal and external innovation
Starts to change the culture of what “innovation” means & costs
Enables greater capacity for public contributions to gov

9. data.gov.au
Free, cloud based, scalable platform for government data.
CKAN on LAMP, Amazon cloud, OSSec, GeoServer, Python
Roadmap
Features
1) Publishing (2013)
•
•
•
•
•
Cleaned up data, improved functionality and
ease of publishing for agencies, did training and
documentation, developed automation support
2) Value realisation (Early 2014)
Providing useful front end tools for data.gov.au
including data visualisation and analysis tools
and reporting/league tabling.
3) Data quality (Late 2014)
Looking at ways to provide agencies the ability
to accept iterative data improvements in a
verifiable way
Good metadata, categorisation, tagging
Manual and automated publishing
API access to government data
Easy to publish, download and
interact with data online
• Basic data visualisation capability
• Federated search could make data
and data services easier to find
Collaboration across
jurisdictions is vital for
good user experience!

10. data.gov.au/stats
`

11. Finding Government Data (latest version online)
Open Government in Australia
Comparatively good
Recent developments:
•
Gov 2.0 Taskforce (2009)
•

12. Different sorts of data
●
●
●
●
●
Unstructured tabular – CSV, XLS, XLSX, txt
Spatial data – data files, WFS, WMS,
Science data – RDIS coming soon!
Real time data – sensor, traffic
Statistics/Census data – ABS.Stat and other data services
Loads of specialist data, not a lot of cross-disciplinary approaches

13. Tools for hacking with data
http://www.govhack.org/howto/
●
●
●
●
●
●
●
●
●
●
●
●
Geographical Data Tools
Tabular Data Tools
Unstructured (Text) Data Tools
Graph (relationships and networks) Data Tools
Publishing tools – CKAN, Socrata, bespoke
Automation – FME, Kettle
Data visualisation – Tableau, SuperDataHub, SpatialKey
Analysis – R, domain specialist software, spook software
API development – Apigee, JSON
Application development
Linked data tools
Metadata tools

14. New and Old Skills Required
Publishing and Automation
•
Project management, reporting
•
Metadata/linked data
•
API developmaent and serving
•
Plumbing between systems
•
Data and info visualisation
•
Analysis and statistics
•
Policy development
•
Public consultation and engagement
•
Online community management
•

15. Some Challenges
•
•
•
•
•
•
•
•
•
•
Legislative
{Open|Big|Linked}
Culture
Data
Systems
Low tolerance for mistakes
Reactive vs proactive
Metadata/semantic context
Too much data
Real time vs historic
Definitions and common references
Limited skills and over specialisation

16. Privacy and confidentiality
•
•
•
•
•
The challenge of raw vs processed data
Custom API approach to confidentialise on fly (eg, ABS)
Deidentification of data – to appropriate level
Aggregation
Leveraging existing processes for researcher (unit level)
access rather than conflating open data discussions
• Privacy Commissioner as point of reference and support
• Avoiding common identifiers across multiple datasets

17. Our Greatest Opportunity
Hacker Culture in Australia

18. GovHack
Develop strong links between government,
research, community and industry
Showcase uses of government data and
clever technical community in Australia
Encourage publishing of government data
Make “innovation” meaningful
July 11 – 13 2014

19. All the pieces are in place,
we need people to put the puzzle together

20. Play, investigate, show off, collaborate...
Most importantly, have fun!
What better place than here?
What better time than now? -- RATM
Questions?