The increasing amount and variety of open and crowdsourced data available in the web leads to new challenges in end-user focused data analysis. This data is characterized by a great structural diversity which causes serious problems regarding their integration. On the other site there is a lack of end-user friendly tools to make productive use of the data available on the web. We want to address the first problem by developing a schema-optional graph-based data model that enables incremental schema augmentation and evolution. The second problem should be adressed by a multi-layered domain-specific language for data mashup construction on schema-optional data.

1. In the Age of Open InformationDo-It-Yourself Analytical Mashups on Schema-optional DataKatrin BraunschweigJulian EberiusMaik ThieleWolfgang Lehner OUTPUT 2011

2. The Roots ofOpen Data The open society is a concept originally developed by philosopher Karl Popper In open societies, government is responsive and tolerant, and political mechanisms are transparent and flexible The state keeps no secrets from itself in the public sense It is a non-authoritarian society in which all are trusted with the knowledge of all

3. 1984 - Freedom of Information Campaign starts up

4. Why Data Shouldbe Open Many scientific data can be deemed to belong to the commons (“the human race”), e.g. the human genome, medical science, environmental data They have an infrastructural role essential for scientific endeavour (e.g. in Geographic Information Systems and maps) Data published in scientific articles are factual and therefore not copyrightable Public money was used to fund the work and so it should be universally available It was created by or at a government institution

5. Open Data – Examples

6. data.gov

7. data.gov.uk

8. data.worldbank.org

9. unData

10. OpenStreetMap

11. CivicApplicationsbased on Open Data

12. Explore How U.S. Budget Proposal

13. Mapnificient

14. Schooloscope

15. Fluglärmkarte (taz.de) Database Journalism

16. Open Data – Challenges an Challenges Lots ofcontributors / maintainers Small informationpieces distributed, decentralised and verylooselycoupled Different degreeofschemainformationand metadata Innovation / unexpectedreuse Nostandardizeddevelopmentprocess Contributions Schema-optional datastore, collaborativeschemaaugmentation (basicoperators) Measuredegreeofschemainformation Non-destructiveschemachanges Capture dataprovenance Visualizationsand interactionpatterns Iterative and guideddevelopment Data and visualizationrecommendation

17. The Big Picture

18. Do-It-Yourself Schema Augmentation Application ReferenceNode AttributeTypes EntityTypes NoType NoType ET1 AT1 ET4 AT2 AT4 ET2 ET3 AT3 Schema Augmentation Automated Schema Extraction AT1 : value AT2 : value AT3 : value AT4 : value E V E V AT TT AT TT ET Relational Table CSV File

19. Do-It-Yourself Analytical Mashups Process Query „number of cafes vs. age distribution per district of Dresden“ Look upfittingdatasets Computesuitablevisualization Computeinteraction / explorationfeatures

20. Do-It-Yourself Analytical Mashups (2) Process Query „number of cafes vs. age distribution per district of Dresden“ Look upfittingdatasets Computesuitablevisualization Computeinteraction / explorationfeatures NLP techniques + Lookup services (e.g. GeoNames) number of cafes vs.age distribution perdistrict of Dresden natural geographic entity value dimensions relations/operations

21. Do-It-Yourself Analytical Mashups (3) Process Query „number of cafes vs. age distribution per district of Dresden“ Look upfittingdatasets Computesuitablevisualization Computeinteraction / explorationfeatures Ambiguity userfeedback OR

22. Do-It-Yourself Analytical Mashups (4) Process Query „number of cafes vs. age distribution per district of Dresden“ Look upfittingdatasets Computesuitablevisualization Computeinteraction / explorationfeatures

23. Do-It-Yourself Analytical Mashups (5) Process Query „number of cafes vs. age distribution per district of Dresden“ Look upfittingdatasets Computesuitablevisualization Computeinteraction / explorationfeatures number of cafes age distribution Too much information for one visualization enableexploration, e.g., clicking a district in themapopenshistogram

24. Demo

25. Map-centricweb application Mobile application 3rd-party applications # # # REST Interface PersistenceLayer Open CivicPlatformforDresden Mobile Application Add new requests by guiding the user through a wizard-style input form Show (own) reports and there current rating and processing actualstate Visualize all reports on a map Subscribe to a set of urban district and notify the user about news Web Application Filter the requests by their category, their creation time (last 24 hours, last week, last month, all) Change the requests state (open, closed, closed) for authorized users Zoom in/out and adapt the type of visualization if the issue density gets very sparse

26. Open CivicPlatformfor Dresden (2)

27. Open CivicPlatformforDresden (3)

28. New York – Example

29. New York – Example (2)

30. In the Age of Open InformationDo-It-Yourself Analytical Mashups on Schema-optional DataKatrin BraunschweigJulian EberiusMaik ThieleWolfgang Lehner OUTPUT 2011