This document discusses open data and big data. It defines open data as publicly available data that anyone can access, use and share, while big data refers to large and complex datasets. Open data could grow into big data as more data is published. The document provides an overview of linked open data standards and technologies like RDF and SPARQL that help structure and connect open datasets. It emphasizes the importance of not just publishing data but also developing applications that make use of open data through hackathons and other initiatives.

1. OPEN DATA, BIG DATA STORY
Ratko Mutavdzic, PROJEKTURA, Experience Architect

2. OPEN DATA VS. BIG DATA
•
•
•
•
•
Open Data =/= Big Data (usually)
Big Data = Open Data (usually)
Open Data could grow to Big Data
Big Data = BUSINESS THEN TRANSPARENCY (Business Conversation)
Open Data = TRANSPARENCY THEN BUSINESS (Government Conversation)
THE NEW REALITY BUT TWO DIFFERENT THINGS

3. OPEN DATA
International Right-To-Know Goal
“to raise global awareness of individuals’ right to access
government information”, and
„to promote access to information as a fundamental human
right”
THE NEW REALITY
2006 EC MEPSIR Study

4. WHY OPEN DATA
TRANSPARENCY
PARTICIPATION
COLLABORATION

5. PUBLIC DATA
OPEN FORMATS
MACHINE READABLE
ACCESSIBLE
WHAT MAKES DATA OPEN
• open format
• publised via industry standards like XML, RDF, HTML, CSV for data, PDF
for documents
• metadata
• published via standards like Dublin Core
• catalouge of open data sources
• http://logd.tw.rpi.edu/
right information is available for people to make a right decisions
... at all levels of the organization

6. PUBLIC INFORMATION POOL
Public Information / Content Pool
CATEGORY
Public Sector
Information
Public Sector
Content
EXAMPLE
geo data, statistical
data, other numbers
geo data, statistical
data, other numbers
OBJECTIVE
INFORMATION
RE-USE
CONTENT
AVAILABILITY
CHARACTERISTICS
source: adapted from OECD, 2006
• transformation of
raw data by value
addition
• frequent
combination of
information types
• education and
cultural value
• limited commercial
exploatation
• content not
transformed

7. KEY COMPONENTS
•
•
Open Data Portal
Open Data API
WHAT IS OPEN DATA SOLUTION
few words of wisdom

8. OPEN DATA IN U.K.
open.data.gov.uk
CKAN, Cloud
Most respected
Open Data
implementation

9. OPEN LINKED DATA

10. AGENDA
•
•
•
What is Open (Linked) Data?
Linked Data Standards and Tools
Linked Open Data in Practice
(LINKED) (OPEN) DATA
few words of wisdom

11. PUBLIC DATA vs. OPEN DATA
•
•
•
difficult to find
difficult to reuse
difficult to integrate
(LINKED) (OPEN) DATA
few words of wisdom

12. WHAT IS OPEN DATA?

13. WHAT IS LINKED OPEN DATA?

14. WHAT IS LINKED OPEN DATA?
ima zabavni život
ima sportska događanja
ima financije
ima sveučilište
OSIJEK
ima državnu upravu
ima zapošljavanje

15. LINKED OPEN DATA
Facilitating data integration through:
• Common data model
• Building relations
2 KEY INGREDIENTS
tbd

16. KEY INGREDIENTS
1.
2.
RDF RESOURCE DESCRIPTION FRAMEWORK (GRAPH BASED DATA)
•
identifies objects (URIs)
•
interlink information (Relationships)
VOCABULARIES (ONTOLOGIES)
•
provide shared understanding of data
•
organize knowledge in a machine comprehensible way
•
give an exploitable meaning to the data
2 KEY INGRIDIENTS
tbd

17. LINKED OPEN DATA
make your stuff available on the Web (whatever format) under an open licence
make it available as structured data (e.g. Excel instead of image scan of a table)
use non-proprierary format (e.g. CSV instead of Excel)
user URI to denote things, so that people can point at your stuff
link your data to other data to provide context
5 STARS OPEN DATA MODEL
Tim Berners-Lee, Linked Data initiative
http://lab.linkeddata.deri.ie/2010/star-scheme-by-example

18. ON WEB, OPEN LICENSE
•
•
ON THE WEB
•
wide access
•
google can index it
•
people can find it themselves
OPEN LINCENCE
•
regulate reuse of data
•
helps maintain provenance
•
strengthens business reuse
1 STAR
http://opendefinition.org/licenses/

19. STRUCTURED DATA
•
2 STAR
MACHINE READABLE

20. FORMATS
•
•
•
•
GOOD
„GOOD”
BAD
BAD, BAD
XLSX, CSV, JSON, MICRODATA
WEB, DOCX
PDF
charts, maps, images
•
SCREENSCRAPING? http://scraperwiki.com
2 STAR
http://opendefinition.org/licenses/

21. NON PROPRIETARY FORMATS
•
•
•
Freedom of how to process, analyse and visualise data
PROPRIETARY
•
DOCX, XLSX, PDF
NON PROPRIETARY
•
CSV, XML, JSON, MICRODATA, RDF
3 STAR
http://opendefinition.org/licenses/

22. USE OF URI
•
Unique identifiers enable others to point to the data
4 STAR
http://opendefinition.org/licenses/

23. LINKING DATA (AND RDF)
•
•
„Linked Data” approach have its use cases in Web Applications with LOT of
Data and little Semantics
Example: definme simple relationship and apply to large, heterogenous data
collections
5 STAR: Link your data to other data to provide context
http://lod-cloud.net

24. RESOURCE DESCRIPTION FRAMEWORK
•
•
•
Web is a global, universal information space for documents
Can we do the same for DATA and make the web into a database?
RDF is the DATA FORMAT for that database
Part fo the 5 STAR story
http://lod-cloud.net

25. RDF 101
small pieces, loosely joined, easy to reuse, easy to recombine, unexpected reuse, iterative

26. TYPICAL DATABASE TABLE
ISBN
TITLE
AUTHOR
PUBLISHERID
PAGES
112349987
Practical RDF
David Nelson Jr.
11692
443
234998021
C# for Dummies
Rick Torrensen
11692
1120
501334301
Calling the Stack
Shelly Monroe
45009
128
...
...
Part of the 5 STAR story
http://lod-cloud.net

27. TYPICAL DATABASE TABLE
ISBN
AUTHOR
PUBLISHERID
PAGES
112349987
Practical RDF
David Nelson Jr.
11692
443
234998021
C# for Dummies
Rick Torrensen
11692
1120
501334301
Calling the Stack
Shelly Monroe
45009
128
...
...
properties
subjects
TITLE
Part of the 5 STAR story
http://lod-cloud.net
Intersection is a
property of the
subject

28. LINKING DATA
title
book
C# for
Dummies
property
subject
The essence of RDF: the „TRIPLE”
value

29. TYPICAL DATABASE TABLE
ISBN
TITLE
AUTHOR
PUBLISHERID
PAGES
112349987
Practical RDF
David Nelson Jr.
11692
443
234998021
C# for Dummies
Rick Torrensen
11692
1120
501334301
Calling the Stack
Shelly Monroe
45009
128
...
...
SELECTING MULTIPLE PROPERTIES

30. LINKING DATA
title
book
C# for
Dummies
isbn
2349908
author
Rick
Torrensen
publisher
publisher
name
Relationship between „things”
Amazon
multiple
properties
graphically:
think in the
terms of
graphs, not
XML or
documents

31. USING THE WEB INFRASTRUCTURE
•
•
•
•
For Web scale database we need to be able to identify things globally and
uniquely
URI (URLs) already provide those capabilities
Name things with URIs, specifically http://
This is THE KEY to linked data
Part of the 5 STAR story
http://lod-cloud.net

32. RDF IN PRACTICE
named resources
http://example.
com/thing
named relations
numeric values and
literals
http://example.com/rel
„text”
http://example.com/other
3.141592
•
•
•
•
The URI identifies the thing you are describing
If two people create data using the same URI then they are describing the same thing
That makes it easy to merge data from different sources together
RDF data can use URIs from many different websites

33. OPEN LINKED DATA IN U.K.
open.data.gov.uk
Cloud
Monitors air and
water quality
Citizens rate quality
via SMS

34. LINKED DATA STANDARDS
Government Linked Data (GLD) WG http://www.w3.org/2011/gld/

35. SPARQL
•
Query language of the semantic web. It lets us:
• Pull values from STRUCTURED and SEMI STRUCTURED data
• Explore data by querying UNKNOWN RELATIONSHIPS
• Perform, COMPLEX JOINS OF DISPARATE DATABASES
• Transforms RDF from one vocabulary to another
Common understaning about „things”
supports the automatic generation of new information

36. SPARQL
# prefix declarations
PREFIX foo: <http://example.com/resources/>
...
# dataset definition
FROM ...
# result clause
SELECT ...
# query pattern
WHERE {
...
}
# query modifiers
ORDER BY ...
Common understaning about „things”
supports the automatic generation of new information

37. SO, WHAT DO WE DO WITH THE DATA?

38. HACKATONS: DATA + APPLICATIONS!
•
•
•
•
value is not in the raw data alone (but the data
needs to be published first!)
applications for use of the data is key to open
data success
size of the application does not guarantee its
value and success
value to the Citizen is the bottom line
BE SURE THAT YOU HAVE APP BUILDING PROCESS
… or paid teams, unpaid volunteers, hackatons, open data camps, student competitions

39. MANY FORMS, SAME PURPOSE
FOR EXAMPLE, INVOLVE… HACKATONS
strange word for a noble cause. building together the future in… 48 hours.

40. PROTOTYPES
OR RESULTS COMING FROM HACKATONS
but also many different scenarios of organization and citizen engagement resulting in apps

41. OPEN GOVERNMENT: PARTICIPATION

42. OPEN DATA ARCHITECTURE

43. ARCHITECTURE FOR OPEN: HYBRID?
Department A
Internal Private
PORTAL
Everybody
is using
External Public
PORTAL
is publishing
Department B
Published Data
Agency
Internal Network
Model that controls sensitive data and supports external scalability and availability. Brigde to PUBLIC.
Keywords: Public and Private Cloud. Provider Datacenter. SLA.
Published Data
External Network

44. CURRENT VIEW ON OPTIONS
solutions on MS
solutions on Linux
?
CKAN
ODGI
MS VM
LINUX VM
PaaS
private infrastructure
Private Cloud
private infrastructure that can be built
on Microsoft or Linux based stack
SOCRATA
CKAN
IaaS (LINUX VM)
public azure infrastructure
Public Cloud (MS AZURE)
public Microsoft Azure cinfrastructure supporting
„pure play” PaaS solutions and VM based solutions
(MS and nonMS)
SOCRATA
CKAN
IaaS (LINUX VM)
public cloud infrastructure
Public Cloud (nonMS)
public cloud infrastrucutre non
Microsoft (usually AWS or
OpenStack or …)

45. CKAN
• fully featured, mature, open source data management solution:
• publish and find datasets
• store and manage data
• engage with users and others
• customize and extend
• rich user base: data.gov.uk, publicdata.eu,…
OPEN SOURCE DATA PORTAL SOFTWARE
CKAN is open source and can be downloaded and used for free

46. BIG DATA

47. CIO
„Big Data is the frontier of a firm’s ability to
store, process, and access (SPA) all of the data
it needs to operate, make decisions, reduce
risks, and serve customers.”
“Predictive analytics solutions allow firms to
discover, evaluate, optimize, and deploy
predictive models by analyzing data sources to
improve business outcomes.”
Source: Forrester, „Evaluating Big Data Predictive Analytics Solutions”, 2012

48. Internet of Things
Internet of Everything
source: „US Unprepared for Internet Device Flood”, Kurt Stammbergerm MOCANA

49. Examples of New MultiStructured Data
Data types collected as big data and/or with advanced analytics
Structured (tables, records)
Complex (hierarchical or legacy)
Unstructured (text, audio, video)
Web logs and clickstreams
Machine-generated (sensors, RFID, devices)
Other
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
source: „Big Data Analytics”, survey of 325 companies, TDWI 2011

50. Key Elements of Big Data
Analytics and
Use
BIG ANALYTICS
DATA USE
Applications
Development
DATA
Analytics
BI and
Visualization
BIG DATA
Structured
Unstructured
Hadoop
Data Management
and Storage
source: „Understanding te elements of Big Data”, Karmasphere, 2011

51. New Questions
Where is BigData coming from... today
New era of computing
VOLUME: SIZE
SOCIAL ANALYTICS: What’s the
social sentiment of my product?
VELOCITY: SPEED
LIVE DATA FEED: How do I optimize
my services based on patterns of
weather, traffic, etc.?
VARIETY: STRUCTURE
ADVANCED ANALYTICS: How do I
better predict future outcomes?
•
•
•
•
twitter
facebook
google
people
12+ TB of tweet data every day
25+ TB of logs data every day
XX+ TB of search logs every day
2+ bilion people on the web today
•
•
•
•
RFID
smart meters
GPS devices
trade
30 billion RFID tags today
76 million smart meters today
100+ millions GPS devices per year
5 mililon trade events per second
•
•
phone cams
cameras
4,6 billion cams today
100+ thousands video feeds from surveillance
Two types of Big Data
• Data in Movements (Streams)
• Data at Rest (Oceans)
... so that is not our ordinary enterprise environment? Well...

52. They are all using BigData approach
and combine that with „socring as a
services mechanisms” like...
So What? Well... Big Data For Finance
KREDITECH
• Looks at 8.000 indicators like
location data, social
graph, behaviooral analytics, ecommerce shopping behavior
and device data...
• So, GPS, likes, friends, locations,
posts, movement, duration on
page, shopping, apps
installed, operating system...

ZESTFINANCE
Credit socring information via big
data, looks at 70.000 signals and
feeds them into 10 spearate
underwriting models
KLOUT
Social Media: Trustworthy Borrowers vs. Defaulters
LENDDO https://www.lenddo.com/



LENDUP https://www.lendup.com/


Looking at applicant’s connection on Facebook and Twitter
Key to get the loan: highly trusted individuals in your social
network
looks at social media activity to ensure that factual data provided
on the online application matches what can be inferred from
Facebook and Twitter.
WONGA https://www.wonga.com/

considers the time of the day and the way a candidate clicks
around the site in determining whether to grant a loan

53. Early operator initiatives will still
involve a strong element of
traditional business intelligence
and analytics: structure records:
Call and Billing
Records, Electronic Data
Records, Location Records...
So What? Well... Big Data for Telcos
Unstructured: Phone Calls, Text
Messages, Social Media posts...
•
Telco: competition in a mature market
•
Two different strategies for growth and mature markets:
•
•
Classical BigData problems: Churn Management (on prepaids)
•
•
•
Growth: aquisition strategy, simple BI needs (reporting)
Mature: differentiation strategy, complex BI needs (data mining)
When to engage (mid of billing cycle) www.globys.com
When not to engage (leave good customers alone) www.venda.com
How telcos can invent business models?
•
•
•
•
IMPROVE SERVICES: Data = Improved Business (Amazon)
MOBILE ADVERTISING: Data = Better Advertising (Google)
SELL ACCESS TO INSIGHTS: Data = Business (comScore)
BECOME GATEKEEPER: Data = Personal Risk (www.reputation.com )

54. OPEN DATA + BIG DATA?
• OPEN DATA
• You can fetch and use any data that exist around. You can connect that
data to any other source and personalize the use.
• BIG DATA
• You can fetch and use any volume of data that is flowing from devices
around you and from your own usage.
• OPEN BIG DATA
• WE CAN PREDICT AND REACT TO ANY ACTION IMMIDIATELLY
IMAGINE THE WORLD...
Where you dont have control over the things that happen around you.

55. INTRO
just a few words about me
so, if
• Ratko Mutavdzic is founder of PROJEKTURA, consulting company that work
with new and emerging technologies and introduce them to the corporate and
enterprise environments. Prior to this one, he spent 15 years Microsoft, starting
in a consulting practice and then leading several different sales and technology
teams.
• He is the author of number of published papers on different aspects of the
technology, successful blogs on new technologies and project
management, and active contributor in a number of social networks exploring
the use and advance of new ways to connect and share innovation and
invention.
• He frequently speaks on conferences, meetings, workshops, coffee shops and
generally our heads... people like to
we all nodat every place wherewe can continue...
explore, challenge, investigate, think and innovate.
• Keywords: change, project, program, portfolio, innovation, startup
note: more contact info on a last slide