The traits of an 'open' dataset -- factors like accuracy, geographic scope and copyright entanglements -- shape the development process in profound ways. I'll share what I've learned building projects around heritage trees, public art and poetry posts in Portland, and extrapolate a blueprint for evaluating and planning open data projects.

1. Similar, But Not The Same:
Designing Projects Around Three Open Datasets
Matt Blair
Open Source Bridge, Portland, Oregon
June 23, 2011

2. • Summary of the three projects
• Planning Open Data Projects
• Slides and PDF online

3. The Three Projects

5. Poetry Boxes

6. What are these things?

7. “A poetry post (or poetry pole or poetry
box) is a wooden pole, usually, mounted on
private property, so that it faces
pedestrians. On top of the pole is a box,
with a glass or clear face and a lid. Inside the
box is a sheet of paper containing a poem
(or, sometimes, prose or a photograph).
Sometimes the pole is absent, the box
mounted to a tree. That’s it.”
– Laura O. Foster

8. A Poetry Box

9. Poetry Posts

10. A Poetry Pole

11. That Poetry “Tree”

12. Project History
• Started: June 2010
• An idea without
much data
• Only 11 locations
until October
• Various iterations...

13. Project Goals
• Map the poetry posts
• Promote the idea of sharing poetry in a
neighborhood context
• Encourage people to get out and walk,
interact with each other

14. Current Status

15. 59 locations
Including 33 photos

16. http://poetrybox.info

17. Data Available for Re-use
GeoJSON in Couch DB

18. I’m not sure what’s next
And that’s part of the story.

19. PDX Trees

20. Project History
• Started: August 2010
• Built around Heritage Tree data released
through Civic Apps
• App released: October 2010

22. Project Goals
• Where are these Heritage Trees?
• Learn more about each
• Get people out to see the trees

23. Public Art PDX

24. Project History
• Started: November 2010
• Built with data from the Regional Arts &
Culture Council (aka RACC)
• Geo-coded by City’s Bureau of Technology
Services for portlandmaps.com

25. http://racc.org/public-art/search

29. portlandmaps.com

30. Project Goals
• Make public art more accessible in situ
• Remind everyone of how lucky we are
• Encourage exploration (& more walking!)

31. App released:
February 2011

35. 451 works of art
At 215 locations

36. Data Available for Re-use
GeoJSON in Couch DB

37. Planning
Open Data
Projects

38. They’re all just maps,
right?

39. They’re all just maps,
right?
What’s the diff?

40. What makes open data
easy -- or difficult?

41. Anticipate problems
and skills needed

42. Make great projects
that excite and engage

43. Seem basic?
Make no assumptions about Open Data.

44. 1. Is there data?

45. Data + Idea = Project

46. True?

47. Lots of great Ideas with no
Open Data to use.

48. Lots of Open Data
that’s just dull as dirt.

49. Data
Ideas
The Open Data Universe

50. Data
Ideas
Work here.
Option #1

51. Option #2:
Assemble your own data

52. If it exists in digital form:
• Screen-scrape
• Repurpose feeds or reporting systems
• Google Refine
• Convince governments/stakeholders to
release it

53. If it’s not in digital form?

54. And arrives like this?

55. Tough to Automate

56. Poetry Posts
Dozens and dozens of emails
in and outside of a Google Group

57. Don’t underestimate
data collection.

58. 2. Data Sources

59. • Who gathered it?
• To what end?
• Are they maintaining it?
• Do they want to share?

60. PDX Trees
One list, managed by one department.

61. Public Art Data
in Portland

62. Data Sources (Nov 2010)
• PDX API: 279 works of art
• Civic Apps CSV file: 366 works of art
• BTS: 300ish works of art (no direct access)
• RACC.org: 1800+ works or art

63. PDXAPI
RACC.org
Civic Apps
BTS?
Where to start?

64. RACC.org
Civic Apps
PDXAPI
And BTS = Civic Apps?

65. But they didn’t match...

66. RACC.org
Permanent
Collection

67. Text = Available Data

68. Text = Available Data
As of November 2010:
~ 370 works of art

69. RACC.org

70. RACC.org
(City of Portland)
(Multnomah County)

71. Done? Not quite...

72. (Caveat: Not To Scale)
Accuracy Not Guaranteed

73. RACC.org
Public Art

74. RACC.org Murals
Public Art

75. RACC.org Murals
TriMet
Public Art

76. RACC.org Murals
TriMet
Convention
Center
Public Art

77. RACC.org Murals
TriMet
Metro
Convention
Center
Public Art

78. RACC.org Murals
Port of
TriMet
Portland
Metro
Convention
Center
Public Art

79. Fountains
RACC.org Murals
Port of
TriMet
Portland
Metro
Convention
Center
Public Art

80. Fountains
RACC.org Murals
Parks
& Rec
Port of
TriMet
Portland
Metro
Convention
Center
Public Art

81. Fountains
RACC.org Murals
Parks
& Rec
Port of
TriMet
Portland
Beaverton?
Metro
Convention
Center
Public Art

82. Fountains
RACC.org Murals
Parks
& Rec
Port of
TriMet
Portland
Beaverton?
Hillsboro? Metro
Convention
Center
Public Art

83. Fountains
RACC.org Murals
Parks
& Rec
Port of
TriMet
Portland
Beaverton?
Hillsboro? Metro
Clark Convention
County? Center
Public Art

84. Fountains
RACC.org Murals
Parks
& Rec
Port of
TriMet
Portland
Performance
Beaverton?
Hillsboro? Metro
Clark Convention
County? Center
Public Art

85. Fountains
RACC.org Murals
Parks
& Rec
Port of
TriMet
Portland
Performance Temporary
Installations
Beaverton?
Hillsboro? Metro
Clark Convention
County? Center
Public Art

86. Fountains
RACC.org Murals Graffiti?
Parks
& Rec
Port of
TriMet
Portland
Performance Temporary
Installations
Beaverton?
Hillsboro? Metro
Clark Convention
County? Center
Public Art

87. “Of course not!”

88. “But what if it’s Banksy?”

89. Fountains
RACC.org Murals Graffiti?
Parks
& Rec
Port of
TriMet
Portland
Performance Temporary
Installations
Beaverton?
Hillsboro? Metro
Clark Convention
County? Center
Community?
Public Art

90. Community Collection

91. Where’s Paul?

92. photo by Cacophony (via Wikipedia)

93. Intersection
Repair
photo by City Repair (via Flickr)

94. Julian Voss-
Andrae’s
Alpha Helix
(at the Linus Pauling House)
photo via julianvossandrae.com

95. Fountains
RACC.org Murals Graffiti?
Parks
& Rec
Port of
TriMet
Portland
Performance Temporary
Installations
Beaverton?
Hillsboro? Metro
Clark Convention
County? Center
Community?
Public Art

96. Fountains
RACC.org Murals Graffiti?
Parks
& Rec
Available
Port of
Data
TriMet
Portland
Performance Temporary
Installations
Beaverton?
Hillsboro? Metro
Clark Convention
County? Center
Community?
Public Art

97. A Community-wide
Database
Build an inter-agency superset of public art

98. Yes, it’s complicated.

99. Yes, it’s complicated.
Yet Portland has some of the best
public art data in the country.

100. Data Source Tips
• Start small
• Find allies and set an example
• Don’t wait for the perfect dataset
• Plan for chaos
• But be ambitious in the long-term!

101. 3. Data Structure

102. Each source has its own:
• metadata
• schema
• volatility
• level of accuracy and currency
• messes to clean up

103. A dynamic mix of...
• noisy data
• shifting standards
• unexpected restrictions
• adapting to community requests
• addressing data provider concerns
• ambiguities of project 'ownership'

104. Less about architecture
than improvisation...

105. Metadata Flexibility
Let early adopters set the standards

106. NoSQL

107. Document Databases

108. CouchDB

109. Eventual Schema

110. “There is always
schema somewhere.”

111. “There is always
schema somewhere.”
If it’s not in your software, you’re forcing
your audience to build it in their heads.

112. Delay Schema Decisions
Push them up to presentation/client layer

113. But what about MVC?

114. But what about MVC?
My model is in my view?!

115. More nimble than
relational database
Especially during development

116. YKmMV

117. 4. Scope and Density

118. Not Just Geographic
Topic or Time

119. Social Clusters

120. Geo-Density and UI

121. PDX
Shrub?

122. PDX
Lichen?

123. Cluster it?

124. Art Mob?

125. Places and Art
Distinct Databases

126. The Map Shows Places
Not Art

127. If a place has
one work of art:
Place = Art

129. If a place has
multiple works of art...

131. Presentation Determines
Data Model

132. Presentation Determines
Data Model
But you just said!?!

133. 5. Stories

135. “While the map makes class and race
differences all the more evident, it's great to
learn about the few murals where I live and
I look forward to using this app on a walking
tour downtown soon.”
– Marshal Kirkpatrick, Read Write Web

138. Incomplete Data

139. Recent Public
Building Projects

140. Economic Activity

141. Voice and
Representation

142. Share Open Data:
Interpretations Emerge

143. A Tool for Advocacy?

144. 6. Accuracy

145. What’s Missing
or Incorrect?

146. What’s Missing
or Incorrect?
Errors and omissions are
an opportunity for participation.

149. > 1 mile

150. Artifact of Geo-Coding
Address != Location

151. Capacity
Who has the time to manually verify all these locations?

152. Crowd-Correction

155. 7. Audience Appeal

156. Public Art
{ "docs": [{
"addrCity": "",
"addrState": " ",
"addrStreet": "",
"addrZip": "",
"artists": "Dan Corson",
"date": "2009",
"dateModified": "2011-04-18 00:00:00",
"description": "Mercurial Sky is an ever-changing array of light played on LED
tubes integrated into the Director Park Canopy. The digital video only emits from the
lighted bars, and provides a sense of movement through an abstract tapestry of light
and color. If you stand farther away, or look in nearby reflections, the images are
compressed and give a clearer view of the video. nn"I filmed images and patterns of
natural phenomena like waves, clouds, fire, earthworms, and jellyfish to bring the
movement and randomness of nature into this mostly hardscaped park."",
"detailPageURL": "http://racc.org/public-art/search/?recid=2909.101",
"dimensions": "duration: 1:23:10",
"discipline": "video",
"fundingSource": "Percent for Art - City of Portland",
"thumbnailURL": "http://data.racc.org/pa_inventory/1844/1844thumb.jpg",
"location": "Director Park Canopy",
"mappableDiscipline": "other",
"medium": "Digital video on DVD",
"recordID": "2909",
"title": "Mercurial Sky",
"dataSource": "RACC",
"collection": "None",
"photoCredit": "RACC",
"artCopyright": "TBD",
"locationVerified": "YES",
"geometry": {
"coordinates": [
-122.681124,
45.518759
],
"type": "Point"
}
}

157. Public Art
{ "docs": [{
"addrCity": "",
"addrState": " ",
"addrStreet": "",
"addrZip": "",
"artists": "Dan Corson",
"date": "2009",
"dateModified": "2011-04-18 00:00:00",
"description": "Mercurial Sky is an ever-changing array of light played on LED
tubes integrated into the Director Park Canopy. The digital video only emits from the
lighted bars, and provides a sense of movement through an abstract tapestry of light
and color. If you stand farther away, or look in nearby reflections, the images are
compressed and give a clearer view of the video. nn"I filmed images and patterns of
natural phenomena like waves, clouds, fire, earthworms, and jellyfish to bring the
movement and randomness of nature into this mostly hardscaped park."",
"detailPageURL": "http://racc.org/public-art/search/?recid=2909.101",
"dimensions": "duration: 1:23:10",
"discipline": "video",
"fundingSource": "Percent for Art - City of Portland",
"thumbnailURL": "http://data.racc.org/pa_inventory/1844/1844thumb.jpg",
"location": "Director Park Canopy",
"mappableDiscipline": "other",
"medium": "Digital video on DVD",
"recordID": "2909",
"title": "Mercurial Sky",
"dataSource": "RACC",
"collection": "None",
"photoCredit": "RACC",
"artCopyright": "TBD",
"locationVerified": "YES",
"geometry": {
"coordinates": [
-122.681124,
45.518759
],
"type": "Point"
}
}

158. Public Art
{ "docs": [{
"addrCity": "",
"addrState": " ",
"addrStreet": "",
"addrZip": "",
"artists": "Dan Corson",
"date": "2009",
"dateModified": "2011-04-18 00:00:00",
"description": "Mercurial Sky is an ever-changing array of light played on LED
tubes integrated into the Director Park Canopy. The digital video only emits from the
lighted bars, and provides a sense of movement through an abstract tapestry of light
and color. If you stand farther away, or look in nearby reflections, the images are
compressed and give a clearer view of the video. nn"I filmed images and patterns of
natural phenomena like waves, clouds, fire, earthworms, and jellyfish to bring the
movement and randomness of nature into this mostly hardscaped park."",
"detailPageURL": "http://racc.org/public-art/search/?recid=2909.101",
"dimensions": "duration: 1:23:10",
"discipline": "video",
"fundingSource": "Percent for Art - City of Portland",
"thumbnailURL": "http://data.racc.org/pa_inventory/1844/1844thumb.jpg",
"location": "Director Park Canopy",
"mappableDiscipline": "other",
"medium": "Digital video on DVD",
"recordID": "2909",
"title": "Mercurial Sky",
"dataSource": "RACC",
"collection": "None",
"photoCredit": "RACC",
"artCopyright": "TBD",
"locationVerified": "YES",
"geometry": {
"coordinates": [
-122.681124,
45.518759
],
"type": "Point"
}
}

159. Heritage Trees
{
"address": "2403 WI/ SW JEFFERSON ST",
"circumfere": "12.300000000000001",
"common_nam": "Deodar cedar",
"diameter": "47",
"geometry": {
"coordinates": [
-122.70463884770101,
45.521710633334202
],
"type": "Point"
},
"gid": "103",
"height": "73",
"notes": "between SW Marconi Ave and SW Tichner Dr",
"objectid": "103",
"owner": "Right Of Way",
"scientific": "Cedrus deodara",
"spread": "73",
"stateid": "1N1E32 100",
"status": "Heritage",
"treeid": "113",
"year": "1996"
}

160. PDX Trees
{
"address": "2403 WI/ SW JEFFERSON ST",
"circumfere": "12.300000000000001",
"common_nam": "Deodar cedar",
"diameter": "47",
"geometry": {
+ ?
"coordinates": [
-122.70463884770101,
45.521710633334202
],
"type": "Point"
},
"gid": "103",
"height": "73",
"notes": "between Marconi Ave and SW Tichner Dr",
"objectid": "103",
"owner": "Right Of Way",
"scientific": "Cedrus deodara",
"spread": "73",
"stateid": "1N1E32 100",
"status": "Heritage",
"treeid": "113",
"year": "1996"
}

161. PDX Trees
{
"address": "2403 WI/ SW JEFFERSON ST",
"circumfere": "12.300000000000001",
"common_nam": "Deodar cedar",
"diameter": "47",
"geometry": {
+
"coordinates": [
-122.70463884770101,
45.521710633334202
],
"type": "Point"
},
"gid": "103",
"height": "73",
"notes": "between Marconi...",
"objectid": "103",
"owner": "Right Of Way",
"scientific": "Cedrus deodara",
"spread": "73",
"stateid": "1N1E32 100",
"status": "Heritage",
"treeid": "113",
"year": "1996"
}

162. PDX Trees
{
"address": "2403 WI/ SW JEFFERSON ST",
"circumfere": "12.300000000000001",
"common_nam": "Deodar cedar",
"diameter": "47",
"geometry": {
+ ??
"coordinates": [
-122.70463884770101,
45.521710633334202
],
"type": "Point"
},
"gid": "103",
"height": "73",
"notes": "between Marconi Ave and SW Tichner Dr",
"objectid": "103",
"owner": "Right Of Way",
"scientific": "Cedrus deodara",
"spread": "73",
"stateid": "1N1E32 100",
"status": "Heritage",
"treeid": "113",
"year": "1996"
}

163. Poetry Posts
1991 SW Mill St Terrace

164. Poetry Posts
1991 SW Mill St Terrace

165. PDX Trees
{
"address": "2403 WI/ SW JEFFERSON ST",
"circumfere": "12.300000000000001",
"common_nam": "Deodar cedar",
"diameter": "47",
"geometry": {
+
"coordinates": [
-122.70463884770101,
45.521710633334202
],
"type": "Point"
},
"gid": "103",
"height": "73",
"notes": "between Marconi...",
"objectid": "103",
"owner": "Right Of Way",
"scientific": "Cedrus deodara",
"spread": "73",
"stateid": "1N1E32 100",
"status": "Heritage",
"treeid": "113",
"year": "1996"
}

166. Sunny October Day?

167. Sunny October Day?
No one will believe it.

168. More Realistic

169. In Rarer Weather, too...

171. But there are 283 Trees!

172. But there are 283 Trees!
I need help.

174. Create a
crowd-
sourced,
longitudinal,
season-
sortable
collection of
tree photos...

175. Will anyone send photos?

176. by Brad B

177. by kateinoregon

178. “Keep
Portland
Green!”
by Dan Flynn

179. (and I still add some...)

180. 320+ Photos Sent In

181. Submitted under
Creative Commons

182. Photos (and data)
available for re-use

183. 8. Intellectual Property

184. I thought this was
“open” data?

185. Restrictions in
Terms of Use

186. Linked Media

187. Linked Media
Who took that photo?
Who owns it?
Who can license it?

188. 9. Data Volatility

189. How fast does the data
change?

190. Poetry Posts
Whenever I update it.

191. Poetry Posts
Website reads from CouchDB

192. Poetry Posts (future)
Multiple clients read/write from CouchDB

193. PDX Trees

194. PDX Trees
Trees are slow movers.

195. PDX Trees
• Data: once or twice a year
• Photos: a couple times a week

196. PDX Trees
• Basic tree data on phone
• Available images for each tree pulled from
API in real-time
• Photos not available offline

197. PDX Trees (future)
• Multiple clients adopting the same pattern?
• Device caching of images?
• Remote delete of flagged images?

198. Public Art PDX
• Basic data changes faster than app update
• Micro-updates (a comma moved)
• New works and collections
• Photos online only (IP)

199. Public Art PDX
• CouchDB is the canonical data store
• App fetches packaged data releases
• App searches data locally
• Photos load from CouchDB

200. DIY Phone to Couch Sync

201. Public Art PDX (future)
Mobile Couchbase!

202. 10. Open Data Roles

203. Who do you need?

204. Standard Roles
• Project Management • Software Engineering
• Graphic Design • Testing
• Information Architecture • Maintenance
• Metadata Design • Coordination
• Data Entry/Import • Marketing
• UI Design • Communication

205. Roles That Deserve
Special Attention

206. Data Collection

207. Fountains
RACC.org Murals Graffiti?
Parks
& Rec
Available
Port of
Data
TriMet
Portland
Performance Temporary
Installations
Beaverton?
Hillsboro? Metro
Clark Convention
County? Center
Community?
Public Art

208. Research & Verification
Not necessarily a technical person

209. Data Scrubbing
Probably Need Tech Skills

210. Curation
What’s in, what’s out and why

211. Authority
(not required, but it helps)

212. Advocacy
Why are we doing this?
Why should you release this data?

213. Public Art PDX 1.0:
Essential Collaborators
• Regional Arts & Culture Council (RACC)
• The Office of Mayor Sam Adams
• City of Portland’s Bureau of Technology
Services
• City Attorney (and pro bono attorneys)
• Bud Clark! (see the video)

214. Public Art PDX 1.0
Collaborators Helped With:
• Graphic Design • Communication
• Metadata Design • Data Collection
• Data Entry/Import • Data Scrubbing
• Testing • Authority
• Marketing • Advocacy

215. Evolving Roles for the
Community Collection
• Metadata Design • Curation
• Data Entry/Import • Photography
• Research and Verification • Communication

216. Public Art PDX 1.x:
Community Collection
Different Phases, Different Roles, New Participants

217. PDF
Online
Text
Text

218. This is a work-in-progress.
Ideas welcome.

219. 11.

220. WhereCamp
September, 2010

221. “The App is not the Thing”

222. “The App is not the Thing”
(redux)

223. Websites =
Presentation

224. Apps = Presentation

225. Presentation Layers are
Ephemeral

226. Presentation Layers are
Ephemeral
They come and go, like fashion

227. This year’s attire

228. What costumes will
your data wear in
five years’ time?

229. Who knows.

230. Data *is* the Thing.

231. Data Lasts.

232. Project Websites
• http://poetrybox.info
• http://pdxtrees.org
• http://publicartpdx.com

233. Thank You
• http://mattblair.net
• Email: elsewisemedia@gmail.com
• Blog: http://elsewisestrategic.com
• github.com/mattblair