1. The influence of training on
position and attribute accuracy
in volunteered geographic
information
GEORGE MASON UNIVERSITY, DEPARTMENT OF GEOGRAPHIC AND
CARTOGRAPHIC SCIENCES, COLLEGE OF SCIENCE
Patricia Pease, MS Candidate, GECA /GISESP
Dr. Matthew Rice, Thesis Director
Dr. Arie Croitoru, Committee Member
Dr. Kevin Curtin, Committee Member
Dr. Michele Campagna, Committee Member (Italy)

2. • Geo-crowdsourcing is emerging geospatial data collection
technique in fieldwork
• Citizen science benefits – savings/expertise
• Offering of specific and targeted training
• Experiment set up with trained and untrained participants to assess
effectiveness of training
• Both treatment groups then reported obstacles found in field
• Trained project moderators previously characterized obstacles
• Detailed quality analysis done by moderators
• Statistical analysis of QA data found no significant difference

3. • VGI – although traced much earlier term named by Goodchild in
2007 (GIS in the hands of citizens)
• Christmas Bird Count well established long-running examples
• Citizen science taps into advantages of funding, expertise, and
interest of people in an area
• Value of VGI declines based on errors, low accuracy or malicious
data
• Open sources such as OpenStreetMap and many cell phone apps
have made more VGI possible
• Democratization of GIS (Brown) (‘earned rather than learned’)

4. • Use and speed of GIS changes
• Much CGD deals with changing, non-static data
• Applications continue to be added and change usability
of GIS
• Goodchild’s concept of ‘platial’ relationaship with
volunteer activity in geography
• Summarization of the American Scientist citizen science
sites – approximately 50 (18 selected)
Continued . . .

5. • FrogWatch
• Wisconsin Bat Monitoring
• Ushahidi
• Secchi
• Marine Debris Tracker
• Astro Drone (European
Space Agency)
• Invaders of Texas
• The Quake Catcher Center
• Project Noah
• Meteor Counter
• PING (precipitation)
• Globe at Night
• eBird
• Dark Sky
• NOVA Energy Lab
• Safecast

6. • Galloway study of training of school age students to identify White
oaks indicated need for sampling methodology training
• Fowler’s study of fleeting weather and atmosphere conditions
required spatial, temporal and altitudinal elements to be included
• Powell also looked into GeoExposures (2011, UK web launch)
where participants can increase geoscience knowledge base with
their data
• Marine Debris Initiative funded project (Martin, 2013) began in
Georgia makes use of VGI to aid environment
• McClendon and Robinson (2013) social media for input of CGD no
training but potential for disaster information

7. • Began in London in 2004 requires internet accessibility
• Haklay (2009) compares OSM to Ordinance Survey (UK’s)
• High quality product from skilled core contributors + large
core participants lack formal training
• Positional accuracy, attribute accuracy and completeness
• Provide good base map data for other open-source projects
• Key issue was completeness – coverage influenced by
participant choices – errors not randomly distributed
• Citizens organize mapping parties to give hands-on experience

8. • Foody (2013) considered land cover identification as important
component of geo-spatial data
• RS generated data followed by analysis = $ costly $
• Unreliability, inconsistencies in accuracy of VGI result in
hesitancy to accept
• Comparisons between volunteer land cover and reference data
• Characterizing performance of ea. volunteer for accuracy can
ID those who could benefit best from training
• Variation in accuracy attributed to background education, skills
and experience of volunteers

9. • Rice et al (2013) began in 2010 to develop interactive website &
tools to use geo-crowdsourcing on GMU campus
• Path access for disabled members of community considered
(vision and mobility impaired)
• Webmap (geo.gmu.edu/vgi/) modeled after OSM to allow
reports of various types of obstructions
• Gazzetteer all named places, geoparsed to location on campus
• Participants recruited on campus to execute reports of obstacles
using the web interface
• Several updates and revisions to accommodate needs

11. • Training for end-users was designed to improve overall use of
tools
• Slide presentation images & exercise for student volunteers
• Attribute categories, types and method of location clarified
• Continued testbed environment development
• Moderation tool further developed to allow for Quality Assurance
of incoming reports
• Test site designed to fill gap in assistive geotechnology, more open
and real-time use

14. • Position of obstacle based on x, y coordinate
• Type of obstacle: obstruction, poor surface, exit/entrance,
crowd/event, construction detour
• Duration: short, medium, long
• Priority: (low, medium, high) based on impact
• Image of obstacle and text description help in moderation of final
report
• All part of final scores developed for Quality Assurance
• Each participant required no more than one hour of time
commitment (including in-class time for training)

15. How It Works
Identify obstacle
that are in your
path
Go to our website
or
Mobile app
2
1
Walk around
campus as you
regularly do
3
4

16. Reporting Obstacles
Create a unique ID
- Use letters, numbers, symbols, & others
- Do NOT use anything that identifies you (i.e. name, last name, G#)
- Use the same ID every time you report
Indicate date & time
- When did you observe the obstacle?

17. Obstacle Types
Obstacle Type
Sidewalk obstruction
Construction detour
Entrance/Exit problems
Poor surface conditions
Crowd/Event

18. Duration & Priority
Duration
Priority
Low (<1 day)
Medium (1-7 days)
Long (>7 days)
Low
Medium
High

19. Images for training with classification of
obstacle – one example of each type shown

20. The image is event/ crowd that changes path access, low
duration, < day; low priority (minor inconvenience)
Image such as this might be uploaded with report to give a
higher completion or overall QA score once moderation done

21. Construction causing sidewalk detour which
could be classified as Medium priority
inconvenience to pedestrians, and Long Duration
(more than 7 days)

22. other examples from presentation
‘Volunteer outreach’
entranceways crowds, events

23. Other detours and poor surface conditions

24. Picture 11
Classification Exercise:
While viewing 15 different
images, such as this, students
in training marked paper
copies with correct answers
for Obstacle type,
Duration and
Priority

25. How participants categorized images during training:

26. Continued . . .

27. Continued . . .

28. Continued . . .

29. Continued . . .

30. Sidewalk Obstrux
32%
Construx
5%
Poor Surface
63%
Obstacle type: pic. 15
Continued . . .

31. • Sampling approach limited (students on campus)
• Schedules kept once signed up (not research team!)
• Low incentive – most not altruistic
• Final sample number low (n = 23)
• Wi-fi reliability problematic to students based on surveys
• Slow upload in many cases, students would stop before done
• Low quality reports for moderation – some not usable

32. • Incoming reports (129) assessed by moderation process
• 23 participants submitted 2 – 8 reports each
• Moderated reports given QA scores based on various criteria:
• Temporal consistency,
• Positional accuracy (both x,y and text description)
• Obstacle Type, Duration and Urgency
• Moderator QA score by 3 moderators, based on all criteria
• QA FINAL SCORE computed as linear combination of all fields

33. • Tests for both location and attributes done using two-sample
(independent) t-test
• X, Y location as QA interval dependent variable and
Trained/Untrained as categorical independent variable
• Also QA Final Score as dependent variable compared to
Trained/Untrained
• Mann-Whitney test done to compare QA moderator score
ordinal scale dependent variable to Trained/Untrained
• H-α Trained group would score higher QA by moderation as
well as Final and overall scores
• H-ο(null) no significant difference between two groups

34. • Both Trained/ Untrained groups QA scores 60-65%
• Trained slightly higher (insignificantly) average total quality
score, obstacle type quality score and moderator-assessed QA
• Untrained slightly higher (insignificantly) average score
accuracy of report location
• Tests suggest that training did not significantly influence
participant ability to position object on map
• Positive correlation between Final score and positional
accuracy

35. Graphed data from scored reports:
Overall scores assigned by moderation process to participants
based on average of all reports, range from 48 to 81 (not
sorted here by Trained /Untrained)

36. Obstacle type scores based
on agreement with ‘correct’
moderator scoring:
0 - no agreement
1 – partial agreement
2 – perfect agreement
Amount of error or
variance of obstacle
location measured in
meters, from largest
(52 meters!) to
smallest (0.5 meters)

37. • Finalized moderation reports, scoring and analysis
• Use of obstacle type agreement, location accuracy, variance,
means, other methods of measuring data accuracy
• Statistical analysis indicated test subjects not significantly
different with respect to positional accuracy and for accuracy
of attribute characteristics
• Sample size problematic/ participant incentives low
• n = 23, with uneven distribution of test subjects

38. • Larger test groups, better planning for reliability of
participants and increased incentives – social setting (?)
• Possibly link training to internet sites
• Develop improvements to site for suitability on campus
• Continued student studies to increase usability
• Possible training developed to be included in GIS classes
• Export best training ideas to use for invasive species, bird
counts, marine debris, seismic hazards, and many other
citizen science applications in crowd-sourcing of
geographic data

39. Abdulmonem, Alabri, and Jane Hunter. "Enhancing the Quality and Trust of
Citizen Science Data." 2010 Sixth IEEE International Conference on e-Science.
Brisbane: IEEE Computer Society, 2010. 81-88.
Brown, Greg. "An empirical evaluation of the spatial accuracy of public
participation GIS (PPGIS) data." Applied Geography 34 (2012a): 289-294.
Brown, Greg. "Public Participation GIS (PPGIS) for Regional and Environmental
Planning: Reflections on a Decade of Empirical Research." URISA Journal 24, no.
2 (2012b): 7-18.
Creative Commons. CC Wiki. December 12, 2013.
http://www.wiki.creativecommons.org/License_versions (accessed December 14,
2013).
Dunn, Erica H., et al. "Enhancing the Scientific Value of The Christmas Bird
Count." The Auk 122, no. 1 (2005): 338-346.
Foody, G. M., et al. "Assessing the Accuracy of Volunteered Geographic
Information arising from Multiple Contributors to an Internet Based Collaborative
Project." Transactions in GIS (John Wiley & Sons Ltd) 1111 (2013): 1-14.

40. Fowler, Amy, J. Duncan Whyatt, Gemma Davies, and Rebecca Ellis. "How Reliable
are Citizen-Derived Scientific Data? Assessing th Quality of Contrail Observations
Made by the General Public." Transactions in GIS 17, no. 4 (2013): 488-506.
Galloway, Aaron W. E., Margaret T. Tudor, and W. Matthew Vander Haegen. "The
Reliability of Citizen Sceince: A Case Study of Oregon White Oak Stand Surveys."
Wildlife Society Bulletin 34, no. 5 (Dec 2006): 1425-1429.
Goodchild, Michael. "Citizens as Sensors: The World of Volunteered Geography."
GeoJournal (National Center for Geographic Information and Analysis) 69, no. 4
(2007): 211-221.
Greenemeier, Larry. "Welcome to Scientific American's Citizen Science Initiative."
Scientific American, May 2, 2011.
Haklay, Mordechai. "How good is volunteered geographical information? A
comparative study of OpenStreetMap and Ordance Survey datasets." Environment
and Planning B: Planning and Design, Sept 13, 2009: 682-703.
Henrich, Joseph Heine, Steven J. and Norenzayan, Ara. "Most people are not
WEIRD." Nature, July 1, 2010: 29.
Howe, Jeff. 2006. http://www.wired.com/wiread/archive/14.06/crowds.html
(accessed December 8, 2013).

41. Jackson, Steven P., William Mullen, Peggy Agouris, Andrew Crooks, Arie Croitoru,
and Anthony Stefanidis. "Assessing Completeness and Spatial Error of Features in
Volunteered Geographic Information ." ISPRS Internation Jouinral of Geo-Information 2
(2013): 507-530.
Martin, Jeannie Miller. "Marine debris removal: One year of effort by the Georgia
Sea Turtle-Center-Marine Debris Initiative." Marine Pollution Bulletin 74 (2013): 165-
169.
McClendon, Susan, and Anthony C. Robinson. "Leveraging Geospatially-Oriented
Social Media Communications in Disaster Response." International Journal of
Information Systems for Crisis Response and Management 5, no. 1 (Jan-Mar 2013): 22-40.
Newman, Greg, Don Zimmerman, Alicia Crall, Melinda Laituru, Jim Graham, and
Linda Stapel. "User-freindly web mapping: lessons from a citizen science website."
International Journal of Geograhpical Information Science (Taylor & Francis) 24, no. 12
(Dec 2010): 1851-1869.
Null, Christopher. Tech Hive. PC World. October 4, 2010.
http://www.techhive.com/article/206702/best_online_mapping_service.html?page
=2 (accessed January 20, 2014).

42. Powell, John, Gemma Nash, and Patrick Bell. "Geo-Exposures: Documenting
temporary geological exposures in Great Britain through a citizen-science web site."
Proceedings of the Geologists' Association. 2013. 638-647.
Rice, Matthew T., Paez, Fanbiana I., Jacobson, R.Daniel, Aburizaiza, Ahmad O.,
Shore, Brandon M. "Supporting Accessibility for Blind and Vision-impaired People
With a Localized Gazetteer and Open Source Geotechnology." Transactions in GIS
(Blackwell Publishing Ltd.) 16, no. 2 (April 2012a): 177-190.
Rice, Matthew T., Caldwell, Douglas R., Paez, Fabiana I., Mulhollen, Aaron, Shore,
Brandon M. Crowdsourced Geospatial Data: A report on the emerging phenomena of
crowdsourced and user-generated geospatial data. contract, Topographic Engineering Center
Technical Report, US Army Corps of Engineers; Engineer Research and
Development Center, Data Level Enterprise Tools Workshop, (2012b), 1-148.
Rice, Matthew T., Curtin, Kevin M., Paez, Fabiana I., Seitz, Christopher R., Qin, Han.
"Crowdsourcing to Support Navigation for the Disabled." Annual, Department of
Geography and Geoinformation Science, George Mason University, Fairfax, (2013),
1-62.

43. Waters, Nigel. "Social Network Analysis." In Handbook of Regional Science, edited by
Manfred M. Fischer and Peter Nijkamp, 725-740. 2014.
Yang, Xiaolin, and Zhongliang Wu. "Civilian Monitoring video records for
earthquake intensity: a potentially unbiased online information source of macro-
seismology." Natural Hazards, 2013: 1765-1781.