The yt project ( http://yt-project.org/ ) is a community developed analysis and visualization system for astrophysical simulation data. In this presentation I talk a bit about what yt is, and then discuss the challenges and strategies for growing a community of practice.

1. The yt Project:
Growing & Engaging a Community of Practice
Matthew Turk
Columbia University

2. What is yt?

3. astro-ph/1011.3514
astro-ph/1112.4482
yt-project.org

4. yt is a platform for analysis and
visualization of astrophysical* simulations

7. install script:
Full dependency stack
Source code
Development environment
GUI
Sample data

8. There are many simulation codes.

9. There are many simulation codes,
but there’s only one sky.

10. data structures,
methods,
assumptions,
IO formats,
units,
variable names,
...

14. Fully-Supported Semi-Supported In-Progress
Enzo Chombo Cactus
FLASH Athena Hydra
Orion ART Gadget
Nyx RAMSES
Uniform Data GDF
Piernik

15. yt is designed to address physical,
not computational,
entities and questions.

16. Orthogonal Rays
1D
Non-orthogonal Rays
Slices
Oblique Slices 2D
Projections
Spheres
Rectangular Prisms
Disks/Cylinders
Inclined Boxes 3D
Clumps
Extracted Regions
Ellipsoids

17. Scripting interface, HTML5 GUI,
in-situ processing, transparent
multi-level parallelism...

18. yt is supposed to get out of the way.

19. from yt.mods import *
pf = load(’galaxy0030/galaxy0030’)
p = SlicePlot(pf, 2, ’Density’, ’c’, (200,’kpc’))
p.save(’Galaxy’)

24. 10 -12
g/cc

25. 6 C.B. Hummels et al.
Figure 1. Sample projections from the medium feedback and cooling suppression run at z = 0.5. Each image represent a column-density

26. Canned Analysis Tasks

27. Absorption Spectrum
Coordinate Transformations
Halo Finding
Mass Functions
Merger Trees
Halo Profiling
Level Sets
Light Cones
Light Rays
Time Series
Star Analysis
Two-Point Analysis

28. “Community”?

29. Traditional View
“Users”
“Developers”

30. Most Scientific Software
“Users”
“Developers”

31. Community of Practice
“Devusers”

32. “Developers”

33. “Developers”
Inspection and verification

34. “Developers”
Inspection and verification
Tracking modifications

35. “Developers”
Inspection and verification
Tracking modifications
Sharing information

36. “Developers”
Inspection and verification
Tracking modifications
Sharing information
Adding functionality

37. “Developers”
Inspection and verification
Tracking modifications
Sharing information
Adding functionality
All are necessary characteristics of the
scientific process as a whole.

38. “Users”

39. “Users”
Uncritical acceptance of code...?

40. “Users”
Uncritical acceptance of code...?
“These are people we give the code
to that don’t care how it works.”

41. Developing as a Team

42. Challenges

43. Reward Structure

44. de facto and de jure

45. de facto and de jure
•Utilization of developed tools
•Respect from community
•Project involvements
•Invitations to speak

46. de facto and de jure
•Funding
•Publications
•Citation count
•Influence

47. Traditional astrophysics does
not favor tool builders.

48. Chores

49. Chores
Documentation,
testing,
outreach,
infrastructure.

50. Tasks not fully-aligned with reward
structure goals present great
motivational challenges.

51. Co-opetition

52. Co-opetition
( )
•Funding
•Publications
•Citation count
•Influence

53. How developer community engagement -- as
seen in cohesion, excitement, energy -- is
affected by funded improvements is unclear.

54. Strategies

55. The Art of Community by Bacon
Producing Open Source Software by Fogel
Team Geek by Fitzpatrick & Collins-Sussman
Organizing Simulation Code Collectives by Sundberg
Scientific Software Production by Howison & Herbsleb

56. You must design the community you want.

57. Technical
& Social

58. Technical
& Social

59. SVN hg
Users &
Repos Users &
Repos
Repository Users &
Repos
Users &
Repos
Users &
Repos
Users &
Users Repos

60. Forky development:
very low barrier to entry;
everything comes in the box.

61. Testing:
answer as well as integration tests
get run every 30 minutes.

62. Code review:
eyes on (nearly) every changeset
mentoring new developers

63. The path upstream must be obvious.

64. •Happy application
•Itch-scratching
•Submission
•Code review & mentoring
•Participation

65. Non-core contributed code:
a place for sharing scripts,
data and images

67. Communication

68. Communication
All business is conducted openly.

69. Communication
Immediate

70. 2 Communication
Low-Latency

71. Communication
High-Latency

72. Technical
& Social

73. H
R
T

74. Humility
R
T

75. Humility
Respect
T

76. Humility
Respect
Trust

77. Humility
Respect
Trust
(Fitzpatrick & Collins-Sussman)

78. The most important thing I try to
remember is that I want to foster a
community of peers, not of elites.

80. “ I’ve noticed something is acting
strangely with ... “

81. “ I’ve noticed something is acting
strangely with ... “
“ You’re probably doing it wrong.
“

82. “ I’ve noticed something is acting
strangely with ... “
“ Can you tell us how you would
expect it to act? “

83. “ I think that there might be a bug in
the way ... “

84. “ I think that there might be a bug in
the way ... “
“ It’s like that for very good reasons.
Don’t touch it. “

85. “ I think that there might be a bug in
the way ... “
“ It behaves the way it does
because ... “

86. Thoughtfulness,
inquiry,
confidence,
letting go.

87. This culture must be seeded directly.

88. This culture must be seeded directly,
because culture will self-propagate.

89. letting go

90. Pride rather than ownership

91. Projects passing between people

92. Too much control can smother.

93. will
Too much control can smother.

94. Successes

96. Developed by working astrophysicists.

99. Tom Abel Jean-Claude Passy
David Collins Thomas Robitaille
Brian Crosby Anna Rosen
Andrew Cunningham Anthony Scopatz
Nathan Goldbaum Devin Silvia
Cameron Hummels Sam Skillman
Ji-hoon Kim Stephen Skory
Steffen Klemer Britton Smith
Kacper Kowalik Geoffrey So
Michael Kuhlen Casey Stark
Eve Lee Elizabeth Tasker
Chris Malone Matthew Turk
Chris Moody Rick Wagner
Andrew Myers John Wise
Jeff Oishi John ZuHone

100. Tom Abel Jean-Claude Passy
David Collins Thomas Robitaille
Brian Crosby Anna Rosen
Andrew Cunningham Anthony Scopatz
Nathan Goldbaum Devin Silvia
Cameron Hummels Sam Skillman
Ji-hoon Kim Stephen Skory
Steffen Klemer Britton Smith
Kacper Kowalik Geoffrey So
Michael Kuhlen Casey Stark
Eve Lee Elizabeth Tasker
Chris Malone Matthew Turk
Chris Moody Rick Wagner
Andrew Myers John Wise
Jeff Oishi John ZuHone

101. Usage on NSF
NICS Nautlius
“Usage of a HPC Data Analysis and
Visualization System,” Szczepanski et al 2012

104. (play movie)

105. “... it seems likely that significant software
contributions to existing scientific software
projects are not likely to be rewarded through
the traditional reputation economy of science.
Together these factors provide a reason to
expect the over-production of independent
scientific software packages, and the under-
production of collaborative projects in which
later academics build on the work of earlier
ones.”
Howison & Herbsleb (2011)

106. Thank you.

107. Tom Abel Jean-Claude Passy
David Collins Thomas Robitaille
Brian Crosby Anna Rosen
Andrew Cunningham Anthony Scopatz
Nathan Goldbaum Devin Silvia
Cameron Hummels Sam Skillman
Ji-hoon Kim Stephen Skory
Steffen Klemer Britton Smith
Kacper Kowalik Geoffrey So
Michael Kuhlen Casey Stark
Eve Lee Elizabeth Tasker
Chris Malone Matthew Turk
Chris Moody Rick Wagner
Andrew Myers John Wise
Jeff Oishi John ZuHone