Presentation given by Jack Humphrey (Engineering Director, Indeed) at Keep Austin Agile 2014. http://keepaustinagile2014.sched.org/event/7faaa9add53a23a59185a595bca50230#.UzCTx61dVt0
For more presentations from Keep Austin Agile 2014, see: http://2014conf.agileaustin.org/?q=presentations
For more presentations from Indeed, see http://engineering.indeed.com/talks.
3. About Me
● I’ve been…
● a software developer for 19 years,
● a manager for 10 of those years,
● at companies including:
○ Trilogy (3 years)
○ Coremetrics (3 years)
○ Google (5 years)
○ Indeed (5 years and counting)
4. About Me
● I’ve worked on products like:
○ a web content management system,
○ enterprise software development frameworks,
○ a large online data collection platform,
○ features for online advertisers, and
○ a few large-scale consumer web applications
6. About Indeed
More job searches worldwide than any other employment website
● Over 100 million unique users
● Over 3 billion searches/month
● Over 24 million jobs
● 50+ countries, 28 languages
Jun-07 Jun-08 Jun-09 Jun-10 Jun-11 Jun-12 Jun-13
Unique Visitors / Month
25. Does everyone understand it and agree?
Does project success line up with the mission?
If not, then a) it’s not success, or b) wrong mission!
What is the overall mission?
26. ● Not a subjective determination
Success should be quantifiable
27. ● Not a subjective determination
● Shipping is not success
Success should be quantifiable
28. ● Not a subjective determination
● Shipping is not success
● Prove success with data
Success should be quantifiable
36. ● Demonstrate success with data
● Evaluate data objectively (unique visitors?)
Avoid Subjectivity
37. ● You may need short-term approximations.
When Success is Hard To Measure
38. ● Best metric for “help people get jobs”: hires
Success for Job Search, take 2
39. ● Best metric for “help people get jobs”: hires
● What’s better than unique visitors & searches?
repeat visitors, bounce rate, click-through rate
Success for Job Search, take 2
40. ● Best metric for “help people get jobs”: hires
● What’s better than unique visitors & searches?
repeat visitors, bounce rate, click-through rate
● What’s even better?
applies (IndeedApply)
Success for Job Search, take 2
41. ● Measurement should be repeatable
● Analysis should be:
○ accurate: based on correct/complete data
○ consistent: metrics don’t change meaning
○ fast: seconds/minutes, not hours/days/weeks
Measure and Learn
44. ● We use Google Analytics
● We build our own frameworks and tools
○ logrepo: capturing what happens
○ ramses, imhotep: detailed analysis in seconds
○ Demo of these tools @IndeedEng on April 30th
http://engineering.indeed.com/talks
Measurement & Analysis Tools
45. Use whatever tools work for you, but make sure:
● measurement is repeatable
● analysis is accurate, consistent, fast
Measurement & Analysis Tools
54. Repeatable and predictable mechanisms for:
● Communication: what, who, how, when
● Priority: do things in the right order
How process can help
55. Repeatable and predictable mechanisms for:
● Communication: what, who, how, when
● Priority: do things in the right order
● Memory: what happened, what next
How process can help
56. Repeatable and predictable mechanisms for:
● Communication: what, who, how, when
● Priority: do things in the right order
● Memory: what happened, what next
● Delivery: get it done as efficiently as possible
How process can help
57. Repeatable and predictable mechanisms for:
● Communication: what, who, how, when
● Priority: do things in the right order
● Memory: what happened, what next
● Delivery: get it done as efficiently as possible
● Quality: it should work intended
How process can help
58. Repeatable and predictable mechanisms for:
● Communication: what, who, how, when
● Priority: do things in the right order
● Memory: what happened, what next
● Delivery: get it done as efficiently as possible
● Quality: it should work intended
● Risk: minimize the chance of bad things
How process can help
60. Can be templates for improving:
● communication
● priority
● delivery
How methodologies can help
61. Can be templates for improving:
● communication
● priority
● delivery
...helping your team iterate on process
How methodologies can help
62. Can help you think about
● What is working well for us?
● What could be better?
How methodologies can help
63. Can help you think about
● What is working well for us?
● What could be better?
… and offer prescribed process changes
How methodologies can help
64. Recently, an experienced engineering manager
described a process change that he loved:
An interesting anecdote
65. Recently, an experienced engineering manager
described a process change that he loved:
“we stopped tracking bugs”
An interesting anecdote
66. Ask yourself, what techniques can help your team:
Focus on what matters.
67. Ask yourself, what techniques can help your team:
● communicate
Focus on what matters.
68. Ask yourself, what techniques can help your team:
● communicate
● prioritize
Focus on what matters.
69. Ask yourself, what techniques can help your team:
● communicate
● prioritize
● remember what has been done
Focus on what matters.
70. Ask yourself, what techniques can help your team:
● communicate
● prioritize
● remember what has been done
● know what needs to be done
Focus on what matters.
71. Ask yourself, what techniques can help your team:
● communicate
● prioritize
● remember what has been done
● know what needs to be done
● build solutions that work, efficiently
Focus on what matters.
72. Ask yourself, what techniques can help your team:
● communicate
● prioritize
● remember what has been done
● know what needs to be done
● build solutions that work, efficiently
● manage risk associated with changes
Focus on what matters.
73. Does your process (or lack thereof)
enable success or just get in the way?
Focus on what matters.
74. Start small and iteratively build process.
...or start big and iteratively reduce/modify process.
Focus on what matters.
79. Idea to implementation, as quickly as possible!
● Software Architecture (Service-Oriented)
● Tools, like automated translation
● Risk Management with Rapid Iteration
Engineering Velocity
@IndeedEng Talk:
http://go.indeed.com/velocity
80. Rapid Iteration… to production (multiple times a week)
Velocity and Delivery
81. Rapid Iteration… to production (multiple times a week)
● Don’t wait to fail: means no big failures
Velocity and Delivery
82. Rapid Iteration… to production (multiple times a week)
● Don’t wait to fail: means no big failures
● Work that’s ready doesn’t have to wait
Velocity and Delivery
83. Rapid Iteration… to production (multiple times a week)
● Don’t wait to fail: means no big failures
● Work that’s ready doesn’t have to wait
● More satisfying for developers
Velocity and Delivery
84. Rapid Iteration… to production (multiple times a week)
● Don’t wait to fail: means no big failures
● Work that’s ready doesn’t have to wait
● More satisfying for developers
● Less deadline pressure
Velocity and Delivery
85. Rapid Iteration… to production (multiple times a week)
● Don’t wait to fail: means no big failures
● Work that’s ready doesn’t have to wait
● More satisfying for developers
● Less deadline pressure
● Developers own quality
Velocity and Delivery
89. ● Encourages modular thinking
● Fewer integration issues
● Fewer changes → faster, comprehensive QA
○ Release multiple times a week for 12 weeks, vs.
○ Release once in 12 weeks
Velocity and Quality
90. ● Encourages modular thinking
● Fewer integration issues
● Fewer changes → faster, comprehensive QA
○ Release multiple times a week for 12 weeks, vs.
○ Release once in 12 weeks
● Encourages automated testing
Velocity and Quality
91. ● Team was small, talented, passionate, focused.
● They iterated quickly.
● Job search was successful.
Success...
92. Process was extremely ad-hoc
● not very consistent (e.g. bug tracking)
● not very repeatable or predictable
● wasn’t creating a memory for the future
...and Mess
93. Process was extremely ad-hoc
● not very consistent (e.g. bug tracking)
● not very repeatable or predictable
● wasn’t creating a memory for the future
… and that was fine. For a while.
...and Mess
94. ...and develop predictable, repeatable ways to:
● Coordinate effort
● Improve communication and transparency
● Create a memory for the future
● Manage risk
● Ensure quality
Time to reflect on process...
107. ● Fix Version deployed to production
→ issues are closed
→ future code changes require new issues
● Every code commit references issue key
Closing the Memory Loop
108. Closing the Memory Loop
JASX-4643
Type: New Feature
Fix Version: 2011-04-21
is incorporated by
UPDOG-76
Type: Initiative
Fix Version: June 2011
111. ● Planning rapid international growth
Iteration: Translation Process
112. ● Planning rapid international growth
● Need a localization strategy that wouldn’t slow
us down
Iteration: Translation Process
113. ● Planning rapid international growth
● Need a localization strategy that wouldn’t slow
us down
● Must coordinate many translators
Iteration: Translation Process
114. ● Planning rapid international growth
● Need a localization strategy that wouldn’t slow
us down
● Must coordinate many translators
● We modeled it in JIRA.
Iteration: Translation Process
124. ● Require a lot of communication and
coordination between roles:
○ Create the release candidate deployable
○ Get it into a testing environment
○ Find the bugs
○ Fix the bugs
○ Declare it ready for production deploy
○ Deploy it to production
Iteration: Production Deploys
125. ● Require a lot of communication and
coordination between roles
● Many people care about current deploy status
Iteration: Production Deploys
126. ● Require a lot of communication and
coordination between roles
● Many people care about current deploy status
● We modeled it in JIRA.
Iteration: Production Deploys
128. ● Clear communication, smooth hand-offs
● Transparency for all stakeholders
● Ability to look at trends in deploys
How JIRA helped Deploys
129. ● Proctor: data-driven framework for A/B testing
○ No full code deploy required for test updates
○ Lighter-weight, but still requires a process for
verification and deployment
Iteration: A/B Testing
130. ● Proctor: data-driven framework for A/B testing
○ No full code deploy required for test updates
○ Lighter-weight, but still requires a process for
verification and deployment
● We modeled it in JIRA.
Iteration: A/B Testing