SlideShare uma empresa Scribd logo
1 de 32
Baixar para ler offline
Bits of Evidence What We Actually Know About Software Development, and Why We Believe It’s True Greg Wilson http://third-bit.com Feb 2010
Once Upon a Time... Seven Years’ War (actually 1754-63) Britain lost 1,512 sailors to enemy action... ...and almost 100,000 to scurvy
Oh, the Irony James Lind (1716-94) 1747: (possibly) the first-ever controlled medical experiment No-one paid attention until a proper Englishman repeated the experiment in 1794... ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
It Took a While to Catch On 1950: Hill & Doll publish a case-control study  comparing smokers with non-smokers 1951: start the British Doctors Study (which runs until 2001)
What They Discovered #1: Smoking causes lung cancer “ ...what happens ‘on average’ is of no help when one is faced with a specific patient...” #2: Many people would rather fail than change
Like Water on Stone 1992: Sackett coins the term “ evidence-based medicine” Randomized double-blind trials are accepted as the gold standard for medical research The Cochrane Collaboration (http://www.cochrane.org/) now archives results from hundreds of medical studies
So Where Are We? “ [Using domain-specific languages] leads to two primary benefits. The first, and simplest, is improved programmer productivity... The second...is...communication with domain experts.” –  Martin Fowler (IEEE Software, July/August 2009)
Say Again? One of the smartest guys in our industry... ...made two substantive claims... ...in an academic journal... ...without a single citation Please note: I’m not disagreeing with his claims —I just want to point out that even the best of us aren’t doing what we expect the makers of acne creams to do.
Um, No “ Debate still continues about how valuable DSLs are in practice. I believe that debate is hampered because not enough people know how to develop DSLs effectively.” I think debate is hampered by  low standards for proof The good news is, things have started to improve
The Times They Are A-Changin’ Growing emphasis on empirical studies in software engineering research since the mid-1990s Papers describing new tools or practices routinely include results from some kind of field study Yes, many are flawed or incomplete, but standards are constantly improving
My Favorite Little Result Aranda & Easterbrook (2005): “Anchoring and Adjustment in Software Estimation” “ How long do you think it will take to make a change to this program?” Control Group: “ I’d like to give an estimate for this project myself, but I admit I have no experience estimating. We’ll wait for your calculations for an estimate.” Group A:  “I admit I have no experience with software projects, but I guess this will take about 2 months to finish. ” Group B:  “...I guess this will take about 20 months... ”
Results The anchor mattered more than experience, how formal the estimation method was, or anything else. Q: Are agile projects similarly afflicted, just on a shorter and more rapid cycle? Group A (lowball) 5.1 months Control Group 7.8 months Group B (highball) 15.4 months
Most Frequently Misquoted Sackman, Erikson, and Grant (1968): “Exploratory experimental studies comparing online and offline programming performance.” Or 10, or 40, or 100, or whatever other large number pops into the head of someone who can’t be bothered to look up the reference... The best programmers are up to 28 times more productive than the worst.
Let’s Pick That Apart ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
So What  Do  We Know? ,[object Object],[object Object],[object Object],I’m not going to tell you Instead, I’d like you to look at the work of Lutz Prechelt Productivity and reliability depend on the length of the program's text, independent of language level.
A Classic Result... Boehm et al (1975): “Some Experience with Automated Aids to the Design of Large-Scale Reliable Software.” ...and many, many more since ,[object Object],[object Object],time number  /  cost
...Which Explains a Lot Pessimists: “If we tackle the hump in the error injection curve, fewer bugs will get to the expensive part of the fixing curve.” Optimists: “If we do lots of short iterations, the total cost of fixing bugs will go down.”
The Real Reason I Care A: I've always believed that there are just fundamental differences between the sexes... B: What data are you basing that opinion on? A: It's more of an unrefuted hypothesis based on personal observation. I have read a few studies on the topic and I found them unconvincing... B: Which studies were those? A: [no reply]
What Real Scientists Do ,[object Object],[object Object],[object Object],[object Object],Ceci & Williams (eds):  Why Aren’t More Women in Science? Top Researchers Debate the Evidence Informed  debate on nature vs. nurture
Greatest Hits ,[object Object],[object Object],[object Object],FIXME: add gratuitous images to liven up these slides.
Greatest Hits (cont.) ,[object Object],[object Object],Gratuitous image. Shouldn’t our development practices  be built around these facts?
More Than Numbers ,[object Object],[object Object],More gratuitous images.
Another Personal Favorite Conway’s Law: A system reflects the organizational structure that built it. Meant as a joke Turns out to be true (Herbsleb et al 1999)
But Wait, There’s More! Nagappan et al (2007) & Bird et al (2009): Physical distance doesn’t affect post-release fault rates Distance in the organizational chart does No, really — shouldn’t our development practices  be built around these facts?
Two Steps Forward... ,[object Object],[object Object],“ Progress” sometimes means saying, “Oops.” El Emam et al (2001): “The Confounding Effect of Class Size on the Validity of Object-Oriented Metrics” Can code metrics predict post-release fault rates? We thought so, but then...
Folk Medicine for Software Systematizing and synthesizing colloquial practice has been very productive in other disciplines…
How Do We Get There? 2007 2008 – 2009
The Book Without a Name Wanted to call the next one  Beautiful Evidence , but Edward Tufte got there first “ What we know and why we think it’s true” (By the way, his book is really good) Knowledge transfer A better textbook Change the debate
A Lot Of Editing In My Future Jorge Aranda Tom Ball Victor Basili Andrew Begel Christian Bird Barry Boehm Marcelo Cataldo Steven Clarke Jason Cohen Rob DeLine Khaled El Emam Hakan Erdogmus Michael Godfrey Mark Guzdial Jo Hannay Ahmed Hassan Israel Herraiz Kim Herzig Barbara Kitchenham Andrew Ko Lucas Layman Steve McConnell Audris Mockus Gail Murphy Nachi Nagappan Tom Ostrand Dewayne Perry Marian Petre Lutz Prechelt Rahul Premraj Dieter Rombach Forrest Shull Beth Simon Janice Singer Diomidis Spinellis Neil Thomas Walter Tichy Burak Turhan Gina Venolia Elaine Weyuker Laurie Williams Andreas Zeller Tom Zimmermann
The Hopeful Result
The  Real  Reason It Matters
Thank you, and good luck

Mais conteúdo relacionado

Mais procurados

DevOps evolution architecting the modern software factory - cloud expo east 2017
DevOps evolution architecting the modern software factory - cloud expo east 2017DevOps evolution architecting the modern software factory - cloud expo east 2017
DevOps evolution architecting the modern software factory - cloud expo east 2017Anand Akela
 
API Strategy Presentation
API Strategy PresentationAPI Strategy Presentation
API Strategy PresentationLawrence Coburn
 
The Next Wave of Reliability Engineering
The Next Wave of Reliability EngineeringThe Next Wave of Reliability Engineering
The Next Wave of Reliability EngineeringMichael Kehoe
 
Monitoring Microservices
Monitoring MicroservicesMonitoring Microservices
Monitoring MicroservicesWeaveworks
 
2021 ZAP Automation in CI/CD
2021 ZAP Automation in CI/CD2021 ZAP Automation in CI/CD
2021 ZAP Automation in CI/CDSimon Bennetts
 
DevOps at Amazon: A Look at Our Tools and Processes
DevOps at Amazon: A Look at Our Tools and ProcessesDevOps at Amazon: A Look at Our Tools and Processes
DevOps at Amazon: A Look at Our Tools and ProcessesAmazon Web Services
 
Snyk Intro - Developer Security Essentials 2022
Snyk Intro - Developer Security Essentials 2022Snyk Intro - Developer Security Essentials 2022
Snyk Intro - Developer Security Essentials 2022Liran Tal
 
APIsecure 2023 - Learning from a decade of API breaches and why application-c...
APIsecure 2023 - Learning from a decade of API breaches and why application-c...APIsecure 2023 - Learning from a decade of API breaches and why application-c...
APIsecure 2023 - Learning from a decade of API breaches and why application-c...apidays
 
A New View of Your Application Security Program with Snyk and ThreadFix
A New View of Your Application Security Program with Snyk and ThreadFixA New View of Your Application Security Program with Snyk and ThreadFix
A New View of Your Application Security Program with Snyk and ThreadFixDenim Group
 
SRE 101 (Site Reliability Engineering)
SRE 101 (Site Reliability Engineering)SRE 101 (Site Reliability Engineering)
SRE 101 (Site Reliability Engineering)Hussain Mansoor
 
PCF Platform Monitoring with Prometheus and Grafana
PCF Platform Monitoring with Prometheus and GrafanaPCF Platform Monitoring with Prometheus and Grafana
PCF Platform Monitoring with Prometheus and GrafanaVMware Tanzu
 
Cybernode.se: Securing the software supply chain (CRA)
Cybernode.se: Securing the software supply chain (CRA)Cybernode.se: Securing the software supply chain (CRA)
Cybernode.se: Securing the software supply chain (CRA)Olle E Johansson
 
High Performance Systems in Go - GopherCon 2014
High Performance Systems in Go - GopherCon 2014High Performance Systems in Go - GopherCon 2014
High Performance Systems in Go - GopherCon 2014Derek Collison
 
The DevOps Dance - Shift Left, Shift Right - Get It Right
The DevOps Dance - Shift Left, Shift Right - Get It RightThe DevOps Dance - Shift Left, Shift Right - Get It Right
The DevOps Dance - Shift Left, Shift Right - Get It RightInflectra
 
Site (Service) Reliability Engineering
Site (Service) Reliability EngineeringSite (Service) Reliability Engineering
Site (Service) Reliability EngineeringMark Underwood
 
Building APIs with the OpenApi Spec
Building APIs with the OpenApi SpecBuilding APIs with the OpenApi Spec
Building APIs with the OpenApi SpecPedro J. Molina
 
APIsecure 2023 - Android Applications and API Hacking, Gabrielle Botbol
APIsecure 2023 - Android Applications and API Hacking, Gabrielle BotbolAPIsecure 2023 - Android Applications and API Hacking, Gabrielle Botbol
APIsecure 2023 - Android Applications and API Hacking, Gabrielle Botbolapidays
 

Mais procurados (19)

DevOps evolution architecting the modern software factory - cloud expo east 2017
DevOps evolution architecting the modern software factory - cloud expo east 2017DevOps evolution architecting the modern software factory - cloud expo east 2017
DevOps evolution architecting the modern software factory - cloud expo east 2017
 
API Strategy Presentation
API Strategy PresentationAPI Strategy Presentation
API Strategy Presentation
 
CI/CD on AWS
CI/CD on AWSCI/CD on AWS
CI/CD on AWS
 
API for Beginners
API for BeginnersAPI for Beginners
API for Beginners
 
The Next Wave of Reliability Engineering
The Next Wave of Reliability EngineeringThe Next Wave of Reliability Engineering
The Next Wave of Reliability Engineering
 
Monitoring Microservices
Monitoring MicroservicesMonitoring Microservices
Monitoring Microservices
 
2021 ZAP Automation in CI/CD
2021 ZAP Automation in CI/CD2021 ZAP Automation in CI/CD
2021 ZAP Automation in CI/CD
 
DevOps at Amazon: A Look at Our Tools and Processes
DevOps at Amazon: A Look at Our Tools and ProcessesDevOps at Amazon: A Look at Our Tools and Processes
DevOps at Amazon: A Look at Our Tools and Processes
 
Snyk Intro - Developer Security Essentials 2022
Snyk Intro - Developer Security Essentials 2022Snyk Intro - Developer Security Essentials 2022
Snyk Intro - Developer Security Essentials 2022
 
APIsecure 2023 - Learning from a decade of API breaches and why application-c...
APIsecure 2023 - Learning from a decade of API breaches and why application-c...APIsecure 2023 - Learning from a decade of API breaches and why application-c...
APIsecure 2023 - Learning from a decade of API breaches and why application-c...
 
A New View of Your Application Security Program with Snyk and ThreadFix
A New View of Your Application Security Program with Snyk and ThreadFixA New View of Your Application Security Program with Snyk and ThreadFix
A New View of Your Application Security Program with Snyk and ThreadFix
 
SRE 101 (Site Reliability Engineering)
SRE 101 (Site Reliability Engineering)SRE 101 (Site Reliability Engineering)
SRE 101 (Site Reliability Engineering)
 
PCF Platform Monitoring with Prometheus and Grafana
PCF Platform Monitoring with Prometheus and GrafanaPCF Platform Monitoring with Prometheus and Grafana
PCF Platform Monitoring with Prometheus and Grafana
 
Cybernode.se: Securing the software supply chain (CRA)
Cybernode.se: Securing the software supply chain (CRA)Cybernode.se: Securing the software supply chain (CRA)
Cybernode.se: Securing the software supply chain (CRA)
 
High Performance Systems in Go - GopherCon 2014
High Performance Systems in Go - GopherCon 2014High Performance Systems in Go - GopherCon 2014
High Performance Systems in Go - GopherCon 2014
 
The DevOps Dance - Shift Left, Shift Right - Get It Right
The DevOps Dance - Shift Left, Shift Right - Get It RightThe DevOps Dance - Shift Left, Shift Right - Get It Right
The DevOps Dance - Shift Left, Shift Right - Get It Right
 
Site (Service) Reliability Engineering
Site (Service) Reliability EngineeringSite (Service) Reliability Engineering
Site (Service) Reliability Engineering
 
Building APIs with the OpenApi Spec
Building APIs with the OpenApi SpecBuilding APIs with the OpenApi Spec
Building APIs with the OpenApi Spec
 
APIsecure 2023 - Android Applications and API Hacking, Gabrielle Botbol
APIsecure 2023 - Android Applications and API Hacking, Gabrielle BotbolAPIsecure 2023 - Android Applications and API Hacking, Gabrielle Botbol
APIsecure 2023 - Android Applications and API Hacking, Gabrielle Botbol
 

Semelhante a Bits of Evidence

Greg Wilson - We Know (but ignore) More Than We Think
Greg Wilson - We Know (but ignore) More Than We ThinkGreg Wilson - We Know (but ignore) More Than We Think
Greg Wilson - We Know (but ignore) More Than We Think#DevTO
 
Why Do Computational Scientists Trust Their So
Why Do Computational Scientists Trust Their SoWhy Do Computational Scientists Trust Their So
Why Do Computational Scientists Trust Their Sojpipitone
 
Possible Essay Questions On Romeo And Juliet
Possible Essay Questions On Romeo And JulietPossible Essay Questions On Romeo And Juliet
Possible Essay Questions On Romeo And JulietJamie Jackson
 
Text, Tags and Thumbnails: Latest Trends in Bioscience Literature Search
Text, Tags and Thumbnails:Latest Trends in Bioscience Literature SearchText, Tags and Thumbnails:Latest Trends in Bioscience Literature Search
Text, Tags and Thumbnails: Latest Trends in Bioscience Literature Searchmarti_hearst
 
Usability Evaluation Considered Harmful (Some of the Time)
Usability Evaluation Considered Harmful (Some of the Time)Usability Evaluation Considered Harmful (Some of the Time)
Usability Evaluation Considered Harmful (Some of the Time)Saul Greenberg
 
Building Interactive Systems for Social Good [Job Talk]
Building Interactive Systems for Social Good [Job Talk]Building Interactive Systems for Social Good [Job Talk]
Building Interactive Systems for Social Good [Job Talk]Matthew Louis Mauriello
 
Software estimation is crap
Software estimation is crapSoftware estimation is crap
Software estimation is crapIan Garrison
 
Examples Of University Application Essay
Examples Of University Application EssayExamples Of University Application Essay
Examples Of University Application EssaySue Jones
 
Validation and mechanism: exploring the limits of evaluation
Validation and mechanism: exploring the limits of evaluationValidation and mechanism: exploring the limits of evaluation
Validation and mechanism: exploring the limits of evaluationAlan Dix
 
Testing Simulation Tools And Simulations.pdfTesting Simulation Tools And Simu...
Testing Simulation Tools And Simulations.pdfTesting Simulation Tools And Simu...Testing Simulation Tools And Simulations.pdfTesting Simulation Tools And Simu...
Testing Simulation Tools And Simulations.pdfTesting Simulation Tools And Simu...Tammy Lacy
 
Into AB experiments
Into AB experimentsInto AB experiments
Into AB experimentsDeven
 
Becoming a software testing expert
Becoming a software testing expertBecoming a software testing expert
Becoming a software testing expertmoosix
 
Annotated Bibliography .Guidelines Annotated Bibliograph.docx
Annotated Bibliography  .Guidelines Annotated Bibliograph.docxAnnotated Bibliography  .Guidelines Annotated Bibliograph.docx
Annotated Bibliography .Guidelines Annotated Bibliograph.docxjustine1simpson78276
 
Effectiveness of Bug Bounty Program in preventing security v
Effectiveness of Bug Bounty Program in preventing security vEffectiveness of Bug Bounty Program in preventing security v
Effectiveness of Bug Bounty Program in preventing security vEvonCanales257
 
Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013
Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013
Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013TEST Huddle
 
Bug debug keynote - Present problems and future solutions
Bug debug keynote - Present problems and future solutionsBug debug keynote - Present problems and future solutions
Bug debug keynote - Present problems and future solutionsRIA RUI Society
 

Semelhante a Bits of Evidence (20)

Greg Wilson - We Know (but ignore) More Than We Think
Greg Wilson - We Know (but ignore) More Than We ThinkGreg Wilson - We Know (but ignore) More Than We Think
Greg Wilson - We Know (but ignore) More Than We Think
 
1325 keynote kohavi
1325 keynote kohavi1325 keynote kohavi
1325 keynote kohavi
 
Why Do Computational Scientists Trust Their So
Why Do Computational Scientists Trust Their SoWhy Do Computational Scientists Trust Their So
Why Do Computational Scientists Trust Their So
 
2014 abic-talk
2014 abic-talk2014 abic-talk
2014 abic-talk
 
Agile numbers
Agile numbersAgile numbers
Agile numbers
 
Possible Essay Questions On Romeo And Juliet
Possible Essay Questions On Romeo And JulietPossible Essay Questions On Romeo And Juliet
Possible Essay Questions On Romeo And Juliet
 
Text, Tags and Thumbnails: Latest Trends in Bioscience Literature Search
Text, Tags and Thumbnails:Latest Trends in Bioscience Literature SearchText, Tags and Thumbnails:Latest Trends in Bioscience Literature Search
Text, Tags and Thumbnails: Latest Trends in Bioscience Literature Search
 
Usability Evaluation Considered Harmful (Some of the Time)
Usability Evaluation Considered Harmful (Some of the Time)Usability Evaluation Considered Harmful (Some of the Time)
Usability Evaluation Considered Harmful (Some of the Time)
 
Building Interactive Systems for Social Good [Job Talk]
Building Interactive Systems for Social Good [Job Talk]Building Interactive Systems for Social Good [Job Talk]
Building Interactive Systems for Social Good [Job Talk]
 
Software estimation is crap
Software estimation is crapSoftware estimation is crap
Software estimation is crap
 
Examples Of University Application Essay
Examples Of University Application EssayExamples Of University Application Essay
Examples Of University Application Essay
 
Validation and mechanism: exploring the limits of evaluation
Validation and mechanism: exploring the limits of evaluationValidation and mechanism: exploring the limits of evaluation
Validation and mechanism: exploring the limits of evaluation
 
Testing Simulation Tools And Simulations.pdfTesting Simulation Tools And Simu...
Testing Simulation Tools And Simulations.pdfTesting Simulation Tools And Simu...Testing Simulation Tools And Simulations.pdfTesting Simulation Tools And Simu...
Testing Simulation Tools And Simulations.pdfTesting Simulation Tools And Simu...
 
TesTrek Notes
TesTrek NotesTesTrek Notes
TesTrek Notes
 
Into AB experiments
Into AB experimentsInto AB experiments
Into AB experiments
 
Becoming a software testing expert
Becoming a software testing expertBecoming a software testing expert
Becoming a software testing expert
 
Annotated Bibliography .Guidelines Annotated Bibliograph.docx
Annotated Bibliography  .Guidelines Annotated Bibliograph.docxAnnotated Bibliography  .Guidelines Annotated Bibliograph.docx
Annotated Bibliography .Guidelines Annotated Bibliograph.docx
 
Effectiveness of Bug Bounty Program in preventing security v
Effectiveness of Bug Bounty Program in preventing security vEffectiveness of Bug Bounty Program in preventing security v
Effectiveness of Bug Bounty Program in preventing security v
 
Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013
Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013
Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013
 
Bug debug keynote - Present problems and future solutions
Bug debug keynote - Present problems and future solutionsBug debug keynote - Present problems and future solutions
Bug debug keynote - Present problems and future solutions
 

Último

IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarPrecisely
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxUdaiappa Ramachandran
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfJamie (Taka) Wang
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDELiveplex
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Brian Pichman
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 

Último (20)

IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity Webinar
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptx
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 

Bits of Evidence

  • 1. Bits of Evidence What We Actually Know About Software Development, and Why We Believe It’s True Greg Wilson http://third-bit.com Feb 2010
  • 2. Once Upon a Time... Seven Years’ War (actually 1754-63) Britain lost 1,512 sailors to enemy action... ...and almost 100,000 to scurvy
  • 3.
  • 4. It Took a While to Catch On 1950: Hill & Doll publish a case-control study comparing smokers with non-smokers 1951: start the British Doctors Study (which runs until 2001)
  • 5. What They Discovered #1: Smoking causes lung cancer “ ...what happens ‘on average’ is of no help when one is faced with a specific patient...” #2: Many people would rather fail than change
  • 6. Like Water on Stone 1992: Sackett coins the term “ evidence-based medicine” Randomized double-blind trials are accepted as the gold standard for medical research The Cochrane Collaboration (http://www.cochrane.org/) now archives results from hundreds of medical studies
  • 7. So Where Are We? “ [Using domain-specific languages] leads to two primary benefits. The first, and simplest, is improved programmer productivity... The second...is...communication with domain experts.” – Martin Fowler (IEEE Software, July/August 2009)
  • 8. Say Again? One of the smartest guys in our industry... ...made two substantive claims... ...in an academic journal... ...without a single citation Please note: I’m not disagreeing with his claims —I just want to point out that even the best of us aren’t doing what we expect the makers of acne creams to do.
  • 9. Um, No “ Debate still continues about how valuable DSLs are in practice. I believe that debate is hampered because not enough people know how to develop DSLs effectively.” I think debate is hampered by low standards for proof The good news is, things have started to improve
  • 10. The Times They Are A-Changin’ Growing emphasis on empirical studies in software engineering research since the mid-1990s Papers describing new tools or practices routinely include results from some kind of field study Yes, many are flawed or incomplete, but standards are constantly improving
  • 11. My Favorite Little Result Aranda & Easterbrook (2005): “Anchoring and Adjustment in Software Estimation” “ How long do you think it will take to make a change to this program?” Control Group: “ I’d like to give an estimate for this project myself, but I admit I have no experience estimating. We’ll wait for your calculations for an estimate.” Group A: “I admit I have no experience with software projects, but I guess this will take about 2 months to finish. ” Group B: “...I guess this will take about 20 months... ”
  • 12. Results The anchor mattered more than experience, how formal the estimation method was, or anything else. Q: Are agile projects similarly afflicted, just on a shorter and more rapid cycle? Group A (lowball) 5.1 months Control Group 7.8 months Group B (highball) 15.4 months
  • 13. Most Frequently Misquoted Sackman, Erikson, and Grant (1968): “Exploratory experimental studies comparing online and offline programming performance.” Or 10, or 40, or 100, or whatever other large number pops into the head of someone who can’t be bothered to look up the reference... The best programmers are up to 28 times more productive than the worst.
  • 14.
  • 15.
  • 16.
  • 17. ...Which Explains a Lot Pessimists: “If we tackle the hump in the error injection curve, fewer bugs will get to the expensive part of the fixing curve.” Optimists: “If we do lots of short iterations, the total cost of fixing bugs will go down.”
  • 18. The Real Reason I Care A: I've always believed that there are just fundamental differences between the sexes... B: What data are you basing that opinion on? A: It's more of an unrefuted hypothesis based on personal observation. I have read a few studies on the topic and I found them unconvincing... B: Which studies were those? A: [no reply]
  • 19.
  • 20.
  • 21.
  • 22.
  • 23. Another Personal Favorite Conway’s Law: A system reflects the organizational structure that built it. Meant as a joke Turns out to be true (Herbsleb et al 1999)
  • 24. But Wait, There’s More! Nagappan et al (2007) & Bird et al (2009): Physical distance doesn’t affect post-release fault rates Distance in the organizational chart does No, really — shouldn’t our development practices be built around these facts?
  • 25.
  • 26. Folk Medicine for Software Systematizing and synthesizing colloquial practice has been very productive in other disciplines…
  • 27. How Do We Get There? 2007 2008 – 2009
  • 28. The Book Without a Name Wanted to call the next one Beautiful Evidence , but Edward Tufte got there first “ What we know and why we think it’s true” (By the way, his book is really good) Knowledge transfer A better textbook Change the debate
  • 29. A Lot Of Editing In My Future Jorge Aranda Tom Ball Victor Basili Andrew Begel Christian Bird Barry Boehm Marcelo Cataldo Steven Clarke Jason Cohen Rob DeLine Khaled El Emam Hakan Erdogmus Michael Godfrey Mark Guzdial Jo Hannay Ahmed Hassan Israel Herraiz Kim Herzig Barbara Kitchenham Andrew Ko Lucas Layman Steve McConnell Audris Mockus Gail Murphy Nachi Nagappan Tom Ostrand Dewayne Perry Marian Petre Lutz Prechelt Rahul Premraj Dieter Rombach Forrest Shull Beth Simon Janice Singer Diomidis Spinellis Neil Thomas Walter Tichy Burak Turhan Gina Venolia Elaine Weyuker Laurie Williams Andreas Zeller Tom Zimmermann
  • 31. The Real Reason It Matters
  • 32. Thank you, and good luck