1. An Approach to Improving Parametric Estimation Models in case of Violation of Assumptions 1 Dept. of Informatica, Sistemi e Produzione University of Rome “Tor Vergata” S. Alessandro Sarcià 1,2 [email_address] Giovanni Cantone 1 Victor R. Basili 2,3 2 Dept. of Computer Science University of Maryland and 2 Fraunhofer Center for ESE Maryland Author Advisors

3. MOTIVATION

4. Predicting software engineering variables accurately is the basis for success of mature organizations. This is still an unsolved problem. Our point of view: Prediction is about estimating values based on mathematical and statistical approaches (no guessing), e.g., regression functions Variables are cost, effort, size, defects, fault proneness, number of test cases and so forth Success refers to delivering software systems on time, on budget, and on quality as initially required. In software estimation , success is about providing estimates as close to the actual values as possible (the error is less than a stated threshold). Focus: We consider a wider meaning of it as keeping prediction uncertainty within acceptable thresholds (risk analysis on the estimation model) Organizations that we refer to are learning organizations that aim at improving their success over time.

5. OBJECTIVES

7. ROADMAP

9. THE PROBLEM

10. Error taxonomy

13. In case of violations, when we estimate the uncertainty on the next estimate the prediction interval may be unreliable (type I – II errors). Violation of Regression assumptions If normality does not hold we cannot use t-Student’s percentiles This is no longer constant This is not the standard error This is not the spread It may be correct Estimate Prediction Interval

14. Violation of Regression assumptions

15. THE SOLUTION

17. The Quality Improvement Paradigm

18. The Estimation Improvement Process

19. The framework

20. Building the BDF Non-linear x-dependent median Class A Class B BDF 0 1 0.5 RE KSLOC (Posterior) Probability RE RE (P1) RE (P2) fixing  A family

21. Inverting the BDF (Sigmoid is smooth and monotonic) Inv(BDF) Fixing the probability RE KSLOC (fixed) 0 0.975 0.5 (Posterior) Probability RE Me UP Fixing a credibility range (95%) 1 0.025 Me DOWN (Bayesian) Error Prediction Interval

22. Analyzing the model behavior 0 Flatter Steeper Biased Biased Unbiased Unbiased KSLOC = 0.95 KSLOC = 0.55 KSLOC = 0.32 KSLOC = 0.11

23. Estimate Prediction Interval (M. Jørgensen ) RE = (Act – Est)/Act To estimate the Estimate Prediction Interval from the Error Prediction Interval, we can substitute and inverting the formula: [Me DOWN , Me UP ] = (Act – Est)/ Act O N+1 DOWN = Act DOWN = Est/(1 – Me DOWN ) O N+1 UP = Act UP = Est/(1 – Me UP ) Estimate Prediction Interval

24. THE APPLICATION

25. Scope Error (similarity analysis with estimated data)

26. Assumption Error (estimated data)

27. Improving the model (actual data) Scope extension

28. Improving the model (actual data) Error magnitude and bias What we need to be worried about is the relative error magnitude not the bias

30. A CASE STUDY

31. The NASA COCOMO data set [PROMISE] UB BS UB BS -0.9 -2.4 Relative Error EXT EXT EXT UB UB UB UB UB UB 77 historical projects (before 1985), 16 projects being estimated (from 1985 to 1987)

32. CONCLUSION & BENEFITS

34. QUESTIONS & FEEDBACKS

35. An Approach to Improving Parametric Estimation Models in case of Violation of Assumptions 1 Dept. of Informatica, Sistemi e Produzione University of Rome “Tor Vergata” S. Alessandro Sarcià 1,2 [email_address] Giovanni Cantone 1 Victor R. Basili 2,3 2 Dept. of Computer Science University of Maryland and 2 Fraunhofer Center for ESE Maryland Author Advisors