As software engineering researchers, we are also zealous tool smiths. Building a research prototype is often a daunting task, let alone building a industry-grade family of tools supporting multiple platforms to ensure the generalizability of our results. In this paper, we give advice to academic and industrial tool smiths on how to design and build an easy-to-maintain architecture capable of supporting multiple integrated development environments (IDEs). Our experiences stem from WatchDog, a multi-IDE infrastructure that assesses developer testing activities in vivo and that over 2,000 registered developers use. To these software engineering practitioners, WatchDog, provides real-time and aggregated feedback in the form of individual testing reports. Project Website: http://www.testroots.org Demonstration Video: https://youtu.be/zXIihnmx3UE

1. How to Catch ’Em All:
WatchDog, a Family of IDE Plug-Ins to Assess
Testing

2. How to Catch ’Em All:
WatchDog, a Family of IDE Plug-Ins to Assess
Testing
Moritz Beller
@Inventitech

3. How to Catch ’Em All:
WatchDog, a Family of IDE Plug-Ins to Assess
Testing
Moritz Beller
@Inventitech
Igor Levaja, Annibale Panichella, Andy Zaidman

4. How to Catch ’Em All:
WatchDog, a Family of IDE Plug-Ins to Assess
Testing
Moritz Beller
@Inventitech
Igor Levaja, Annibale Panichella, Andy Zaidman
Georgios Gousios

5. Caveat: Not Science Science
Image: https://static.pexels.com/photos/34845/pexels-photo.jpg

6. Caveat: Not Science Science
Image: https://static.pexels.com/photos/34845/pexels-photo.jpg

7. Caveat: Not Science Science
Discussion Item!
Image: https://static.pexels.com/photos/34845/pexels-photo.jpg

8. Image: https://twitter.com/anonymousprofs
TODO: Add background with
Sun
Academic Priming (or Blindness?)

9. Image: https://twitter.com/anonymousprofs
TODO: Add background with
Sun
Academic Priming (or Blindness?)

10. Image: https://twitter.com/anonymousprofs
TODO: Add background with
Sun
Academic Priming (or Blindness?)

11. Anonymous professor at TU Munich
If you don't use Eclipse,
you will fail the course.
Image: https://twitter.com/anonymousprofs
TODO: Add background with
Sun
Academic Priming (or Blindness?)

12. Anonymous professor at TU Munich
If you don't use Eclipse,
you will fail the course.
Image: https://twitter.com/anonymousprofs
TODO: Add background with
Sun
Academic Priming (or Blindness?)

13. Anonymous professor at TU Munich
If you don't use Eclipse,
you will fail the course.
Image: https://twitter.com/anonymousprofs
TODO: Add background with
Sun
Academic Priming (or Blindness?)

14. Anonymous professor at TU Munich
If you don't use Eclipse,
you will fail the course.
Image: https://twitter.com/anonymousprofs
Prototype
TODO: Add background with
Sun
Academic Priming (or Blindness?)

15. WatchDog In One Slide

16. WatchDog In One Slide

17. WatchDog In One Slide
How Much Do Developers Test? Do They Know This?

18. WatchDog In One Slide
Do Developers Follow Test-Driven Development?
How Much Do Developers Test? Do They Know This?

19. WatchDog In One Slide
How Do Developers Test?
Do Developers Follow Test-Driven Development?
How Much Do Developers Test? Do They Know This?

20. Image: http://www.xda-developers.com/windows-developer-fee-gone/

21. Traveling Salesman (Problem)

22. Contacted Companies

23. Contacted Companies

24. Contacted Companies

25. Contacted Companies

26. Contacted Companies

27. Contacted Companies

28. Evolution of My Sales Pitch

29. Evolution of My Sales Pitch
Brooks said that testing
takes 50% of project effort.
But does it really? Do you
know how much you are
testing?

30. Evolution of My Sales Pitch
Brooks said that testing
takes 50% of project effort.
But does it really? Do you
know how much you are
testing?
You can win an iPad!

31. Evolution of My Sales Pitch
Brooks said that testing
takes 50% of project effort.
But does it really? Do you
know how much you are
testing?
You can win an iPad!
Help science!

32. Evolution of My Sales Pitch
Brooks said that testing
takes 50% of project effort.
But does it really? Do you
know how much you are
testing?
You can win an iPad!
Help science!
Help me!

33. Bootstrapping Research With Companies

34. Bootstrapping Research With Companies
●
Not a good idea to come with an existing problem (and a solution).

35. Bootstrapping Research With Companies
●
Not a good idea to come with an existing problem (and a solution).
●
We were not fixing one of the companies' immediate problems. You need a
company that wants to know this in their own interest.

36. Bootstrapping Research With Companies
●
Not a good idea to come with an existing problem (and a solution).
●
We were not fixing one of the companies' immediate problems. You need a
company that wants to know this in their own interest.
●
Privacy concerns can stop everything, at any stage (German “Betriebsrat”).

37. Bootstrapping Research With Companies
●
Not a good idea to come with an existing problem (and a solution).
●
We were not fixing one of the companies' immediate problems. You need a
company that wants to know this in their own interest.
●
Privacy concerns can stop everything, at any stage (German “Betriebsrat”).
Very hard to do general research that everybody agrees on has value but
does not fix any one's company specific problem. How can we do such research?

38. Bootstrapping Research With Companies
●
Not a good idea to come with an existing problem (and a solution).
●
We were not fixing one of the companies' immediate problems. You need a
company that wants to know this in their own interest.
●
Privacy concerns can stop everything, at any stage (German “Betriebsrat”).
Very hard to do general research that everybody agrees on has value but
does not fix any one's company specific problem. How can we do such research?
Like so many research advances in SE, it is hard to quantify its value
even for super-cool things (e.g., Code Recommenders).

39. That Didn't Work. What About OSS?

40. That Didn't Work. What About OSS?
●
I was super-desperate.

41. That Didn't Work. What About OSS?
●
I was super-desperate.
●
Approached developers individually, but at scale.

42. That Didn't Work. What About OSS?
●
I was super-desperate.
●
Approached developers individually, but at scale.
●
I approached people …

43. That Didn't Work. What About OSS?
●
I was super-desperate.
●
Approached developers individually, but at scale.
●
I approached people …
●
16,000 of them.

44. Reactions to 16,000 Emails

45. Reactions to 16,000 Emails
“Spammer!”

46. Reactions to 16,000 Emails
“Spammer!”
Even though not spam in the law sense (IANAL!),
developers will perceive your mail as such.

47. Reactions to 16,000 Emails
“Spammer!”
“I'm using IntelliJ IDEA, so I won't be able to participate.”
Even though not spam in the law sense (IANAL!),
developers will perceive your mail as such.

48. Reactions to 16,000 Emails
“Spammer!”
“I'm using IntelliJ IDEA, so I won't be able to participate.”
“Your research sounds verry [sic!] interesting to me. My Problem is
that i am using Intellij for my project.”
Even though not spam in the law sense (IANAL!),
developers will perceive your mail as such.

49. Reactions to 16,000 Emails
“Spammer!”
“I'm using IntelliJ IDEA, so I won't be able to participate.”
“Your research sounds verry [sic!] interesting to me. My Problem is
that i am using Intellij for my project.”
Even though not spam in the law sense (IANAL!),
developers will perceive your mail as such.
Some academics might be blind on what is going on “out there.”
How can we change this?

50. The Academic Pitfall

51. The Academic Pitfall
Write papers, not code!
Good supervisor's advice.

52. The Academic Pitfall
Write papers, not code!
Good supervisor's advice.
How many of our findings in the SE community are evaluated in 1 company, OSS project, … ?

53. How To Catch 'Em All

54. How To Catch 'Em All

55. How To Catch 'Em All
You can support an array of environments with little more than 1x the effort.

56. WatchDog Architecture

57. WatchDog Architecture

58. WatchDog Architecture

59. WatchDog Architecture

60. WatchDog Architecture

61. WatchDog Architecture

62. WatchDog Architecture

63. WatchDog Architecture
By relying on existing low-interference solutions, we can reduce LOC and effort to maintain.

64. Create Value For Your Users!

65. Create Value For Your Users!

66. Create Value For Your Users!

67. OK, So What Else Did We Do?

68. OK, So What Else Did We Do?
●
Providing a high-profile project website.

69. OK, So What Else Did We Do?
●
Providing a high-profile project website.
●
Raffling off prizes.

70. OK, So What Else Did We Do?
●
Providing a high-profile project website.
●
Raffling off prizes.
●
Writing articles in magazines and blogs relevant
to Java and Eclipse developers (Eclipse Magazin,
Jaxenter, EclipsePlanet, Heise News).

71. OK, So What Else Did We Do?
●
Providing a high-profile project website.
●
Raffling off prizes.
●
Writing articles in magazines and blogs relevant to Java and
Eclipse developers (Eclipse Magazin, Jaxenter, EclipsePlanet,
Heise News).
●
Giving talks and presentations at developer conferences (Dutch
Testing Day, EclipseCon).

72. OK, So What Else Did We Do?
●
Providing a high-profile project website.
●
Raffling off prizes.
●
Writing articles in magazines and blogs relevant to Java and Eclipse
developers (Eclipse Magazin, Jaxenter, EclipsePlanet, Heise News).
●
Giving talks and presentations at developer conferences (Dutch
Testing Day, EclipseCon).
●
Participating in a YouTube Java Developer series.

73. OK, So What Else Did We Do?
●
Providing a high-profile project website.
●
Raffling off prizes.
●
Writing articles in magazines and blogs relevant to Java and Eclipse
developers (Eclipse Magazin, Jaxenter, EclipsePlanet, Heise News).
●
Giving talks and presentations at developer conferences (Dutch Testing Day,
EclipseCon).
●
Participating in a YouTube Java Developer series.
●
Penetrating social media (Reddit, Hackernews, Twitter, Facebook).

74. OK, So What Else Did We Do?
●
Providing a high-profile project website.
●
Raffling off prizes.
●
Writing articles in magazines and blogs relevant to Java and Eclipse developers
(Eclipse Magazin, Jaxenter, EclipsePlanet, Heise News).
●
Giving talks and presentations at developer conferences (Dutch Testing Day,
EclipseCon).
●
Participating in a YouTube Java Developer series.
●
Penetrating social media (Reddit, Hackernews, Twitter, Facebook).
●
Putting our plugin in a well-established marketplace.

75. Is There a Single “Silver Bullet”?
To catch 'em all?

76. Is There a Single “Silver Bullet”?
To catch 'em all?
I think, yes.

77. Is There a Single “Silver Bullet”?
To catch 'em all?
I think, yes.
But YMMV.

78. Has It Worked?

79. Has It Worked?
3,000

80. Has It Worked?
3,000

81. Moritz Beller
@Inventitech
How to Catch ’Em All:
WatchDog, a Family of IDE Plug-Ins to Assess
Testing

82. Moritz Beller
@Inventitech
How to Catch ’Em All:
WatchDog, a Family of IDE Plug-Ins to Assess
Testing
Very hard to do general research that everybody agrees on has value but
does not fix any one's company specific problem. How can we do such research?

83. Moritz Beller
@Inventitech
How to Catch ’Em All:
WatchDog, a Family of IDE Plug-Ins to Assess
Testing
Very hard to do general research that everybody agrees on has value but
does not fix any one's company specific problem. How can we do such research?
How can we contact developers individually without spamming them?

84. Moritz Beller
@Inventitech
How to Catch ’Em All:
WatchDog, a Family of IDE Plug-Ins to Assess
Testing
How many of our findings in the SE community are evaluated in 1 company, OSS project, …
Very hard to do general research that everybody agrees on has value but
does not fix any one's company specific problem. How can we do such research?
How can we contact developers individually without spamming them?