This presentation was delivered by Adrien Ball at the Open Science in Practice (OSIP) summer school at EPFL Lausanne on September 2019 (http://osip2019.epfl.ch/). It presents some lessons that were learned in the process of open sourcing the Snips NLU python library.

1. EXPERIENCES FROM GOING
OPEN SOURCE
BY ADRIEN BALL

2. SNIPS

3. SNIPS
END-TO-END SPEECH TO MEANING SOFTWARE
Wake word Speech to text
Natural Language
Understanding

4. SNIPS
NATURAL LANGUAGE UNDERSTANDING
Speech to text
Natural Language
Understanding

5. SNIPS
NATURAL LANGUAGE UNDERSTANDING
Natural Language
Understanding

6. OPEN SOURCE AT SNIPS
MOTIVATIONS
▸ Transparency
▸ Visibility
▸ Reproducibility

8. PACKAGING FOR
OPEN SOURCE

9. PACKAGING FOR OPEN SOURCE
OBJECTIVES FOR THE COMMUNITY
▸ Understand
▸ Use
▸ Contribute

10. PACKAGING FOR OPEN SOURCE
REQUIREMENTS
▸ Documentation
▸ Continuous Integration and build automation
▸ APIs and versioning

11. PACKAGING FOR OPEN SOURCE
GOOD PRACTICES
▸ Documentation
▸ Continuous Integration and build automation
▸ APIs and versioning

12. PACKAGING FOR OPEN SOURCE
DOCUMENTATION
▸ Hard and painful to maintain
▸ More documentation => More outdated documentation
▸ Less documentation => Less explanations

13. https://xkcd.com/1343/

14. DOCUMENTATION
WELL DESIGNED APIS

15. DOCUMENTATION
DOCSTRINGS

16. DOCUMENTATION
DOCSTRINGS

17. DOCUMENTATION
DOCTESTS

18. PACKAGING FOR OPEN SOURCE
GOOD PRACTICES
▸ Documentation
▸ Continuous Integration and build automation
▸ APIs and versioning

19. PACKAGING FOR OPEN SOURCE
CONTINUOUS INTEGRATION AND BUILD AUTOMATION
▸ Continuous integration:
▸ Always be merging into a branch
▸ Merge frequently
▸ Build automation:
▸ Enforce tests and checks to pass before merging

20. PACKAGING FOR OPEN SOURCE
BUILD AUTOMATION, WHAT FOR ?
▸ the project can be installed or built on the targeted
platforms
▸ the code is doing what it is expected to do
▸ you haven't introduced regressions
▸ the documentation is not outdated
▸ automate whatever is error prone, and can be automated

21. PACKAGING FOR OPEN SOURCE
GOOD PRACTICES
▸ Documentation
▸ Continuous Integration and build automation
▸ APIs and versioning

22. PACKAGING FOR OPEN SOURCE
APIS AND VERSIONING
▸ Python: everything is public!
▸ Public API = Conventions + Doc

23. PACKAGING FOR OPEN SOURCE
SEMANTIC VERSIONING
1.3.2

24. PACKAGING FOR OPEN SOURCE
SEMANTIC VERSIONING
1 3 2
major minor patch
Bump when you
Examples
make incompatible API
changes
- removed function
- additional mandatory
param
- changed returned
type
Impact on
client code
no longer works
add functionality in a
backwards compatible
manner
- new API
- new optional param
additional capabilities
make backwards
compatible bug fixes
improved behavior
- internal bugs

25. MACHINE LEARNING
AND OPEN SOURCE

26. MACHINE LEARNING AND OPEN SOURCE
SPECIFIC CHALLENGES
▸ Managing resources
▸ Testing a Machine Learning pipeline
▸ Reproducibility
▸ Modularity and Extensibility

27. MACHINE LEARNING AND OPEN SOURCE
SPECIFIC CHALLENGES
▸ Managing resources
▸ Testing a Machine Learning pipeline
▸ Reproducibility
▸ Modularity and Extensibility

28. MACHINE LEARNING AND OPEN SOURCE
MANAGING RESOURCES
Input Output
resources
ML Pipeline

29. MACHINE LEARNING AND OPEN SOURCE
MANAGING RESOURCES
Input Output
ML Pipeline
▸ Heavier library
▸ Updating the resources requires a release
▸ No user-defined resources

30. MACHINE LEARNING AND OPEN SOURCE
MANAGING RESOURCES
Input Output
resources
ML Pipeline

31. MACHINE LEARNING AND OPEN SOURCE
MANAGING RESOURCES
Input Output
resources
ML Pipeline

32. MACHINE LEARNING AND OPEN SOURCE
MANAGING RESOURCES IN SNIPS NLU

33. MACHINE LEARNING AND OPEN SOURCE
MANAGING RESOURCES IN SNIPS NLU

34. MACHINE LEARNING AND OPEN SOURCE
SPECIFIC CHALLENGES
▸ Managing resources
▸ Testing a Machine Learning pipeline
▸ Reproducibility
▸ Modularity and Extensibility

35. ▸ Traditional testing:
▸ Testing in ML ?
MACHINE LEARNING AND OPEN SOURCE
TESTING A MACHINE LEARNING PIPELINE

36. MACHINE LEARNING AND OPEN SOURCE
TESTING A MACHINE LEARNING PIPELINE

37. MACHINE LEARNING AND OPEN SOURCE
TESTING DONE WRONG

38. MACHINE LEARNING AND OPEN SOURCE
TESTING DONE RIGHT

40. MACHINE LEARNING AND OPEN SOURCE
TESTING A MACHINE LEARNING PIPELINE

42. MACHINE LEARNING AND OPEN SOURCE
HANDLE RANDOMNESS IN TESTS

43. MACHINE LEARNING AND OPEN SOURCE
SPECIFIC CHALLENGES
▸ Managing resources
▸ Testing a Machine Learning pipeline
▸ Reproducibility
▸ Modularity and Extensibility

44. MACHINE LEARNING AND OPEN SOURCE
REPRODUCIBILITY FROM A PRODUCT PERSPECTIVE
Data
Training
Evaluation selected data

45. Suffering
Reproducibility
0% 100%
MACHINE LEARNING AND OPEN SOURCE
REPRODUCIBILITY FROM A DEBUGGING PERSPECTIVE

46. MACHINE LEARNING AND OPEN SOURCE
REPRODUCIBILITY FOR BENCHMARKS
Data
+
Code
🤓
🤓
🤓
0.95 0.87 0.92
0.98 0.91 0.88
0.89 0.83 0.92

47. MACHINE LEARNING AND OPEN SOURCE
REPRODUCIBILITY WITH RANDOM SEEDS

48. MACHINE LEARNING AND OPEN SOURCE
REPRODUCIBILITY THROUGH CONFIGURATIONS
56
3.0
True
code

49. MACHINE LEARNING AND OPEN SOURCE
REPRODUCIBILITY THROUGH CONFIGURATIONS
42
1.5
False
code

50. MACHINE LEARNING AND OPEN SOURCE
REPRODUCIBILITY THROUGH CONFIGURATIONS
42
1.5
False
x
y
z
param_1:
param_2:
param_3:
code
config

51. MACHINE LEARNING AND OPEN SOURCE
REPRODUCIBILITY THROUGH CONFIGURATIONS
Data
+
Code
+
Config
🤓
🤓
🤓
0.95 0.87 0.92
0.98 0.91 0.88
0.89 0.83 0.92

53. MACHINE LEARNING AND OPEN SOURCE
SPECIFIC CHALLENGES
▸ Managing resources
▸ Testing a Machine Learning pipeline
▸ Reproducibility
▸ Modularity and Extensibility

54. MACHINE LEARNING AND OPEN SOURCE
MODULARITY AND EXTENSIBILITY
Input Output
LogReg
SVM
PIPELINE
AVAILABLE
COMPONENTS

55. MACHINE LEARNING AND OPEN SOURCE
MODULARITY AND EXTENSIBILITY
Input Output
LogReg
AVAILABLE
COMPONENTS
SVM
PIPELINE

56. MACHINE LEARNING AND OPEN SOURCE
MODULARITY AND EXTENSIBILITY
Input Output
LogReg
SVM
PIPELINE
AVAILABLE
COMPONENTS

57. MACHINE LEARNING AND OPEN SOURCE
MODULARITY AND EXTENSIBILITY
Input Output
LogReg
SVM
PIPELINE
AVAILABLE
COMPONENTS

58. MACHINE LEARNING AND OPEN SOURCE
MODULARITY AND EXTENSIBILITY
Input Output
LogReg
SVM
PIPELINE
AVAILABLE
COMPONENTS

59. MACHINE LEARNING AND OPEN SOURCE
MODULARITY AND EXTENSIBILITY
FILE SYSTEMRAM

63. MACHINE LEARNING AND OPEN SOURCE
REGISTRABLE COMPONENTS
Declarative syntax for
pipeline and components

64. MACHINE LEARNING AND OPEN SOURCE
REGISTRABLE COMPONENTS
Components Registry

65. EXPERIENCES FROM GOING OPEN SOURCE
TAKEAWAYS
▸ writing tests save you time, not the opposite
▸ test the right things
▸ make your outputs reproducible
▸ use abstractions to improve modularity and clarity

66. THANK YOU
@adrien_ball
github.com/snipsco/snips-nlu